A computer vision-based rock-soil deformation measurement method and remote monitoring system

Abstract

The application discloses a kind of rock-soil deformation measurement method and remote monitoring system based on computer vision, and deformation measurement method includes: target data set construction, model training, image calibration and monitoring point selection, target matching and search parameter determination, whole pixel search, subpixel search, real coordinate conversion, displacement calculation, fused the advantages of rapid positioning of deep learning and DIC accurate search, solved the failure of traditional DIC method in harsh environment and the problem that it is difficult to track instantaneous large deformation represented by surface subsidence, and maintains the continuity of monitoring under temporary occlusion;Remote monitoring system is composed of image acquisition module, image transmission module and deformation calculation and visualization display module, and the online monitoring and management of rock-soil structure deformation are realized through deformation calculation software and visualization management platform, the cost and deployment difficulty of deformation monitoring are reduced, the accuracy and efficiency of deformation calculation are improved, it is strong in universality, easy to engineering promotion.

Description

Technical Field

[0001] This invention belongs to the field of geotechnical engineering deformation monitoring technology, specifically relating to a geotechnical deformation measurement method and remote monitoring system based on computer vision. Background Technology

[0002] During engineering construction and geological disaster investigation, it is necessary to accurately observe the degree of change in the soil and rock structure such as the surface and slopes, and to statistically update the geographic information database to provide decision support for engineering construction management and emergency rescue.

[0003] Traditional monitoring of soil and rock structure deformation mainly employs leveling technology. While this technique offers high accuracy and ease of operation, it suffers from drawbacks such as heavy fieldwork, slow speed, low coverage density, and difficulty in accurately determining the spatial distribution characteristics of settlement. The development of camera sensors has also spurred the rapid advancement of photogrammetry. Photogrammetry involves photographing the object being measured, using an optical camera to capture images, and then digitally processing these images to obtain the object's shape, size, location, characteristics, and interrelationships. Visual measurement technology, as a non-contact modern optical measurement technique, features high accuracy, long-distance multi-point measurement, high frequency, and full-field measurement capabilities. Currently, the main visual measurement algorithms include feature point matching, optical flow estimation, and digital image correlation (DIC). DIC, in particular, searches for monitoring points by matching the correlation of regions, and through interpolation, it can refine the resolution to sub-image level, exhibiting high accuracy, adaptability, and reliability. However, current visual measurement methods still have limitations: they are greatly affected by environmental interference such as lighting, occlusion, shadows, rain and fog, which can easily lead to monitoring failure and make it difficult to maintain stable monitoring. Therefore, it is also difficult to guarantee the stability of long-term monitoring in natural environments. Moreover, most monitoring research focuses only on improving monitoring algorithms and has not formed a mature and reliable system. Summary of the Invention

[0004] The purpose of this invention is to provide a computer vision-based method for measuring soil and rock deformation and a remote monitoring system, which can effectively solve the problems of existing visual measurement methods being greatly affected by harsh environments, having difficulty in monitoring large instantaneous deformations, having poor stability, and being immature.

[0005] To achieve the above objectives, this invention provides a computer vision-based method for measuring soil and rock deformation, comprising the following steps:

[0006] Step 1: Construction of target dataset: Obtain target images in indoor simulation and actual scenarios, construct the minimum bounding rectangle to label the target images, and use the inner ring number of the target as its category label. Complete the construction of the target dataset through data augmentation, and then divide it into training set and validation set according to the preset ratio.

[0007] Step 2, Model Training: A deep learning target detection model is used to train the target dataset. Cross-entropy loss and DFL+CIOU loss are jointly optimized for target classification and localization to obtain the target recognition model.

[0008] Step 3, Image calibration and monitoring point selection: In a set of time-series deformed images containing targets, the first image in this set of time-series images is used as the reference image, and any other image is used as the displacement analysis image. The monitoring points and calibration points of each target in the reference image are determined. The plane where the target is located is set as the plane with Z=0 in the world coordinate system. The unit deformation matrix H of the plane where each target is located is calculated by using the pixel coordinates of the four calibration points and the world coordinates.

[0009] Step 4: Target Matching and Search Parameter Determination: Using a target recognition model, obtain the minimum bounding rectangle and category of the target in the time-series deformation image. Match the targets in the baseline image with the targets in the displacement analysis image based on their categories. If a match is successful and a matching target i exists, dynamically calculate the search box range (X) of target i in the displacement analysis image for DIC calculation based on the minimum bounding rectangle. min :X max, Y min :Y max If the target i is matched with the square frame size W, it is assumed that the target i was lost or temporarily occluded during the capture of the displacement analysis image.

[0010] Step 5, Integer Pixel Search: If target i is successfully matched in Step 4, construct a square template matching box u with side length W, centered on the monitoring point of target i in the reference image, and construct a matching box u with a range of (X) in the displacement analysis image. min :X max, Y min :Y max The search box is defined, and a matching box v with side length W is constructed in the displacement analysis image with any pixel in the search box as the center. The correlation coefficient R between the two matching boxes is calculated. All pixels in the search box are traversed, and the pixel corresponding to the maximum value of the correlation coefficient R is the integer pixel position of the target i monitoring point in the displacement analysis image.

[0011] Step 6, Subpixel Search: Based on the integer pixel position coordinates of the target i monitoring point in the displacement analysis image obtained in Step 5, obtain the coordinates (X,Y) and correlation coefficient R of nine points in the eight-neighbor pixel matrix of the pixel point. Based on these nine points (X,Y,R), construct a set of three-dimensional data points and use the data point set to fit a quadratic correlation surface. By solving the extreme points of the surface, obtain the subpixel coordinates of the target i monitoring point.

[0012] Step 7, Real coordinate transformation: Based on the homography transformation matrix H corresponding to each target plane obtained in Step 3, the sub-pixel coordinates of the monitoring point of target i are transformed into world coordinates;

[0013] Step 8, Displacement Calculation: Calculate the actual displacement value of target i at this time based on the world coordinates in Step 7.

[0014] As a further aspect of the present invention: In step 1, the target image is designed as a circle, divided into an outer ring and an inner ring. The inner ring is the main body of the target, including a background, lines, calibration circles, dots, and numbers. The background area is equal to that of the main body of the target. The dots are monitoring points set at the center of the main body of the target. The numbers are located at the center of the main body of the target and are used to distinguish multiple targets. There are four lines, all of which pass through the dots and have a length that is exactly the diameter of the main body of the target, which is used to increase the complexity of the background texture. There are four calibration circles, which are used for the calibration of the target plane. The center of each circle is the calibration point located at the two ends of the two vertically intersecting lines.

[0015] As a further aspect of the present invention: In step 3, the pixel position of the calibration point is obtained through two methods: when the radius of the calibration circle in the time-series deformed image is ≥10 pixels, the range of four calibration circles is manually constructed, and then the Hough circle detection method is used to detect the center of these four calibration circles, thereby obtaining the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10 pixels, the center of the calibration circle is manually selected as the calibration point.

[0016] As a further aspect of the present invention: In step 4, when the reference image and the target i in the displacement analysis image are successfully matched, the range of the search box in the displacement analysis image (Xmin:Xmax,Ymin:Ymax) and the size W of the square matching box adapted to targets of different sizes are dynamically calculated, as shown in formula (1):

[0017]

[0018] In the formula: (x, y) are the coordinates of the top-left corner of the minimum bounding rectangle; w is the width of the minimum bounding rectangle; h is the height of the minimum bounding rectangle; n is the scaling factor, X min ,X max These represent the two boundaries of the search box in the X direction and Y direction of the displacement analysis image, respectively. min ,Y max These represent the two boundaries of the search box in the Y direction of the displacement analysis image, respectively, and W is the side length of the square matching box;

[0019] In step 5, the correlation coefficient R between the two matching boxes is calculated, as shown in formula (2):

[0020]

[0021] In the formula: R is the correlation coefficient; The average grayscale value of the template matching box; The average gray value of the search matching box; u(x,y) is the average gray value of the template matching box at point (x,y); v(x,y) is the average gray value of the search matching box at point (x,y);

[0022] In step 6, the equation of the quadratic correlation surface is shown in formula (3):

[0023] R(x,y)=a0+a1x+a2y+a3x 2 +a4xy+a5y 2 (3)

[0024] In the formula: R(x,y) is the correlation coefficient at any image coordinate point (x,y), and a0~a5 are surface coefficients;

[0025] In step 7, the sub-pixel coordinates are converted to world coordinates using formula (4):

[0026]

[0027] In the formula: (X,Y) is the position of any point in the plane where target i is located in the image coordinate system; (x,y) is the coordinate of the point (X,Y) projected onto the Z=0 plane in the world coordinate system (the plane where target i is located in the world coordinate system);

[0028] H is the homography matrix; s is the scaling factor;

[0029] In step 8, the actual displacement value is calculated using formula (5):

[0030]

[0031] In the formula: Δx B Δy represents the X-direction displacement of target i at the moment of the displacement analysis image; B Δs represents the Y-direction displacement of target i at the moment of the displacement analysis image; Δs represents the total displacement of target i at the moment of the displacement analysis image. The true coordinates of target i monitoring point in the baseline image; represents the true coordinates of target i monitoring point in the displacement analysis image.

[0032] As a further solution of the present invention: In step 8, if the target i is lost or temporarily occluded at the moment of displacement analysis image, the position of the target i at the moment of displacement analysis image is roughly estimated by formula (6), and then the real displacement value of the target i is calculated according to formula (5). In the programming, the error log with the words "target i is lost or temporarily occluded" is returned until the target i is successfully matched again in the subsequent image, and the original calculation process is restored.

[0033]

[0034] In the formula: The calculated true coordinates of target i monitoring point in the displacement analysis image; This represents the true coordinates of target i in the previous image of the displacement analysis image; Let be the average displacement velocity of target i in the X direction during the previous time interval; ΔT represents the average displacement velocity of target i in the Y direction during the previous time interval; ΔT is the time interval between the displacement analysis image and the previous image.

[0035] To achieve the above objectives, the present invention also provides a remote monitoring system, comprising:

[0036] The image acquisition module includes a network camera, a lighting device, and a self-made target device. The self-made target device is installed in the deformation monitoring area, and the network camera is installed in a stable area away from the deformation monitoring area to capture images of the target and ensure that the camera's field of view covers the entire target arrangement area. The lighting device is fixed to the network camera and also illuminates the monitoring area at night. The light is turned on or off via a relay.

[0037] The image transmission module includes a smart gateway and a cloud server. The smart gateway connects directly to the network camera and relay via Ethernet cable, and also connects to the network via both 4G SIM card and Ethernet cable. The smart gateway is located on a local area network (LAN), while the cloud server is located on the public network. Using Fast Reverse Proxy (FRP) intranet penetration technology, the ports controlling network camera image capture, focus adjustment, and relay switching are mapped to corresponding ports on the public network, establishing a data transmission channel. The smart gateway carries the FRP client program, which automatically starts upon power-on and network connection. The cloud server carries the FRP server program, requiring a data download bandwidth of >10M / s.

[0038] The deformation calculation and visualization module includes a deformation calculation server, deformation calculation software, and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform, which are used for online monitoring, visualization, and early warning of deformation of soil and rock structures.

[0039] As a further aspect of the present invention: the self-made target device is made of steel and is shaped as a right-angled triangular prism with hollowed-out sides. The remaining three complete surfaces—the bottom, the vertical front, and the oblique back—are 20mm thick. The bottom surface contains four anchor holes, and the vertical front is perpendicular to the bottom surface. The target image is engraved on the vertical front, and its vertical displacement represents the ground subsidence at that location.

[0040] As a further aspect of the present invention: in the deformation calculation and visualization module, the specific analysis process for deformation calculation and visualization is as follows:

[0041] Process 1): The deformation calculation software connects to the mapped public network port, controls the network camera to capture initial images, loads them onto the software page, and saves the images to the designated folder A. The center point of each target in the initial image is selected as a monitoring point using the mouse, and labels are added to the monitoring points according to the numbers at the target centers. Based on this, the method for obtaining the calibration point positions is set according to the size of the calibration circle in the image for each target, and the selection is made to construct the calibration circle range or directly select the calibration point. Simultaneously, the software automatically generates a txt format data file for each monitoring point in folder A to save the calculation results, and also automatically generates the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels. Then, an image is automatically captured every N seconds and saved to folder A. Next, the deformation calculation software searches all images in folder A and adds the most recently captured image to the software's list of images to be analyzed.

[0042] Process 2): The deformation analysis software uses the built-in deformation measurement algorithm to perform deformation analysis on the new image, and obtains the X-direction displacement value, Y-direction displacement value, total displacement value and error log of each target at the current moment. At this time, the software saves the calculation results of each monitoring point to the corresponding data file under folder A, and at the same time saves the calculation results of each monitoring point to the corresponding data table in the database. The deformation curve built into the software will be updated synchronously.

[0043] Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, it is connected to the data table corresponding to each measuring point in the database. Deformation alarm thresholds are set according to different engineering needs. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, the new data is loaded into the platform for visualization display, and risk warnings are issued according to the set displacement thresholds.

[0044] As a further aspect of the present invention: In process 1), before each image capture, the control relay is closed to turn on the lighting; after the image capture is completed, the relay is opened to turn off the lighting; the captured images are automatically named according to the shooting time, and the basic format of the image name is: "year-month-day & hour-minute-second.bmp"; each data table in the database contains seven fields: ID (int), current time (datatime), interval time with the previous image (double), X-direction displacement (double), Y-direction displacement (double), total displacement (double), and error log (varchar); where "ID" represents the number of images currently being analyzed, "interval time with the previous image" is in seconds, "X-direction displacement", "Y-direction displacement", and "total displacement" are in mm, and "error log" contains error information that occurs during the analysis process.

[0045] As a further aspect of the present invention: In process 2), the deformation measurement algorithm adopts the computer vision-based soil deformation measurement method described in claim 1; after calculating the deformation of the current analysis image, the next image in the analysis list is immediately calculated. If there are no remaining images in the analysis list, the program pauses until a new image is loaded into the analysis list and then restarts; after obtaining a new set of deformation data, for different targets, the program will construct strings and lists according to six data: "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement" and "error log", and save them to the data file and data table. The "error log" only contains two types: "none" and "target lost or temporarily occluded", which correspond to the successful and unsuccessful target matching in the computer vision-based soil deformation measurement method, respectively.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention fully leverages the advantages of contactless monitoring using computer vision. Simultaneously, it employs deep learning and DIC algorithms for coarse and fine searches respectively, effectively improving the adaptability of computer vision algorithms to harsh environments. This ensures the continuity of monitoring even when targets are temporarily obscured, and also allows for the detection of instantaneous large deformations, such as surface subsidence. Based on this, the constructed remote monitoring system achieves stable and accurate long-term monitoring and stable remote data transmission. With a bandwidth of 10M / s, it achieves an image transmission speed exceeding 2 frames per second when transmitting 3840×2160 resolution images, demonstrating fast transmission speed. Furthermore, it possesses advantages such as high precision, economic reliability, strong adaptability to harsh environments, convenient operation, simple deployment, and ease of promotion. Attached Figure Description

[0048] Figure 1This is a flowchart of the computer vision-based method for measuring soil and rock deformation in this invention.

[0049] Figure 2 This is a schematic diagram of the target image style in this invention.

[0050] Figure 3 This is a framework diagram of the remote monitoring system in this invention.

[0051] Figure 4 This is a schematic diagram of the structure of the self-made target device in this invention.

[0052] Figure 5 This is an analysis flowchart of the deformation calculation and visualization module in this invention.

[0053] In the image: 1. Background, 2. Outer ring, 3. Lines, 4. Marker circle, 5. Dots, 6. Numbers. Detailed Implementation

[0054] The present invention will be further illustrated by the following examples.

[0055] like Figure 1 As shown, a computer vision-based method for measuring soil and rock deformation includes the following steps:

[0056] Step 1: Construction of the target dataset: A total of 400 target images were acquired from indoor simulations and real-world scenarios, covering three types of harsh environments: heavy fog, rain, and strong light. This allows the trained model to better adapt to harsh environments in real-world situations. The minimum bounding rectangle was constructed, and the target images were labeled using the LabelImg tool. The inner ring number of the target was used as its category label. To enhance the model's recognition performance, the dataset was expanded to 5000 images through random rotation of ±30°, random scaling from 0.7 to 1.5 times, random cropping, and random fusion of multiple images. The dataset was then divided into training and validation sets in a 4:1 ratio.

[0057] Furthermore, the target image is designed as a circle, divided into an outer ring 2 and an inner ring. The inner ring is the main body of the target, including a background 1, lines 3, calibration circles 4, dots 5, and numbers 6. The area of ​​the background 1 is equal to that of the main body of the target. Dot 5 is a monitoring point located at the center of the main body of the target. Number 6 is located at the center of the main body of the target and is used to distinguish multiple targets. Four lines 3 are arranged, all passing through dots 4 and with a length exactly equal to the diameter of the main body of the target, used to increase the texture complexity of the background 1. Four calibration circles 4 are included, used for calibrating the target plane, with their centers at the two ends of the two vertically intersecting lines. For convex display, as shown... Figure 2 As shown, outer ring 2 and line 3 are gray, background 1 is black, calibration circle 4 is blue, calibration point is green, dot 5 is red, and number 6 is white.

[0058] Step 2, Model Training: The YOLO V8 model is used to train the target dataset. The model used is a network structure that includes Backbone feature extraction, Neck feature fusion, and Head no-anchor-box detection. Cross-entropy loss and DFL+CIOU loss are used for joint optimization of target classification and localization to obtain the target recognition model.

[0059] Step 3, Image Calibration and Monitoring Point Selection: In a set of time-series deformed images containing targets, the first image in this set of time-series images (hereinafter referred to as Image A for convenience) is used as the reference image. The center point of each target in Image A is selected as the monitoring point for subsequent tracking. The center point of the four calibration circles in each target is selected as the calibration point of the target. The plane where the target is located is set to the Z=0 plane in the world coordinate system, and the center point of the target is taken as the center point of this plane. At this time, the pixel coordinates and world coordinates of these four calibration points are all known. The homography transformation matrix H of the plane where each target is located can be calculated using the findHomography function in the opnecv image processing library for subsequent coordinate transformation. Any subsequent image other than Image A (hereinafter referred to as Image B for convenience) is used as the displacement analysis image.

[0060] Furthermore, the pixel positions of the calibration points are obtained through two methods: when the radius of the calibration circle in the time-series deformed image is ≥10 pixels, the range of four calibration circles is manually constructed, and then the Hough circle detection method is used to detect the center of these four calibration circles, thereby obtaining the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10 pixels, the center of the calibration circle is manually selected as the calibration point.

[0061] Step 4: Target Matching and Search Parameter Determination: Using a target recognition model, obtain the minimum bounding rectangle and category of the targets in the time-series deformation images. Match the targets in image A with the targets in image B based on their categories. Taking any target in image A as an example, assuming the category of this target is i, denoted as target i, if there is also a unique target in image B with category i, then they are considered to be the same target, indicating a successful match. Then, dynamically calculate the search box range (X) of target i in the displacement analysis image for DIC calculation based on the minimum bounding rectangle. min :X max, Y min :Y max If the target i is matched with the square bounding box size W, it is assumed that the target i was lost or temporarily occluded when image B was captured.

[0062] Furthermore, as shown in formula (1):

[0063]

[0064] In the formula: (x, y) are the coordinates of the top-left corner of the minimum bounding rectangle; w is the width of the minimum bounding rectangle; h is the height of the minimum bounding rectangle; n is the scaling factor, which is flexibly set according to the size of the target; X min ,X max These represent the two boundaries of the search box in the X direction and Y direction of the displacement analysis image, respectively. min ,Y max These represent the two boundaries of the search box in the Y direction of the displacement analysis image, respectively, and W is the side length of the square matching box.

[0065] The search box's range (X, y) in image B is dynamically calculated based on the top-left corner coordinates (x, y), width w, and height h of the smallest bounding rectangle in image B. min :X max ,Y min :Y max The algorithm also incorporates the size W of the square matching box, which solves the problem that the search box size needs to be set manually in the traditional DIC algorithm and that the search box positioning fails when there is a large instantaneous displacement. It can also flexibly construct matching boxes of different scales according to the different sizes of the target in the image.

[0066] Step 5, Integer Pixel Search: If target i is successfully matched in step 4, construct a square template matching box u with side length W, centered on the monitoring point of target i in image A, and construct a matching box u with a range of (X) in image B. min :X max, Y min :Y max For each pixel in the search box, a matching box v with side length W is constructed in image B. The correlation coefficient R between the two matching boxes is calculated. All pixels in the search box are traversed. The pixel corresponding to the maximum value of the correlation coefficient R is the integer pixel position of the target i monitoring point in image B.

[0067] Furthermore, the correlation coefficient R between the two matching boxes is calculated, as shown in formula (2):

[0068]

[0069] In the formula: R is the correlation coefficient; The average grayscale value of the template matching box; The average gray value of the search matching box is denoted as u(x,y); the average gray value of the template matching box at point (x,y) is denoted as u(x,y); and the average gray value of the search matching box at point (x,y) is denoted as v(x,y).

[0070] Step 6, Subpixel Search: Based on the integer pixel position coordinates of the target i monitoring point in image B obtained in Step 5, obtain the coordinates (X,Y) and correlation coefficient R of all points (nine points) in the eight-neighbor pixel matrix of the pixel point. Based on these nine points (X,Y,R), construct a set of three-dimensional data points and use the data point set to fit a quadratic correlation surface. By solving the extreme points of the surface, obtain the subpixel coordinates of the target i monitoring point.

[0071] Furthermore, the equation of the quadratic correlation surface is shown in formula (3):

[0072] R(x,y)=a0+a1x+a2y+a3x 2 +a4xy+a5y 2 (3)

[0073] In the formula: R(x,y) is the correlation coefficient at any image coordinate point (x,y), and a0~a5 are surface coefficients.

[0074] Step 7, Real Coordinate Transformation: Based on the homography transformation matrix H corresponding to each target plane obtained in Step 3, the sub-pixel coordinates of the monitoring point of target i are transformed into world coordinates.

[0075] Furthermore, the sub-pixel coordinates are transformed into world coordinates using formula (4):

[0076]

[0077] In the formula: [X,Y,1] T Let [x, y, 1] be the position of any point in the plane containing target i in the image coordinate system; T Let (X,Y) be the coordinates of the point in the world coordinate system.

[0078] H is the homography matrix; s is the scaling factor.

[0079] Step 8, Displacement Calculation: Calculate the actual displacement value of target i at this time based on the world coordinates in Step 7.

[0080] Furthermore, the actual displacement value is calculated using formula (5):

[0081]

[0082] In the formula: Δx B Let Δy be the displacement of target i in the X direction at time B in image B; B Δs represents the Y-direction displacement of target i at time B in image B; Δs represents the total displacement of target i at time B in image B. Let i be the true coordinates of the target i monitoring point in image A; Let be the true coordinates of the target i monitoring point in image B.

[0083] Furthermore, in step 8, if target i is lost or temporarily occluded at time B in image B, the position of target i at time B is roughly estimated by formula (6), and then the actual displacement value of target i is calculated according to formula (5). In the programming, an error log with the words "target i is lost or temporarily occluded" is returned until target i is successfully matched again in subsequent images, and the original calculation process is restored.

[0084]

[0085] In the formula: The calculated true coordinates of the target i monitoring point in image B; The true coordinates of the target i monitoring point in the previous image of image B; Let be the average displacement velocity of target i in the X direction during the previous time interval; ΔT represents the average displacement velocity of target i in the Y direction during the previous time interval; ΔT is the time interval between image B and the previous image.

[0086] like Figure 3 As shown, a remote monitoring system includes:

[0087] The image acquisition module includes a network camera (model: Dahua Network Camera DH-IPC-HF8843F-ZRL, resolution: 3840×2160, lens focal length: 10mm-40mm), a lighting device, and a self-made target device. The self-made target device is installed in the deformation monitoring area, with one device every 5m, for a total of 5 devices, covering an area of ​​20m. The deformation of the soil and rock structure is characterized by monitoring the displacement of the target. The network camera is installed in a stable area 30m away from the deformation monitoring area. A solid circular column is poured with concrete, and the camera is fixed to the top of the column to capture images of the target, ensuring that the camera's field of view covers the entire target area. The lighting device is fixed to the network camera and also illuminates the monitoring area at night. The light is turned on or off via a relay.

[0088] Furthermore, the structure of the self-made target device is as follows: Figure 4 As shown, it is made of steel and is shaped like a right-angled triangular prism with open sides to reduce the weight of the device. The remaining three complete surfaces (bottom surface DCEF, vertical front surface ABDC, and oblique back surface ABFE) are 20mm thick to allow the target structure to withstand the effects of strong winds. The bottom surface DCEF contains four anchor holes to anchor the target device to the ground. The vertical front surface ABDC is perpendicular to the ground, and the target image is engraved on the vertical front surface ABDC. Its vertical displacement at this time represents the ground subsidence at this location.

[0089] The image transmission module includes a smart gateway and a cloud server. The smart gateway connects directly to the network camera and relay via a network cable, and also connects to the network via both a 4G card and a network cable. The smart gateway is located on a local area network (LAN), while the cloud server is located on the public network. Through Fast Reverse Proxy (FRP) intranet penetration technology, the ports controlling the network camera's image capture, focus adjustment, and relay switching are mapped to corresponding ports on the public network, establishing a data transmission channel. The smart gateway carries the FRP client program, which can automatically start after power-on and network connection. The cloud server carries the FRP server program, and the data download bandwidth needs to be >10M / s.

[0090] Furthermore, the smart gateway uses the OpenWRT system and carries the FRP client program. It can automatically start the FRP client program after power-on and network connection, ensuring that the information transmission channel can immediately restore connection in the event of a temporary power outage or network interruption, thereby achieving stability for long-term monitoring. The cloud server uses the Ubuntu system and carries the FRP server program, with a data download bandwidth of 10M / s. Assuming in this embodiment, the local area network address and port controlling the camera's automatic image capture, focus adjustment, and relay switching are 192.168.1. The public IP address of the cloud server is 128.224.57.102. After NAT traversal, the ports for the camera's automatic image capture, focus adjustment, and relay switch are mapped to 128.224.57.102:37777, 128.224.57.102:7100, and 128.224.57.102:1000, respectively. Other remote LANs can then access these addresses and ports to control the camera and relay.

[0091] The deformation calculation and visualization module includes a deformation calculation server, deformation calculation software, and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform, which are used for online monitoring, visualization, and early warning of deformation of soil and rock structures.

[0092] Furthermore, the analysis process of the deformation calculation and visualization module is as follows: Figure 5 As shown, the specific details are as follows:

[0093] Process 1): The deformation calculation software connects to the mapped public network port, controls the network camera to capture initial images, loads them onto the software page, and saves the images to a designated image folder (hereinafter referred to as Folder A). The center point of each target in the initial image is selected as a monitoring point using the mouse, and labels are added to the monitoring points according to the numbers at the target centers. Based on this, the method for obtaining the calibration point positions is set according to the size of the calibration circle in the image for each target, and the range of the calibration circle or the calibration point is selected directly according to this method. Simultaneously, the software automatically generates a txt format data file for each monitoring point in Folder A to save the calculation results, and also automatically generates the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels. Then, an image is automatically captured every N seconds and saved to Folder A. Next, the deformation calculation software searches all images in Folder A and adds the most recently captured image to the software's list of images to be analyzed.

[0094] Specifically, before each image capture, the control relay closes to turn on the lighting; after the image capture is complete, the relay opens to turn off the lighting. Captured images are automatically named based on the capture time, with the basic format being: "year-month-day & hour-minute-second.bmp". Each data table in the database contains six fields: ID (int), current time (datetime), time interval from the previous image (double), X-direction displacement (double), Y-direction displacement (double), and total displacement (double). "ID" represents the number of images currently being analyzed; "time interval from the previous image" is in seconds; and "X-direction displacement," "Y-direction displacement," and "total displacement" are in mm.

[0095] Process 2): The deformation analysis software uses the built-in deformation measurement algorithm to perform deformation analysis on the newly added image, and obtains the X-direction displacement value, Y-direction displacement value, total displacement value and error log of each target at the current moment. At this time, the software saves the calculation results of each monitoring point to the corresponding data file under folder A, and at the same time saves the calculation results of each monitoring point to the corresponding data table in the database. The deformation curve built into the software will be updated synchronously.

[0096] Specifically, the deformation measurement algorithm adopts the computer vision-based soil and rock deformation measurement method provided by this invention. After calculating the deformation in the current analysis image, the program immediately calculates the next image in the analysis list. If there are no remaining images in the analysis list, the program pauses until a new image is loaded into the analysis list and then restarts. After obtaining a new set of deformation data, for different targets, the program constructs strings and lists according to six data: "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement", and "error log", and saves them to data files and data tables. The "error log" only contains two types: "none" and "target lost or temporarily occluded", which correspond to the successful and unsuccessful target matching in the computer vision-based soil and rock deformation measurement method, respectively.

[0097] Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, it is connected to the data table corresponding to each measuring point in the database. Deformation alarm thresholds are set according to different engineering needs. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, the new data is loaded into the platform for visualization display, and risk warnings are issued according to the set displacement thresholds.

[0098] In the implementation of the remote measurement system, the self-made target devices can be installed on the roadside slope in two rows, one row in the middle of the slope and the other row at the top of the slope. Each row has three target devices, with each target spaced five meters apart. The network camera is installed on the lampposts opposite the road, aimed at the monitoring area and the focus is adjusted so that the camera can cover the entire target coverage area.

Claims

1. A method for measuring soil and rock deformation based on computer vision, characterized in that, Includes the following steps: Step 1: Construction of target dataset: Obtain target images in indoor simulation and actual scenarios, construct the minimum bounding rectangle to label the target images, and use the inner ring number of the target as its category label. Complete the construction of the target dataset through data augmentation, and then divide it into training set and validation set according to the preset ratio. Step 2, Model Training: A deep learning target detection model is used to train the target dataset. Cross-entropy loss and DFL+CIOU loss are jointly optimized for target classification and localization to obtain the target recognition model. Step 3, Image calibration and monitoring point selection: In a set of time-series deformed images containing targets, the first image in this set of time-series images is used as the reference image, and any other image is used as the displacement analysis image. The monitoring points and calibration points of each target in the reference image are determined. The plane where the target is located is set to the plane with Z=0 in the world coordinate system. The unit deformation matrix H of the plane where each target is located is calculated by using the pixel coordinates of the four calibration points and the world coordinates. Step 4: Target Matching and Search Parameter Determination: Utilize the target recognition model to obtain the minimum bounding rectangle and category of the target in the time-series deformation image. Based on the category, match the targets in the reference image with the targets in the displacement analysis image. If a match is successful, it indicates the existence of identical targets. i The target is then dynamically calculated based on the minimum bounding rectangle. i The search box range (X) used for DIC calculation in the displacement analysis image. min X max, Y min Y max If the target is matched with the square bounding box size W, then the target is considered to be within the range of displacement analysis images taken at that time. i Lost or temporarily obscured; Step 5, Integer Pixel Search: If the target in Step 4 i Match successful, target in the baseline image i Using the monitoring point as the center, construct a square template matching box with a side length of W. u And construct a range of (X) in the displacement analysis image. min X max, Y min :Y max For each pixel within the search box, a matching search box with side length W is constructed in the displacement analysis image. v Calculate the correlation coefficient between the two matching boxes. R ; Iterate through all pixels within the search box and calculate the relevance coefficient. R The pixel that corresponds to the maximum value is the target. i The monitoring point is located at an integer pixel position in the displacement analysis image; Step 6, Subpixel Search: Based on the target obtained in Step 5 i The monitoring point is located at the integer pixel position coordinates of the displacement analysis image. The coordinates (X, Y) and correlation coefficient R of nine points in the eight-neighbor pixel matrix of that pixel are obtained. A set of three-dimensional data points is constructed based on these nine points (X, Y, R), and this data point set is used to fit a quadratic correlation surface. The target is obtained by solving for the extreme points of the surface. i Subpixel coordinates of the monitoring point; Step 7, Real Coordinate Transformation: Based on the homography transformation matrix H obtained in Step 3 for each target plane, transform the target... i The sub-pixel coordinates of the monitoring points are converted into world coordinates; Step 8, Displacement Calculation: Calculate the target displacement at this point based on the world coordinates from Step 7. i The actual displacement value.

2. The method for measuring soil and rock deformation based on computer vision according to claim 1, characterized in that, In step 1, the target image is designed as a circle, divided into an outer ring (2) and an inner ring. The inner ring is the main body of the target, including a background (1), lines (3), calibration circles (4), dots (5), and numbers (6). The area of ​​the background (1) is the same as that of the main body of the target. The dots (5) are monitoring points set at the center of the main body of the target. The numbers (6) are located at the center of the main body of the target and are used to distinguish multiple targets. There are four lines (3), all of which pass through the dots and have a length that is exactly the diameter of the main body of the target, which are used to increase the texture complexity of the background (1). There are four calibration circles (4), which are used for the calibration of the target plane. The center of the circle is the calibration point located at the two ends of the two vertically intersecting lines.

3. The method for measuring soil and rock deformation based on computer vision according to claim 2, characterized in that, In step 3, the pixel positions of the calibration points are obtained through two methods: when the radius of the calibration circle in the time-series deformed image is ≥10 pixels, the range of four calibration circles is manually constructed, and then the Hough circle detection method is used to detect the center of these four calibration circles to obtain the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10 pixels, the center of the calibration circle is manually selected as the calibration point.

4. The method for measuring soil and rock deformation based on computer vision according to claim 1, characterized in that, In step 4, when the reference image and the target i in the displacement analysis image are successfully matched, the range of the search box in the displacement analysis image (Xmin: Xmax, Ymin: Ymax) and the size W of the square matching box adapted to targets of different sizes are dynamically calculated, as shown in formula (1): ； In the formula: (x, y) are the coordinates of the top-left corner of the smallest bounding rectangle; w The width of the minimum bounding rectangle; h The height of the minimum bounding rectangle; n X is the scaling factor. min , X max These represent the two boundaries of the search box in the X direction and Y direction of the displacement analysis image, respectively. min , Y max These represent the two boundaries of the search box in the Y direction of the displacement analysis image, respectively, and W is the side length of the square matching box; In step 5, the correlation coefficient between the two matching boxes is calculated. R As shown in formula (2): ； In the formula: R is the correlation coefficient; The average grayscale value of the template matching box; To retrieve the average grayscale value of the matching box; u (x, y) Match the template box in ( x , y The average gray value of the point; v ( x , y ) represents the average gray value of the search matching box at point (x, y); In step 6, the equation of the quadratic correlation surface is shown in formula (3): ； In the formula: For any image coordinate point ( x , y The correlation coefficient at position ) ~ For surface coefficients; In step 7, the sub-pixel coordinates are converted to world coordinates using formula (4): ； In the formula: (X, Y) represents the target. i The position of any point in the plane in the image coordinate system; x , y The point (X, Y) is projected onto the Z=0 plane in the world coordinate system (target). i Coordinates of a plane in the world coordinate system; H is the homography matrix; s It is a scaling factor; In step 8, the actual displacement value is calculated using formula (5): ； In the formula: For target i The X-direction displacement of the monitoring point at the moment of the displacement analysis image; For target i The displacement of the monitoring point in the Y direction at the moment of the displacement analysis image; For target i The total displacement of the monitoring point at the moment of the displacement analysis image; , ) as target i The actual coordinates of the monitoring point in the baseline image; ( , ) as target i The actual coordinates of the monitoring point in the displacement analysis image.

5. The method for measuring soil and rock deformation based on computer vision according to claim 4, characterized in that, In step 8, if the target i In the displacement analysis image, there are instances of data loss or temporary occlusion. The target is roughly estimated using formula (6). i The target position is determined at the moment of displacement analysis image, and then calculated according to formula (5). i The actual displacement value is returned in the program as the "target". i Error logs displaying the message "missing or temporarily occluded" appeared until the target was visible in subsequent images. i Once the rematch is successful, resume the original calculation process; ； In the formula: ( , The calculated target i The actual coordinates of the monitoring point in the displacement analysis image; ( , ) as target i The true coordinates of the monitoring point in the previous image of the displacement analysis image; For target i The average displacement velocity in the X direction during the previous time interval; For target i The average displacement velocity in the Y direction during the previous time interval; This represents the time interval between the displacement analysis image and the previous image.

6. A remote monitoring system, characterized in that, To implement the computer vision-based method for measuring soil and rock deformation as described in any one of claims 1-5, the method comprises: The image acquisition module includes a network camera, a lighting device, and a self-made target device. The self-made target device is installed in the deformation monitoring area, and the network camera is installed in a stable area away from the deformation monitoring area to capture images of the target and ensure that the camera's field of view covers the entire target arrangement area. The lighting device is fixed to the network camera and also illuminates the monitoring area at night. The light is turned on or off via a relay. The image transmission module includes a smart gateway and a cloud server. The smart gateway connects directly to the network camera and relay via Ethernet cable, and also connects to the network via both 4G SIM card and Ethernet cable. The smart gateway is located on a local area network (LAN), while the cloud server is located on the public network. Using Fast Reverse Proxy (FRP) intranet penetration technology, the ports controlling network camera image capture, focus adjustment, and relay switching are mapped to corresponding ports on the public network, establishing a data transmission channel. The smart gateway carries the FRP client program, which automatically starts upon power-on and network connection. The cloud server carries the FRP server program, requiring a data download bandwidth of >10M / s. The deformation calculation and visualization module includes a deformation calculation server, deformation calculation software, and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform, which are used for online monitoring, visualization, and early warning of deformation of soil and rock structures.

7. A remote monitoring system according to claim 6, characterized in that, The homemade target device is made of steel and is shaped like a right-angled triangular prism with hollowed-out sides. The remaining three complete surfaces—the bottom, the vertical front, and the oblique back—are 20mm thick. The bottom surface contains four anchor holes, and the vertical front is perpendicular to the ground. The target image is engraved on the vertical front, and its vertical displacement represents the ground subsidence at that location.

8. A remote monitoring system according to claim 6, characterized in that, In the deformation calculation and visualization module, the specific analysis process for deformation calculation and visualization is as follows: Process 1): The deformation calculation software connects to the mapped public network port, controls the network camera to capture initial images, loads them onto the software page, and saves the images to the designated folder A. The center point of each target in the initial image is selected as a monitoring point using the mouse, and labels are added to the monitoring points according to the numbers at the target centers. Based on this, the method for obtaining the calibration point positions is set according to the size of the calibration circle in each target in the image, and the selection is made to construct the calibration circle range or directly select the calibration point. Simultaneously, the software automatically generates a txt format data file for each monitoring point in folder A to save the calculation results, and also automatically generates the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels. Then, an image is automatically captured every N seconds and saved to folder A. Next, the deformation calculation software searches all images in folder A and adds the latest captured image to the software's analysis list. Process 2): The deformation analysis software uses the built-in deformation measurement algorithm to perform deformation analysis on the newly added image, and obtains the X-direction displacement value, Y-direction displacement value, total displacement value and error log of each target at the current moment. At this time, the software saves the calculation results of each monitoring point to the corresponding data file under folder A, and at the same time saves the calculation results of each monitoring point to the corresponding data table in the database. The deformation curve built into the software will be updated synchronously. Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, it is connected to the data table corresponding to each measuring point in the database. Deformation alarm thresholds are set according to different engineering needs. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, the new data is loaded into the platform for visualization display, and risk warnings are issued according to the set displacement thresholds.

9. A remote monitoring system according to claim 8, characterized in that, In process 1), before each image capture, the control relay closes to turn on the lighting. After the image capture is completed, the relay opens to turn off the lighting. The captured images are automatically named according to the capture time, and the basic format of the image name is: "year-month-day & hour-minute-second.bmp". Each data table in the database contains seven fields: ID (int), current time (datetime), interval time with the previous image (double), X-direction displacement (double), Y-direction displacement (double), total displacement (double), and error log (varchar). Among them, "ID" represents the number of images currently being analyzed, "interval time with the previous image" is in seconds, "X-direction displacement", "Y-direction displacement", and "total displacement" are in mm, and "error log" contains error information that occurs during the analysis process.

10. A remote monitoring system according to claim 8, characterized in that, In process 2), the deformation measurement algorithm adopts the computer vision-based soil and rock deformation measurement method described in claim 1. After calculating the deformation of the current analysis image, the next image in the analysis list is immediately calculated. If there are no remaining images in the analysis list, the program pauses until a new image is loaded into the analysis list and then restarts. After obtaining a new set of deformation data, for different targets, the program will construct strings and lists according to six data: "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement" and "error log", and save them to the data file and data table. The "error log" only contains two types: "none" and "target lost or temporarily occluded", which correspond to the successful and unsuccessful target matching in the computer vision-based soil and rock deformation measurement method, respectively.

Images