Rock deformation measuring method based on computer vision and remote monitoring system

Abstract

The invention discloses a rock-soil deformation measurement method and remote monitoring system based on computer vision, and the deformation measurement method comprises the steps: construction of a target data set, model training, image calibration and monitoring point selection, target matching and search parameter determination, integer pixel search, sub-pixel search, real coordinate conversion, and displacement calculation. The advantages of deep learning rapid positioning and DIC accurate searching are fused, the problems that a traditional DIC method loses efficacy in a severe environment and instantaneous large deformation represented by surface collapse is difficult to track are solved, and monitoring continuity under the temporary shielding condition is kept; the remote monitoring system is composed of an image acquisition module, an image transmission module and a deformation calculation and visual display module, online monitoring and management of rock-soil structure deformation are realized through deformation calculation software and a visual management platform, the cost and deployment difficulty of deformation monitoring are reduced, the accuracy and efficiency of deformation calculation are improved, and the system is suitable for popularization and application. The universality is high, and engineering popularization is easy.

Description

Technical Field

[0001] The present invention belongs to the technical field of geotechnical engineering deformation monitoring, and in particular relates to a geotechnical deformation measurement method and a remote monitoring system based on computer vision. Background Art

[0002] During engineering construction and geological disaster investigation, it is necessary to accurately observe the degree of change in rock and soil structures such as the surface and slopes, statistically update the geographic information database, and provide decision support for engineering construction management and emergency rescue.

[0003] Traditionally, leveling techniques have been used to monitor deformation in geotechnical structures. While this technique offers high measurement accuracy and ease of operation, it suffers from limitations such as heavy fieldwork, slow speed, low coverage density, and difficulty accurately capturing the spatial distribution of settlement. The development of camera sensors has also led to rapid advancements in photogrammetry. By photographing the object being measured, the images captured by an optical camera are digitally processed to reveal its shape, size, position, characteristics, and their interrelationships. Visual measurement, as a modern, non-contact optical measurement technique, offers high measurement accuracy, long-range multi-point measurement, high frequency, and full-field measurement capabilities. Currently, leading visual measurement algorithms include feature point matching, optical flow estimation, and digital image correlation (DIC). DIC searches for monitoring points by matching correlations across regions. Using interpolation, it can refine resolution to sub-image levels, offering high accuracy, adaptability, and reliability. However, current visual measurement methods still have limitations: they are greatly affected by environmental interference such as lighting, occlusion, shadows, rain and fog, which can easily lead to monitoring failure and make it impossible to maintain stable monitoring. Therefore, it is difficult to ensure the stability of long-term monitoring in natural environments; and most monitoring research only focuses on improving monitoring algorithms, and has not formed a mature and reliable system. Summary of the Invention

[0004] The purpose of the present invention is to provide a rock and soil deformation measurement method and remote monitoring system based on computer vision, which can effectively solve the problems of existing visual measurement methods such as being greatly disturbed by harsh environments, difficulty in monitoring instantaneous large deformations, poor stability and immature systems.

[0005] To achieve the above object, the present invention provides a rock and soil deformation measurement method based on computer vision, comprising the following steps:

[0006] Step 1: Target dataset construction: Obtain target images in indoor simulation and actual scenes, construct a minimum enclosing rectangle to annotate the target images, and use the inner ring number of the target as its category label. Complete the construction of the target dataset through data expansion, and then divide it into training set and validation set according to the preset ratio;

[0007] Step 2: Model training: Use a deep learning target detection model to train the target dataset. For target classification and positioning, use cross entropy loss and DFL+CIOU loss to perform joint optimization to obtain a target recognition model.

[0008] Step 3: Image calibration and monitoring point selection: In a set of time-series deformed images containing targets, the first image in this set of time-series images is used as the reference image, and any other image is used as the displacement analysis image. The monitoring points and calibration points of each target in the reference image are determined, and the plane where the target is located is set to the plane with Z = 0 in the world coordinate system. The unit transformation matrix H of the plane where each target is located is calculated by using the pixel coordinates of the four calibration points and the world coordinates;

[0009] Step 4: Target matching and search parameter determination: The target recognition model is used to obtain the minimum bounding rectangle and category of the target in the time-series deformation image. The target in the reference image is matched with the target in the displacement analysis image according to the category. If the match is successful and the same target i exists, the search box range (X min :X max, Y min :Y max ) matches the square frame size W, otherwise it is considered that the target i is lost or temporarily blocked when the displacement analysis image is taken;

[0010] Step 5, whole pixel search: If the target i is successfully matched in step 4, a square template matching box u with a side length of W is constructed with the monitoring point of target i in the reference image as the center, and a range of (X min :X max, Y min :Y max ), with any pixel point in the search box as the center, similarly construct a retrieval matching box v with a side length of W in the displacement analysis image, and calculate the correlation coefficient R between the two matching boxes; traverse all pixels in the search box, and the pixel point corresponding to the maximum value of the correlation coefficient R is the integer pixel position of the target i monitoring point in the displacement analysis image;

[0011] Step 6, Sub-pixel Search: Based on the integer pixel position coordinates of target i monitoring point in the displacement analysis image obtained in step 5, obtain the coordinates (X, Y) and correlation coefficient R of nine points in the eight-neighborhood pixel matrix of the pixel point. Based on these nine points (X, Y, R), a set of three-dimensional data points is constructed. This data point set is used to fit a quadratic correlation surface, and the sub-pixel coordinates of target i monitoring point are obtained by solving the extreme points of the surface.

[0012] Step 7, real coordinate conversion: According to step 3, the homography transformation matrix H corresponding to each target plane is obtained, and the sub-pixel coordinates of the target i monitoring point are converted into world coordinates;

[0013] Step 8: Displacement calculation: Calculate the actual displacement value of target i at this time based on the world coordinates in step 7.

[0014] As a further solution of the present invention: in step 1, the target image is designed to be circular, divided into an outer ring and an inner ring. The inner ring is the target body, including background, lines, calibration circles, dots, and numbers; the background area is equal to the target body; the dots are monitoring points, set at the center of the target body; the numbers are located in the center of the target body, which are used to distinguish multiple targets; there are four lines, all of which pass through the dots and have a length just equal to the diameter of the target body, which are used to increase the complexity of the background texture; there are four calibration circles in total, which are used for calibrating the target plane, and the center of the circle is the calibration point located at both ends of the upper and lower vertical intersecting lines.

[0015] As a further solution of the present invention: in step 3, the pixel position of the calibration point is obtained by two methods: when the radius of the calibration circle in the time series deformed image is ≥10piexls, the range of four calibration circles is manually constructed, and then the center of the four calibration circles is detected by the Hough circle detection method to obtain the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10piexls, the center of the calibration circle is manually selected as the calibration point.

[0016] As a further solution of the present invention: in step 4, when the reference image successfully matches the target i in the displacement analysis image, the range (Xmin:Xmax, Ymin:Ymax) of the search box in the displacement analysis image and the size W of the square matching box adapted to targets of different sizes are dynamically calculated, as shown in formula (1):

[0017]

[0018] Where: (x, y) is the coordinate of the upper left corner of the minimum bounding rectangle; w is the width of the minimum bounding rectangle; h is the height of the minimum bounding rectangle; n is the scaling factor, X min ,X max They represent the two boundaries of the search box in the X direction of the displacement analysis image, and the Y min ,Y max They represent the two boundaries of the search box in the Y direction of the displacement analysis image, and W is the side length of the square matching box;

[0019] In step 5, the correlation coefficient R between the two matching boxes is calculated as shown in formula (2):

[0020]

[0021] R is a correlation coefficient; is the average gray value of the template matching frame; is the average gray value of the search matching frame; u(x, y) is the average gray value of the template matching frame at the point (x, y); v(x, y) is the average gray value of the search matching frame at the point (x, y);

[0022] In step 6, the equation of the quadratic correlation surface is shown in formula (3):

[0023] R(x, y) = a0 + a1x + a2y + a3x 2 +a4xy + a5y 2 (3)

[0024] R(x, y) is the correlation coefficient at any image coordinate point (x, y), and a0-a5 are surface coefficients;

[0025] In step 7, the sub-pixel coordinates are converted into world coordinates by formula (4):

[0026]

[0027] (X, Y) is the position of any point in the plane of target i in the image coordinate system; (x, y) is the coordinate of the point (X, Y) projected to the Z=0 plane (the plane of target i in the world coordinate system) in the world coordinate system;

[0028] H is a homography matrix; s is a scale factor;

[0029] In step 8, the real displacement value is calculated by formula (5):

[0030]

[0031] Δx B is the X-direction displacement of the monitoring point of target i at the time of the displacement analysis image; Δy B is the Y-direction displacement of the monitoring point of target i at the time of the displacement analysis image; Δs is the total displacement of the monitoring point of target i at the time of the displacement analysis image; is the real coordinate value of the monitoring point of target i in the reference image; is the real coordinate value of the monitoring point of target i in the displacement analysis image.

[0032] As a further solution of the present invention: in step 8, if target i is lost or temporarily blocked at the moment of the displacement analysis image, the position of target i at the moment of the displacement analysis image is roughly estimated by formula (6), and then the true displacement value of target i is calculated according to formula (5), and an error log with the words "target i is lost or temporarily blocked" is returned in the programming until target i is successfully re-matched in the subsequent image, and the original calculation process is resumed;

[0033]

[0034] Where: is the calculated true coordinate value of the target i monitoring point in the displacement analysis image; is the real coordinate value of the target i monitoring point in the previous image of the displacement analysis image; is the average displacement velocity of target i in the X direction during the previous interval; is the average displacement velocity of target i in the Y direction during the previous interval; ΔT is the interval between the displacement analysis image and the previous image.

[0035] To achieve the above object, the present invention further provides a remote monitoring system, comprising:

[0036] The image acquisition module includes a network camera, a lighting lamp, and a self-made target device. The self-made target device is installed in the deformation monitoring area, and the network camera is installed in a stable area away from the deformation monitoring area. It is aimed at the target to take pictures, and the camera's field of view covers the entire target arrangement range. The lighting lamp is fixed to the network camera and also aims at the monitoring area to illuminate the night, and is controlled to turn on or off by a relay.

[0037] The image transmission module includes an intelligent gateway and a cloud server. The intelligent gateway is directly connected to the network camera and relay via a network cable, and is connected to the network via a 4G card and a network cable. The intelligent gateway is located in the local area network, and the cloud server is located in the public network. Through the fast reverse proxy (FRP) intranet penetration technology, the ports that control the network camera's image capture, focus adjustment, and relay switching are mapped to the corresponding ports on the public network, establishing a data transmission channel. The intelligent gateway is equipped with an FRP client program and can automatically start the FRP client program after powering on and connecting to the network. The cloud server is equipped with an FRP server program, and the data download bandwidth must be greater than 10M / s.

[0038] The deformation calculation and visualization display module includes a deformation calculation server, deformation calculation software and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform to respectively perform online monitoring, visualization display and early warning of geotechnical structure deformation.

[0039] As a further solution of the present invention: the homemade target device is made of steel material, and its shape is a right-angled triangular prism with hollowed-out sides. The thickness of the remaining three complete surfaces, namely the bottom surface, the vertical front surface, and the oblique back surface, is set to 20 mm. The bottom surface contains four anchor holes, and the vertical front surface is perpendicular to the bottom surface. The target image is engraved on the vertical front surface. At this time, its vertical displacement represents the surface subsidence here.

[0040] As a further solution of the present invention: in the deformation calculation and visualization display module, the specific analysis process of deformation calculation and visualization display is as follows:

[0041] Process 1): The deformation calculation software controls the network camera to capture the initial image by connecting to the mapped public network port, loads it into the software page, and saves the image to the specified folder A; click the center point of each target in the initial image with the mouse as the monitoring point, and add a label to the monitoring point according to the number of the target center. On this basis, according to the size of the calibration circle of each target in the image, set the method for obtaining the calibration point position, and choose to construct the calibration circle range or directly select the calibration point according to the acquisition method; At the same time, the software will automatically generate a txt format data file for each monitoring point in folder A to save the calculation results, and will also automatically generate the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels; Then, an image is automatically captured every N seconds and saved in folder A; Then, the deformation calculation software retrieves all images in folder A and adds the latest captured image to the software's list to be analyzed;

[0042] Process 2): The deformation analysis software uses its built-in deformation measurement algorithm to perform deformation analysis on the newly added image. The software obtains the current X- and Y-direction displacement values ​​of each target, as well as the total displacement value and error log. The software then saves the calculation results for each monitoring point to the corresponding data file in folder A and to the corresponding data table in the database. The built-in deformation curve in the software is also updated synchronously.

[0043] Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, a connection is established with the data table corresponding to each measurement point in the database. Deformation alarm thresholds are set according to different project requirements. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, it is loaded into the platform for visualization and risk warning is issued according to the set displacement threshold.

[0044] As a further solution of the present invention: in process 1), before each image is captured, the control relay is closed to turn on the lighting lamp, and after the image capture is completed, the relay is disconnected to turn off the lighting lamp; the captured image is automatically named according to the shooting time, and the basic format of the image name is: "year-month-day&hour-minute-second.bmp"; each data table in the database contains seven fields: ID (int), current time (datatime), interval time with the previous image (double), X-direction displacement (double), Y-direction displacement (double), total displacement (double), and error log (varchar); among which "ID" represents the number of images currently being analyzed, "interval time with the previous image" is in seconds, "X-direction displacement", "Y-direction displacement", and "total displacement" are in mm, and "error log" is the error information generated during the analysis process.

[0045] As a further solution of the present invention: in process 2), the deformation measurement algorithm adopts the geotechnical deformation measurement method based on computer vision described in claim 1; after calculating the deformation of the current analysis image, the next image in the list to be analyzed is calculated immediately. If there are no remaining images in the list to be analyzed, the program is suspended until a new image is loaded into the list to be analyzed and the program is restarted; after obtaining a new set of deformation data, for different targets, the program will construct a character string and a list according to the six data of "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement" and "error log", and save them to the data file and data table, among which the "error log" only includes two types: "none" and "target lost or temporarily blocked", which correspond to the successful and unsuccessful target matching in the geotechnical deformation measurement method based on computer vision.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention gives full play to the advantages of non-contact monitoring by computer vision, and at the same time performs coarse search and fine search based on deep learning algorithm and DIC algorithm respectively, effectively improving the adaptability of computer vision algorithm to harsh environment, ensuring the monitoring continuity of temporary target occlusion, and can also monitor instantaneous large deformation represented by surface collapse; on this basis, the constructed remote monitoring system realizes stable and accurate long-term monitoring and remote and stable transmission of data. Under the bandwidth of 10M / s, when transmitting 3840×2160 resolution images, the image transmission speed exceeds 2 frames / second, and the transmission speed is fast; and it has the advantages of high precision, economy and reliability, strong adaptability to harsh environment, convenient operation, simple layout and easy promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1This is a flow chart of the rock and soil deformation measurement method based on computer vision in the present invention.

[0049] Figure 2 Schematic diagram of the target image format in the present invention.

[0050] Figure 3 This is a diagram of the implementation framework of the remote monitoring system in the present invention.

[0051] Figure 4 It is a structural schematic diagram of the homemade target device in the present invention.

[0052] Figure 5 It is an analysis flow chart of the deformation calculation and visualization display module in the present invention.

[0053] In the figure: 1. Background, 2. Outer ring, 3. Lines, 4. Calibration circle, 5. Dots, 6. Numbers. DETAILED DESCRIPTION

[0054] The present invention will be further described below by way of examples.

[0055] like Figure 1 As shown, a method for measuring rock and soil deformation based on computer vision includes the following steps:

[0056] Step 1. Construction of target dataset: A total of 400 target images in indoor simulation and actual scenes are obtained, covering three types of harsh environments: heavy fog, rain, and strong light, so that the trained model can better adapt to the harsh environment in the actual environment. The minimum enclosing rectangle is constructed and the LabelImg tool is used to annotate the target image, and the inner ring number of the target is used as its category label. In order to enhance the recognition performance of the model, the dataset is expanded to 5000 by means of random rotation of ±30°, random scaling of 0.7-1.5 times, random cropping, and random fusion of multiple images. The construction of the dataset is completed and divided into training set and validation set in a ratio of 4:1.

[0057] Furthermore, the target image is designed to be circular, divided into an outer ring 2 and an inner ring. The inner ring is the target body, including background 1, lines 3, calibration circles 4, dots 5, and numbers 6. The area of ​​background 1 is the same as that of the target body. Dot 5 is a monitoring point, set at the center of the target body. Number 6 is located in the center of the target body and is used to distinguish multiple targets. There are four lines 3, all of which pass through dot 4 and have a length that is exactly the diameter of the target body, which is used to increase the texture complexity of background 1. There are four calibration circles 4, which are used for calibration of the target plane. The center of the circle is the calibration point, which is located at the two ends of the upper and lower vertical intersecting lines. For convex display, such as Figure 2 As shown, the outer ring 2 and the line 3 are gray, the background 1 is black, the calibration circle 4 is blue, the calibration point is green, the dot 5 is red, and the number 6 is white.

[0058] Step 2. Model training: The YOLO V8 model is used to train the target dataset. The model used is a network structure that includes Backbone feature extraction, Neck feature fusion, and Head anchor-free box detection. Cross entropy loss and DFL+CIOU loss are used for joint optimization of target classification and positioning to obtain the target recognition model.

[0059] Step 3. Image calibration and monitoring point selection: In a set of time-series deformed images containing targets, the first image in this set of time-series images (hereinafter referred to as image A for convenience) is used as the reference image, and the center point of each target in image A is selected as the monitoring point for subsequent tracking. The center points of the four calibration circles in each target are selected as the calibration points of the target. The plane where the target is located is set to the plane with Z = 0 in the world coordinate system, and the center point of the target is taken as the center point of the plane. At this time, the pixel coordinates and world coordinates of these four calibration points are all known. The findHomography function in the opnecv image processing library can be used to calculate the homography transformation matrix H of the plane where each target is located for subsequent coordinate transformation; any subsequent image other than image A (hereinafter referred to as image B for convenience) is the displacement analysis image.

[0060] Furthermore, the pixel positions of the calibration points are obtained by two methods: when the radius of the calibration circle in the time-series deformed image is ≥10 piexls, the range of four calibration circles is manually constructed, and then the center of these four calibration circles is detected using the Hough circle detection method to obtain the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10 piexls, the center of the calibration circle is manually selected as the calibration point.

[0061] Step 4: Target matching and search parameter determination: The target recognition model is used to obtain the minimum bounding rectangle and category of the target in the time-series deformation image. The target in image A is matched with the target in image B according to the category. Taking any target in image A as an example, assuming that the category of the target is i, it is recorded as target i. If there is also a unique target of category i in image B, then the two are considered to be the same target. If the match is successful and the same target i exists, the search box range (X min :X max, Y min :Y max ) matches the square box size W, otherwise it is considered that the target i is lost or temporarily blocked when the image B is taken.

[0062] Furthermore, as shown in formula (1):

[0063]

[0064] Where: (x, y) is the coordinate of the upper left corner of the minimum bounding rectangle; w is the width of the minimum bounding rectangle; h is the height of the minimum bounding rectangle; n is the scaling factor, which can be flexibly set according to the size of the target; X min ,X max They represent the two boundaries of the search box in the X direction of the displacement analysis image, and the Y min ,Y max They represent the two boundaries of the search box in the Y direction of the displacement analysis image, and W is the side length of the square matching box.

[0065] The range of the search box in image B (X, Y) is dynamically calculated based on the coordinates of the upper left corner (x, y), width w, and height h of the minimum bounding rectangle in image B. min :X max ,Y min :Y max ) and the size W of the square matching box, which solves the problem that the search box size needs to be manually set in the traditional DIC algorithm and the search box positioning fails when large instantaneous displacement occurs. At the same time, it can also flexibly construct matching boxes of different scales according to the different sizes of the target in the image.

[0066] Step 5, whole pixel search: If the target i is successfully matched in step 4, a square template matching box u with a side length of W is constructed with the monitoring point of target i in image A as the center, and a range of (X min :X max, Y min :Y max ), with any pixel point in the search box as the center, also construct a retrieval matching box v with a side length of W in image B, and calculate the correlation coefficient R between the two matching boxes; traverse all pixels in the search box, and the pixel point corresponding to the maximum value of the correlation coefficient R is the integer pixel position of the target i monitoring point in image B.

[0067] Furthermore, the correlation coefficient R between the two matching boxes is calculated as shown in formula (2):

[0068]

[0069] Where: R is the correlation coefficient; is the average grayscale value of the template matching box; is the average grayscale value of the retrieval matching box; u(x,y) is the average grayscale value of the template matching box at point (x,y); v(x,y) is the average grayscale value of the retrieval matching box at point (x,y).

[0070] Step 6, sub-pixel search: based on the target i monitoring point in the image B of the whole pixel position coordinates obtained in step 5, the coordinates (X, Y) and the correlation coefficient R of all points (nine points) in the eight neighborhood pixel matrix of the pixel point are obtained, a set of three-dimensional data points is constructed based on the nine points (X, Y, R), and a quadratic correlation surface is fitted by using the data point set, and the sub-pixel coordinates of the target i monitoring point are obtained by solving the extreme points of the surface;

[0071] Further, the equation of the quadratic correlation surface is shown in formula (3):

[0072] R(x,y)=a0+a1x+a2y+a3x 2 +a4xy+a5y 2 (3)

[0073] In the formula, R(x,y) is the correlation coefficient at any image coordinate point (x,y), and a0-a5 are the surface coefficients.

[0074] Step 7, real coordinate conversion: according to the homographic transformation matrix H corresponding to each target plane obtained in step 3, the sub-pixel coordinates of the target i monitoring point are converted into world coordinates.

[0075] Further, the sub-pixel coordinates are converted into world coordinates by formula (4):

[0076]

[0077] In the formula, [X,Y,1] T is the position of any point in the target i plane in the image coordinate system; [x,y,1] T is the coordinate corresponding to the world coordinate system of the point (X,Y);

[0078] is the homographic matrix H; and s is a scale factor.

[0079] Step 8, displacement calculation: the real displacement value of the target i at this time is calculated according to the world coordinates in step 7.

[0080] Further, the real displacement value is calculated by formula (5):

[0081]

[0082] In the formula, Δx B is the X-direction displacement of the target i monitoring point at the time of image B; Δy B is the Y-direction displacement of the target i monitoring point at the time of image B; and Δs is the total displacement of the target i monitoring point at the time of image B; is the real coordinate value of the target i monitoring point in image A; is the real coordinate value of the monitoring point of target i in image B.

[0083] Furthermore, in step 8, if target i is lost or temporarily blocked at time B in the image, the position of target i at time B is roughly estimated using formula (6), and the true displacement value of target i is calculated according to formula (5). An error log indicating "target i is lost or temporarily blocked" is returned in the programming until target i is successfully re-matched in the subsequent image, and the original calculation process is resumed.

[0084]

[0085] Where: is the calculated true coordinate value of the monitoring point of target i in image B; is the real coordinate value of the monitoring point of target i in the previous image of image B; is the average displacement velocity of target i in the X direction during the previous interval; is the average displacement speed of target i in the Y direction during the previous interval; ΔT is the interval between image B and the previous image.

[0086] like Figure 3 As shown, a remote monitoring system includes:

[0087] The image acquisition module includes a network camera (model: Dahua network camera DH-IPC-HF8843F-ZRL, resolution: 3840×2160, lens focal length: 10mm-40mm), a lighting lamp and a self-made target device. The self-made target device is installed in the deformation monitoring area, one is arranged every 5 meters, and a total of 5 are arranged, covering a range of 20 meters. The deformation of the geotechnical structure is characterized by monitoring the displacement of the target. The network camera is installed in a stable area 30 meters away from the deformation monitoring area. A solid circular column is cast by concrete, and the camera is fixed on the top of the circular column. It is aimed at the target for shooting, and the camera's field of view covers the entire target arrangement range; the lighting lamp is fixed on the network camera and also aimed at the monitoring area for night lighting. The light is controlled to turn on or off by a relay.

[0088] Furthermore, the structure of the homemade target device is as follows Figure 4 As shown, it is made of steel material and has a right-angled triangular prism shape with hollowed-out sides to reduce the weight of the device. The thickness of the remaining three complete surfaces (bottom surface DCEF, vertical front surface ABDC, and oblique back surface ABFE) is set to 20 mm, so that the target structure can withstand the influence of strong winds. The bottom surface DCEF contains 4 anchor holes to anchor the target device to the ground. The vertical front surface ABDC is perpendicular to the ground, and the target image is engraved on the vertical front surface ABDC. At this time, its vertical displacement represents the surface settlement here.

[0089] The image transmission module includes an intelligent gateway and a cloud server. The intelligent gateway is directly connected to the network camera and relay via a network cable, and is connected to the network through two modes: a 4G card and a network cable. The intelligent gateway is located in the local area network, and the cloud server is located in the public network. Through the fast reverse proxy (FRP) intranet penetration technology, the ports for controlling the network camera's image capture, focus adjustment, and relay switch are mapped to the corresponding ports on the public network to establish a data transmission channel. Among them, the intelligent gateway is equipped with the FRP client program and can automatically start the FRP client program after booting and connecting to the network. The cloud server is equipped with the FRP server program, and the data download bandwidth must be greater than 10M / s.

[0090] Furthermore, the smart gateway uses the OpenWRT system and is equipped with an FRP client program. This program automatically starts upon power-up and connection to the internet, ensuring that the information transmission channel can be immediately restored in the event of a temporary power outage or network disconnection, thereby ensuring long-term monitoring stability. The cloud server uses the Ubuntu system and is equipped with an FRP server program, with a data download bandwidth of 10M / s. In this embodiment, the LAN address and port for controlling the camera's automatic image capture, focus adjustment, and relay switching are assumed to be 192.168.1. 2:37777, 192.168.1.2:7100, and 192.168.1.3:1000. The public IP of the cloud server is 128.224.57.102. After intranet penetration, the camera's automatic capture, focus adjustment, and relay switch ports are mapped to 128.224.57.102:37777, 128.224.57.102:7100, and 128.224.57.102:1000, respectively. At this time, other remote LANs can control the camera and relay by accessing these addresses and ports.

[0091] The deformation calculation and visualization display module includes a deformation calculation server, deformation calculation software and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform to respectively perform online monitoring, visualization display and early warning of geotechnical structure deformation.

[0092] Furthermore, the analysis process of the deformation calculation and visualization display module is as follows Figure 5 The specific details are as follows:

[0093] Process 1): The deformation calculation software controls the network camera to capture the initial image by connecting to the mapped public network port, loads it into the software page, and saves the image to the specified image folder (hereinafter referred to as folder A); click the center point of each target in the initial image with the mouse as the monitoring point, and add a label to the monitoring point according to the number of the target center. On this basis, according to the size of the calibration circle of each target in the image, the method for obtaining the calibration point position is set, and the calibration circle range is constructed or the calibration point is directly selected according to the method; at the same time, the software will automatically generate a txt format data file for each monitoring point in folder A to save the calculation results, and will also automatically generate the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels; then, an image is automatically captured every N seconds and saved in folder A; then, the deformation calculation software retrieves all images in folder A and adds the latest captured image to the software's list for analysis.

[0094] Specifically, before each image is captured, the control relay is closed to turn on the light. After the image is captured, the relay is disconnected to turn off the light. The captured image is automatically named according to the shooting time. The basic format of the image name is: "year-month-day&hour-minute-second.bmp". Each data table in the database contains six fields: ID (int), current time (datatime), time interval from the previous image (double), X-direction displacement (double), Y-direction displacement (double), and total displacement (double). Among them, "ID" represents the number of images currently being analyzed, "time interval from the previous image" is in seconds, and "X-direction displacement", "Y-direction displacement", and "total displacement" are in mm.

[0095] Process 2): The deformation analysis software uses its built-in deformation measurement algorithm to perform deformation analysis on the newly added image. The software obtains the current X- and Y-direction displacement values, the total displacement value, and the error log for each target. The software then saves the calculation results for each monitoring point to the corresponding data file in folder A. The software also saves the calculation results for each monitoring point to the corresponding data table in the database, and the built-in deformation curve in the software is updated synchronously.

[0096] Specifically, the deformation measurement algorithm adopts the geotechnical deformation measurement method based on computer vision provided by the present invention; after calculating the deformation in the current analysis image, the next image in the list to be analyzed is calculated immediately. If there are no remaining images in the list to be analyzed, the program is suspended until a new image is loaded into the list to be analyzed and the program is restarted; after obtaining a new set of deformation data, for different targets, the program will construct character strings and lists according to the six data of "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement" and "error log", and save them to data files and data tables, among which the "error log" only includes two types: "none" and "target lost or temporarily blocked", which correspond to the successful and unsuccessful target matching in the geotechnical deformation measurement method based on computer vision.

[0097] Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, a connection is established with the data table corresponding to each measurement point in the database. Deformation alarm thresholds are set according to different project requirements. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, it is loaded into the platform for visualization and risk warning is issued according to the set displacement threshold.

[0098] During the specific implementation of the remote measurement system, the self-made target device can be installed on the road slope in two rows, one row in the middle of the slope and one row at the top of the slope. There are three target devices in each row, and each target is five meters apart. The network camera is installed on the lamppost opposite the road, aimed at the monitoring area and the focal length is adjusted so that the camera can cover the entire target coverage range.

Claims

1. A rock and soil deformation measurement method based on computer vision, characterized in that: The following steps are involved: Step 1: Target dataset construction: Obtain target images in indoor simulation and actual scenes, construct a minimum enclosing rectangle to annotate the target images, and use the inner ring number of the target as its category label. Complete the construction of the target dataset through data expansion, and then divide it into training set and validation set according to the preset ratio; Step 2: Model training: Use a deep learning target detection model to train the target dataset. For target classification and positioning, use cross entropy loss and DFL+CIOU loss to perform joint optimization to obtain a target recognition model. Step 3. Image calibration and monitoring point selection: In a set of time-series deformed images containing targets, take the first image in this set of time-series images as the reference image and any other image as the displacement analysis image. Determine the monitoring points and calibration points of each target in the reference image, set the plane where the target is located at Z=0 in the world coordinate system, and calculate the unit transformation matrix H of the plane where each target is located through the pixel coordinates and world coordinates of the four calibration points. Step 4: Target matching and search parameter determination: The target recognition model is used to obtain the minimum bounding rectangle and category of the target in the time-series deformation image. The target in the reference image is matched with the target in the displacement analysis image according to the category. If the match is successful and the same target i exists, the search box range (X min :X max, Y min :Y max ) matches the square frame size W, otherwise it is considered that the target i is lost or temporarily blocked when the displacement analysis image is taken; Step 5, whole pixel search: If the target i is successfully matched in step 4, a square template matching box u with a side length of W is constructed with the monitoring point of target i in the reference image as the center, and a range of (X min :X max, Y min :Y max ), with any pixel point in the search box as the center, similarly construct a retrieval matching box v with a side length of W in the displacement analysis image, and calculate the correlation coefficient R between the two matching boxes; traverse all pixels in the search box, and the pixel point corresponding to the maximum value of the correlation coefficient R is the integer pixel position of the target i monitoring point in the displacement analysis image; Step 6, Sub-pixel Search: Based on the integer pixel position coordinates of target i monitoring point in the displacement analysis image obtained in step 5, obtain the coordinates (X, Y) and correlation coefficient R of nine points in the eight-neighborhood pixel matrix of the pixel point. Based on these nine points (X, Y, R), a set of three-dimensional data points is constructed. This data point set is used to fit a quadratic correlation surface, and the sub-pixel coordinates of target i monitoring point are obtained by solving the extreme points of the surface. Step 7, real coordinate conversion: According to step 3, the homography transformation matrix H corresponding to each target plane is obtained, and the sub-pixel coordinates of the target i monitoring point are converted into world coordinates; Step 8: Displacement calculation: Calculate the actual displacement value of target i at this time based on the world coordinates in step 7.

2. The method for measuring rock and soil deformation based on computer vision according to claim 1, characterized in that: In step 1, the target image is designed to be circular and divided into two parts: an outer ring (2) and an inner ring. The inner ring is the target body, including the background (1), lines (3), calibration circles (4), dots (5), and numbers (6). The area of ​​the background (1) is equal to that of the target body. The dot (5) is a monitoring point, which is set at the center of the target body. The number (6) is located at the center of the target body and is used to distinguish multiple targets. There are four lines (3) in total, all of which pass through the dots and have a length just equal to the diameter of the target body, which are used to increase the texture complexity of the background (1). There are four calibration circles (4) in total, which are used for calibration of the target plane. The center of the circle is the calibration point located at the two ends of the upper and lower vertical intersecting lines.

3. The method for measuring rock and soil deformation based on computer vision according to claim 2, characterized in that: In step 3, the pixel positions of the calibration points are obtained by two methods: when the radius of the calibration circle in the time-series deformed image is ≥10piexls, the range of four calibration circles is manually constructed, and then the center of these four calibration circles is detected using the Hough circle detection method to obtain the pixel coordinates of the four calibration points; when the radius of the blue calibration circle is <10piexls, the center of the calibration circle is manually selected as the calibration point.

4. The method for measuring rock and soil deformation based on computer vision according to claim 1, characterized in that: In step 4, when the reference image successfully matches the target i in the displacement analysis image, the range of the search box in the displacement analysis image (Xmin:Xmax, Ymin:Ymax) and the size W of the square matching box adapted to targets of different sizes are dynamically calculated, as shown in formula (1): Where: (x, y) is the coordinate of the upper left corner of the minimum bounding rectangle; w is the width of the minimum bounding rectangle; h is the height of the minimum bounding rectangle; n is the scaling factor, X min ,X max They represent the two boundaries of the search box in the X direction of the displacement analysis image, and the Y min ,Y max They represent the two boundaries of the search box in the Y direction of the displacement analysis image, and W is the side length of the square matching box; In step 5, the correlation coefficient R between the two matching boxes is calculated as shown in formula (2): Where: R is the correlation coefficient; is the average grayscale value of the template matching box; is the average grayscale value of the retrieval matching box; u(x,y) is the average grayscale value of the template matching box at point (x,y); v(x,y) is the average grayscale value of the retrieval matching box at point (x,y); In step 6, the equation of the quadratic correlation surface is shown in formula (3): R(x,y)=a0+a1x+a2y+a3x 2 +a4xy+a5y 2 (3) Where: R(x,y) is the correlation coefficient at any image coordinate point (x,y), a0~a5 are surface coefficients; In step 7, the sub-pixel coordinates are converted to world coordinates using formula (4): Where: (X, Y) is the position of any point in the plane where target i is located in the image coordinate system; (x, y) is the coordinates of the point (X, Y) projected onto the Z=0 plane in the world coordinate system (the plane of target i in the world coordinate system); is the homography matrix H; s is the scale factor; In step 8, the true displacement value is calculated using formula (5): Where: Δx B is the X-direction displacement of target i monitoring point at the moment of displacement analysis image; Δy B is the Y-direction displacement of target i monitoring point at the time of displacement analysis image; Δs is the total displacement of target i monitoring point at the time of displacement analysis image; is the real coordinate value of the target i monitoring point in the reference image; is the real coordinate value of the target i monitoring point in the displacement analysis image.

5. The method for measuring rock and soil deformation based on computer vision according to claim 4, characterized in that: In step 8, if target i is lost or temporarily blocked at the moment of displacement analysis image, the position of target i at the moment of displacement analysis image is roughly estimated by formula (6), and then the real displacement value of target i is calculated according to formula (5), and an error log with the words "target i is lost or temporarily blocked" is returned in the programming until target i is successfully re-matched in the subsequent image, and the original calculation process is resumed; Where: is the calculated true coordinate value of the target i monitoring point in the displacement analysis image; is the real coordinate value of the target i monitoring point in the previous image of the displacement analysis image; is the average displacement velocity of target i in the X direction during the previous interval; is the average displacement velocity of target i in the Y direction during the previous interval; ΔT is the interval between the displacement analysis image and the previous image.

6. A remote monitoring system, characterized in that: include: The image acquisition module includes a network camera, a lighting lamp, and a self-made target device. The self-made target device is installed in the deformation monitoring area, and the network camera is installed in a stable area away from the deformation monitoring area. It is aimed at the target to take pictures, and the camera's field of view covers the entire target arrangement range. The lighting lamp is fixed to the network camera and also aims at the monitoring area to illuminate the night, and is controlled to turn on or off by a relay. Image transmission module, including intelligent gateway and cloud server. The intelligent gateway is directly connected to the network camera and relay via network cable, and is connected to the network via 4G card and network cable. The smart gateway is located in the local area network, and the cloud server is located in the public network. Through the fast reverse proxy (FRP) intranet penetration technology, the ports for controlling the network camera's image capture, focus adjustment, and relay switch are mapped to the corresponding ports on the public network, establishing a data transmission channel. The smart gateway is equipped with the FRP client program and can automatically start the FRP client program after it is turned on and connected to the Internet. The cloud server is equipped with FRP server program, and the data download bandwidth must be >10M / s; The deformation calculation and visualization display module includes a deformation calculation server, deformation calculation software and a visualization management platform. The deformation calculation server is equipped with both the deformation calculation software and the visualization management platform to respectively perform online monitoring, visualization display and early warning of geotechnical structure deformation.

7. A remote monitoring system according to claim 6, characterized in that: The homemade target device is made of steel material and has the shape of a right-angled triangular prism with hollowed-out sides. The thickness of the remaining three complete surfaces, the bottom surface, the vertical front surface, and the oblique back surface, is set to 20 mm. The bottom surface contains four anchor holes, and the vertical front surface is perpendicular to the ground. The target image is engraved on the vertical front surface. At this time, its vertical displacement represents the surface subsidence here.

8. A remote monitoring system according to claim 6, characterized in that: In the deformation calculation and visualization display module, the specific analysis process of deformation calculation and visualization display is as follows: Process 1): The deformation calculation software controls the network camera to capture the initial image by connecting to the mapped public network port, loads it into the software page, and saves the image to the specified folder A; click the center point of each target in the initial image with the mouse as the monitoring point, and add a label to the monitoring point according to the number of the target center. On this basis, according to the size of the calibration circle of each target in the image, set the method for obtaining the calibration point position, and choose to construct the calibration circle range or directly select the calibration point according to the acquisition method; At the same time, the software will automatically generate a txt format data file for each monitoring point in folder A to save the calculation results, and will also automatically generate the same number of data tables as the monitoring points in the database. The names of each data file and data table are consistent with the monitoring point labels; Then, an image is automatically captured every N seconds and saved in folder A; Then, the deformation calculation software retrieves all images in folder A and adds the latest captured image to the software's list to be analyzed; Process 2): The deformation analysis software uses its built-in deformation measurement algorithm to perform deformation analysis on the newly added image. The software obtains the current X- and Y-direction displacement values ​​of each target, as well as the total displacement value and error log. The software then saves the calculation results for each monitoring point to the corresponding data file in folder A and to the corresponding data table in the database. The built-in deformation curve in the software is also updated synchronously. Process 3): The visualization management platform is built using WebGIS. The actual latitude and longitude coordinates of each monitoring point are marked on the electronic map. At the same time, a connection is established with the data table corresponding to each measurement point in the database. Deformation alarm thresholds are set according to different project requirements. After the settings are completed, the visualization management platform also searches the database every N seconds. If there is new data, it is loaded into the platform for visualization and risk warning is issued according to the set displacement threshold.

9. A remote monitoring system according to claim 8, characterized in that: In process 1), before each image is captured, a control relay is closed to turn on the light. After the image is captured, the relay is disconnected to turn off the light. The captured image is automatically named based on the capture time. The basic format of the image name is: "year-month-day&hour-minute-second.bmp". Each data table in the database contains seven fields: ID (int), current time (datatime), interval time since the previous image (double), X-direction displacement (double), Y-direction displacement (double), total displacement (double), and error log (varchar). Among them, "ID" represents the number of images currently being analyzed, "interval time since previous image" is in seconds, "X-direction displacement", "Y-direction displacement", and "total displacement" are in millimeters, and "error log" contains error information encountered during the analysis process.

10. A remote monitoring system according to claim 8, characterized in that: In process 2), the deformation measurement algorithm adopts the geotechnical deformation measurement method based on computer vision described in claim 1; after calculating the deformation of the current analysis image, the next image in the list to be analyzed is calculated immediately. If there are no remaining images in the list to be analyzed, the program is suspended until a new image is loaded into the list to be analyzed and the program is restarted; after obtaining a new set of deformation data, for different targets, the program will construct a string and a list according to the six data of "number of current analysis images", "current time", "interval time with the previous image", "X-direction displacement", "Y-direction displacement", "total displacement" and "error log", and save them to the data file and data table, among which the "error log" only includes two types: "none" and "target lost or temporarily blocked", which correspond to the successful and unsuccessful target matching in the geotechnical deformation measurement method based on computer vision.

Images