Large-view-field high-precision monitoring method and system

Abstract

The invention provides a large-view-field high-precision monitoring method and system, and relates to the technical field of real-time monitoring. The detection method combines a machine vision technology and a DIC technology based on deep learning to optimize a monitoring process, can perform high-precision deformation monitoring on multiple points of a large view field, and is suitable for deformation monitoring of reservoir dam abutment rock mass; before detection, a convolutional neural network is used to carry out defogging operation and intelligent analysis on a surface image of a to-be-detected area, a key target area is automatically identified, and errors caused by manual intervention are effectively avoided; after the target area is determined, displacement field distribution of the target area is calculated by adopting a digital image correlation algorithm based on deep learning, and deformation measurement with sub-pixel-level precision is realized; through fusion of deep learning and a traditional image processing technology, the influence of manual parameter setting is reduced, the problem of decorrelation during matching is solved, and the precision and reliability of displacement calculation are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time monitoring, and particularly relates to a large field of view high-precision monitoring method and system. Background Art

[0002] Concrete rock dams are widely used in the construction of reservoir rock dams due to their abundant materials, low cost, and good performance. However, over time, due to the long-term action of seepage erosion and material loss, the rock dam is prone to deformation, and finally the risk of dam failure is triggered. Therefore, it is necessary to give early warning of the deformation of the rock mass at the dam shoulder. In the existing deformation measurement schemes for the rock mass at the dam shoulder, one is the contact monitoring technology such as burying piezometers, infiltration line pipes, deformation displacement gauges, etc., and the other is the non-contact detection technology such as GNSS (Global Navigation Satellite System), GPS (Global Positioning System), etc.

[0003] Digital image correlation is a non-contact detection technology and is widely used in the fields of military, aviation, biology, architecture, etc. due to its advantages such as high precision, anti-interference, and wide measurement range. In previous monitoring schemes, Zhang Lirong et al. proposed applying traditional DIC technology and three-dimensional reconstruction technology in the video real-time monitoring system and method for emergency disposal of barrier lakes, constructing a three-dimensional model of the dam surface, analyzing surface deformation, tracking the change trend of cracks through a crack detection algorithm, and generating dam stability indicators. Traditional DIC uses a calibrated camera to record the speckle patterns before and after the deformation of the measured object, and sets parameters such as sub-region size, calculation step size, shape function, correlation function, etc. in advance for correlation calculation. The obtained correlation coefficient is used as the basis for the next IC-GN iteration to update the shape of the sub-region and determine whether it is the best matching point.

[0004] In 2020, Boukhtache et al. referred to the optical flow estimation method, based on computer-simulated speckle patterns, fine-tuned multiple existing optical flow estimation models such as FlowNet, SpyNet, PWC-Net, etc., and selected the best-performing FlowNet model for improvement. By introducing the U-Net structure, the StrainNet model suitable for the speckle pattern was constructed. This model fully utilized the information obtained by the encoder part of the convolutional neural network during the regression prediction process and achieved a measurement accuracy comparable to that of the traditional DIC method. On the basis of StrainNet-f and StrainNet-h, Boukhtache et al. further evolved an improved network designed specifically for small displacement measurement. Compared with the previous versions, this network has higher measurement accuracy in the case of small deformations. Compared with the previous versions, StrainNet-l has higher accuracy in the case of small deformations.

[0005] Regarding the real-time monitoring of the deformation of the rock mass at the dam shoulder of a reservoir, traditional non-contact deformation monitoring techniques rely on the placement and reading of detection devices. Not only are the wiring and installation complex, but the measurement accuracy and range are also restricted by the real environment, so the measurement error is relatively large. In the non-contact method based on satellite measurement, satellite signals are easily affected by environmental factors such as weather and obstacles, and the installation and maintenance costs are high. The installation of satellite signal receivers and related equipment requires professional skills and equipment investment, increasing the monitoring cost.

[0006] In the traditional DIC technology method, the main defects include:

[0007] 1. Dependence on manual parameter selection. Traditional digital image correlation algorithms rely on parameters selected manually, such as the order of the shape function, the size of the sub-region, the step size, etc. When the selected parameters are inappropriate, the calculation accuracy will be seriously affected.

[0008] 2. Long calculation time. In a large field of view environment, when the range of the speckle pattern is large, the matching speed of each pixel point will be very slow, seriously affecting the requirements of real-time monitoring.

[0009] 3. Need to manually select the region of interest. Traditional digital image correlation algorithms need to manually select the region of interest, that is, the speckle region, which affects the calculation efficiency.

[0010] 4. When the image contrast is low and the image is blurred, the de-correlation will occur.

[0011] In other digital image correlation algorithms based on deep learning, blurred images in a large field of view are not considered as one of the contents. The reference images generated by the datasets are mostly clear images obtained by real shooting or computer simulation, so the calculation error is relatively large. Summary of the Invention

[0012] Aiming at the deficiencies of the existing technology, the present invention provides a large field of view high-precision monitoring method and system, which solves the problems of low calculation efficiency and low precision in real-time monitoring of a large field of view.

[0013] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0014] A large field of view high-precision monitoring method, the detection method includes:

[0015] S1. Draw a number of speckle patterns in the real-time monitoring area of the object to be measured, and set standard cross scales in the speckle patterns;

[0016] S2. Collect images of the real-time monitoring area through a camera;

[0017] S3. Dehaze the first captured image, identify the speckle pattern, crop the speckle pattern, and record the position of the speckle pattern in the image;

[0018] S4. Capture an image every interval of time t, perform dehazing on the subsequent images, and crop the speckle pattern from the subsequent images according to the recorded position of the speckle pattern. The cropped speckle patterns are saved separately according to the regions to be measured;

[0019] S5. Use the trained DIC model to calculate the regions to be measured. Compare and calculate the currently obtained speckle pattern with the previous speckle pattern and the first speckle pattern in the corresponding region respectively to obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured region; Transfer the average value of the relative pixel displacement and the average value of the absolute pixel displacement of each region to be measured to the control module;

[0020] S6. Convert the pixel displacement to the actual displacement in the three-dimensional world through the mapping relationship between the standard cross scale of the region to be measured and the pixel unit;

[0021] S7. Repeat S4 - S6 to perform real-time monitoring on all regions to be measured. When the actual displacement is greater than the specified displacement, issue a warning.

[0022] Preferably, the training method of the DIC model includes:

[0023] T1. Collect speckle pictures through actual experiments, simulations, and other data sets to generate a reference image data set containing various speckle patterns;

[0024] T2. Simulate complex deformation displacement fields through Hermite element interpolation and cubic spline interpolation. The interpolation formula is as follows:

[0025]

[0026] where u c is the displacement of the whole image;

[0027] E is the number of units divided by the speckle pattern being operated on;

[0028] e is the serial number of the current unit;

[0029] N e (x) is the Hermite polynomial shape function of the unit e;

[0030] q e represents the degree of freedom of the unit node;

[0031] S j (x) = a j x 3 + b jx 2 +c j x + d j , (j = 1, 2, ..., n)

[0032] where S j (x) is the displacement of different pixel points;

[0033] j is different intervals in the image;

[0034] a j , b j , c j , d j are the parameters to be fitted in the current interval;

[0035] The interpolation points are a randomly generated array with a displacement less than 2 pixels;

[0036] T3. Build a deep learning-based DIC model. Based on the ResNet structure, input the reference image and the deformed image respectively for convolution to expand the number of channels;

[0037] T4. Convolve the expanded feature image as shown in the following formula:

[0038]

[0039] where f and g represent the reference image and the deformed image respectively;

[0040] w1 and w2 represent the pixel points of the reference image and the deformed image respectively;

[0041] k represents the size of the image block;

[0042] C(w1, w2) represents the matching degree after the convolution operation of different pixel points;

[0043] Obtain a four-dimensional correlation layer representing the matching degree of different regions, adjust its size to combine with the original image as a new input for subsequent convolution;

[0044] T5. The subsequent part of the encoder includes five convolutional operation units, and the corresponding decoder part contains five decoding units. Convolve the smaller-scale encoder, the same-scale encoder, and the larger-scale decoder corresponding to each decoding unit to the same size as the current decoding unit and merge them to form the input of the next decoder module;

[0045] T6. Train the built network in the proposed dataset. Use the global loss function to represent the total error of the training network, and the formula is as follows:

[0046]

[0047] Among them, H and W are the height and width of the image respectively;

[0048] x f , x g , y f , y g respectively represent the magnitudes of the true displacement field and the predicted displacement field in the x and y directions;

[0049] L error represents the total error between the predicted displacement field and the true displacement field.

[0050] Preferably, in the S1, the speckle pattern is drawn with a pigment that is not easily decomposed, and a ceramic chip engraved with a standard cross scale is inlaid in the speckle pattern.

[0051] Preferably, the defogging process is implemented by the GCA Net model. The GCA Net model learns the residual between the original image and the foggy image and adds it to the input blurred image to achieve the defogging effect.

[0052] Preferably, the speckle pattern recognition is implemented by the YOLOv8 model. In the dataset constructed by the YOLOv8 model, the samples are images containing multiple speckle regions, the labels are all the speckle regions in the images, and the selected regions are the inscribed rectangles of each speckle region in the figure.

[0053] Preferably, the Hermite element interpolation requires two nodes, and each node contains three degrees of freedom, as shown in the following formula:

[0054]

[0055] Among them, u j is the displacement of the unit node;

[0056] The j subscript represents the node number;

[0057] ξ represents the local unit coordinate axis;

[0058] Thus, it can be obtained that the displacement of any node on this axis can be represented by these six degrees of freedom. The formula for the one-dimensional Hermite element is as follows:

[0059]

[0060] Among them, is the displacement formed by the i-th node in the j-th degree of freedom;

[0061] The two-dimensional Hermite polynomial shape functions are all products of the above one-dimensional shape functions. The two-dimensional interpolation consists of four nodes, each node has 9 degrees of freedom, and there are a total of 36 shape functions. Its general formula is:

[0062]

[0063]

[0064] Among them, is related to the degrees of freedom of element node i .

[0065] w and h are the half-width and half-height of the element respectively;

[0066] i′ and i″ are the corresponding node numbers in the one-dimensional Hermite element respectively;

[0067] From this, it can be deduced that the displacement u(x) at any point x within a single element can be expressed as:

[0068] u(x) = N e (x)q e

[0069] Among them, N e (x) is the Hermite polynomial shape function of the element e where it is located;

[0070] q e is the degrees of freedom of the element node;

[0071] After determining the displacement within each element, the displacement u of the entire image can be obtained through the formula c :

[0072]

[0073] Among them, E is the number of elements into which the speckle pattern to be operated is divided;

[0074] e represents the serial number of the currently located element.

[0075] Preferably, the cubic spline interpolation sets the interval [a, b] in a one-dimensional scenario, and interpolation nodes a = x1 < x1 <... < x n = b are set on the interval, and the corresponding function values are y1, y2,..., y n . If there exists a function S(x) that satisfies S(x j ) = y j , and it is a polynomial not higher than cubic within the interval and has second-order continuous derivatives within the interval [a, b], it is called a cubic spline interpolation function;

[0076] Within the interval [x j , x j+1 , a cubic polynomial in the following form can be determined:

[0077] S j (x) = a j x3 +b j x 2 +c j x + d j , (j = 1, 2, ..., n)

[0078] where a j , b j , c j , d j are to be determined. At the same time, the cubic spline interpolation polynomial needs to satisfy the continuity of the original function, the first-order function, and the second-order function, and satisfy the following formulas:

[0079] S(x j ) = y j , S(x j - 0) = S(x j + 0), j = 2, 3, ..., n - 1

[0080] S′(x j - 0) = S′(x j + 0), S″(x j - 0) = S″(x j + 0), j = 2, 3, ..., n - 1

[0081] The interpolation nodes on the image are selected by setting grid points at intervals, as shown in the following formula:

[0082]

[0083] where x i , y i represent different grid points;

[0084] d x_grid , d y_grid represent the spacings of the grid points in the x and y directions respectively;

[0085] d edge represents the distances of the initial grid point from the left edge and the upper edge;

[0086] After such calculations, by adding the four corner points of the image to the sequence of interpolation points, the complete interpolation nodes can be obtained;

[0087] The deformation amount of the interpolation nodes is given by the following formula:

[0088]

[0089] where x and y are the coordinates of each pixel point in the reference image;

[0090] v, v represent the displacement amounts in the x and y directions respectively;

[0091] W and H are the width and height of the image;

[0092] a and b are random coefficients when generating displacements for each image. The value range of a is [0, 4], and the value of b is 1 or 2;

[0093] d max is the maximum displacement determined artificially;

[0094] d random is the randomly added perturbation.

[0095] A large field of view high-precision monitoring system, the monitoring system includes: a speckle pattern drawing module, an image acquisition module, a first image processing module, a subsequent image processing module, a DIC model processing module, a displacement conversion module, and an alarm module;

[0096] The speckle pattern drawing module is used to draw a number of speckle patterns in the real-time monitoring area of the object to be measured, and standard cross scales are set in the speckle patterns;

[0097] The image acquisition module is used to acquire images of the real-time monitoring area through a camera;

[0098] The first image processing module is used to perform defogging, speckle pattern recognition, and speckle pattern cropping on the first acquired image, and record the position of the speckle pattern in the image;

[0099] The subsequent image processing module is used to acquire an image every interval of time t, perform defogging processing on the subsequent images, and crop the speckle patterns from the subsequent images according to the recorded positions of the speckle patterns. The cropped speckle patterns are saved separately according to the areas to be measured;

[0100] The DIC model processing module is used to calculate the areas to be measured using the trained DIC model, and perform comparative calculations on the currently acquired speckle pattern with the previous speckle pattern and the first speckle pattern in the corresponding area respectively to obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured area; transfer the mean value of the relative pixel displacement of each area to be measured to the control module;

[0101] The displacement conversion module is used to convert the pixel displacement to the actual displacement in the three-dimensional world through the mapping relationship between the standard cross scale and the pixel unit of the area to be measured;

[0102] The alarm module is used to repeat S4 - S6 to perform real-time monitoring on all areas to be measured, and issue a warning when the actual displacement is greater than the rated displacement.

[0103] The present invention provides a large field of view high-precision monitoring method and system. Compared with the prior art, it has the following beneficial effects:

[0104] In the present invention, the detection method combines machine vision technology with DIC technology based on deep learning to optimize the monitoring process, and can perform high-precision deformation monitoring on large fields of view at multiple points, which is applicable to the deformation monitoring of the rock mass of the reservoir dam shoulder; before detection, the surface image of the area to be measured is first dehazed by a convolutional neural network and intelligently analyzed to automatically identify the key target area, effectively avoiding the errors caused by manual intervention; after determining the target area, a digital image correlation algorithm based on deep learning is used to calculate the displacement field distribution of the target area, realizing deformation measurement with sub-pixel accuracy; through the integration of deep learning and traditional image processing technology, the influence of artificially set parameters is reduced, the problem of decorrelation during matching is solved, and the accuracy and reliability of displacement calculation are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0106] Figure 1 It is a schematic diagram of drawing a speckle pattern on the real-time monitoring area in the embodiment of the present invention.

[0107] Figure 2 It is an example speckle pattern of the dataset in the DIC model training in the embodiment of the present invention.

[0108] Figure 3 It is a schematic diagram of the Hermite element interpolation displacement field in the embodiment of the present invention.

[0109] Figure 4 It is a schematic diagram of the cubic spline interpolation displacement field in the embodiment of the present invention.

[0110] Figure 5 It is a schematic diagram of the network structure of the DIC model in the embodiment of the present invention.

[0111] Figure 6 It is a schematic diagram of the structure of the full-size skip connection in the embodiment of the present invention.

[0112] Figure 7 It is a flowchart of the detection method in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0113] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0114] By providing a large field of view high-precision monitoring method and system in the embodiments of this application, the problems of low computational efficiency and low precision in real-time monitoring of a large field of view are solved.

[0115] To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0116] Embodiment:

[0117] As Figures 1-7 shown, the present invention provides a large field of view high-precision monitoring method, and the detection method includes:

[0118] S1. Draw a number of speckle patterns in the real-time monitoring area of the object to be measured, and set standard cross scales in the speckle patterns;

[0119] S2. Collect images of the real-time monitoring area through a camera;

[0120] S3. Perform defogging, speckle pattern recognition, and speckle pattern cropping on the first collected image, and record the position of the speckle pattern in the image;

[0121] S4. Collect an image every interval of time t, perform defogging processing on the subsequent images, crop the speckle patterns from the subsequent images according to the recorded positions of the speckle patterns, and save the cropped speckle patterns separately according to the areas to be measured;

[0122] S5. Use the trained DIC model to calculate the areas to be measured, compare and calculate the currently obtained speckle pattern with the previous speckle pattern and the first speckle pattern in the corresponding area respectively, and obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured area; transfer the average value of the relative pixel displacement amount and the average value of the absolute pixel displacement amount of each area to be measured to the control module;

[0123] S6. Convert the pixel displacement amount to the actual displacement amount in the three-dimensional world through the mapping relationship between the standard cross scale and the pixel unit of the area to be measured;

[0124] S7. Repeat S4 - S6 to perform real-time monitoring on all areas to be measured. When the actual displacement amount is greater than the rated displacement amount, issue a warning;

[0125] The training method of the DIC model includes:

[0126] T1. Generate a reference image dataset containing various speckle patterns by collecting speckle images through actual experiments, simulation generation, and other datasets;

[0127] T2. Simulate complex deformation displacement fields through Hermite element interpolation and cubic spline interpolation, add blurred images to simulate real environments, and improve the generalization ability of the network. The interpolation formula is as follows:

[0128]

[0129] where u c is the displacement of the entire image;

[0130] E is the number of elements divided in the speckle pattern being operated on;

[0131] e is the serial number of the current element;

[0132] N e (x) is the Hermite polynomial shape function of the element e;

[0133] q e represents the degrees of freedom of the element nodes;

[0134] S j (x) = a j x 3 + b j x 2 + c j x + d j , (j = 1, 2,..., n)

[0135] where S j (x) is the displacement of different pixel points;

[0136] j is different intervals in the image;

[0137] a j , b j , c j , d j represent the parameters to be fitted in the current interval;

[0138] The interpolation points are randomly generated arrays with displacements less than 2 pixels;

[0139] T3. Build a deep learning-based DIC model. Based on the ResNet structure, input the reference image and the deformed image respectively for convolution to expand the number of channels;

[0140] T4. Convolve the expanded feature image as shown in the following formula:

[0141]

[0142] Among them, f and g respectively represent the reference image and the deformed image;

[0143] w1 and w2 respectively represent the pixel points of the reference image and the deformed image;

[0144] k represents the size of the image block;

[0145] C(w1, w2) represents the matching degree after the convolution operation of different pixel points;

[0146] A four-dimensional correlation layer is obtained, representing the matching degree of different regions. Adjust its size to combine it with the original image as a new input for subsequent convolution;

[0147] T5. The subsequent part of the encoder includes five convolution operation units, and the corresponding decoder part contains five decoding units. The smaller-scale encoder, the same-scale encoder, and the larger-scale decoder corresponding to each decoding unit are convolved to the same size as the current decoding unit and merged to form the input of the next decoder module;

[0148] T6. The constructed network is trained in the proposed dataset, and a global loss function is used to represent the total error of the training network. The formula is as follows:

[0149]

[0150] Among them, H and W are respectively the height and width of the image;

[0151] x f ,x g ,y f ,y g respectively represent the sizes of the true displacement field and the predicted displacement field in the x and y directions;

[0152] L error represents the total error between the predicted displacement field and the true displacement field.

[0153] In digital image correlation technology, the first step is to obtain an image of the space where the object to be measured is located through a camera. In order to improve the measurement accuracy in this application, as Figure 1 shown, a speckle pattern is drawn using a non-decomposable pigment such as paint in the real-time monitoring area.

[0154] In digital image correlation, it is necessary to calibrate the camera first to establish the calculation relationship between the three-dimensional real distance and the two-dimensional image coordinates, and reduce the influence of camera distortion on actual measurement. However, the calibration method of the traditional DIC method is too complex. Therefore, in this method, the calibration process is omitted. A ceramic piece engraved with standard cross scales is inlaid in each speckle pattern. The actual length of each scale is known. In subsequent calculations, the length of the scale in the image is detected through an edge detection algorithm to construct the mapping relationship between the pixel unit and the actual distance.

[0155] Suppose the five regions to be monitored are Figure 1 In this case, the camera will collect multiple images in chronological order. According to the traditional DIC process, it is necessary to manually select the region of interest at this time. In this figure, the five speckle regions are dispersed. If five regions are manually selected, the calculation efficiency will be affected. If full-field matching is performed without selection, it will cause a large waste of resources. Therefore, image segmentation is required. However, in the case of a large field of view, defocusing is inevitable, and in the reservoir environment, there will be a large amount of water mist, both of which will cause the image to be blurred. Therefore, before image segmentation, it is necessary to perform dehazing on the image. To avoid grid artifacts, GCANet is selected to perform dehazing on the image. By directly learning the residual between the original image and the hazy image and adding it to the input blurred image, the purpose of dehazing is achieved, and the contrast of the image is enhanced. There are many irrelevant lines in the space. Therefore, using the edge detection method to select the speckle region may cause false detection. We still choose to use the deep learning method to select the speckle region. Therefore, a dataset needs to be constructed. The samples in the dataset are images containing multiple speckle regions, and the labels are all the speckle regions in the image. Note that the selected regions are the inscribed rectangles of the respective speckle regions in the figure. YOLOv8 has improved the detection accuracy compared to the previous version and can adapt to scenarios with high requirements for small object detection. It is a relatively mature network. Therefore, YOLOv8 is used to select the speckle region, and the marked region values are used to crop the image to obtain five groups of corresponding speckle patterns before and after deformation.

[0156] After all five regions are selected, in traditional DIC, relevant matching operations will be carried out. Relevant matching is the core content of the DIC algorithm, and it relies too much on the selection of artificial parameters. For example, when the sub-region sizes are selected inconsistently, the accuracy of the measured results will be different. Digital image correlation based on deep learning solves this problem. It takes two speckle patterns before and after deformation as inputs and directly outputs the displacement field, avoiding the process of artificially selecting parameters and shape functions. However, in existing digital image correlation algorithms based on deep learning, high precision and high efficiency are pursued, and the displacement under a large field of view is not calculated. In the real environment of a reservoir, its measurement range can reach dozens of meters or even hundreds of meters, while the actual possible displacement is only on the order of centimeters. Corresponding to the image, there is a displacement of one or two pixels on an image of millions of pixels. Although the image after image preprocessing has improved contrast, there will still be some blurring. So we made a specific dataset. The data sources include simulated speckles generated by programs, artificially sprayed speckles, and laser speckles. The obtained images are as Figure 2 shown. Among them, the artificially sprayed speckles and laser speckles come from experiments on DIC in the laboratory, as Figure 2 shown. The artificially sprayed speckles have larger speckle particles and stronger speckle randomness; the laser speckles have smaller particles and are evenly distributed. The combination of the two can describe different types of clear speckles. The simulated speckle pattern uses a Gaussian filter to smooth the noise to generate a speckle effect. As the parameters change, the speckle will show a blurred effect. The dataset will contain these three categories of images, and all images will be cropped to a size of 128×128 pixels to improve the training speed. The displacement field of the dataset is generated by computer-generated information and added to the reference image to create the true displacement field and deformed image in the dataset. In existing research, in order to simulate real displacements, researchers have established different mathematical models to enrich the deformation modes in the dataset, such as using random functions, Gaussian functions, second-order shape functions, etc. In order to simulate the diversity of deformations, the Hermite interpolation and cubic spline interpolation are selected for the displacement field in the dataset to construct the displacement field. The Hermite element interpolation can obtain any complex full-field deformation within the element by randomly generating the displacement and strain degree-of-freedom values at the unit nodes. The size of the deformed unit of the obtained displacement field is related to the set unit size. Applying it to the deformed image to generate the reference image can characterize the complex situation generated by the speckle displacement in the real situation. In the one-dimensional case, two nodes are required for interpolation, and each node contains three degrees of freedom, as shown in the formulas respectively:

[0157]

[0158] where, u jis the displacement of the unit node, the subscript j represents the node number, and ξ represents the local unit coordinate axis. It can be obtained that the displacement of any node on this axis can be expressed by these six degrees of freedom. The formula of the one-dimensional Hermite element is as follows:

[0159]

[0160] where ξ represents the local unit coordinate axis. They represent the displacement formed by the jth degree of freedom of the ith node. The two-dimensional Hermite polynomial shape functions are the product of the above one-dimensional shape functions. The two-dimensional interpolation consists of four nodes, each with 9 degrees of freedom, a total of 36 shape functions, and its general formula is:

[0161]

[0162] in, The degrees of freedom of the element node i Where w and h are the half-width and half-height of the unit, respectively, and i′, i″ are the corresponding node numbers in the one-dimensional Hermite unit. It can be deduced that the displacement u(x) of any point x in a single unit can be expressed as:

[0163] u(x)=N e (x)q e

[0164] Among them, N e (x) is the Hermite polynomial shape function of the unit e, q e It is expressed as the unit node degree of freedom. After determining the displacement in each unit, the displacement u of the entire graph can be obtained by the formula c :

[0165]

[0166] Where E represents the number of units divided by the operated speckle pattern, and e represents the serial number of the current unit. The displacement field map constructed by Hermite is as follows: Figure 3 As shown, the size of the displacement block is related to the width and height of the unit. When the width and height are small, a dense small-range displacement block will be presented, and the displacement change will be more drastic. When the width and height are large, the displacement change will be relatively gentle.

[0167] Cubic spline interpolation is a smooth piecewise interpolation method that constructs a cubic polynomial function between each data point so that the entire interpolation curve or surface has continuous first-order and second-order derivatives at the data point. It is suitable for scenarios with high precision and continuity. In a one-dimensional scenario, let the interval [a, b], and set the interpolation node a=x1 on the interval <x1<...<xn = b, and the corresponding function values are y1, y2,..., y n , if there exists a function S(x) that satisfies S(x j ) = y j , and are all polynomials of degree not higher than three within the interval and have second-order continuous derivatives within the interval [a, b], then it is called a cubic spline interpolation function.

[0168] Within the interval [x j , x j+1 , a cubic polynomial in the following form can be determined:

[0169] S j (x) = a j x 3 + b j x 2 + c j x + d j , (j = 1, 2,..., n)

[0170] where a j , b j , c j , d j are to be determined. At the same time, the cubic spline interpolation polynomial needs to satisfy the continuity of the original function, the first-order function, and the second-order function, that is, it satisfies the following formulas:

[0171] S(x j ) = y j , S(x j -0) = S(x j +0), j = 2, 3,..., n - 1

[0172] S′(x j -0) = S′(x j +0), S″(x j -0) = S″(x j +0), h = 2, 3,..., n - 1

[0173] In this way, the above two formulas together give \(n + 3(n - 2)=4n - 6\) conditions, and \(4(n - 1)\) coefficients need to be determined. Therefore, to determine a unique cubic interpolation function, theoretically, 2 additional boundary conditions are required. Commonly used boundary conditions include: specifying the first derivative value at the endpoints, specifying the second derivative value at the endpoints, or the given function being a periodic function. When programming, it is assumed that the two ends of the curve are the same polynomials, that is, the first and the second piecewise polynomials are the same at the starting point of the curve, and the last and the second-to-last piecewise polynomials are the same at the ending point of the curve. In this way, the starting condition is that the cubic coefficients of the first piecewise polynomial are the same as those of the second piecewise polynomial, and the ending condition is that the cubic coefficients of the last piecewise polynomial are the same as those of the second-to-last piecewise polynomial.

[0174] The interpolation nodes on the image can be selected by setting grid points at intervals, as shown in the following formula:

[0175]

[0176] where \(x\) i , \(y\) i represent different grid points, \(d\) x_grid , \(d\) y_grid respectively represent the intervals of the grid points in the \(x\) and \(y\) directions, and \(d\) edge represents the distances of the initial grid points from the left edge and the upper edge. After such calculations, by adding the four corner points of the image to the sequence of interpolation points, a complete set of interpolation nodes can be obtained.

[0177] The deformation amounts of the interpolation nodes can be given by different formulas. As used in this paper, displacement amounts are randomly added using trigonometric functions and linear functions, as shown below.

[0178]

[0179] where \(x\) and \(y\) are the coordinates of each pixel point in the reference image, \(u\) and \(v\) respectively represent the displacement amounts in the \(x\) and \(y\) directions, \(W\) and \(H\) are the width and height of the image, \(a\) and \(b\) are random coefficients for generating displacements for each image. The value range of \(a\) is \([0, 4]\), the value of \(b\) is 1 or 2, \(d\) max is the maximum displacement amount determined manually, and \(d\) random is the randomly added perturbation. By setting the nodes and their deformation amounts, and using interpolation to obtain the full-field displacement and applying it to the speckle pattern, a set of images before and after deformation and their theoretical deformation values can be obtained. When the maximum deformation amount is set to 2 pixels, the obtained displacement field is as shown in Figure 4 . Thus, the construction of the dataset is completed.

[0180] To improve the measurement accuracy of the neural network in micro-deformation under a large field of view, an improved neural network structure based on CNN is proposed. On the basis of ResNet, a Correlation module is added to obtain the prior information of image displacement and provide guidance for image detail matching. At the same time, a full-field skip connection structure is used so that each layer of the network can be combined with the previous encoding block and decoding block to capture semantic information at different levels. The specific network principle diagram is as shown in Figure 5 Figure []. The last block e6 of the encoder and the first block d6 of the decoder are of the same layer. The image before displacement or deformation is called the reference image, and the changed one is called the deformed image. The information directly read from the image, including the original speckle position, size, gray level, etc. of the reference image and the deformed image, is called the original information. In the operation, the reference image and the deformed image are first input, and the number of channels of the gray image is increased through convolution to associate the original image information with more channels to retain more original information. Then, the correlation calculation is performed on the two images to obtain a new set of data, which is called the prior information and provides guidance for subsequent calculations. These prior information and the original image information are combined and sent into the encoder together. The encoder is constructed based on the relatively mature ResNet network, which uses shortcut connections to solve the problem of model degradation in deep networks, reducing the computational amount while maintaining the model accuracy. In the decoder part, the original skip connections are cancelled and instead a full-size skip connection is adopted to receive more information from the upper and lower layers, thereby improving the accuracy.

[0181] The Correlation module is similar to the correlation matching module in traditional DIC. When performing the correlation matching calculation in traditional DIC, after selecting the sub-region size, the correlation degree between the corresponding regions of the reference image and the deformed image is calculated, which is called the correlation function. For example, the zero-mean normalized sum of squared differences criterion (ZNSSD):

[0182]

[0183] where, represent the mean gray values of the reference image and the deformed image respectively, and f i , g i represent the gray values of each pixel in the reference sub-region and the deformed sub-region respectively, is the normalization function of the selected sub-region, which characterizes the dispersion degree of the sub-region gray values.

[0184] C ZNSSDThe smaller the value, the higher the correlation between the selected sub-region and the target sub-region. In traditional DIC, the most crucial step in full-field calculation is to find the extreme value of the correlation criterion composed of the shape function and the correlation function to determine the position of the optimal solution. In previous DL-DIC studies, researchers mostly directly convolved the reference image and the deformed image to extract features without considering this prior condition. In this study, a Correlation module is added, and its calculation result represents the matching degree of two image patches, extracting, combining, and activating the original information of the two images. The Correlation calculation process is essentially a convolution operation. However, different from using a specific convolution kernel in CNN for operation, here a patch on the reference image is convolved with a patch on the deformed image. Therefore, this operation does not contain training parameters. Its calculation process is as shown in the formula:

[0185]

[0186] Among them, w1 and w2 represent the pixel points of the reference image and the target image respectively, and k represents the size of the image patch. After the calculation is completed, a four-dimensional correlation layer is obtained, and its size is adjusted to be the same as the size of the original input image. At this time, the obtained data contains the correlations of different regions of the original two input images, representing the matching degrees of different regions. The obtained data is combined with the original images as the new input. Since the matching degrees of different regions are different, the weights they occupy in the subsequent calculation will also be different, so as to play a role in providing guiding information. Finally, this information is encoded together so that the obtained result can be more accurate.

[0187] To enable the network to obtain more detailed information, this study adopts full-size skip connections in the decoder part. This connection enables each layer to be combined with the previous encoding block and decoding block, directly combining the high-level semantics and low-level semantics from feature maps of different scales, thereby fusing multi-scale information. In the encoder part, this study names the six convolution operations e1 to e6 in the order of operation; similarly, in the decoder part, the five convolution block operations are named d1 to d5. Since e6 is the last block of the encoder and directly performs an upsampling operation after that without a corresponding decoding block of the same size, the information of e6 can be regarded as the first block d6 of the decoder to participate in the calculation.

[0188] Such as Figure 5As shown, there is a merging operation in front of the convolution block of each decoder. Taking d3 as an example, its sources include: 1. Smaller-scale encoders e1 and e2: This link retains the detail information and integrates fine-grained semantics. In actual operation, non-overlapping maximum pooling is used with magnifications of 4 times and 2 times respectively to unify the size of the feature map; 2. Encoder e3 of the same scale: This link is similar to an ordinary skip connection to preserve the original information; 3. Larger-scale decoders d6, d5, d4: This link is to integrate coarse-grained semantic information, and upsampling is required to ensure the uniformity of the feature map size. In this study, the bilinear upsampling method is used to restore the image to maximize the integrity of the edge information during the upsampling process. Before connection, all modules need to be convolved with a 3×3 convolution kernel to unify the number of channels. The specific operation diagram is shown as follows. Figure 6 At this point, the six layers of feature maps are superimposed and concatenated to form a new feature map, and a new convolution operation is performed to form the next decoder module.

[0189] At this point, the deep learning-based digital image correlation method has been constructed, and the network parameters are obtained through training, which are used to calculate the cropped images mentioned above. The overall monitoring solution operation process is as follows: Figure 7 shown.

[0190] The present invention provides a large-field-of-view high-precision monitoring system, which includes: a speckle pattern drawing module, an image acquisition module, a first-image processing module, a subsequent image processing module, a DIC model processing module, a displacement conversion module, and an alarm module;

[0191] The speckle pattern drawing module is used to draw a number of speckle patterns in the real-time monitoring area of the object under test, and a standard cross scale is set in the speckle pattern;

[0192] The image acquisition module is used to acquire images of the real-time monitoring area through a camera;

[0193] The first image processing module is used to perform defogging, speckle pattern recognition, and speckle pattern cropping on the first image collected, and record the position of the speckle pattern in the image;

[0194] The subsequent image processing module is used to collect an image at each interval t, perform defogging on the subsequent image, and cut out the speckle pattern from the subsequent image according to the recorded speckle pattern position, and the cut out speckle pattern is saved respectively according to the area to be measured;

[0195] The DIC model processing module is used to calculate the area to be measured using the trained DIC model, and perform comparative calculations on the currently acquired speckle pattern with the previous speckle pattern and the first speckle pattern of the corresponding area respectively, so as to obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured area; transfer the mean value of the relative pixel displacement of each measured area to the control module;

[0196] The displacement conversion module is used to convert the pixel displacement into the actual displacement in the three-dimensional world through the mapping relationship between the standard cross scale of the measured area and the pixel unit;

[0197] The alarm module is used to repeat S4 - S6 to monitor all measured areas in real time, and issue a warning when the actual displacement is greater than the rated displacement.

[0198] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0199] 1. In the embodiment of the present invention, the detection method combines machine vision technology and DIC technology based on deep learning to optimize the monitoring process, and can perform high-precision deformation monitoring on large fields of view and multiple positions, which is suitable for the deformation monitoring of the rock mass of the reservoir dam shoulder; before detection, first use a convolutional neural network to perform defogging operation on the surface image of the area to be measured and perform intelligent analysis to automatically identify the key target area, effectively avoiding errors caused by manual intervention; after determining the target area, use the digital image correlation algorithm based on deep learning to calculate the displacement field distribution of the target area and achieve deformation measurement with sub-pixel accuracy; through the fusion of deep learning and traditional image processing technology, reduce the influence of artificially set parameters, solve the problem of decoherence during matching, and effectively improve the accuracy and reliability of displacement calculation.

[0200] 2. In the embodiment of the present invention, the detection method consists of three neural networks and edge detection in machine vision in terms of algorithms, and the method is simple and the cost is low.

[0201] 3. In the embodiment of the present invention, the detection method can realize the simultaneous selection and calculation of multiple regions of interest. Even if parts that need to be monitored are added later, calculation can be added by drawing speckles in the target area.

[0202] 4. In the embodiment of the present invention, the detection method improves the measurement accuracy of the network through the improvement of the digital image correlation method based on neural networks.

[0203] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0204] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large field of view high-precision monitoring method, characterized in that, The detection method includes: S1. Draw a number of speckle patterns in the real-time monitoring area of the object to be measured, and set a standard cross scale in the speckle pattern; S2. Collect images of the real-time monitoring area through a camera; S3. Perform defogging, speckle pattern recognition, and speckle pattern cropping on the first collected image, and record the position of the speckle pattern in the image; S4. Collect an image every interval of time t, perform defogging on the subsequent images, and crop the speckle pattern from the subsequent images according to the recorded position of the speckle pattern. The cropped speckle patterns are saved separately according to the areas to be measured; S5. Use the trained DIC model to calculate the areas to be measured, compare and calculate the currently obtained speckle pattern with the previous speckle pattern and the first speckle pattern in the corresponding area respectively, and obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured area; Transmit the average value of the relative pixel displacement and the average value of the absolute pixel displacement of each area to be measured to the control module; S6. Convert the pixel displacement to the actual displacement in the three-dimensional world through the mapping relationship between the standard cross scale in the area to be measured and the pixel unit; S7. Repeat S4 - S6 to perform real-time monitoring on all areas to be measured. When the actual displacement is greater than the rated displacement, issue a warning.

2. The large field of view high-precision monitoring method according to claim 1, wherein The training method of the DIC model includes: T1. Collect speckle pictures through actual experiments, simulations, and other data sets to generate a reference image data set containing various speckle patterns; T2. Simulate complex deformation displacement fields through Hermite element interpolation and cubic spline interpolation. The interpolation formula is as follows: Among them, u c is the displacement of the entire figure; E is the number of units divided by the speckle pattern being operated on; e is the serial number of the current unit; N e (x) is the Hermite polynomial shape function of the unit e where it is located; q e Expressed as the degrees of freedom of the element nodes; S j f(x) = a j x 3 + b j x 2 + c j x + d j , (j = 1, 2,..., n) where S j (x) is the displacement of different pixel points; j is different intervals in the picture; a j , b j , c j , d j are the parameters to be fitted in the current interval; The interpolation points are randomly generated arrays with a displacement less than 2 pixels; T3. Build a DIC model based on deep learning, based on the ResNet structure, and perform convolutions on the reference image and the deformed image respectively to expand the number of channels; T4. Perform convolutions on the expanded feature images, as shown in the following formula: Among them, f and g represent the reference image and the deformed image respectively; w1 and w2 represent the pixel points of the reference image and the deformed image respectively; k represents the size of the image block; C(w1, w2) represents the matching degree after convolution operations on different pixel points; Obtain a four-dimensional correlation layer representing the matching degree of different regions, adjust its size to combine with the original image as a new input for subsequent convolutions; T5. The subsequent encoder includes five convolutional operation units, and the corresponding decoder part contains five decoding units. Convolve the smaller-scale encoder, the same-scale encoder, and the larger-scale decoder corresponding to each decoding unit to the same size as the current decoding unit and merge them to form the input of the next decoder module; T6. Train the built network in the proposed data set, and use the global loss function to represent the total error of the training network. The formula is as follows: Among them, H and W are the height and width of the image respectively; x f ,x g ,y f ,y g respectively represent the magnitudes of the true displacement field and the predicted displacement field in the x and y directions; L error Represents the total error between the predicted displacement field and the true displacement field.

3. The large field of view high-precision monitoring method according to claim 1, characterized in that In S1, the speckle pattern is drawn with paint that is not easily decomposed, and a ceramic piece engraved with a standard cross scale is inlaid in the speckle pattern.

4. The large field of view high-precision monitoring method according to claim 1, characterized in that The defogging process is achieved through the GCANet model. The GCANet model learns the residual between the original image and the foggy image and adds it to the input blurred image to achieve the defogging effect.

5. The large field of view high-precision monitoring method according to claim 1, characterized in that, The speckle pattern recognition is implemented using the YOLOv8 model. In the dataset constructed by the YOLOv8 model, the samples are images containing multiple speckle regions, the labels are all the speckle regions in the images, and the selected regions are the inscribed rectangles of each speckle region in the figure.

6. The large field of view high-precision monitoring method according to claim 2, characterized in that The Hermite element interpolation requires two nodes, and each node contains three degrees of freedom, as shown in the following formula: where u i is the displacement of the element node; The subscript j represents the node number; ξ represents the local element coordinate axis; From this, it can be obtained that the displacement of any node on this axis can be represented by these six degrees of freedom. The formula for the one-dimensional Hermite element is as follows: wherein, the displacement formed at the j-th degree of freedom of the i-th node; The two-dimensional Hermite polynomial shape functions are all products of the above one-dimensional shape functions. The two-dimensional interpolation consists of four nodes, each node has 9 degrees of freedom, and there are a total of 36 shape functions. Its general formula is: Among them, is related to the degrees of freedom of element node i ; w and h are the half-width and half-height of the element respectively; i′ and i″ are the corresponding node numbers in the one-dimensional Hermite element respectively; From this, it can be deduced that the displacement u(x) of any point x within a single element can be expressed as: u(x) = N e (x)q e where, N e (x) is the Hermite polynomial shape function of the element e where it is located; q e is the degree of freedom of the element node; After determining the displacements within each element, the displacement u of the entire figure can be obtained through the formula c : Among them, E is the number of elements into which the operated speckle pattern is divided; e represents the serial number of the current element.

7. The large field of view high-precision monitoring method according to claim 2, characterized in that The cubic spline interpolation sets an interval [a, b] in a one-dimensional scenario, and interpolation nodes a = x1 < x1 <... < x n = b are set on the interval, and the corresponding function values are y1, y2,..., y n . If there exists a function S(x) that satisfies S(x j ) = y j , and is a polynomial of no higher than cubic degree within the interval and has second-order continuous derivatives within the interval [a, b], it is called a cubic spline interpolation function; On the interval [x j , x j+1 , a cubic polynomial in the following form can be determined: S j f(x) = a j x 3 + b j x 2 + c j x + d j , (j = 1, 2,..., n) where a j , b j , c j , d j to be determined. At the same time, the cubic spline interpolation polynomial needs to satisfy the continuity of the original function, the first-order function, and the second-order function, and satisfy the following formula: S(x j ) = y j , S(x j - 0) = S(x j + 0), j = 2, 3,..., n - 1 S′(x j - 0) = S′(x j + 0), S″(x j - 0) = S″(x j + 0), j = 2, 3, ..., n - 1 The interpolation nodes on the image are selected by setting grid points at intervals, as shown in the following formula: where x i , y i represent different grid points; d x_grid ,d y_grid represent the grid point spacings in the x and y directions, respectively; d edge represents the distances of the initial grid point from the left edge and the upper edge; After such calculation, by adding the four corner points of the image to the sequence of interpolation points, the complete interpolation nodes can be obtained; The deformation amount of the interpolation nodes is given by the following formula: Among them, x and y are the coordinates of each pixel point in the reference image; u and v respectively represent the displacement amounts in the x and y directions; W and H are the width and height of the image; a and b are the random coefficients when generating displacements for each image. The value range of a is [0, 4], and the value range of b is 1 or 2; d max is the maximum displacement determined artificially; d random is a randomly added perturbation.

8. A large field of view high-precision monitoring system, characterized in that, The monitoring system includes: a speckle pattern drawing module, an image acquisition module, a first image processing module, a subsequent image processing module, a DIC model processing module, a displacement conversion module, and an alarm module; The speckle pattern drawing module is used to draw a number of speckle patterns in the real-time monitoring area of the object to be measured, and standard cross scales are set in the speckle patterns; The image acquisition module is used to collect images of the real-time monitoring area through a camera; The first image processing module is used to perform defogging, speckle pattern recognition, and speckle pattern cropping on the first collected image, and record the position of the speckle pattern in the image; The subsequent image processing module is used to collect an image every interval of time t, perform defogging processing on the subsequent images, and crop the speckle pattern from the subsequent images according to the recorded position of the speckle pattern. The cropped speckle patterns are saved separately according to the regions to be measured; The DIC model processing module is used to calculate the area to be measured using the trained DIC model, and perform comparative calculations on the currently obtained speckle pattern with the previous speckle pattern and the first speckle pattern in the corresponding area respectively to obtain the relative pixel displacement and the absolute pixel displacement relative to the initial value of the currently measured area; The mean value of the relative pixel displacement amounts of each area to be measured is transmitted to the control module; The displacement conversion module is used to convert the pixel displacement into the actual displacement in the three-dimensional world through the mapping relationship between the standard cross scale of the area to be measured and the pixel unit; The alarm module is used to repeat S4 - S6 to monitor all areas to be measured in real time. When the actual displacement is greater than the rated displacement, a warning is issued.

Images