Deformation monitoring method and system based on machine vision technology

Abstract

The invention discloses a deformation monitoring method and system based on a machine vision technology, and relates to the technical field of crossing of structural health monitoring and computer vision, and the method comprises the steps: improving the angular point positioning precision in a calibration stage through a sub-pixel-level Harris angular point detection algorithm; robust recognition and initial positioning of an artificial marker in a first frame of image are realized by using an X-Feat convolutional neural network model, an anti-interference template is generated in combination with histogram equalization and Gaussian filtering, the stability of the system is enhanced, a local search area is delimited by taking an initial positioning point as a center, and a deep learning and interpolation algorithm is combined, so that the anti-interference performance of the system is improved. High-precision tracking and displacement calculation of the marker center are realized, the monitoring resolution is effectively improved, pixel-level displacement is accurately converted into physical space displacement through inverse transformation of a homography matrix, and dual threshold criteria of mode length and change rate are introduced for dynamic early warning, so that the engineering practicability and safety response capability of a monitoring result are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of structural health monitoring and computer vision, in particular to a deformation monitoring method based on machine vision technology. BACKGROUND

[0002] In the field of safety monitoring of major engineering structures such as bridges, dams and high-rise buildings, deformation detection is a key link for evaluating the health status of the structure. Non-contact monitoring technology based on machine vision acquires structure surface images through camera equipment, calculates feature point displacement using image analysis algorithms, and realizes remote, full-field and real-time dynamic monitoring. It has become an important development direction of structural health monitoring. This technology combines artificial markers and image matching algorithms to track the position changes of the markers in the image sequence, and then calculates the actual deformation of the structure. Typical technologies include template matching, digital image correlation method and feature extraction and matching method combined with deep learning. In addition, geometric calibration is used to realize the conversion from pixel coordinates to physical coordinates. However, existing technologies such as CN11378043A use traditional template matching algorithms, which only support pixel-level positioning and are prone to lose tracking under conditions of light changes or occlusion. Other traditional technologies introduce perspective distortion in the camera imaging process due to the tilt of the viewing angle, resulting in distortion of the target position in the image and affecting the accuracy of displacement measurement. Moreover, the positioning of the corner points by conventional calibration methods is not accurate enough. In the image matching process, the conversion from image displacement to physical displacement relies on the homography relationship, resulting in measurement errors in the physical dimension. Furthermore, the existing system's warning mechanism mainly uses static threshold judgment, which is difficult to effectively identify sudden or accelerating dangerous deformations. SUMMARY

[0003] In view of the above existing problems, the present application is proposed.

[0004] Therefore, the present application provides a deformation monitoring method based on machine vision technology to solve the problem of distortion of the target position in the image caused by the tilt of the viewing angle in the camera imaging process, which affects the accuracy of displacement measurement. Moreover, the positioning of the corner points by conventional calibration methods is not accurate enough. In the image matching process, traditional algorithms are sensitive to changes in light, background interference and image blur, and have poor matching stability. The conversion from image displacement to physical displacement relies on the homography relationship, resulting in measurement errors in the physical dimension. Furthermore, the existing system's warning mechanism mainly uses static threshold judgment, which is difficult to effectively identify sudden or accelerating dangerous deformations.

[0005] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a deformation monitoring method based on machine vision technology, which comprises: acquiring a video stream through a high-definition variable-focus infrared camera; Based on the calibration board image contained in the video stream, the geometric mapping relationship between the image plane and the structure surface is calculated, and the perspective transformation of the video stream is performed using the geometric mapping relationship to eliminate imaging distortion and obtain the corrected image sequence. The corrected image sequence is input into a lightweight convolutional neural network model. The semi-dense matching capability of the convolutional neural network model is used to identify artificial markers, and the position of the artificial marker in the first frame image is used as the initial localization point. Centered on the initial positioning point, a region containing the central features of the marker is extracted from the first frame image as a template image for matching calculation in subsequent image frames; In subsequent image frame matching calculations, the local search region is delineated with the initial positioning point as a reference. A lightweight convolutional neural network model and image processing techniques are used to compare the template image with the local search region to obtain the precise displacement of the marker center. Based on the geometric mapping relationship, the precise displacement is converted into the actual physical displacement of the structure, and a monitoring curve of displacement changing over time is generated. The monitoring curve is compared with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered.

[0006] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the steps include: calculating the geometric mapping relationship between the image plane and the structural surface based on the calibration plate image contained in the video stream, and using the geometric mapping relationship to perform perspective transformation on the video stream to eliminate imaging distortion and obtain a corrected image sequence. The specific steps are as follows: A rectangular calibration plate with a regular geometric pattern is fixedly installed on the surface of the structure to be monitored, and the physical dimensions and corner positions of the calibration plate are obtained. Using a high-definition variable-focus infrared camera, images containing the calibration board are captured from multiple different perspectives to form a calibration image set. Corner detection is performed on each image in the calibration image set to identify the pixel coordinates of all corner points on the calibration board, thus forming an image coordinate set. Based on the physical structure of the calibration plate, obtain the world coordinates of each corner point in the coordinate system of the structural surface, form a set of world coordinates, and establish the projection equation between the image coordinates and the world coordinates. The homography matrix is ​​calculated by solving a system of linear equations consisting of multiple sets of corresponding points. The homography matrix expresses the planar projection transformation relationship between the image plane and the structural surface. For any frame of the original image in the video stream, the homography matrix Perform coordinate transformation; For any pixel in the original image, according to the homography matrix Calculate the new position of the pixel in the corrected image; By using bilinear interpolation, the pixel values ​​of the original image are resampled to new positions to generate a corrected image without perspective distortion. The coordinate transformation and resampling operations are repeatedly performed on each frame of data in the video stream to output a continuous sequence of corrected images.

[0007] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the steps of inputting the corrected image sequence into a lightweight convolutional neural network model, identifying artificial markers through the semi-dense matching capability of the convolutional neural network model, and determining the center position of the artificial marker in the first frame image as the initial positioning point are as follows: The first frame of the corrected image is input into the X-Feat convolutional neural network model; The X-Feat convolutional neural network model processes images through feature extraction, generates multi-scale feature maps, and outputs feature point heatmaps by combining the feature point detection branch of the X-Feat convolutional neural network model. ; The reliability heatmap is output from the reliability evaluation branch of the X-Feat convolutional neural network model. The reliability heatmap Each pixel value in the image represents the stability of the corresponding feature point under different imaging conditions. In feature point heatmap The non-maximum suppression algorithm is applied to select the pixels with the highest local response as the candidate feature point set; Query candidate feature point set in reliability heatmap The corresponding values ​​in the data are used to retain high-confidence feature points; Calculate the mean of the pixel coordinates of all high-confidence feature points, and use the mean coordinates as the geometric center of the artificial marker; The pixel coordinates of the geometric center are used as the initial positioning point.

[0008] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the step of extracting a region containing the central features of the marker from the first frame image, centered on the initial positioning point, as a template image for matching calculation in subsequent image frames, includes the following specific steps: Using the initial positioning point as the center, extract a square region as the initial template; The initial template is subjected to histogram equalization and Gaussian filtering is applied to smooth image noise; The processed image is used as a template image for subsequent image matching calculations.

[0009] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the following steps are taken: In subsequent images, a local search region is delineated with the initial positioning point as a reference; a lightweight convolutional neural network model and image processing technology are used to compare the template image with the local search region to obtain the precise displacement of the marker center. In subsequent image matching calculations, a square region with a side length greater than that of the template image is defined as the local search region, centered on the initial location point. The template image is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the template image in the descriptor map. ; The image within the local search region is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the image within the local search region in the descriptor map. ; Descriptor set in the local search region In this process, the Euclidean distance between each descriptor vector of the template image and all local descriptor vectors is calculated, and the local descriptor vector with the smallest Euclidean distance is taken as the matching result, forming a preliminary set of matching points; For each matching point in the set of matching points, sub-pixel level interpolation is performed using the grayscale values ​​of the pixels surrounding the matching point to refine the coordinate position of the matching point. Calculate the weighted average of the coordinates of all refined matching points to obtain the precise location of the marker center in the current image, and use the offset of the precise location from the initial positioning point as the precise displacement. .

[0010] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the specific steps of converting the precise displacement into the actual physical displacement of the structure based on the geometric mapping relationship and generating a monitoring curve of displacement changing over time are as follows: Obtain the homography matrix and with homography matrix Calculate the inverse matrix The inverse matrix is ​​used to convert pixel displacements back to physical space; Using the inverse homography matrix pixel displacement vector Perform a coordinate transformation to obtain the world coordinate displacement vector of the structural surface. The expression is: ; Here For pixel displacement vectors, Let be the physical displacement vector in the world coordinate system of the structure surface. It is the inverse of the homography matrix; Store the calculated physical displacement vector Stored in association with the current timestamp; The conversion and comparison are repeatedly performed on each frame of the video stream, the physical displacement data is continuously calculated and recorded, and the physical displacement data points arranged in chronological order are connected into a line to generate a displacement-time monitoring curve.

[0011] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the step of comparing the monitoring curve with a preset safety threshold, and triggering an early warning command when the physical displacement exceeds the safety threshold, specifically includes the following steps: Let the safety threshold vector be defined. To obtain the maximum allowable displacement values ​​of the structure in the horizontal and vertical directions, obtain the latest physical displacement vector on the displacement-time monitoring curve. ; Calculate the physical displacement vector Length of the module The expression is: ; in and They are respectively The horizontal and vertical components.

[0012] Determine the modulus Is it greater than the safety threshold vector? Length of the module ; like If so, the continuous judgment process will be initiated; Get continuous Calculate the rate of change of displacement from the physical displacement data of the frame. The formula is: ; Here and For the first Maximum and minimum values ​​in frame data For continuous Frame displacement value, This refers to the inter-frame time interval. If the rate of change of displacement Greater than the preset rate of change threshold If the signal is detected, it is determined to be a dangerous state, and a high-level warning signal is generated and sent out through the communication interface.

[0013] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the sparse matching capability of the lightweight convolutional neural network model includes sub-pixel-level displacement determination functionality, and the specific steps are as follows: Centered on the initial matching point, extract a... small window, The number of pixels is odd, and a two-dimensional Gaussian weighting is applied to the grayscale values ​​of the pixels within the window to generate a weighted grayscale matrix. ; Calculate the weighted gray matrix centroid coordinates The formula is: ; Here, The row and column indices of the pixels within the window. , This indicates double summation, i.e. Iterate through and accumulate all elements in the image window; centroid coordinates As a subpixel-level precise matching position, the offset between the precise position and the initial positioning point is calculated, which is used as the final precise displacement.

[0014] As a preferred embodiment of the deformation monitoring method based on machine vision technology described in this invention, the corner detection of the calibration board adopts a sub-pixel level Harris corner detection algorithm, and the specific steps are as follows: Gaussian filtering is applied to the grayscale image of any image in the calibration image set, and the gradient magnitude and direction of each pixel in the image are calculated to construct a gradient map; The Harris corner response function is applied to the gradient map to generate an initial corner response map, and non-maximum suppression is applied to the initial corner response map to obtain an integer pixel-level corner candidate set. For each corner point in the candidate set, construct a quadratic polynomial interpolation function in the neighborhood of the response function; The true sub-pixel coordinates of corner points are calculated using an interpolation function, as follows: ; Here, Integer pixel coordinates; It is the first-order partial derivative; It is the second-order partial derivative; The sub-pixel coordinates of all corner points are used as the final corner coordinates of the calibration board for accurate calculation of the homography matrix.

[0015] Secondly, the present invention provides a deformation monitoring system based on machine vision technology, comprising: The system includes a data acquisition module, a calibration module, a positioning module, a generation module, a displacement calculation module, a conversion module, an early warning and decision-making module, and a communication module. The acquisition module acquires a video stream containing artificial markers on the structure to be monitored via a high-definition variable-focus infrared camera. The correction module calculates the geometric mapping relationship between the image plane and the structural surface based on the calibration board image contained in the video stream, and uses the geometric mapping relationship to perform perspective transformation on the video stream to eliminate imaging distortion and output the corrected image sequence. The positioning module inputs the corrected image sequence into a lightweight convolutional neural network model, identifies artificial markers through the semi-dense matching capability of the convolutional neural network model, and determines the center position of the artificial marker in the first frame image as the initial positioning point. The generation module takes the initial positioning point determined in the previous step as the center, extracts the region containing the center feature of the marker in the first frame image, and generates the final template image after histogram equalization and Gaussian filtering. The displacement calculation module delineates a local search region in subsequent frames using the initial positioning point determined in the previous step as a reference, matches the template image with the feature descriptor of the local search region, and refines the matching position through sub-pixel interpolation to calculate the precise displacement of the marker center. The conversion module converts the precise displacement output by the displacement calculation module into the actual physical displacement of the structure based on the geometric mapping relationship calculated by the geometric correction module, and generates a monitoring curve of displacement changing over time. The early warning decision module compares the monitoring curve generated by the conversion module with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered. The communication module transmits the early warning signal generated by the early warning decision module to the remote monitoring center via wired or wireless means.

[0016] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the deformation monitoring method based on machine vision technology as described in the first aspect of the present invention.

[0017] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deformation monitoring method based on machine vision technology as described in the first aspect of the present invention.

[0018] The beneficial effects of this invention are as follows: It improves the corner positioning accuracy during the calibration stage by using a sub-pixel-level Harris corner detection algorithm; it achieves robust recognition and initial positioning of artificial markers in the first frame image using an X-Feat convolutional neural network model; and it enhances system stability by combining histogram equalization and Gaussian filtering to generate anti-interference templates. Furthermore, it achieves high-precision tracking and displacement calculation of marker centers by combining deep learning and interpolation algorithms, effectively improving monitoring resolution and measurement accuracy. Finally, it accurately converts pixel-level displacement into physical space displacement through the inverse transformation of the homography matrix and introduces dual threshold criteria of modulus and rate of change for dynamic early warning, improving the engineering practicality and safety response capability of the monitoring results. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the deformation monitoring method based on machine vision technology in Example 1.

[0021] Figure 2 This is a schematic diagram of the deformation monitoring system based on machine vision technology in Example 1. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Example 1, referring to Figure 1 and Figure 2 This embodiment of the invention provides a deformation monitoring method and system based on machine vision technology, comprising the following steps: S1. Based on the calibration board image contained in the video stream, calculate the geometric mapping relationship between the image plane and the structural surface, and use the geometric mapping relationship to perform perspective transformation on the video stream to eliminate imaging distortion and obtain a corrected image sequence. Furthermore, a rectangular calibration plate with a regular geometric pattern is fixedly installed on the surface of the structure to be monitored to obtain the physical dimensions and corner positions of the calibration plate; Using a high-definition variable-focus infrared camera, images containing the calibration board are captured from multiple different perspectives to form a calibration image set. Corner detection is performed on each image in the calibration image set to identify the pixel coordinates of all corner points on the calibration board, thus forming an image coordinate set. Based on the physical structure of the calibration plate, obtain the world coordinates of each corner point in the coordinate system of the structural surface, form a set of world coordinates, and establish the projection equation between the image coordinates and the world coordinates. The homography matrix is ​​calculated by solving a system of linear equations consisting of multiple sets of corresponding points. The homography matrix expresses the planar projection transformation relationship between the image plane and the structural surface. For any frame of the original image in the video stream, the homography matrix Perform coordinate transformation; For any pixel in the original image, according to the homography matrix Calculate the new position of the pixel in the corrected image; By using bilinear interpolation, the pixel values ​​of the original image are resampled to new positions to generate a corrected image without perspective distortion. The coordinate transformation and resampling operations are repeatedly performed on each frame of data in the video stream to output a continuous sequence of corrected images.

[0026] It should be noted that by acquiring calibration plate images from multiple perspectives and calculating the homography matrix, the geometric mapping relationship between the image plane and the structural surface can be accurately established, effectively eliminating perspective distortion caused by camera tilt or lens distortion, and providing a reliable geometric correction basis for subsequent high-precision displacement measurement.

[0027] S2. Input the corrected image sequence into a lightweight convolutional neural network model. Use the semi-dense matching capability of the convolutional neural network model to identify artificial markers and determine the center position of the artificial marker in the first frame image as the initial localization point. Furthermore, the first frame of the corrected image is input into the X-Feat convolutional neural network model; The X-Feat convolutional neural network model processes images through feature extraction, generates multi-scale feature maps, and outputs feature point heatmaps by combining the feature point detection branch of the X-Feat convolutional neural network model. ; The reliability heatmap is output from the reliability evaluation branch of the X-Feat convolutional neural network model. The reliability heatmap Each pixel value in the image represents the stability of the corresponding feature point under different imaging conditions. In feature point heatmap The non-maximum suppression algorithm is applied to select the pixels with the highest local response as the candidate feature point set; Query candidate feature point set in reliability heatmap The corresponding values ​​in the data are used to retain high-confidence feature points; Calculate the mean of the pixel coordinates of all high-confidence feature points, and use the mean coordinates as the geometric center of the artificial marker; The pixel coordinates of the geometric center are used as the initial positioning point.

[0028] It should be noted that by utilizing the feature point heatmap and reliability assessment mechanism of a lightweight convolutional neural network, it is possible to stably identify feature regions on markers under complex imaging conditions, determine the center position by multi-point averaging, and improve the accuracy and anti-interference capability of initial positioning.

[0029] S3. Using the initial positioning point as the center, extract the region containing the central features of the marker in the first frame image as a template image for matching calculation in subsequent image frames.

[0030] Furthermore, a square region is selected as the initial template, centered on the initial positioning point; The initial template is subjected to histogram equalization and Gaussian filtering is applied to smooth image noise; The processed image is used as a template image for subsequent image matching calculations.

[0031] It should be noted that histogram equalization and Gaussian filtering of the template image can enhance image contrast and suppress the influence of noise, improve the matching robustness of the template under different lighting conditions, and ensure the continuity and stability of the subsequent tracking process.

[0032] S4. In subsequent images, a local search region is defined with the initial positioning point as a reference. A lightweight convolutional neural network model and image processing techniques are used to compare the template image with the local search region to obtain the precise displacement of the marker center.

[0033] Furthermore, in subsequent image matching calculations, a square region with a side length greater than that of the template image is defined as the local search region, centered on the initial location point. The template image is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the template image in the descriptor map. ; The image within the local search region is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the image within the local search region in the descriptor map. ; Descriptor set in the local search region In this process, the Euclidean distance between each descriptor vector of the template image and all local descriptor vectors is calculated, and the local descriptor vector with the smallest Euclidean distance is taken as the matching result, forming a preliminary set of matching points; For each matching point in the set of matching points, sub-pixel level interpolation is performed using the grayscale values ​​of the pixels surrounding the matching point to refine the coordinate position of the matching point. Calculate the weighted average of the coordinates of all refined matching points to obtain the precise location of the marker center in the current image, and use the offset of the precise location from the initial positioning point as the precise displacement. .

[0034] It should be noted that defining a local search region in subsequent frames and matching it with a depth descriptor can achieve accurate positioning while ensuring computational efficiency. Combined with subpixel interpolation, this further improves the accuracy of displacement resolution and effectively captures minute deformation changes.

[0035] S5. Based on the geometric mapping relationship, the precise displacement is converted into the actual physical displacement of the structure, and a monitoring curve of displacement changing with time is generated.

[0036] Furthermore, obtain the homography matrix. and with homography matrix Calculate the inverse matrix The inverse matrix is ​​used to convert pixel displacements back to physical space; Using the inverse homography matrix pixel displacement vector Perform a coordinate transformation to obtain the world coordinate displacement vector of the structural surface. The expression is: ; Here For pixel displacement vectors, Let be the physical displacement vector in the world coordinate system of the structure surface. It is the inverse of the homography matrix; Store the calculated physical displacement vector Stored in association with the current timestamp; The conversion and comparison are repeatedly performed on each frame of the video stream, the physical displacement data is continuously calculated and recorded, and the physical displacement data points arranged in chronological order are connected into a line to generate a displacement-time monitoring curve.

[0037] It should be noted that the pixel displacement is converted into physical space displacement based on the inverse homography transformation, which ensures that the monitoring results have practical engineering significance. The generated displacement-time curve truly reflects the dynamic deformation process of the structure, which is convenient for safety assessment and trend analysis.

[0038] S6. Compare the monitoring curve with the preset safety threshold. When the physical displacement exceeds the safety threshold, trigger an early warning command.

[0039] Furthermore, let's define a safety threshold vector. To obtain the maximum allowable displacement values ​​of the structure in the horizontal and vertical directions, obtain the latest physical displacement vector on the displacement-time monitoring curve. ; Calculate the physical displacement vector Length of the module The expression is: ; in and They are respectively The horizontal and vertical components.

[0040] Determine the modulus Is it greater than the safety threshold vector? Length of the module ; like If so, the continuous judgment process will be initiated; Get continuous Calculate the rate of change of displacement from the physical displacement data of the frame. The formula is: ; Here and For the first Maximum and minimum values ​​in frame data For continuous Frame displacement value, This refers to the inter-frame time interval. If the rate of change of displacement Greater than the preset rate of change threshold If the signal is detected, it is determined to be a dangerous state, and a high-level warning signal is generated and sent out through the communication interface.

[0041] It should be noted that using both modulus length judgment and displacement change rate as dual criteria for early warning decision-making not only focuses on whether the current displacement exceeds the limit, but also considers the deformation development trend, which can more scientifically distinguish between normal disturbances and potential dangers, and improve the reliability and practicality of the early warning system.

[0042] S7, the sparse matching capability of the lightweight convolutional neural network model includes subpixel-level displacement determination function.

[0043] Furthermore, taking the initial matching point as the center, extract a... small window, The number of pixels is odd, and a two-dimensional Gaussian weighting is applied to the grayscale values ​​of the pixels within the window to generate a weighted grayscale matrix. ; Calculate the weighted gray matrix centroid coordinates The formula is: ; Here, The row and column indices of the pixels within the window. , This indicates double summation, i.e. Iterate through and accumulate all elements in the image window; centroid coordinates As a subpixel-level precise matching position, the offset between the precise position and the initial positioning point is calculated as the final precise displacement.

[0044] It should be noted that by constructing a Gaussian weighted grayscale matrix and calculating the centroid coordinates, sub-pixel-level matching point refinement is achieved, effectively breaking through the pixel grid limitation and improving the displacement measurement resolution, which is especially suitable for long-term monitoring scenarios with high accuracy requirements.

[0045] S8. The corner detection of the calibration board adopts the sub-pixel level Harris corner detection algorithm.

[0046] Furthermore, Gaussian filtering is applied to the grayscale image of any image in the calibration image set, and the gradient magnitude and direction of each pixel in the image are calculated to construct a gradient map; The Harris corner response function is applied to the gradient map to generate an initial corner response map, and non-maximum suppression is applied to the initial corner response map to obtain an integer pixel-level corner candidate set. For each corner point in the candidate set, construct a quadratic polynomial interpolation function in its 3×3 pixel neighborhood. The true sub-pixel coordinates of corner points are calculated using an interpolation function, as follows: ; Here, Integer pixel coordinates; It is the first-order partial derivative; It is the second-order partial derivative; The sub-pixel coordinates of all corner points are used as the final corner coordinates of the calibration board for accurate calculation of the homography matrix.

[0047] It should be noted that the sub-pixel-level Harris corner detection algorithm is used to accurately locate the corners of the calibration board, which greatly improves the coordinate accuracy of the corresponding points during the calibration process, thereby improving the accuracy of the homography matrix calculation and laying a solid foundation for the high-precision measurement of the entire system.

[0048] This embodiment also provides a deformation monitoring system based on machine vision technology, including: The system includes a data acquisition module, a calibration module, a positioning module, a generation module, a displacement calculation module, a conversion module, an early warning and decision-making module, and a communication module. The acquisition module acquires a video stream containing artificial markers on the structure to be monitored via a high-definition variable-focus infrared camera. The correction module calculates the geometric mapping relationship between the image plane and the structural surface based on the calibration board image contained in the video stream, and uses the geometric mapping relationship to perform perspective transformation on the video stream to eliminate imaging distortion and output the corrected image sequence. The positioning module inputs the corrected image sequence into a lightweight convolutional neural network model, identifies artificial markers through the semi-dense matching capability of the convolutional neural network model, and determines the center position of the artificial marker in the first frame image as the initial positioning point. The generation module takes the initial positioning point determined in the previous step as the center, extracts the region containing the center feature of the marker in the first frame image, and generates the final template image after histogram equalization and Gaussian filtering. The displacement calculation module delineates a local search region in subsequent frames using the initial positioning point determined in the previous step as a reference, matches the template image with the feature descriptor of the local search region, and refines the matching position through sub-pixel interpolation to calculate the precise displacement of the marker center. The conversion module converts the precise displacement output by the displacement calculation module into the actual physical displacement of the structure based on the geometric mapping relationship calculated by the geometric correction module, and generates a monitoring curve of displacement changing over time. The early warning decision module compares the monitoring curve generated by the conversion module with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered. The communication module transmits the early warning signal generated by the early warning decision module to the remote monitoring center via wired or wireless means.

[0049] The data flow between the various functional modules works closely together to form a complete automated processing flow. From raw data collection to final safety decision-making and information output, the entire system has a clear architecture and well-defined division of labor. It supports continuous unattended operation and is suitable for long-term structural health monitoring, greatly improving the efficiency and reliability of monitoring work.

[0050] This embodiment also provides a computer device applicable to a deformation monitoring method based on machine vision technology, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the deformation monitoring method based on machine vision technology as proposed in the above embodiment.

[0051] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0052] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the deformation monitoring method based on machine vision technology as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0053] In summary, this invention improves the corner location accuracy during the calibration stage through a sub-pixel-level Harris corner detection algorithm, achieves robust recognition and initial positioning of artificial markers in the first frame image using an X-Feat convolutional neural network model, and enhances system stability by combining histogram equalization and Gaussian filtering to generate anti-interference templates. By combining deep learning and interpolation algorithms, high-precision tracking and displacement calculation of marker centers are achieved, effectively improving monitoring resolution and measurement accuracy. The inverse transformation of the homography matrix accurately converts pixel-level displacement into physical space displacement, and the introduction of dual threshold criteria of modulus and rate of change for dynamic early warning improves the engineering practicality and safety response capability of the monitoring results.

[0054] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A deformation monitoring method based on machine vision technology, characterized in that: include: Video streams are captured using a high-definition variable-focus infrared camera; Based on the calibration board image contained in the video stream, the geometric mapping relationship between the image plane and the structure surface is calculated, and the perspective transformation of the video stream is performed using the geometric mapping relationship to eliminate imaging distortion and obtain the corrected image sequence. The corrected image sequence is input into a lightweight convolutional neural network model. The semi-dense matching capability of the convolutional neural network model is used to identify artificial markers, and the position of the artificial marker in the first frame image is used as the initial localization point. Centered on the initial positioning point, a region containing the central features of the marker is extracted from the first frame image as a template image for matching calculation in subsequent image frames; In subsequent image frame matching calculations, the local search region is delineated with the initial positioning point as a reference. A lightweight convolutional neural network model and image processing techniques are used to compare the template image with the local search region to obtain the precise displacement of the marker center. Based on the geometric mapping relationship, the precise displacement is converted into the actual physical displacement of the structure, and a monitoring curve of displacement changing over time is generated. The monitoring curve is compared with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered.

2. The deformation monitoring method based on machine vision technology as described in claim 1, characterized in that: The steps are as follows: Based on the calibration board image contained in the video stream, the geometric mapping relationship between the image plane and the structural surface is calculated, and the perspective transformation of the video stream is performed using the geometric mapping relationship to eliminate imaging distortion and obtain a corrected image sequence. A rectangular calibration plate with a regular geometric pattern is fixedly installed on the surface of the structure to be monitored, and the physical dimensions and corner positions of the calibration plate are obtained. Using a high-definition variable-focus infrared camera, images containing the calibration board are captured from multiple different perspectives to form a calibration image set. Corner detection is performed on each image in the calibration image set to identify the pixel coordinates of all corner points on the calibration board, thus forming an image coordinate set. Based on the physical structure of the calibration plate, obtain the world coordinates of each corner point in the coordinate system of the structural surface, form a set of world coordinates, and establish the projection equation between the image coordinates and the world coordinates. The homography matrix is ​​calculated by solving a system of linear equations consisting of multiple sets of corresponding points. The homography matrix expresses the planar projection transformation relationship between the image plane and the structural surface. For any frame of the original image in the video stream, the homography matrix Perform coordinate transformation; For any pixel in the original image, according to the homography matrix Calculate the new position of the pixel in the corrected image; By using bilinear interpolation, the pixel values ​​of the original image are resampled to new positions to generate a corrected image without perspective distortion. The coordinate transformation and resampling operations are repeatedly performed on each frame of data in the video stream to output a continuous sequence of corrected images.

3. The deformation monitoring method based on machine vision technology as described in claim 2, characterized in that: The corrected image sequence is input into a lightweight convolutional neural network model. The model's semi-dense matching capability is used to identify artificial landmarks, and the center position of the artificial landmark in the first frame is determined as the initial localization point. The specific steps are as follows: The first frame of the corrected image is input into the X-Feat convolutional neural network model; The X-Feat convolutional neural network model processes images through feature extraction, generates multi-scale feature maps, and outputs feature point heatmaps by combining the feature point detection branch of the X-Feat convolutional neural network model. ; The reliability heatmap is output from the reliability evaluation branch of the X-Feat convolutional neural network model. The reliability heatmap Each pixel value in the image represents the stability of the corresponding feature point under different imaging conditions; In feature point heatmap The non-maximum suppression algorithm is applied to select the pixels with the highest local response as the candidate feature point set; Query candidate feature point set in reliability heatmap The corresponding values ​​in the data are used to retain high-confidence feature points; Calculate the mean of the pixel coordinates of all high-confidence feature points, and use the mean coordinates as the geometric center of the artificial marker; The pixel coordinates of the geometric center are used as the initial positioning point.

4. The deformation monitoring method based on machine vision technology as described in claim 3, characterized in that: The step of using the initial positioning point as the center, extracting a region containing the central features of the marker from the first frame image as a template image for matching calculations in subsequent image frames, specifically involves the following steps: Using the initial positioning point as the center, extract a square region as the initial template; The initial template is subjected to histogram equalization and Gaussian filtering is applied to smooth image noise; The processed image is used as a template image for subsequent image matching calculations.

5. The deformation monitoring method based on machine vision technology as described in claim 4, characterized in that: In subsequent images, a local search region is defined with the initial positioning point as a reference. A lightweight convolutional neural network model and image processing techniques are used to compare the template image with the local search region to obtain the precise displacement of the marker center. The specific steps are as follows: In subsequent image matching calculations, a square region with a side length greater than that of the template image is defined as the local search region, centered on the initial location point. The template image is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the template image in the descriptor map. ; The image within the local search region is input into the X-Feat convolutional neural network model to obtain the feature descriptor set of the image within the local search region in the descriptor map. ; Descriptor set in the local search region In this process, the Euclidean distance between each descriptor vector of the template image and all local descriptor vectors is calculated, and the local descriptor vector with the smallest Euclidean distance is taken as the matching result, forming a preliminary set of matching points; For each matching point in the set of matching points, sub-pixel level interpolation is performed using the grayscale values ​​of the pixels surrounding the matching point to refine the coordinate position of the matching point. Calculate the weighted average of the coordinates of all refined matching points to obtain the precise location of the marker center in the current image, and use the offset of the precise location from the initial positioning point as the precise displacement. .

6. The deformation monitoring method based on machine vision technology as described in claim 5, characterized in that: Based on the geometric mapping relationship, the precise displacement is converted into the actual physical displacement of the structure, generating a monitoring curve of displacement changing over time. The specific steps are as follows: Obtain the homography matrix and with homography matrix Calculate the inverse matrix The inverse matrix is ​​used to convert pixel displacements back to physical space; Using the inverse homography matrix pixel displacement vector Perform a coordinate transformation to obtain the world coordinate displacement vector of the structural surface. The expression is: ； Here For pixel displacement vectors, Let be the physical displacement vector in the world coordinate system of the structure surface. It is the inverse of the homography matrix; Store the calculated physical displacement vector Stored in association with the current timestamp; The conversion and comparison are repeatedly performed on each frame of the video stream, the physical displacement data is continuously calculated and recorded, and the physical displacement data points arranged in chronological order are connected into a line to generate a displacement-time monitoring curve.

7. The deformation monitoring method based on machine vision technology as described in claim 6, characterized in that: The process involves comparing the monitoring curve with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered. The specific steps are as follows: Let the safety threshold vector be defined. To obtain the maximum allowable displacement values ​​of the structure in the horizontal and vertical directions, obtain the latest physical displacement vector on the displacement-time monitoring curve. ; Calculate the physical displacement vector Length of the module The expression is: ； in and They are respectively Horizontal and vertical components; Determine the modulus Is it greater than the safety threshold vector? Length of the module ; like If so, the continuous judgment process will be initiated; Get continuous Calculate the rate of change of displacement from the physical displacement data of the frame. The formula is: ； Here and For the first Maximum and minimum values ​​in frame data For continuous Frame displacement value, This refers to the inter-frame time interval. If the rate of change of displacement Greater than the preset rate of change threshold If the signal is detected, it is determined to be a dangerous state, and a high-level warning signal is generated and sent out through the communication interface.

8. The deformation monitoring method based on machine vision technology as described in claim 7, characterized in that: The sparse matching capability of the lightweight convolutional neural network model includes sub-pixel-level displacement determination, and the specific steps are as follows: Centered on the initial matching point, extract a... small window, The number of pixels is odd, and a two-dimensional Gaussian weighting is applied to the grayscale values ​​of the pixels within the window to generate a weighted grayscale matrix. ; Calculate the weighted gray matrix centroid coordinates The formula is: ； Here, The row and column indices of the pixels within the window. , This indicates a double summation, i.e. Iterate through and accumulate all elements in the image window; centroid coordinates As a subpixel-level precise matching position, the offset between the precise position and the initial positioning point is calculated as the final precise displacement.

9. The deformation monitoring method based on machine vision technology as described in claim 8, characterized in that... The calibration board uses a sub-pixel level Harris corner detection algorithm for corner detection. The specific steps are as follows: Gaussian filtering is applied to the grayscale image of any image in the calibration image set, and the gradient magnitude and direction of each pixel in the image are calculated to construct a gradient map; The Harris corner response function is applied to the gradient map to generate an initial corner response map, and non-maximum suppression is applied to the initial corner response map to obtain an integer pixel-level corner candidate set. For each corner point in the candidate set, construct a quadratic polynomial interpolation function in the neighborhood of the response function; The true sub-pixel coordinates of corner points are calculated using an interpolation function, as follows: ； Here, Integer pixel coordinates; It is the first-order partial derivative; It is the second-order partial derivative; The sub-pixel coordinates of all corner points are used as the final corner coordinates of the calibration board for accurate calculation of the homography matrix.

10. A deformation monitoring system based on machine vision technology, based on the deformation monitoring method based on machine vision technology according to any one of claims 1 to 9, characterized in that: It includes a data acquisition module, a calibration module, a positioning module, a generation module, a displacement calculation module, a conversion module, an early warning decision module, and a communication module; The acquisition module acquires a video stream containing artificial markers on the structure to be monitored via a high-definition variable-focus infrared camera. The correction module calculates the geometric mapping relationship between the image plane and the structural surface based on the calibration board image contained in the video stream, and uses the geometric mapping relationship to perform perspective transformation on the video stream to eliminate imaging distortion and output the corrected image sequence. The positioning module inputs the corrected image sequence into a lightweight convolutional neural network model, identifies artificial markers through the semi-dense matching capability of the convolutional neural network model, and determines the center position of the artificial marker in the first frame image as the initial positioning point. The generation module takes the initial positioning point determined in the previous step as the center, extracts the region containing the center feature of the marker in the first frame image, and generates the final template image after histogram equalization and Gaussian filtering. The displacement calculation module delineates a local search region in subsequent frames using the initial positioning point determined in the previous step as a reference, matches the template image with the feature descriptor of the local search region, and refines the matching position through sub-pixel interpolation to calculate the precise displacement of the marker center. The conversion module converts the precise displacement output by the displacement calculation module into the actual physical displacement of the structure based on the geometric mapping relationship calculated by the geometric correction module, and generates a monitoring curve of displacement changing over time. The early warning decision module compares the monitoring curve generated by the conversion module with a preset safety threshold. When the physical displacement exceeds the safety threshold, an early warning command is triggered. The communication module transmits the early warning signal generated by the early warning decision module to the remote monitoring center via wired or wireless means.

Images