Super-resolution reconstruction deformation monitoring method and system

Abstract

The invention discloses a super-resolution reconstruction deformation monitoring method and system, and the method comprises the steps: obtaining a current low-resolution image and a historical reference low-resolution image of a target object to be subjected to deformation monitoring, inputting the images into a super-resolution reconstruction model, and obtaining corresponding high-resolution images, extracting feature points of the high-resolution image by using a scale invariant feature transformation algorithm; and matching the two feature points to obtain an effectively matched descriptor, further obtaining a matched feature point corresponding to the feature point, obtaining feature point position offset data of the current high-resolution image according to the matched feature point, and further obtaining a deformation monitoring result of the target object to be subjected to deformation monitoring. The method can improve the image resolution through the super-resolution reconstruction model under the condition that the imaging is blurred when the distance between the camera with the limited resolution and the target object is relatively long, and achieves the high-precision deformation monitoring on equipment with limited resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a deformation monitoring method and system based on super-resolution reconstruction. BACKGROUND

[0002] In the field of infrastructure, geological disaster warning, visual-based deformation monitoring is an important means to ensure the safety of structures and prevent disasters. In the prior art, high-resolution cameras are used to capture images of the target objects to be monitored, and the deformation of bridges, dams and high-rise buildings is determined based on image information. However, it is found that the prior art still has the following shortcomings:

[0003] 1. Limited spatial resolution: When monitoring the overall deformation of large bridges, dams and high-rise buildings, a long shooting distance is required to cover the entire structure, resulting in a small target size in the image;

[0004] 2. Limited sensor hardware performance: The resolution of the obtained image or data is limited, making it difficult to capture small deformation information;

[0005] 3. Large consumption of computing resources: High-precision image processing and analysis usually require a large amount of computing resources, which is not suitable for deployment on resource-limited devices. SUMMARY

[0006] The purpose of the present application is to provide a deformation monitoring method and system based on super-resolution reconstruction, which can improve the image resolution through a super-resolution reconstruction model under the condition of limited camera resolution and blurred imaging of the target object at a long distance, and realize real-time and high-precision deformation monitoring on resource-limited devices.

[0007] The present application adopts the following technical scheme: a deformation monitoring method based on super-resolution reconstruction, comprising the following steps:

[0008] S1, using a pinhole camera model to obtain a current low-resolution image of a target object to be deformed, inputting the image into a super-resolution reconstruction model to obtain a current high-resolution image of the image, and using a scale-invariant feature transformation algorithm to extract feature points of the high-resolution image.

[0009] S2, obtaining a historical reference low-resolution image of the target object to be deformed, inputting the image into a super-resolution reconstruction model to obtain a historical reference high-resolution image of the image, and using a scale-invariant feature transformation algorithm to extract historical reference feature points of the image.

[0010] S3, matching the feature points in step S1 with the historical reference feature points in step S2 to obtain effective matching descriptors.

[0011] S4, obtaining the matching feature points corresponding to the feature points in step S1 according to the matched descriptors, and obtaining the feature point position offset data of the current high-resolution image according to the current pixel position of the matching feature points in the current high-resolution image and the historical reference pixel position in the historical reference high-resolution image.

[0012] S5, obtaining the deformation monitoring result of the target object to be monitored according to the feature point position offset data of the current high-resolution image.

[0013] Further, obtaining the high-resolution image includes the following contents:

[0014] The super-resolution reconstruction model includes an interpolation module, a depth separable convolution layer, an attention layer, and an adaptive residual module; wherein the attention layer includes a channel attention layer and a spatial attention layer.

[0015] Selecting a monitoring area in the low-resolution image to obtain a monitoring area image in which deformation monitoring is required in the image, and performing denoising, image enhancement, and color correction processing on the monitoring area image to obtain a processed monitoring area image.

[0016] Inputting the processed monitoring area image into the super-resolution reconstruction model, performing initial bilinear interpolation operation by the interpolation module to obtain a basic high-resolution feature image, performing spatial convolution operation and point-by-point convolution operation on the basic high-resolution feature image by the depth separable convolution layer to obtain corresponding local features, performing global pooling feature weighting processing by the channel attention layer and average pooling and maximum pooling by the spatial attention layer on the local features to obtain two-dimensional attention features, and performing feature fusion on the two-dimensional attention features and the basic high-resolution feature image by the adaptive residual module to obtain a high-resolution image.

[0017] Further, the feature point extraction includes the following contents:

[0018] Convolve the high-resolution image I(x i ,y i ) with different scale Gaussian kernels to generate scale space images L(x i ,y i ,σ), and the specific expression is:

[0019] L(x i ,y i ,σ)=G(x i ,y i ,σ)*I(x i ,y i )

[0020] Wherein, * represents convolution operation; G(x i ,y i,σ) represents a two-dimensional Gaussian kernel, σ represents the standard deviation of the two-dimensional Gaussian kernel, x i represents the horizontal pixel position of the i-th feature point in the high-resolution image, y i represents the vertical pixel position of the i-th feature point in the high-resolution image.

[0021] The difference pyramid is used to difference the adjacent scale space images to obtain the difference Gaussian image, and the specific expression is:

[0022] D(x i ,y i ,σ)=L(x i ,y i ,kσ)-L(x i ,y i ,σ)

[0023] wherein D(x i ,y i ,σ) represents the difference Gaussian image of pixel point coordinates (x i ,y i ), L(x i ,y i ,kσ) represents the scale space image of the i-th feature point in the high-resolution image with pixel point coordinates (x i ,y i ) and Gaussian kernel standard deviation kσ, and k represents the scale factor.

[0024] The local extreme points of the difference Gaussian image in the spatial and scale directions are extracted, and Taylor expansion is used to fit the local extreme points to obtain the image coordinates of the feature points, and the specific expression is:

[0025]

[0026] wherein, represents the pixel coordinates of the i-th feature point in the high-resolution image, represents the first derivative of the difference Gaussian image, represents the second derivative of the difference Gaussian image.

[0027] The gradient direction histogram of the neighborhood of the feature point is calculated, and the main direction of the feature point is determined, and the specific expression is:

[0028]

[0029] wherein θ represents the main direction of the feature point, L xi represents the gradient of the i-th feature point in the x direction in the scale space image, represents the gradient of the i-th feature point in the y direction in the scale space image,

[0030] Divide the neighborhood of the i-th feature point into 4×4 regions. Construct a main direction θ relative to the i-th feature point for each region. i The histogram of the eight gradient directions is used to obtain the descriptor d of the i-th feature point. i .

[0031] The i-th feature point is determined based on its descriptor and pixel coordinates.

[0032] Furthermore, in step S3, the descriptors that yield valid matches include the following:

[0033] Define the descriptor of the i-th feature point in the current high-resolution image as: Define the descriptor of the j-th feature point in the historical benchmark high-resolution image as: Calculate using Euclidean distance and The similarity metric match(i) is calculated using the following formula:

[0034]

[0035] Define the distances between the nearest neighbor and the second nearest neighbor as match(i). first and match(i) second When match(i) second / match(i) first When the value is less than 0.8, the matching result is valid, indicating that the i-th feature point in the current high-resolution image has matched the j-th feature point in the historical benchmark high-resolution image. and For a valid match, use the descriptor.

[0036] Furthermore, in step S4, the feature point position offset data of the current high-resolution image includes the following:

[0037] The pixel coordinates of the j-th valid matching descriptor in the historical benchmark high-resolution image are: The pixel coordinates of the i-th valid matching descriptor in the current high-resolution image are: The feature point position offset is obtained by comparing the pixel coordinates of the historical benchmark high-resolution image and the current high-resolution image. The specific formula is as follows:

[0038]

[0039] ΔP(i)=(Δx i ,Δy i )

[0040] Where, Δx irepresents the pixel-level offset of the i-th feature point in the current high-resolution image in the horizontal direction; Δy i represents the pixel-level offset of the i-th feature point in the current high-resolution image in the vertical direction; ΔP(i) represents the pixel displacement vector of the i-th feature point in the current high-resolution image, i = 1, 2, …, N, and N represents the number of matched point pairs.

[0041] Further, in step S5, obtaining the deformation monitoring result includes the following contents:

[0042] Based on the vertical distance H between the pinhole camera and the target object to be deformed, the single-pixel size s of the image sensor, and the focal length f of the pinhole camera, the conversion coefficient k between the pixel offset and the actual physical displacement is obtained conv ,

[0043] Based on k conv , the feature point position offset data of the current high-resolution image is converted into the deformation in the physical space, and the specific formula is:

[0044]

[0045] wherein, represents the horizontal direction physical displacement of the i-th feature point of the target object to be deformed, represents the vertical direction physical displacement of the i-th feature point of the target object to be deformed.

[0046] The physical displacements of all feature points are obtained, and the average values of the horizontal direction physical displacements and the vertical direction physical displacements of all feature points are obtained, respectively, to obtain the deformation monitoring result of the target object to be deformed.

[0047] Further, the present application also proposes a system of a super-resolution reconstruction deformation monitoring method, which comprises:

[0048] A high-resolution image acquisition module is configured to acquire the current low-resolution image of the target object to be deformed by using a pinhole camera model, input the image into a super-resolution reconstruction model to obtain the current high-resolution image of the image, acquire the historical reference low-resolution image of the target object to be deformed, and input the image into the super-resolution reconstruction model to obtain the historical reference high-resolution image of the image.

[0049] A feature point acquisition module is configured to extract the feature points of the current high-resolution image and the historical reference high-resolution image by using a scale-invariant feature transformation algorithm, respectively.

[0050] A feature point matching module is configured to match the feature points of the current high-resolution image and the feature points of the historical reference high-resolution image to obtain the effective matched descriptors.

[0051] The feature point position offset data acquisition module is configured to obtain, according to the valid matched descriptors, matching feature points corresponding to the feature points of the current high-resolution image, and obtain feature point position offset data of the current high-resolution image according to current pixel positions of the matching feature points in the current high-resolution image and historical reference pixel positions of the matching feature points in the historical reference high-resolution image.

[0052] The deformation monitoring result acquisition module is configured to calculate a deformation amount of the target object to be monitored in an actual physical space according to the feature point position offset data of the current high-resolution image and in combination with a pixel-physical conversion coefficient, and obtain a deformation monitoring result of the target object to be monitored.

[0053] Further, the present application also proposes an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps of the super-resolution reconstruction deformation monitoring method when executing the computer program.

[0054] Further, the present application also proposes a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the super-resolution reconstruction deformation monitoring method.

[0055] Compared with the prior art, the present application has the following technical effects:

[0056] 1. Improved image quality: The present application significantly improves the detail clarity of the original low-quality image through the super-resolution reconstruction model, and enhances the expression ability of key areas such as edges and textures.

[0057] 2. High deformation monitoring accuracy: The present application can effectively extract key feature points and perform feature matching, thereby improving the accuracy of deformation monitoring.

[0058] 3. Wide applicability: The present application can run on embedded or edge computing devices, and is suitable for resource-limited situations such as construction sites, and has a wide engineering application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the overall implementation flowchart of the present application.

[0060] Figure 2 is the flowchart of obtaining a high-resolution image of the present application.

[0061] Figure 3 is the result image of the embodiment of the present application.

[0062] Figure 4 is the structure diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0063] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of the present application.

[0064] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The case of "target", "original" and the like is similar, and will not be described here again. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0065] The present application will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical scheme of the present application, and cannot limit the protection scope of the present application.

[0066] To achieve the above-mentioned purpose, the present application proposes a deformation monitoring method for super-resolution reconstruction, as shown in Figure 1 The specific steps are as follows:

[0067] S1, in a set monitoring period, using a visual sensor installed at a fixed position to obtain a current low-resolution image of a target object to be deformed, inputting the image into a super-resolution reconstruction model to obtain a current high-resolution image of the image, and using a scale-invariant feature transformation algorithm to extract feature points of the high-resolution image. For example, the target object to be deformed can be at least one of a building, a bridge and equipment, etc. that need to be deformed, and can also be an ancient building and / or a dangerous building, a bridge, etc. that is prone to deformation.

[0068] S2, obtaining a historical reference low-resolution image of a target object to be deformed, inputting the image into a super-resolution reconstruction model to obtain a historical reference high-resolution image of the image, and using a scale-invariant feature transformation algorithm to extract historical reference feature points of the image.

[0069] wherein obtaining the high-resolution image comprises the following contents, as shown in Figure 2

[0070] The super-resolution reconstruction model comprises an interpolation module, a depth separable convolution layer, an attention layer, and an adaptive residual module; wherein the attention layer comprises a channel attention layer and a spatial attention layer.

[0071] The low-resolution image is manually selected to monitor the region, and a monitoring region image of the image with deformation monitoring demand is obtained. The monitoring region image is processed by denoising, image enhancement and color correction to improve the accuracy and stability of subsequent processing, and a processed monitoring region image is obtained.

[0072] The processed monitoring region image is input into the super-resolution reconstruction model, and an initial bilinear interpolation operation is performed by the interpolation module to obtain a basic high-resolution feature image. The image is subjected to spatial convolution operation and point-by-point convolution operation by the depth separable convolution layer, information integration is realized between channels, the feature extraction capability is maintained while the parameter quantity and the calculation complexity are significantly reduced, and the corresponding local feature is obtained. The local feature is subjected to feature weighting processing by the channel attention layer, average pooling and maximum pooling by the spatial attention layer, and two-dimensional attention features are obtained. The adaptive residual module is used to fuse the two-dimensional attention features and the basic high-resolution feature image to obtain a high-resolution image.

[0073] The feature point extraction comprises the following contents:

[0074] The scale-invariant feature transform algorithm can extract key points with scale, rotation and illumination invariance from images, and is a highly stable feature extraction method in structural monitoring image analysis.

[0075] The high-resolution image I(x i ,y i ) is convolved with a Gaussian kernel of different scales to generate a scale space image L(x i ,y i ,σ), and the specific expression is:

[0076] L(x i ,y i ,σ)=G(x i ,y i ,σ)*I(x i ,y i )

[0077] Wherein, * represents convolution operation; G(x i ,y i ,σ) represents a two-dimensional Gaussian kernel, ​σ denotes the standard deviation of the two-dimensional Gaussian kernel, x i represents the horizontal pixel position of the i-th feature point in the high-resolution image, y i represents the vertical pixel position of the i-th feature point in the high-resolution image.

[0078] The difference pyramid is used to difference the adjacent scale space images to obtain the difference Gaussian image, and the specific expression is:

[0079] D(x i ,y i ,σ)=L(x i ,y i ,kσ)-L(x i ,y i ,σ)

[0080] where D(x i ,y i ,σ) represents the difference Gaussian image of pixel point coordinates (x i ,y i ), L(x i ,y i ,kσ) represents the i-th feature point in the high-resolution image. The pixel point coordinates (x i ,y i ) are the scale space image with a Gaussian kernel standard deviation of kσ, and k represents the scale factor. The size of kσ can be controlled to obtain spatial images of different scales.

[0081] The local extreme points of the difference Gaussian image in space and scale direction are extracted, and Taylor expansion is used to fit the local extreme points to obtain the image coordinates of the feature points, and the specific expression is:

[0082]

[0083] where, represents the pixel coordinates of the i-th feature point in the high-resolution image, represents the first derivative of the difference Gaussian image, represents the second derivative of the difference Gaussian image.

[0084] The gradient direction histogram of the feature point neighborhood is calculated, and the main direction of the feature point is determined, and the specific expression is:

[0085]

[0086] where θ represents the main direction of the feature point, L xi represents the gradient of the i-th feature point in the x direction in the scale space image, represents the gradient of the i-th feature point in the y direction in the scale space image,

[0087] Divide the i-th feature point neighborhood into 4x4 regions, and construct a histogram of 8 gradient directions relative to the main direction θ i of the i-th feature point for each region, obtaining 16x8=128 descriptors d i of the i-th feature point.

[0088] Based on the descriptor of the i-th feature point and the pixel coordinates of the i-th feature point, determine the i-th feature point for subsequent inter-image matching.

[0089] S3, match the feature points in step S1 and the historical reference feature points in step S2 to obtain valid matching descriptors; the specific content is:

[0090] Define the descriptor of the i-th feature point in the current high-resolution image as Define the descriptor of the j-th feature point in the historical reference high-resolution image as Calculate the similarity measure match(i) of and using the Euclidean distance, and the specific formula is:

[0091]

[0092] Use the ratio test method to exclude ambiguous matches, and define the distances of the nearest neighbor and the second nearest neighbor as match(i) first and match(i) second When match(i) second / match(i) first <0.8, the matching result is valid, indicating that the i-th feature point in the current high-resolution image is matched with the j-th feature point in the historical reference high-resolution image, and are valid matching descriptors.

[0093] S4, obtain the matching feature points corresponding to the feature points in step S1 according to the valid matching descriptors, and obtain the feature point position offset data of the current high-resolution image according to the current pixel position of the matching feature points in the current high-resolution image and the historical reference pixel position of the matching feature points in the historical reference high-resolution image; the specific content is:

[0094] The pixel coordinates of the j-th matching valid descriptor in the historical reference high-resolution image are The pixel coordinates of the i-th matching valid descriptor in the current high-resolution image are Obtain the feature point position offset amount by the pixel coordinate difference between the historical reference high-resolution image and the current high-resolution image, and the specific formula is:

[0095]

[0096] ΔP(i)=(Δx i ,Δy i )

[0097] Where, Δx i Δy represents the pixel-level offset of the i-th feature point in the current high-resolution image along the horizontal direction. i ΔP(i) represents the pixel-level offset of the i-th feature point in the current high-resolution image in the vertical direction; ΔP(i) represents the pixel displacement vector of the i-th feature point in the current high-resolution image, i = 1, 2, ..., N, where N represents the number of matching point pairs.

[0098] S5. Based on the feature point position offset data of the current high-resolution image, obtain the deformation monitoring results of the target object to be deformed; the specific content is as follows:

[0099] Based on the vertical distance H between the pinhole camera and the target object to be deformed, the single pixel size s of the image sensor, and the focal length f of the pinhole camera, the conversion coefficient k between pixel offset and actual physical displacement is obtained. conv ,

[0100] Based on k conv The feature point offset data of the current high-resolution image is converted into deformation in physical space. The specific formula is as follows:

[0101]

[0102] in, This represents the horizontal physical displacement of the i-th feature point of the target object to be deformed. This represents the vertical physical displacement of the i-th feature point of the target object to be deformed.

[0103] The physical displacements of all feature points are obtained, and the average values ​​of the horizontal and vertical physical displacements of all feature points are calculated to obtain the deformation monitoring results of the target object to be deformed.

[0104] This invention is applicable to the monitoring of object deformation, and is especially applicable to the monitoring of deformation of objects at a distance.

[0105] Example:

[0106] like Figure 3 As shown, a visual sensor is used to acquire the current low-resolution image. The hardware layout of the visual sensor needs to be optimized according to the actual structural position of the object being monitored.

[0107] The visual sensor includes an industrial camera, a high-definition lens, and necessary auxiliary equipment, is installed on an adjustable mounting bracket, a gimbal, or a tripod, ensures that the imaging view covers the entire monitoring area, and reduces view distortion. To improve imaging quality and system stability, a waterproof cover, a dust cover, a light shield, a temperature control device, or other external protection components can be selected.

[0108] The visual sensor should be fixed at a position with stable structure and small vibration, and the shooting direction should be perpendicular to the target surface or be set according to the best observation angle, to ensure that the target area in the image is complete and not blocked. The focal length of the lens should be selected according to the monitoring distance and the size of the area, to meet the required image resolution and detail performance.

[0109] The visual sensor is connected to the upper processing device through an image acquisition interface (such as USB 3.0, Gigabit Ethernet GigE, PoE power supply interface, etc.), to realize real-time transmission and processing of images.

[0110] The internal structure of the visual sensor includes a camera module, a lens system, and an image acquisition and processing unit. The camera module can use CMOS (Complementary Metal-Oxide-Semiconductor) or CCD (Charge-Coupled Device) type image sensors, which have high sensitivity and dynamic range, and are suitable for visual perception of structural surface micro-deformation; the lens system is configured with focal length and aperture according to the monitoring scene, and can be a fixed focus or electric zoom lens, to meet the clear imaging requirement at different distances.

[0111] The image acquisition and processing unit integrates an image signal processor, which is used to perform preliminary image denoising, gamma correction, exposure control, and other operations, to improve the image signal-to-noise ratio and stabilize the output quality. The entire visual sensor unit can be powered externally, supports remote control function, and realizes automatic image acquisition, parameter adjustment, and state monitoring.

[0112] In the obtained low-resolution image, a graphical interface is provided for an operator to manually select a monitoring area of interest. The user is allowed to draw a selection box in the image according to key parts of the structure (such as crack concentration area, connection node, component edge, etc.), to accurately crop out an image part containing potential deformation area, i.e., a monitoring area image.

[0113] A curve graph of the deformation amount of the monitored object changing with time is obtained, in which the horizontal coordinate is time in hours, and the vertical coordinate is deformation amount in millimeters. Figure 3The horizontal displacement x (labeled as a blue solid circle) and the vertical displacement y (labeled as an orange solid square) are 10 hours of data, with a monitoring data point every half hour, for a total of 21 data points. The maximum value of the horizontal displacement x is 0.127 mm, the minimum value is -0.130 mm, and the average value is -0.002 mm; the maximum value of the vertical displacement y is 0.110 mm, the minimum value is -0.107 mm, and the average value is -0.001 mm. From the overall trend, the deformation variables in both directions are maintained within the range of ±0.2 mm, and periodically fluctuate with a small fluctuation amplitude.

[0114] The embodiment of the present application also provides a deformation monitoring system for super-resolution reconstruction, comprising a high-resolution image acquisition module, a feature point acquisition module, a feature point matching module, a feature point position offset data acquisition module, a deformation monitoring result acquisition module and a computer program capable of running on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided by the embodiment of the present application, has the corresponding function modules and beneficial effects of the method. Technical details not described in detail in the present embodiment can be referred to the method provided by the embodiment of the present application.

[0115] As shown in Figure 4 The embodiment of the present application also provides an electronic device 10, which is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and the like. The electronic device 10 can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and the like. The components of the present application, their connections, and their functions, as well as the overall architecture, are by way of example only and not intended to limit the present application described and / or claimed herein.

[0116] The electronic device 10 includes at least one processor 11, and a memory, such as a ROM (Read-Only Memory) 12, a RAM (Random Access Memory) 13, etc., which is in communication with the at least one processor 11, wherein the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0117] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16 (e.g., a keyboard, a mouse, etc.), an output unit 17 (e.g., various types of displays, speakers, etc.), a storage unit 18 (e.g., a magnetic disk, an optical disk, etc.), and a communication unit 19 (e.g., a network card, a modem, a wireless communication transceiver, etc.). The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit, a graphics processing unit, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, a digital signal processor, and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the object deformation detection method.

[0119] It should be noted that the processor executes the computer program to correspond to the specific steps of the method provided by the embodiments of the present application, has the function modules and beneficial effects corresponding to the execution method. Technical details not described in detail in the embodiments can refer to the method provided by the embodiments of the present application.

[0120] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program.

[0121] The computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable signal medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disks read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] The computer programs can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flow diagrams and / or the block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0123] It should be noted that the computer program corresponds to the specific steps of the method provided by the embodiments of the present application when executed by the processor, and has the corresponding function modules and beneficial effects of the executed method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0124] The above only describes the preferred embodiments of the present application. It should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. A method of deformation monitoring by super-resolution reconstruction, characterized in that, The method comprises the following steps: S1, obtaining a current low-resolution image of a target object to be monitored for deformation by using a pinhole camera model, inputting the image into a super-resolution reconstruction model to obtain a current high-resolution image of the image, and extracting feature points of the high-resolution image by using a scale-invariant feature transform algorithm; S2, obtaining a historical reference low-resolution image of the target object to be monitored for deformation, inputting the image into the super-resolution reconstruction model to obtain a historical reference high-resolution image of the image, and extracting historical reference feature points of the image by using the scale-invariant feature transform algorithm; S3, matching the feature points in step S1 with the historical reference feature points in step S2 to obtain effective matching descriptors; S4, obtaining matching feature points corresponding to the feature points in step S1 according to the effective matching descriptors, and obtaining feature point position offset data of the current high-resolution image according to current pixel positions of the matching feature points in the current high-resolution image and historical reference pixel positions of the matching feature points in the historical reference high-resolution image; S5, obtaining a deformation monitoring result of the target object to be monitored for deformation according to the feature point position offset data of the current high-resolution image.

2. The method of deformation monitoring by super-resolution reconstruction according to claim 1, characterized in that, The high-resolution image comprises the following contents: The super-resolution reconstruction model comprises an interpolation module, a depth separable convolution layer, an attention layer, and an adaptive residual module; The attention layer comprises a channel attention layer and a spatial attention layer; A monitoring region in the low-resolution image is selected to obtain a monitoring region image in the image that requires deformation monitoring, and the monitoring region image is processed by denoising, image enhancement, and color correction to obtain a processed monitoring region image; The processed monitoring region image is input into the super-resolution reconstruction model, an initial bilinear interpolation operation is performed by using the interpolation module to obtain a basic high-resolution feature image, the image is subjected to spatial convolution operation and point-by-point convolution operation by using the depth separable convolution layer to obtain corresponding local features, the local features are subjected to global pooling feature weighting processing by using the channel attention layer, average pooling and maximum pooling by using the spatial attention layer to obtain two-dimensional attention features, and the two-dimensional attention features are fused with the basic high-resolution feature image by using the adaptive residual module to obtain the high-resolution image.

3. The method of deformation monitoring by super-resolution reconstruction according to claim 1, characterized in that, The feature point extraction comprises the following contents: Convolve the high-resolution image I(x i ,y i ) with different scales of Gaussian kernels to generate scale space images L(x i ,y i ,σ), which is expressed as: L(x i ,y i ,σ) = G(x i ,y i ,σ) * I(x i ,y i ) where * denotes a convolution operation; G(x i ,y i ,σ) denotes a two-dimensional Gaussian kernel, σ denotes a standard deviation of the two-dimensional Gaussian kernel, x i denotes a horizontal pixel position of the i-th feature point in the high-resolution image, y i denotes a vertical pixel position of the i-th feature point in the high-resolution image; Difference Gaussian images are obtained by performing difference operation on adjacent scale space images by using a difference pyramid, and the specific expression is as follows: D(x i ,y i ,σ) = L(x i ,y i ,kσ) - L(x i ,y i ,σ) where D(x i ,y i ,σ) represents a difference of Gaussian image of pixel point coordinates (x i ,y i ), L(x i ,y i ,kσ) represents a scale space image of pixel point coordinates (x i ,y i ) of the i-th feature point in the high-resolution image with a Gaussian kernel standard deviation of kσ, and k represents a scale factor; Local extreme points of the difference Gaussian images in the spatial and scale directions are extracted, Taylor expansion is used to fit the local extreme points to obtain image coordinates of the feature points, and the specific expression is as follows: wherein, represents the pixel coordinates of the i-th feature point in the high-resolution image, represents the first derivative of the difference of Gaussians image, represents the second derivative of the difference of Gaussians image; The gradient direction histogram of the neighborhood of the feature points is calculated, and the main direction of the feature points is determined, and the specific expression is as follows: wherein θ denotes a principal direction of the feature point, denotes a gradient of the i-th feature point in the x-direction in the scale space image, denotes a gradient of the i-th feature point in the y-direction in the scale space image, Divide the neighborhood of the i-th feature point into 4x4 regions, and construct a histogram of 8 gradient directions relative to the main direction θ of the i-th feature point for each region to obtain a descriptor d i of the i-th feature point i ; The i-th feature point is determined based on the descriptor of the i-th feature point and the pixel coordinates of the i-th feature point.

4. The method of deformation monitoring by super-resolution reconstruction according to claim 3, characterized in that, In step S3, the effective matching descriptors comprise the following contents: The descriptor of the i-th feature point in the current high-resolution image is defined as The descriptor of the j-th feature point in the historical reference high-resolution image is defined as The similarity measure match(i) is calculated using the Euclidean distance and The specific formula is: match(i) = min(match(i), match(i-1)) first and match(i) second , match(i) = min(match(i), match(i-1)) second / match(i) first <0.8, the matching result is valid, indicating that the i-th feature point in the current high-resolution image is matched with the j-th feature point in the historical reference high-resolution image, and is the descriptor of the valid matching.

5. The method of super-resolution reconstructed deformation monitoring according to claim 4, wherein, In step S4, the feature point position offset data of the current high-resolution image comprises the following contents: Pixel coordinates of the jth matching valid descriptor in the history reference high-resolution image are Pixel coordinates of the ith matching valid descriptor in the current high-resolution image are The feature point position offset is obtained by the pixel coordinate difference between the history reference high-resolution image and the current high-resolution image, and the specific formula is: ΔP(i) = (Δx i ,Δy i ) where Δx i represents the pixel-level offset of the i-th feature point in the current high-resolution image in the horizontal direction; Δy i represents the pixel-level offset of the i-th feature point in the current high-resolution image in the vertical direction; ΔP(i) represents the pixel displacement vector of the i-th feature point in the current high-resolution image, i = 1, 2,..., N, and N represents the number of matched point pairs.

6. The method of super-resolution reconstructed deformation monitoring according to claim 1, wherein, In step S5, the deformation monitoring result comprises the following contents: Based on the vertical distance H between the pinhole camera and the target object to be deformation monitored, the image sensor single-pixel size s and the pinhole camera focal length f, the conversion coefficient k between the pixel offset and the actual physical displacement is obtained conv , Based on k conv The feature point position offset data of the current high-resolution image is converted into the deformation variable in the physical space, and the specific formula is: wherein, represents a horizontal direction physical displacement of an i-th feature point of a target object to be deformation monitored, represents a vertical direction physical displacement of an i-th feature point of a target object to be deformation monitored, Δx i represents a pixel level offset of an i-th feature point in a current high resolution image in a horizontal direction, Δy i represents a pixel level offset of an i-th feature point in a current high resolution image in a vertical direction; The physical displacements of all feature points are obtained, and the horizontal direction physical displacement and the vertical direction physical displacement of all feature points are averaged respectively to obtain the deformation monitoring result of the target object to be monitored.

7. System for applying the deformation monitoring method of super-resolution reconstruction of claim 1, characterized by the fact that, The method comprises the following steps: a high-resolution image acquisition module is configured to acquire a current low-resolution image of a target object to be monitored by using a pinhole camera model, and input the image into a super-resolution reconstruction model to obtain a current high-resolution image of the image; a historical reference low-resolution image of the target object to be monitored is acquired, and the image is input into the super-resolution reconstruction model to obtain a historical reference high-resolution image of the image; a feature point acquisition module is configured to extract feature points of the current high-resolution image and the historical reference high-resolution image by using a scale-invariant feature transform algorithm; a feature point matching module is configured to match the feature points of the current high-resolution image and the feature points of the historical reference high-resolution image to obtain effective matching descriptors; a feature point position offset data acquisition module is configured to obtain matching feature points corresponding to the feature points of the current high-resolution image according to the effective matching descriptors, and obtain feature point position offset data of the current high-resolution image according to current pixel positions of the matching feature points in the current high-resolution image and historical reference pixel positions of the matching feature points in the historical reference high-resolution image; a deformation monitoring result acquisition module is configured to calculate a deformation amount of the target object to be monitored in an actual physical space according to the feature point position offset data of the current high-resolution image and in combination with a pixel-physical conversion coefficient, and obtain a deformation monitoring result of the target object to be monitored.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the super-resolution reconstruction deformation monitoring method in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The computer program is run by the processor to execute the super-resolution reconstruction deformation monitoring method in any one of claims 1 to 6.

Images