Building structure image correction and restoration method and device

Abstract

The invention relates to a building structure image correction and restoration method and device. The method comprises the following steps: shooting a target object by using a camera equipped with a distance sensor to obtain a to-be-corrected image; a plurality of reference lines perpendicular to the ground are marked on the to-be-corrected image, the to-be-measured size of the target object is calculated through the reference lines, the lengths of the two adjacent edges of the target object are obtained, and the aspect ratio of the target object is obtained; a homography matrix is determined, perspective transformation is applied to the to-be-corrected image by using the obtained homography matrix, and a corrected image is generated; the actual distance between the target object and the camera is determined through the distance sensor arranged on the camera, so that the actual height corresponding to the reference line of the shot object is determined through the focal length of the camera, and the actual size of the target object is determined. Compared with the prior art, the method has the advantages that the structure size can be accurately restored, the adaptability is enhanced, the measurement error is minimized and the like on the premise that the aspect ratio information of the structure does not exist.

Description

Technical Field

[0001] The present invention relates to an image processing technology, and in particular to a method and a device for correcting and restoring an image of a building structure. Background Art

[0002] In actual engineering application scenarios, engineers or structural inspectors often need to capture images of building structures with cracks and target objects for further analysis and evaluation. In this process, it is often difficult to shoot with the camera completely perpendicular to the target plane. Whether it is the complexity of the on-site environment or the limitations of the shooting equipment, the camera shooting direction may not be strictly perpendicular to the target plane. In this case, the captured image may be distorted in perspective.

[0003] That is, the objects in the image do not match the actual size, which makes the objects in the photo appear "larger when near and smaller when far away", which will lead to errors in the subsequent measurement of the crack size.

[0004] When correcting distorted images, it is very important to determine the exact shape of the target object. Existing image correction methods usually use the bounding box of the distorted original image as the target shape for correction. When the shape of the target object is relatively common, such as an A4 paper, a playing card, or an ID card, it is very simple to determine its target shape. However, when facing target objects of unconventional sizes, the correction caused by this method still has distortion in the horizontal direction and cannot be accurately measured. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for correcting and restoring an architectural structure image in order to overcome the defects of the prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a method for correcting and restoring an image of a building structure is provided, which combines distance sensor data with image processing technology to restore the original aspect ratio of the target object. The method comprises the following steps:

[0008] S1. Use a camera equipped with a distance sensor to shoot the target object to obtain an image to be corrected;

[0009] S2. Mark multiple reference lines perpendicular to the ground on the image to be corrected, where the actual distance between two reference lines is the width of the target object; the length of one reference line segment is the height of the target object; a rectangle is determined by the reference lines to represent the target object, which is used to assist in calculating the homography matrix in the subsequent image correction stage;

[0010] S3, calculating the size of the target object to be measured through the reference line, and obtaining the lengths of two adjacent sides of the target object, that is, obtaining the aspect ratio of the target object;

[0011] S4, using the image coordinates and real coordinates of four points to calculate the homography matrix; the four points are the four corner points of the object to be measured, the image coordinates are directly obtained from the image, and the real coordinates are the coordinates calculated by the above S3;

[0012] S5. Apply perspective transformation to the image to be rectified using the homography matrix obtained in S4 to generate a rectified image.

[0013] As a preferred solution, in S3, when the size of the target object to be measured is calculated by the reference line, when there is only one reference line, that is, the length of the reference line segment is the height of the target object to be measured, the actual length of the height of the target object is the actual length of the reference line. The specific calculation process includes:

[0014] The pixel length of the reference line on the image is known, and the length of the line segment projected from the reference line to the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor;

[0015] The length of the line segment projected from the reference line to the camera sensor is known, and the actual length of the reference line, ie, the actual height length of the target object, is calculated using the distance from the reference line to the lens plane and the lens focal length.

[0016] Furthermore, the actual height of the target object is equal to the product of the length of the line segment projected by the reference line onto the camera sensor and the focal length of the lens divided by the distance from the reference line to the lens plane.

[0017] Furthermore, the length of the line segment projected from the reference line onto the camera sensor is equal to the product of the pixel length of the reference line on the image and the diagonal length of the sensor divided by the pixel diagonal size of the image.

[0018] As a preferred solution, in S3, the measured size of the target object is calculated by reference lines. When there are two reference lines, the distance between the two reference lines is the width of the target object. The actual length of the width of the target object is the actual distance between the two reference lines. The actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the reference line and the lens distance and the square of the distance between the two reference lines in the x-axis direction.

[0019] Furthermore, the difference between the distances between the reference lines and the lens is the difference between the distances between the two reference lines and the lens plane, which is the absolute value of the difference between the distances between the two reference lines and the lens plane.

[0020] As a preferred solution, the process of determining a rectangle uses edge detection to reduce manual work, and the edge detection process includes adding three vertical reference lines in the image, applying a Gaussian blur algorithm to minimize noise, converting the original image to a grayscale image, and executing a Canny edge detection algorithm.

[0021] Furthermore, during the execution of the Canny edge detection algorithm, once the Canny detection algorithm detects the edge of the beam, the detected edge of the beam and the three vertical reference lines are combined to form a quadrilateral, that is, the shape of the target object is obtained, which represents the original image corresponding to the homography matrix.

[0022] As a preferred solution, when the camera photographs the target object, the top and bottom edges of the measured surface of the target object are horizontal, the surface itself is perpendicular to the ground, and the camera lens is perpendicular to the ground.

[0023] According to another aspect of the present invention, there is provided a device for correcting and restoring an image of a building structure, and the device operates by applying the method for correcting and restoring an image of a building structure as described above.

[0024] Compared with the prior art, the present invention:

[0025] 1. Improve measurement accuracy. Traditional shooting methods often lead to large errors in measurement results due to perspective distortion. The present invention combines sensor distance data with image processing technology, accurately corrects the image through the homography matrix, eliminates the influence of perspective distortion, thereby improving measurement accuracy and ensuring that the measurement results are closer to the actual value;

[0026] 2. Enhanced adaptability. In the traditional correction and restoration process, the target object is often a common object such as A4 paper, playing cards or ID cards, with simple and limited shapes. In the present invention, the actual length of the reference line is calculated from the pixel length of the reference line on the image, using the pixel diagonal size of the image, the diagonal length of the sensor, the distance from the reference line to the lens plane and the focal length of the lens, that is, the actual length of the target object's measured size. The original width and height of the target object are further calculated and used to restore the distorted image, breaking the size limitation, so that it is suitable for objects of complex and uncommon sizes, and significantly improving the adaptability of image restoration.

[0027] 3. Minimize the error in measuring cracks on beams. The edge of the beam is detected using the Canny detection algorithm. The edges obtained by the vertical reference line and the edge detection can form a quadrilateral, which represents the original image corresponding to the homography matrix. Combined with the target image of known size, the matrix itself can be derived. The entire plane also maintains the correct aspect ratio, eliminating all distortion. This process ensures that the error in measuring cracks on beams can be minimized.

[0028] 4. Easy to operate. The distance sensor data is combined with image processing technology to eliminate perspective distortion in the image, realizing an integrated automated process of on-site shooting, perspective correction and size recognition. This eliminates the tedious post-processing and correction process of a large number of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the building structure image correction and restoration method;

[0030] Figure 2 Schematic diagram of the light path when the camera photographs an object;

[0031] Figure 3 is a planar schematic diagram of an image on a camera sensor;

[0032] Figure 4 A schematic diagram of the perspective correction process proposed by the present invention;

[0033] Figure 5 This is a schematic diagram of the transformation result after applying the homography matrix to the original image;

[0034] Figure 6 Schematic diagram of the comparison between the original image and the corrected image;

[0035] Figure 7 This is a schematic diagram comparing the correction results of the traditional algorithm, the correction results of the algorithm proposed in the present invention, and the real picture. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] Example

[0038] In this embodiment, in order to accurately determine the actual size of the object and restore its aspect ratio, a method for correcting and restoring an architectural structure image is proposed, which combines distance sensor data with image processing technology to restore the original aspect ratio of the target object. Figure 1 shown.

[0039] In this embodiment, in order to verify the performance of the invented building structure image correction and restoration method, a site setting is carried out. The selected building structure in this embodiment is a concrete structure, that is, the target object model of the experiment is a five-story reinforced concrete frame structure model with a total height of 5.0 meters (including the base) and a plane size of 3.6m×3.0m. Cracks of varying degrees of damage appear at the beam-column nodes. The method proposed in the present invention is intended to be applied to the surface cracks, and attempts are made to detect, extract and calculate the crack size, which is then compared with the actual measurement results to verify the accuracy.

[0040] In this embodiment, five live images of the target object are taken using a camera equipped with a distance sensor, and the lens plane is adjusted to be perpendicular to the ground. When each image is taken, a lightweight distance sensor fixed to the camera is used to measure and record the corresponding object distance. A reference line is selected on the surface of the target object.

[0041] In this embodiment, the object distance between the reference line and the lens plane is used to calculate their projection length on the camera sensor; the optical path diagram of the camera and the distance sensor imaging based on the pinhole imaging model is as follows: Figure 2 As shown, the planar schematic diagram of the image on the camera sensor is as follows Figure 3 shown.

[0042] Where h is the actual length of the reference line; f is the focal length of the lens; d0 is the object distance between reference line 0 and the lens plane; d1 is the object distance between reference line 1 and the lens plane; w 01 is the distance between the two reference lines in the x-axis direction, which is used to calculate the tilt length between the reference lines; h0 and h1 are the lengths of the line segments projected onto the camera sensor; d ′ 1 (dashed red line) is the slope length between the reference lines in the real world.

[0043] In this example, the subsequent analysis relies on the following conditions: the top and bottom edges of the measured surface are horizontal; the surface itself is perpendicular to the ground; and the camera lens is also perpendicular to the ground. These requirements are usually easy to meet, because many cameras are equipped with built-in angle sensors, or use external levels to achieve accurate positioning.

[0044] Combining the horizontal distance to the target object measured by the camera distance sensor and the resolution of the image, the pixel length of the reference line on the image is converted into its length in the real world to obtain the length of two adjacent sides of the target object;

[0045] In this embodiment, a reference line is set, that is, the line segment of the target object to be measured is marked as the reference line, and the actual length of the target object to be measured is the actual length of the reference line. The specific calculation process includes:

[0046] The pixel length of the reference line on the image is known, and the length of the line segment projected from the reference line to the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor;

[0047] The length of the line segment projected from the reference line to the camera sensor is known, and the actual length of the reference line, that is, the actual length of the target object to be measured, is calculated using the distance from the reference line to the lens plane and the focal length of the lens. Specifically, the following mathematical relationship is obtained:

[0048]

[0049] Thus:

[0050]

[0051] Where h is the actual length of the reference line, h i i is the pixel length of the reference line on the image, h i is the length of the line segment projected from the reference line i onto the camera sensor; r is the pixel diagonal size of the image; l is the diagonal length of the sensor; d i is the object distance between reference line i and the lens plane; f is the focal length of the lens.

[0052] In this embodiment, the focal length of the camera lens, sensor size and image resolution can generally be found in the product manual or on the Internet. 2 , with a diagonal length of 43.27mm), the user manual will usually state the "equivalent focal length" to compare the diagonal field of view (DFOV) with other cameras with different sensor sizes. To convert between actual focal length and equivalent focal length, the ratio between the camera's sensor size and the standard sensor size is essential. Once the conversion ratio is obtained, the equivalent focal length can be derived from the actual length, and vice versa. For example, the following are the specifications of the main camera of the mobile phone used in this embodiment:

[0053] Table 1 Camera parameters

[0054]

[0055] In this embodiment, according to the above conclusion, the equivalent focal length of the camera on the mobile phone is:

[0056]

[0057] where f ′ is the equivalent focal length, f is the focal length of the lens

[0058] In some cases, camera manufacturers may only provide the equivalent focal length of their products.

[0059] In addition to converting the ratio between pixel length and actual length, the actual length of the target object to be measured can also be calculated in the following way:

[0060] Set two reference lines, the distance between the two reference lines is the size of the target object to be measured, the actual length of the size of the target object to be measured is the actual distance between the two reference lines, and the actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the reference line and the lens distance and the square of the distance between the two reference lines in the x-axis direction. Reference line 0 and reference line 1, the distance between the two reference lines is the actual length of the size of the target object to be measured, and there is the following calculation method:

[0061]

[0062] in

[0063] Δd i =|d0-d i |.

[0064] Where d0 is the object distance between reference line 0 and the lens plane; d1 is the object distance between reference line 1 and the lens plane; w 01 is the distance between the two reference lines in the x-axis direction, which is used to calculate the tilt length between the reference lines; h0 and h1 are the lengths of the line segments projected onto the camera sensor by reference lines 0 and 1; d1′ and d2′; Figure 1 is the distance between the two reference lines on the beam, d ′ 1 (dashed red line) is the slope length between the reference lines in the real world, along the length of the beam.

[0065] In this embodiment, the lengths of two pairs of adjacent sides of the rectangle on the plane are obtained, which satisfies the necessary conditions for determining the homography matrix. The matrix is ​​applied to the entire image to generate the rectified image. The side of the beam is a plane containing all the necessary reference lines, so this method is feasible.

[0066] In this embodiment, edge detection is used to determine the exact shape of the target object, and a function called cv2.Canny is used to detect edges in the input image. The process includes applying a Gaussian blur algorithm to minimize noise, converting the original image to a grayscale image, and executing the Canny edge detection algorithm. Before these steps begin, the algorithm adds three vertical reference lines to the image as a pre-step. Once the Canny detection algorithm detects the edges of the beam, these edges can form a quadrilateral (a trapezoid, to be precise), representing the original image corresponding to the homography matrix, and combined with the target image of known size, the matrix itself can be derived. The brief flow of the algorithm is as follows Figure 4 As shown, 4(a) is the original image, 4(b) is the image after the edge is extracted using the Canny edge detection algorithm and three reference lines are added, 4(c) is the detected reference rectangle, and 4(d) is the rectified image. According to the above argument, Figure 4 The red rectangle in (c) has been accurately measured to its true size and has the same aspect ratio as its actual size. Therefore, the entire plane it lies on also maintains the correct aspect ratio, eliminating all distortion.

[0067] In this embodiment, the principle of homography matrix is ​​mainly used to restore and present the original image and its correct aspect ratio. When the perspective transformation is applied to a plane, the formula can be written as

[0068]

[0069] Where (x, y) is the coordinates of the point on the original plane, and (X, Y, Z) is the homogeneous coordinates of the point on the new plane. 3×3 The submatrix A in 2×2 represents affine transformation, T 2×1 represents the translation transformation, V T Represents a perspective transformation.

[0070] The transformation result after applying the homography matrix to the original image is as follows Figure 5 shown. Figure 5 Each subfigure in shows the result of applying the homography matrix to the original shape.

[0071] Expand the formula, we have

[0072] X=a 11 ×x+a 12 ×y+t x ,

[0073] Y=a 21 ×x+a 22 ×y+t y ,

[0074] Z=v1×x+v2×y+s.

[0075] Based on the transformation rules between Cartesian coordinate system and homogeneous coordinate system, that is,

[0076]

[0077] We can get the coordinates of the point in the new plane through the following transformation relationship:

[0078]

[0079] In this embodiment, in order to obtain the homography matrix, at least four pairs of coordinates of points on the original and target planes are required (equivalent to 8 equations). Each numerator and denominator in the two equations above can be divided by the parameter s to eliminate it, so that only 8 unknown parameters are retained, thereby determining all the coefficients of the matrix. In engineering practice, rectangles are one of the most commonly used shapes; therefore, the lengths of two adjacent sides of the target rectangle can be obtained to derive the matrix, and finally the obtained homography matrix is ​​used to apply perspective transformation to the entire image to generate a corrected image.

[0080] In this embodiment, the camera pinhole imaging model, rangefinder and homography matrix are used to design and theoretically derive the correction of structural images and the restoration of aspect ratio, and it is concluded that under certain conditions, structural photos can be accurately corrected with the help of rangefinder and camera built-in parameters.

[0081] Table 2 Accuracy verification of image correction algorithm

[0082]

[0083] The final result after algorithm processing and correction is shown in Table 2. The comparison diagram of the original image and the corrected image is shown in Figure 6 shown.

[0084] In this embodiment, the calculated beam height is compared with the height actually measured with a tape measure; the relative angles of the upper and lower edges of the beam in the image restored by the algorithm are measured. The results show that the algorithm performs well with only slight errors, with the maximum error of 4.43% in height measurement and the maximum error of 1.2° in parallelism. The results obtained in this embodiment are compared with the correction results of the traditional algorithm and the real image taken perpendicular to the plane. Figure 7 As shown, it can be seen that the accuracy of the present invention is significantly better than the traditional correction method.

[0085] In this example, the error mainly occurs in the edge detection stage; in this stage, the algorithm draws a red reference trapezoid on the measurement plane as the original shape to be corrected; in the final image, the trapezoid is corrected to a square by the homography matrix. This means that the error generated by this method can be further reduced if its edges are aligned with the edges of the beam with higher accuracy. The parallelism error is the angle between the upper and lower edges in the corrected image.

[0086] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for correcting and restoring a building structure image, characterized in that: The distance sensor data is combined with image processing technology to restore the original aspect ratio of the target object. The method steps include: S1. Use a camera equipped with a distance sensor to shoot the target object to obtain an image to be corrected; S2. Mark multiple reference lines perpendicular to the ground on the image to be corrected, where the actual distance between two reference lines is the width of the target object; the length of one reference line segment is the height of the target object; a rectangle is determined by the reference lines to represent the target object, which is used to assist in calculating the homography matrix in the subsequent image correction stage; S3, calculating the size of the target object to be measured through the reference line, and obtaining the lengths of two adjacent sides of the target object, that is, obtaining the aspect ratio of the target object; S4, using the image coordinates and real coordinates of four points to calculate the homography matrix; the four points are the four corner points of the object to be measured, the image coordinates are directly obtained from the image, and the real coordinates are the coordinates calculated by the above S3; S5. Apply perspective transformation to the image to be rectified using the homography matrix obtained in S4 to generate a rectified image.

2. A method for correcting and restoring a building structure image according to claim 1, characterized in that: In the step S3, the measured size of the target object is calculated by the reference line. When there is only one reference line, that is, the length of the reference line segment is the height of the target object to be measured, and the actual length of the height of the target object is the actual length of the reference line. The specific calculation process includes: The pixel length of the reference line on the image is known, and the length of the line segment projected from the reference line to the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor; The length of the line segment projected from the reference line onto the camera sensor is known, and the actual length of the reference line, ie, the actual height length of the target object, is calculated using the distance from the reference line to the lens plane and the focal length of the lens.

3. A method for correcting and restoring a building structure image according to claim 2, characterized in that: The actual height of the target object is equal to the product of the length of the line segment projected from the reference line onto the camera sensor and the focal length of the lens divided by the distance from the reference line to the lens plane.

4. The method for correcting and restoring a building structure image according to claim 2, characterized in that: The length of the line segment projected onto the camera sensor by the reference line is equal to the product of the pixel length of the reference line on the image and the diagonal length of the sensor divided by the pixel diagonal size of the image.

5. The method for correcting and restoring a building structure image according to claim 1, characterized in that: In S3, the measured size of the target object is calculated by the reference lines. When there are two reference lines, the actual length of the width of the target object is the actual distance between the two reference lines. The actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the reference line and the lens distance and the square of the distance between the two reference lines in the x-axis direction.

6. A method for correcting and restoring a building structure image according to claim 5, characterized in that: The distance difference between the reference line and the lens is the distance difference between the two reference lines and the lens plane, which is the absolute value of the distance difference between the two reference lines and the lens plane.

7. The method for correcting and restoring a building structure image according to claim 1, characterized in that: The process of determining a rectangle uses edge detection to reduce artifacts, and the edge detection process includes adding three vertical reference lines in the image, applying a Gaussian blur algorithm to minimize noise, converting the original image to a grayscale image, and performing a Canny edge detection algorithm.

8. The method for correcting and restoring a building structure image according to claim 7, characterized in that: During the execution of the Canny edge detection algorithm, once the Canny detection algorithm detects the edge of the beam, the detected edge of the beam and three vertical reference lines together form a quadrilateral, that is, the shape of the target object is obtained, representing the original image corresponding to the homography matrix.

9. The method for correcting and restoring a building structure image according to claim 1, characterized in that: When the camera takes a picture of the target object, the top and bottom edges of the measured surface of the target object are horizontal, the surface itself is perpendicular to the ground, and the lens of the camera is perpendicular to the ground.

10. A building structure image correction and restoration device, characterized in that: The device works by applying the building structure image correction and restoration method as described in any one of claims 1-9.

Images