A method and device for correcting and restoring building structure images

Abstract

The present invention relates to a method and apparatus for correcting and restoring an image of a building structure. The method uses a camera equipped with a distance sensor to photograph a target object to obtain an image to be corrected; multiple reference lines perpendicular to the ground are marked on the image to be corrected, and the target object's dimensions to be measured are calculated using the reference lines to obtain the lengths of two adjacent sides of the target object, i.e., the aspect ratio of the target object; a homography matrix is ​​established and the obtained homography matrix is ​​used to apply a perspective transformation to the image to be corrected to generate a corrected image; the actual distance between the target object and the camera is determined using a distance sensor provided on the camera, thereby determining the actual height corresponding to the reference line of the photographed object using the camera's focal length, and thereby determining the actual dimensions of the target object. Compared to the prior art, the present invention has the advantages of being able to accurately restore structural dimensions without structural aspect ratio information, enhancing adaptability, and minimizing measurement errors.

Description

Technical Field

[0001] The present invention relates to an image processing technology, and in particular to a method and device for correcting and restoring an architectural structure image. Background Art

[0002] In real-world engineering applications, engineers and structural inspectors often need to capture images of cracked structures and target objects for further analysis and evaluation. This process often presents the challenge of keeping the camera perfectly perpendicular to the target plane. Whether due to the complexity of the site environment or limitations of the camera equipment, the camera's orientation may not be perfectly perpendicular to the target plane. In this case, the captured image may exhibit perspective distortion.

[0003] That is, the objects in the image do not match their actual sizes, appearing as "larger when near and smaller when far away" in the photo, which will lead to errors in the subsequent measurement of crack size.

[0004] When correcting distorted images, it's crucial to accurately determine the target object's shape. Existing image correction methods typically use the bounding box of the distorted original image as the target shape for correction. This is straightforward for common target shapes, such as an A4 sheet of paper, playing cards, or ID cards. However, for objects of unusual sizes, this correction still results in horizontal distortion, which cannot be accurately measured. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for correcting and restoring building structure images in order to overcome the defects of the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a method for correcting and restoring an image of a building structure is provided. The method combines distance sensor data with image processing technology to restore the original aspect ratio of the target object. The method comprises the following steps:

[0008] S1. Use a camera equipped with a distance sensor to shoot the target object to obtain an image to be corrected;

[0009] S2. Mark multiple reference lines perpendicular to the ground on the image to be corrected, where the actual distance between two reference lines is the width of the target object; the length of one reference line segment is the height of the target object; a rectangle is determined by the reference lines to represent the target object, which is used to assist in calculating the homography matrix in the subsequent image correction stage;

[0010] S3. Calculate the size of the target object to be measured using the reference line to obtain the lengths of two adjacent sides of the target object, that is, obtain the aspect ratio of the target object;

[0011] S4. Calculate a homography matrix using the image coordinates and real coordinates of four points; the four points are the four corner points of the object to be measured, the image coordinates are obtained directly from the image, and the real coordinates are the coordinates calculated in S3 above;

[0012] S5. Apply perspective transformation to the image to be rectified using the homography matrix obtained in S4 to generate a rectified image.

[0013] As a preferred solution, in S3, when the size of the target object to be measured is calculated using the reference line, when there is only one reference line, that is, the length of the reference line segment is the height of the target object to be measured, the actual length of the height of the target object is the actual length of the reference line. The specific calculation process includes:

[0014] Given the pixel length of the reference line on the image, the length of the line segment projected onto the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor.

[0015] The length of the line segment projected from the reference line onto the camera sensor is known, and the actual length of the reference line, ie, the actual height of the target object, is calculated using the distance from the reference line to the lens plane and the focal length of the lens.

[0016] Furthermore, the actual height of the target object is equal to the product of the length of the line segment projected by the reference line onto the camera sensor and the focal length of the lens divided by the distance from the reference line to the lens plane.

[0017] Furthermore, the length of the line segment projected by the reference line onto the camera sensor is equal to the product of the pixel length of the reference line on the image and the diagonal length of the sensor divided by the pixel diagonal size of the image.

[0018] As a preferred solution, in S3, the measured size of the target object is calculated by reference lines. When there are two reference lines, the distance between the two reference lines is the width of the target object. The actual length of the width of the target object is the actual distance between the two reference lines. The actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the distance between the reference line and the lens and the square of the distance between the two reference lines in the x-axis direction.

[0019] Furthermore, the difference in distance between the reference line and the lens is the difference in distance between the two reference lines and the lens plane, which is the absolute value of the difference in distance between the two reference lines and the lens plane.

[0020] As a preferred solution, the process of determining a rectangle uses edge detection to reduce labor, and the edge detection process includes adding three vertical reference lines in the image, applying a Gaussian blur algorithm to minimize noise, converting the original image into a grayscale image, and executing a Canny edge detection algorithm.

[0021] Furthermore, in the process of executing the Canny edge detection algorithm, once the Canny detection algorithm detects the edge of the beam, the detected edge of the beam and the three vertical reference lines are combined to form a quadrilateral, that is, the shape of the target object is obtained, which represents the original image corresponding to the homography matrix.

[0022] As a preferred solution, when the camera photographs the target object, the top and bottom edges of the measured surface of the target object are horizontal, the surface itself is perpendicular to the ground, and the camera lens is perpendicular to the ground.

[0023] According to another aspect of the present invention, a building structure image correction and restoration device is provided, which operates using the above-mentioned building structure image correction and restoration method.

[0024] Compared with the prior art, the present invention:

[0025] 1. Improve measurement accuracy. Traditional photography methods often result in large measurement errors due to perspective distortion. This invention combines sensor distance data with image processing technology to accurately correct the image through the homography matrix, eliminating the influence of perspective distortion, thereby improving measurement accuracy and ensuring that the measurement results are closer to the actual value.

[0026] 2. Enhanced adaptability. In traditional correction and restoration processes, the target objects are often common objects such as A4 paper, playing cards, or ID cards, with simple and limited shapes. However, in this invention, the actual length of the reference line (i.e., the actual length of the target object's measured dimension) is calculated from the pixel length of the reference line in the image using the image's pixel diagonal size, the sensor's diagonal length, the distance from the reference line to the lens plane, and the lens focal length. Furthermore, the original width and height of the target object are calculated and used to restore the distorted image, breaking the size limitation and making it applicable to objects of complex and uncommon sizes, significantly improving the adaptability of image restoration.

[0027] 3. Minimize errors in beam crack measurement. The Canny detection algorithm detects the beam's edges. Vertical reference lines and the edges detected form a quadrilateral, representing the homography matrix corresponding to the original image. Combining this with a target image of known dimensions allows the matrix itself to be derived. The correct aspect ratio is maintained across the entire plane, eliminating any distortion. This process ensures that errors in beam crack measurement are minimized.

[0028] 4. Easy to operate. By combining distance sensor data with image processing technology, the system eliminates perspective distortion in images, enabling an automated process of on-site shooting, perspective correction, and size recognition. This eliminates the tedious post-processing and correction of large amounts of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the building structure image correction and restoration method;

[0030] Figure 2 Schematic diagram of the light path when the camera shoots an object;

[0031] Figure 3 is a planar schematic diagram of an image on a camera sensor;

[0032] Figure 4 This is a schematic diagram of the perspective correction process proposed by the present invention;

[0033] Figure 5 Schematic diagram of the transformation result after applying the homography matrix to the original image;

[0034] Figure 6 Schematic diagram of the comparison between the original image and the corrected image;

[0035] Figure 7 Schematic diagram comparing the correction results of the traditional algorithm, the correction results of the algorithm proposed in this invention, and the real picture. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0037] Example

[0038] In this embodiment, in order to accurately determine the actual size of the object and restore its aspect ratio, a building structure image correction and restoration method is proposed, which combines distance sensor data with image processing technology to restore the original aspect ratio of the target object. The process of the building structure image correction and restoration method is as follows: Figure 1 shown.

[0039] In this embodiment, in order to verify the performance of the invented building structure image correction and restoration method, a site was set up. The building structure selected in this embodiment is a concrete structure, that is, the target object model of the experiment is a five-story reinforced concrete frame structure model with a total height of 5.0 meters (including the base) and a plane size of 3.6m×3.0m. Cracks of varying degrees of damage appeared at the beam-column nodes. The method proposed in the present invention was intended to be applied to the surface cracks, and attempts were made to detect, extract and calculate the crack size, which was then compared with the actual measurement results to verify the accuracy.

[0040] In this embodiment, five live images of a target object were captured using a camera equipped with a distance sensor, with the lens plane adjusted to be perpendicular to the ground. For each image, a lightweight distance sensor attached to the camera was used to measure and record the corresponding object distance. A reference line was selected on the surface of the target object.

[0041] In this embodiment, the object distance between the reference line and the lens plane is used to calculate their projected lengths on the camera sensor; the optical path diagram of the camera and distance sensor imaging based on the pinhole imaging model is as follows: Figure 2 As shown, the planar diagram of the image on the camera sensor is as follows Figure 3 shown.

[0042] Where h is the actual length of the reference line; f is the focal length of the lens; d0 is the object distance between reference line 0 and the lens plane; d1 is the object distance between reference line 1 and the lens plane; w 01 is the distance between the two reference lines in the x-axis direction, which is used to calculate the tilt length between the reference lines; h0 and h1 are the lengths of the line segments projected onto the camera sensor; d ′ 1 (dashed red line) is the slope length between the reference lines in the real world.

[0043] In this example, subsequent analysis relies on the following conditions: the top and bottom edges of the measured surface are horizontal; the surface itself is perpendicular to the ground; and the camera lens is also perpendicular to the ground. These requirements are generally easy to meet, as many cameras are equipped with built-in angle sensors or can be accurately positioned using an external level.

[0044] Combining the horizontal distance to the target object measured by the camera distance sensor and the image resolution, the pixel length of the reference line in the image is converted to its length in the real world to obtain the length of two adjacent sides of the target object;

[0045] In this embodiment, a reference line is set, that is, the line segment of the target object's dimension to be measured is marked as the reference line, and the actual length of the target object's dimension to be measured is the actual length of the reference line. The specific calculation process includes:

[0046] Given the pixel length of the reference line on the image, the length of the line segment projected onto the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor.

[0047] Given the length of the line segment projected onto the camera sensor, the actual length of the reference line, i.e., the actual length of the target object to be measured, is calculated using the distance from the reference line to the lens plane and the focal length of the lens. Specifically, the following mathematical relationship is derived:

[0048]

[0049] Thus:

[0050]

[0051] Where h is the actual length of the reference line, h i i is the pixel length of the reference line on the image, h i is the length of the line segment projected from the reference line i onto the camera sensor; r is the pixel diagonal size of the image; l is the diagonal length of the sensor; d i is the object distance between the reference line i and the lens plane; f is the focal length of the lens.

[0052] In this embodiment, the focal length of the camera lens, sensor size and image resolution can generally be found in the product manual or on the Internet. 2 , with a diagonal length of 43.27mm), the user manual will usually indicate the "equivalent focal length" to compare the diagonal field of view (DFOV) with other cameras with different sensor sizes. To convert between actual focal length and equivalent focal length, the ratio between the camera's sensor size and the standard sensor size is essential. Once the conversion ratio is obtained, the equivalent focal length can be derived from the actual length, and vice versa. For example, the following are the specifications of the main camera of the mobile phone used in this example:

[0053] Table 1 Camera parameters

[0054]

[0055] In this embodiment, based on the above conclusions, the equivalent focal length of the camera on the mobile phone is:

[0056]

[0057] where f ′ is the equivalent focal length, f is the focal length of the lens

[0058] In some cases, camera manufacturers may only provide the equivalent focal length of their products.

[0059] In addition to converting the ratio between pixel length and actual length, the actual length of the target object to be measured can also be calculated in the following way:

[0060] Set up two reference lines. The distance between the two reference lines is the target object's dimension to be measured. The actual length of the target object's dimension to be measured is the actual distance between the two reference lines. The actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the reference lines and the lens and the square of the distance between the two reference lines in the x-axis direction. Reference line 0 and reference line 1, the distance between the two reference lines is the actual length of the target object's dimension to be measured, which can be calculated as follows:

[0061]

[0062] in

[0063] Δd i =|d0-d i |.

[0064] Where d0 is the object distance between reference line 0 and the lens plane; d1 is the object distance between reference line 1 and the lens plane; w 01 is the distance between the two reference lines in the x-axis direction, which is used to calculate the tilt length between the reference lines; h0 and h1 are the lengths of the line segments projected onto the camera sensor by reference lines 0 and 1; d1′ and d2′; Figure 1 is the distance between the two reference lines on the beam, d ′ 1 (dashed red line) is the tilt length between the reference lines in the real world, along the length of the beam.

[0065] In this embodiment, obtaining the lengths of two pairs of adjacent sides of a rectangle on a plane satisfies the necessary conditions for determining the homography matrix. Applying this matrix to the entire image generates the rectified image. This approach is feasible because the side of the beam is a plane that contains all the necessary reference lines.

[0066] In this embodiment, edge detection is used to determine the exact shape of the target object, and a function called cv2.Canny is used to detect edges in the input image. The process includes applying a Gaussian blur algorithm to minimize noise, converting the original image to a grayscale image, and executing the Canny edge detection algorithm. Before these steps begin, the algorithm adds three vertical reference lines to the image as a pre-step. Once the Canny detection algorithm detects the edges of the beam, these edges can be formed into a quadrilateral (a trapezoid, to be precise), representing the original image corresponding to the homography matrix, and combined with the target image of known size, the matrix itself can be derived. The brief flow of the algorithm is as follows Figure 4 As shown in Figure 4(a), 4(b) is the original image, 4(c) is the image after edges are extracted using the Canny edge detection algorithm and three reference lines are added, 4(d) is the detected reference rectangle, and 4(d) is the rectified image. Figure 4 The red rectangle in (c) is accurately measured and has the same aspect ratio as its actual size. Therefore, the entire plane it lies on also maintains the correct aspect ratio, eliminating all distortion.

[0067] In this embodiment, the principle of the homography matrix is ​​mainly used to restore and present the original image and its correct aspect ratio. When the perspective transformation is applied to a plane, the formula can be written as

[0068]

[0069] Where (x, y) is the coordinate of the point on the original plane, and (X, Y, Z) is the homogeneous coordinate of the point on the new plane. 3×3 The submatrix A in 2×2 represents affine transformation, T 2×1 represents the translation transformation, V T Represents a perspective transformation.

[0070] The transformation result after applying the homography matrix to the original image is as follows Figure 5 shown. Figure 5 Each subfigure in shows the result of applying the homography matrix to the original shape.

[0071] Expand the formula, we have

[0072] X=a 11 ×x+a 12 ×y+t x ,

[0073] Y=a 21 ×x+a 22 ×y+t y ,

[0074] Z=v1×x+v2×y+s.

[0075] Based on the transformation rules of Cartesian coordinate system and homogeneous coordinate system, that is,

[0076]

[0077] We can get the corresponding coordinates of the point in the new plane through the following transformation relationship:

[0078]

[0079] In this embodiment, to obtain the homography matrix, the coordinates of at least four pairs of points on the original and target planes are required (equivalent to eight equations). The parameter s can be eliminated by dividing each numerator and denominator in the two equations above by it, thereby retaining only eight unknown parameters and determining all the coefficients of the matrix. In engineering practice, rectangles are one of the most commonly used shapes; therefore, the matrix can be derived by obtaining the lengths of two adjacent sides of the target rectangle. Finally, the obtained homography matrix is ​​used to apply a perspective transformation to the entire image to generate a corrected image.

[0080] In this embodiment, the camera pinhole imaging model, rangefinder, and homography matrix are used to design and theoretically derive the correction of structural images and aspect ratio restoration, and it is concluded that under certain conditions, structural photos can be accurately corrected with the help of the rangefinder and the built-in parameters of the camera.

[0081] Table 2 Accuracy verification of image correction algorithm

[0082]

[0083] The final results after algorithm processing and correction are shown in Table 2. The comparison diagram of the original image and the corrected image is shown in Figure 6 shown.

[0084] In this example, the calculated beam height was compared with the actual height measured using a tape measure. The relative angles of the upper and lower edges of the beam in the image restored by the algorithm were measured. The results show that the algorithm performs well, with only minor errors. The maximum error in height measurement is 4.43%, and the maximum error in parallelism is 1.2°. A schematic diagram comparing the results obtained in this example with the correction results of the traditional algorithm and the real image taken perpendicular to the plane is shown in the figure below. Figure 7 As shown, it can be seen that the accuracy of the present invention is significantly better than that of the traditional correction method.

[0085] In this example, errors primarily occur during the edge detection phase. During this phase, the algorithm draws a red reference trapezoid on the measurement plane as the original shape to be corrected. In the final image, this trapezoid is corrected to a square using the homography matrix. This means that if its edges are aligned more accurately with the beam edges, the error introduced by this method can be further reduced. Parallelism error is the angle between the upper and lower edges in the corrected image.

[0086] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for correcting and restoring an architectural structure image, characterized in that: The distance sensor data is combined with image processing technology to restore the original aspect ratio of the target object. The method steps include: S1. Use a camera equipped with a distance sensor to shoot the target object to obtain an image to be corrected; S2. Mark multiple reference lines perpendicular to the ground on the image to be corrected, where the actual distance between two reference lines is the width of the target object; the length of one reference line segment is the height of the target object; a rectangle is determined by the reference lines to represent the target object, which is used to assist in calculating the homography matrix in the subsequent image correction stage; S3. Calculate the size of the target object to be measured using the reference line to obtain the lengths of two adjacent sides of the target object, that is, obtain the aspect ratio of the target object; S4. Calculate a homography matrix using the image coordinates and real coordinates of four points; the four points are the four corner points of the object to be measured, the image coordinates are obtained directly from the image, and the real coordinates are the coordinates calculated in S3 above; S5. Apply perspective transformation to the image to be rectified using the homography matrix obtained in S4 to generate a rectified image.

2. The method for correcting and restoring a building structure image according to claim 1, wherein: In the step S3, the size of the target object to be measured is calculated using the reference line. When there is only one reference line, that is, the length of the reference line segment is the height of the target object to be measured, and the actual length of the height of the target object is the actual length of the reference line. The specific calculation process includes: Given the pixel length of the reference line on the image, the length of the line segment projected onto the camera sensor is calculated using the pixel diagonal size of the image and the diagonal length of the sensor. The length of the line segment projected from the reference line onto the camera sensor is known, and the actual length of the reference line, ie, the actual height of the target object, is calculated using the distance from the reference line to the lens plane and the focal length of the lens.

3. The method for correcting and restoring an architectural structure image according to claim 2, wherein: The actual height of the target object is equal to the product of the length of the line segment projected by the reference line onto the camera sensor and the focal length of the lens divided by the distance from the reference line to the lens plane.

4. The method for correcting and restoring an architectural structure image according to claim 2, wherein: The length of the line segment projected from the reference line onto the camera sensor is equal to the product of the pixel length of the reference line on the image and the diagonal length of the sensor divided by the pixel diagonal size of the image.

5. The method for correcting and restoring a building structure image according to claim 1, wherein: In S3, the measured size of the target object is calculated using the reference lines. When there are two reference lines, the actual length of the width of the target object is the actual distance between the two reference lines. The actual distance between the two reference lines is specifically equal to the square root of the sum of the square of the difference between the distance between the reference line and the lens and the square of the distance between the two reference lines in the x-axis direction.

6. The method for correcting and restoring a building structure image according to claim 5, characterized in that: The distance difference between the reference line and the lens is the distance difference between the two reference lines and the lens plane, which is the absolute value of the difference between the distances between the two reference lines and the lens plane.

7. The method for correcting and restoring a building structure image according to claim 1, wherein: The process of determining a rectangle uses edge detection to reduce labor, and the edge detection process includes adding three vertical reference lines in the image, applying a Gaussian blur algorithm to minimize noise, converting the original image to a grayscale image, and performing a Canny edge detection algorithm.

8. The method for correcting and restoring an architectural structure image according to claim 7, wherein: During the execution of the Canny edge detection algorithm, once the Canny detection algorithm detects the edge of the beam, the detected edge of the beam and the three vertical reference lines are combined to form a quadrilateral, that is, the shape of the target object is obtained, which represents the original image corresponding to the homography matrix.

9. The method for correcting and restoring an architectural structure image according to claim 1, wherein: When the camera photographs the target object, the top and bottom edges of the measured surface of the target object are horizontal, the surface itself is perpendicular to the ground, and the camera lens is perpendicular to the ground.

10. A building structure image correction and restoration device, characterized in that: The device operates by applying the architectural structure image correction and restoration method as described in any one of claims 1 to 9.

Images