Concrete member surface bubble detection method based on machine vision and deep learning

Abstract

This invention discloses a method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning. The method includes acquiring RGB images and depth maps of the surface of the concrete component to be inspected; inputting the RGB images into a trained air bubble detection model, outputting a detection box for each bubble; synchronizing the coordinate parameters of the detection boxes to the depth map; calculating continuous bubble boundary points with a single pixel width based on the range determined by each detection box and the normalized gradient method of the array depth map; using the number of detection boxes as the number of bubbles; and calculating the area, maximum diameter, and depth of a single bubble, as well as the bubble area ratio within a single detection area, based on the bubble boundary points. This invention achieves integrated intelligent analysis throughout the entire process, from automatic identification and location of air bubbles on the surface of concrete components to accurate calculation of geometric parameters such as area, maximum diameter, depth, and bubble area ratio, significantly improving the efficiency and accuracy of on-site quality acceptance.

Description

Technical Field

[0001] This invention relates to the field of concrete bubble detection technology, and specifically to a method for detecting bubbles on the surface of concrete components based on machine vision and deep learning. Background Technology

[0002] Detection of defects in the appearance of concrete is an important part of the quality acceptance of concrete structure engineering. Current standards divide the detection of air bubbles into qualitative and quantitative tests. Qualitative detection of air bubbles generally refers to the manual observation methods given in the "Code for Acceptance of Construction Quality of Concrete Structures". Air bubbles belong to the appearance defects defined in Table 8.1.2 of this code, and are classified as serious defects or general defects depending on their presence on fair-faced concrete components or other concrete components with significant decorative effects. The presence of air bubbles on the surface of fair-faced concrete components with significant decorative effects is considered a serious defect, while air bubbles on the surface of other concrete components that do not affect their function are considered general defects. Quantitative detection of air bubbles mainly refers to the "Technical Specification for Application of Fair-faced Concrete", which requires dimensional measurement of the air bubble area ratio, maximum diameter, and maximum depth on the surface of fair-faced concrete components.

[0003] In the current technology, the need for automated detection of air bubbles on concrete surfaces often relies on computer image processing algorithms, deep learning object detection algorithms, or deep learning semantic segmentation algorithms for qualitative or quantitative detection.

[0004] Chinese patent applications CN201910411299.2 and CN201911415677.0 both employ computer image processing algorithms for bubble detection. These algorithms primarily involve thresholding the pixels of an image, manually adjusting the pixel thresholds to obtain a binary image that is either black or white, thus assigning different colors to pixels representing bubbles and non-bubble areas. When calculating the bubble area, the method uses the pixel area ratio (the ratio of pixel area to actual physical area). The total number of pixels classified as bubbles is used as the bubble area, and the bubble area ratio can be further calculated based on the actual physical area of ​​the captured image. When calculating the number of bubbles, the method divides all bubble pixels into several connected components based on connectivity analysis, considering whether the bubble pixels in the binary image are adjacent. Then, contours are formed based on the boundary pixels of each connected component, generating a closed sequence of boundary points. This sequence represents the contour of the bubble represented by that connected component, and the total number of contours in the image is the number of bubbles. However, the drawback of using computer image processing algorithms for bubble detection is that the threshold segmentation operation of pixels cannot obtain the bubble depth, and the manually set threshold is greatly affected by image noise. Without fixed imaging equipment, it needs to be constantly adjusted according to the shooting environment, resulting in low practicality in engineering sites.

[0005] Chinese patent application CN202410526103.5 employs the YOLO v5 deep learning object detection algorithm for qualitative bubble detection. This algorithm automatically identifies bubble regions in images through model training and outputs bubble location information using bounding boxes to achieve qualitative bubble detection. However, the required dataset of bubbles on concrete component surfaces relies on online searches, cast test blocks, and on-site collection. Due to the influence of factors such as variable imaging distance, lighting, and angle, the detection model's efficiency in capturing bubble features under diverse environments is unstable. Furthermore, this method does not perform quantitative detection of geometric parameters such as bubble area and maximum diameter, nor can it measure bubble depth.

[0006] Chinese patent application CN202410901154.1 employs a deep learning semantic segmentation algorithm for bubble detection. This algorithm accurately extracts bubble regions by performing fine pixel-level segmentation on concrete surface images. It then outputs the area of ​​a single bubble based on the pixel area ratio and calculates the maximum diameter of a single bubble using methods such as minimum bounding rectangle. However, deep learning semantic segmentation algorithms require fine pixel-level annotation when building the dataset. This patent uses a microscope camera in a crack width measuring instrument to acquire images containing single bubbles, which has low sampling efficiency and is time-consuming and labor-intensive to annotate. Furthermore, since the training set only contains semantic segmentation annotations for single bubbles, the model may tend to identify only a small number of bubbles within a given region rather than all bubbles present simultaneously, thus limiting its effectiveness and flexibility in practical applications. Finally, this method also cannot calculate bubble depth. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning.

[0008] To achieve the above objectives, this invention provides a method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning, comprising: RGB images and depth maps of the concrete component to be inspected are acquired. The RGB images and depth maps have the same resolution so that each pixel has a consistent coordinate correspondence in the RGB images and depth maps. The RGB image is input into a trained bubble detection model to initially identify bubbles in the RGB image and output a detection box for each bubble; The coordinate parameters of the detection boxes are synchronized to the depth map, and continuous bubble boundary points with a single pixel width are calculated based on the range determined by each detection box. The number of detection frames is taken as the number of bubbles, and the area, maximum diameter and depth of a single bubble and the bubble area ratio in a single detection area are calculated based on the bubble boundary points.

[0009] Furthermore, the detection frame is rectangular in shape and includes... , , and Four coordinate parameters, among which, and This indicates the coordinates of the center point of the detection frame. and This indicates the width and height of the detection frame.

[0010] Furthermore, the bubble boundary points are calculated using the array depth map normalized gradient method, as detailed below: For the depth matrix determined by the rectangular detection box The point with the greatest depth is taken as the reference point, and the remaining points within the detection box are taken as judgment points. The coordinates of the judgment points are represented as ( , ), its depth is The gradients of the eight neighboring points around the judgment point are calculated as follows: ; in, To determine the gradient of points to the right of the given point, To determine the gradient of the point to the upper right of the given point, To determine the gradient of points above a given point, To determine the gradient of the point to the upper left of the given point, and to determine the gradient of the point to the upper right of the given point, To determine the gradient of the point to the left of the point, To determine the gradient of the point to the lower left of the given point, To determine the gradient of points below a given point, To determine the gradient of the point to the lower right of the given point, The depth of the point to the right of the reference point. The depth of the point to the upper right of the reference point. The depth of the point above the reference point. The depth of the point to the upper left of the reference point. The depth of the point to the left of the reference point. The depth of the point to the lower left of the reference point. The depth of the point below the reference point. The depth of the point to the lower right of the reference point; take M gradients where the angle between the line containing the judgment point and the line containing the judgment point and the reference point is less than 90°, and the line containing the judgment point and the reference point are respectively located on the line containing the judgment point and the reference point. , By fusing M gradients through root mean square The gradient of the judgment point The details are as follows: ; gradient Greater than the threshold The corresponding point is regarded as a bubble point; if at least one of the four adjacent points in the four directions of the bubble point is not a bubble point, then the bubble point is regarded as a bubble boundary point and is retained for display.

[0011] Furthermore, the threshold The calculation method is as follows: ; in, The minimum value among all the depth values ​​of the judgment points at the boundary of the detection box. The depth value of the reference point. Scaling factor .

[0012] Furthermore, the area of ​​the bubble is calculated as follows: ; in, The area of ​​the bubble is calculated, where, The total number of bubble boundary points, ( , ) represents the coordinates of the bubble boundary point.

[0013] Furthermore, Bubble area ratio within a single detection area The calculation method is as follows: ; in, For the first in the detection area The area of ​​each bubble, , These are the width and length of the inner wall of the detection device used to acquire the RGB image and depth map, respectively.

[0014] Furthermore, the maximum diameter of the bubble is calculated as follows: For any planar polygon formed by the boundary points of the bubble, the centroid coordinates are calculated by decomposing the triangles. , )for: ; ; Using the centroid as the origin, calculate the centered coordinates of the bubble boundary points and construct the covariance matrix. for: ; in, Let be the dispersion of the bubble boundary points in the x-direction relative to the centroid. , Let be the dispersion of the bubble boundary points in the y-direction relative to the centroid. , The degree of correlation between the bubble boundary points and the centroid in the x and y directions. ; For the covariance matrix The decomposition yields two feature vectors. The feature vector with the larger feature value is the direction of the primary centroidal axis, and the other feature vector is the secondary centroidal axis. The distances between the primary and secondary centroidal axes and the intersection point of the polygon are calculated, and the larger value is taken as the maximum diameter of the bubble.

[0015] Furthermore, the depth of the bubble is calculated as follows: Establish a set of bubble interior points based on the coordinate relationship between the bubble boundary points and the reference point. , Given the total number of points inside the bubble, calculate the average depth of the reference point and its eight adjacent points as the bubble depth.

[0016] Furthermore, the bubble detection model includes a backbone network unit, a neck unit, a detection head unit, and a detection box optimization module; The backbone network unit is used to extract shallow feature maps from RGB images. Mid-layer feature map and deep feature maps ; The neck unit is used to extract features from shallow feature maps. Mid-layer feature map and deep feature maps The detection features are extracted as follows: ; ; ; in, To obtain from shallow feature maps The first detection feature extracted from it, To obtain the mid-layer feature map The second detection feature extracted from it, To extract from deep feature maps The third detection feature extracted from it. , , These are extraction and transformation modules for the first detection feature, the second detection feature, and the third detection feature, respectively. The detection head unit is used to output a set of candidate detection boxes based on the first detection feature, the second detection feature, and the third detection feature. for: ; in, , , These are three sizes of detection heads for detecting the head unit. This indicates that the candidate detection results output by the detection heads at each scale are merged; The detection box optimization module is used to optimize the candidate detection box set output by the detection head unit. The edge margin is calculated based on the width, height, and confidence of the candidate detection boxes, and adaptive box expansion is performed on the candidate detection boxes to obtain and output the final detection boxes.

[0017] Furthermore, the detection box optimization module obtains the final detection box in the following specific way: Calculate marginal remnants for: ; in, This is a truncation function. Based on the margin of the base, This is the size adjustment factor. To prevent constants with a denominator of zero, and The candidate detection box set The p-th candidate detection box Width and height, , and Candidate detection boxes The coordinates of the center point, Candidate detection boxes The prediction confidence level This is the confidence level adjustment coefficient. , These are the minimum and maximum values ​​of the margin, respectively; Based on the edge leave Obtain the final detection box for: ; ; ; ; .

[0018] Beneficial effects: This invention first uses an improved target detection model to initially locate the bubble. Then, based on the point cloud data acquired by a binocular camera, gradient calculation is used to accurately define the three-dimensional boundary contour of the bubble, effectively overcoming the shortcomings of traditional image recognition methods, such as being greatly affected by lighting and shadows and having blurred boundaries. The contour enclosed by the boundary point cloud is used as a polygon, and geometric algorithms are comprehensively applied to calculate parameters such as the bubble's area and maximum diameter. The bubble depth is calculated by analyzing the depth values ​​of the deepest point and its neighborhood, thus forming a complete and high-precision quantitative analysis method for bubble geometric parameters. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the bubble detection model extracting the detection frame according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of identifying bubbles and generating detection boxes on an RGB image; Figure 4 This is a schematic diagram showing the detection bounding box synchronized onto the depth map; Figure 5 This is a schematic diagram illustrating the calculation of the gradient of the eight adjacent points around the judgment point; Figure 6 This is a schematic diagram of the gradient of the corresponding points selected based on the included angle; Figure 7 It is a schematic diagram of the contour formed by the calculated bubble boundary points. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0021] like Figure 1 As shown, this embodiment of the invention provides a method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning, including: RGB images and depth maps of the concrete component surface to be inspected are acquired. The RGB images and depth maps have the same resolution to ensure consistent coordinate correspondence for each pixel in both maps. Specifically, the RGB images and depth maps of the concrete component surface are acquired using a detection device. This device includes a housing and an image acquisition module mounted on top of the housing. This image acquisition module contains an RGB imaging module and a binocular imaging module, which can obtain RGB images and depth maps with pixel alignment. The housing is preferably rectangular, preferably 200mm × 200mm. The effective pixel area of ​​the concrete component surface contained in the RGB images and depth maps is preferably greater than or equal to 4 million pixels, preferably 6-8 million pixels, to meet a detection accuracy of at least 0.1mm. Two industrial LED strip light sources can be installed on the top of the housing, preferably symmetrically positioned on both sides of the image acquisition module, forming forward direct illumination during use to obtain a good field of view. The length of a single light source should be less than the length of the detection equipment, ideally around 100-150mm, and the power should generally be around 10W. The operating voltage is typically 12V or 24V, supplied by a power module. It offers advantages such as moderate brightness, good stability, easy installation, good heat dissipation, long service life, and low price, making it suitable for the long-term stable operation requirements of the confined space inside the detection equipment.

[0022] The RGB image is input into a trained bubble detection model to initially identify bubbles in the RGB image, and a detection box is output for each bubble. See also Figure 3 The detection frame is preferably rectangular, and it includes... , , and Four coordinate parameters, among which, and This indicates the coordinates of the center point of the detection frame. and This indicates the width and height of the detection frame. The detection frame shown above completely covers the bubble.

[0023] The coordinate parameters of the detection boxes are synchronized to the depth map, based on the depth matrix of the area defined by each detection box. Calculate the bubble boundary points. The effect after synchronizing the detection box to the depth map is shown below. Figure 4 As shown. Within the local area defined by the detection box, further precise filtering of the point cloud belonging to the bubble is needed, classifying each point as one of the bubble boundary point, bubble outer point, or bubble inner point. Since a single point corresponds one-to-one with a single pixel in the depth map, the first feature of this point is that its projection on the xOy plane (i.e., parallel to the concrete surface) is arranged in a regular array, which can be regarded as having a size of The matrix; in addition, the second feature is that since the spacing between image pixels is fixed and consistent, the spacing between adjacent points in the horizontal and vertical directions in the point cloud data is the same; finally, the third feature is that the point with the largest depth in the target detection box is taken as the reference point, and the gradient magnitude of the bubble edge region and the points inside it will increase significantly in the direction towards the reference point due to the abrupt change in depth value.

[0024] See Figure 5 and Figure 6 Given the three features mentioned above, the range determined by a single rectangular detection box... Depth matrix In the middle, we consider using the array depth map normalized gradient method to filter bubble boundary points, as follows: For the depth matrix Take the point with the greatest depth as the reference point, and use the remaining points within the detection box (excluding points on the edges of the detection box) as decision points. Represent the coordinates of the decision points as ( , ), its depth is The gradients of the eight adjacent points around the judgment point are calculated as follows: ; in, To determine the gradient of points to the right of the given point, To determine the gradient of the point to the upper right of the given point, To determine the gradient of points above a given point, To determine the gradient of the point to the upper left of the given point, and to determine the gradient of the point to the upper right of the given point, To determine the gradient of the point to the left of the point, To determine the gradient of the point to the lower left of the given point, To determine the gradient of points below a given point, To determine the gradient of the point to the lower right of the given point, The depth of the point to the right of the reference point. The depth of the point to the upper right of the reference point. The depth of the point above the reference point. The depth of the point to the upper left of the reference point. The depth of the point to the left of the reference point. The depth of the point to the lower left of the reference point. The depth of the point below the reference point. The depth of the point to the lower right of the reference point; take M gradients where the angle between the line containing the judgment point and the line containing the judgment point and the reference point is less than 90°, and the line containing the judgment point and the reference point are respectively located on the line containing the judgment point and the reference point. , By fusing M gradients through root mean square The gradient of the judgment point The details are as follows: ; gradient Points with a value greater than a threshold T are considered bubble points. If at least one of the four adjacent points in the four directions (up, down, left, right) of a bubble point is a non-bubble point, then that bubble point is considered a bubble boundary point and retained for display. The bubble boundary obtained in this way is a continuous closed boundary with a width of one pixel, and the set of all bubble boundary points is the point set of this continuous closed boundary. The contour formed by the bubble boundary points is as follows: Figure 7 As shown.

[0025] The above threshold The calculation method is as follows: ; in, The minimum value among all the depth values ​​of the judgment points at the boundary of the detection box. The depth value of the reference point. Scaling factor .

[0026] The number of detection frames is taken as the number of bubbles, and the area, maximum diameter and depth of a single bubble and the bubble area ratio in a single detection area are calculated based on the bubble boundary points.

[0027] Specifically, when calculating the area of ​​a bubble, the irregular bubble boundary is treated as a planar polygon with finite sides. The Gaussian area formula can then be used to calculate the polygon's area as the detected bubble area. The Gaussian area formula is applicable to the calculation of polygon areas; its essence is to... The edge polygon (whether concave or convex) is decomposed into The algorithm calculates the area of ​​each triangle using the cross product of vectors, and then sums the results to obtain the final area. This algorithm is simple, accurate, and can improve the efficiency of bubble area detection. Assume the bubble boundary points are set, then... bubble boundary points If the arrangement forms a closed polygon, then the area A of this polygon is calculated as follows: ; in, The area of ​​the bubble is calculated, where, The total number of bubble boundary points, ( , ) represents the coordinates of the bubble boundary point.

[0028] Bubble area ratio within a single detection area The calculation method is as follows: ; in, For the first in the detection area The area of ​​each bubble, , These are the width and length of the inner wall of the detection device used to acquire the RGB image and depth map, respectively.

[0029] Bubble diameter is a crucial parameter for assessing the severity of bubbles and its impact on concrete performance. Therefore, measuring the maximum bubble diameter is essential for quantitatively evaluating concrete surface defects. This invention uses the maximum distance between the centroidal axis of a polygon and the intersection point of the polygon as the maximum bubble diameter, transforming the complex and irregular boundaries of bubbles into a more standardized geometric feature description. Compared to existing methods that calculate the maximum diameter based on maximum pixel distance, minimum bounding rectangle, or minimum bounding circle, this algorithm ensures that the overall distribution of the bubble profile is considered when measuring the maximum diameter, thereby improving the accuracy and consistency of dimensional calculations and providing a reliable mathematical foundation for the automated detection of maximum bubble diameter. Specifically, the maximum bubble diameter is calculated as follows: For the centroidal axis of any planar polygon formed by the boundary points of the bubble, the coordinates of its centroid are first calculated by decomposing the triangle. , )for: ; ; Using the centroid as the origin, calculate the centered coordinates of the bubble boundary points and construct the covariance matrix. for: ; in, Let be the dispersion of the bubble boundary points in the x-direction relative to the centroid. , Let be the dispersion of the bubble boundary points in the y-direction relative to the centroid. , The degree of correlation between the bubble boundary points and the centroid in the x and y directions. ; For the covariance matrix The decomposition yields two feature vectors. The feature vector with the larger feature value is the direction of the primary centroidal axis, and the other feature vector is the secondary centroidal axis. The distances between the primary and secondary centroidal axes and the intersection point of the polygon are calculated, and the larger value is taken as the maximum diameter of the bubble.

[0030] The depth of the bubble is calculated as follows: Establish a set of bubble interior points based on the coordinate relationship between the bubble boundary points and the reference point. , To determine the total number of points within the bubble, the depth of the bubble is calculated as the average depth of the reference point and its eight adjacent points. Specifically, the average depth of the reference point and its eight adjacent points... The calculation method is as follows: ; in, Let be the depth value of the i-th point.

[0031] See Figure 2 As a preferred embodiment, the bubble detection model of this application is based on an improvement of YOLO v11. It includes a backbone network unit, a neck unit, a detection head unit, and a detection box optimization module. The detection head unit includes a high-resolution detection branch for small-sized dense bubbles, in order to improve the detection capability of small-sized bubbles.

[0032] Let the shallow, middle, and deep feature maps output by the backbone network be respectively... , and The feature transformation modules corresponding to the neck unit are respectively , and .

[0033] The detection features extracted from the neck unit at each scale can then be expressed as: ; ; ; in, To obtain from shallow feature maps The first detection feature extracted is a high-resolution detection feature, which is used to enhance the detection capability for small-sized bubbles. To obtain the mid-layer feature map The second detection feature extracted from it, To extract from deep feature maps The third detection feature extracted from the second detection feature and third detection features They are used for detecting bubbles at medium and large scales, respectively.

[0034] Let the small-scale, medium-scale, and large-scale detection heads in the detection head unit be respectively... , and The output candidate detection box set It can be represented as: ; in, This indicates that the candidate detection results output by the detection heads at each scale are merged.

[0035] The detection box optimization module is used to determine the edge margin based on the width, height and confidence of the initial detection box output by the detection head unit, and to perform adaptive box expansion processing on the initial detection box to obtain an optimized detection box suitable for subsequent array depth map normalized gradient method to extract bubble boundaries.

[0036] Let the set of candidate detection boxes output by the detection head unit be... Candidate detection boxes in for: ; in, and These represent candidate detection boxes. The coordinates of the center point, and These represent candidate detection boxes. Width and height, Represents candidate detection boxes The prediction confidence level is determined. To improve the detection box's ability to fully enclose the boundary region of a single bubble, an adaptive edge leave for the detection box is defined. for: ; in, This is a truncation function. Based on the margin of the base, This is the size adjustment factor. To prevent constants with a denominator of zero, and The candidate detection box set The p-th candidate detection box Width and height, , and Candidate detection boxes The coordinates of the center point, Candidate detection boxes The prediction confidence level This is the confidence level adjustment coefficient. , These are the minimum and maximum values ​​of the edge allowance, respectively. Based on the edge allowance... The final detection frame can be obtained. : ; in: ; ; ; .

[0037] The bubble detection model uses a bubble-weighted loss function during training. The bubble-weighted loss function determines the sample weights based on the actual detection box area and weights the classification loss, detection box regression loss, and distribution regression loss to improve the recognition accuracy of small bubbles.

[0038] Let the first The width and height of each real detection box are respectively and Then its actual detection frame area It can be represented as: ; Define the bubble weights for the i-th sample. for: ; in, This represents the small bubble reinforcement coefficient. It is an exponential function. This is an area scale parameter; when the actual detection box area is small... The larger the size, the greater the contribution of small-sized bubble samples to the training loss.

[0039] Let the first The classification loss, bounding box regression loss, and distribution regression loss for each sample are respectively... , and The weighted classification loss Weighted detection frame regression loss and weighted distributed regression loss They are represented as follows: ; ; ; in, The total number of samples.

[0040] Therefore, the total loss function It can be represented as: ; in, , and These are the weight coefficients for classification loss, bounding box regression loss, and distribution regression loss, respectively.

[0041] When training the bubble detection model, three data augmentation strategies are used to build a bubble defect dataset for the collected image data. Specifically, three methods are considered: arbitrary angle rotation, flipping, and scaling at a specific ratio. Arbitrary angle rotation and flipping do not change the size and geometric features of the bubbles in the original image; while scaling at a specific ratio changes the size of the bubbles in the original image. Since the device's shooting distance is fixed, the scaling factor cannot be arbitrarily set to avoid scaling the bubble size to too large or too small a ratio. The scaling factor of the original image can be limited to 0.9-1.1 to ensure the accuracy of subsequent detection. In the third step, the training set, validation set, and test set are divided according to a ratio (the ratio factor is not specific), and training is performed based on a selected target detection algorithm. When the accuracy of the test set is ≥95%, the training is considered successful. Then, the bubble detection model can be loaded into the device for actual bubble sampling and detection.

[0042] In summary, compared with existing computer image processing algorithms, deep learning object detection algorithms, and deep learning semantic segmentation algorithms, this application achieves integrated qualitative and quantitative detection of bubbles through object detection and 3D point cloud analysis. It detects the presence of bubbles based on a bubble extraction model and accurately quantifies bubble size based on 3D point cloud data, avoiding the massive annotation work involved in semantic segmentation algorithms. At the same time, it obtains the depth information of bubbles, filling the gap in the field of concrete surface bubble detection and providing a new detection method for the field of concrete component appearance quality acceptance.

[0043] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning, characterized in that, include: RGB images and depth maps of the surface of the concrete component to be inspected are acquired. The RGB images and depth maps have the same resolution so that each pixel has a consistent coordinate correspondence in the RGB images and depth maps. The RGB image is input into a trained bubble detection model to initially identify bubbles in the RGB image and output a detection box for each bubble; The coordinate parameters of the detection boxes are synchronized to the depth map, and continuous bubble boundary points with a single pixel width are calculated based on the range determined by each detection box. The number of detection frames is taken as the number of bubbles, and the area, maximum diameter and depth of a single bubble and the bubble area ratio in a single detection area are calculated based on the bubble boundary points.

2. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 1, characterized in that, The detection frame is rectangular and includes , , and Four coordinate parameters, among which, and This indicates the coordinates of the center point of the detection frame. and This indicates the width and height of the detection frame.

3. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 2, characterized in that, The bubble boundary points are calculated using the array depth map normalized gradient method, as detailed below: For the depth matrix determined by the rectangular detection box The point with the greatest depth is taken as the reference point, and the remaining points within the detection box are taken as judgment points. The coordinates of the judgment points are represented as ( , ), its depth is The gradients of the eight neighboring points around the judgment point are calculated as follows: ； in, To determine the gradient of points to the right of the given point, To determine the gradient of the point to the upper right of the given point, To determine the gradient of points above a given point, To determine the gradient of the point to the upper left of the given point, and to determine the gradient of the point to the upper right of the given point, To determine the gradient of the point to the left of the point, To determine the gradient of the point to the lower left of the given point, To determine the gradient of points below a given point, To determine the gradient of the point to the lower right of the given point, The depth of the point to the right of the reference point. The depth of the point to the upper right of the reference point. The depth of the point above the reference point. The depth of the point to the upper left of the reference point. The depth of the point to the left of the reference point. The depth of the point to the lower left of the reference point. The depth of the point below the reference point. The depth of the point to the lower right of the reference point; take M gradients where the angle between the line containing the judgment point and the line containing the judgment point and the reference point is less than 90°, and the line containing the judgment point and the reference point are respectively located on the line containing the judgment point and the reference point. , By fusing M gradients through root mean square The gradient of the judgment point The details are as follows: ； gradient Greater than the threshold The corresponding point is regarded as a bubble point; if at least one of the four adjacent points in the four directions of the bubble point is not a bubble point, then the bubble point is regarded as a bubble boundary point and is retained for display.

4. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 3, characterized in that, The threshold The calculation method is as follows: ； in, The minimum value among all the depth values ​​of the judgment points at the boundary of the detection box. The depth value of the reference point. Scaling factor .

5. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 3, characterized in that, The area of ​​the bubble is calculated as follows: ； in, The area of ​​the bubble is calculated, where, The total number of bubble boundary points, ( , ) represents the coordinates of the bubble boundary point.

6. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 5, characterized in that, Bubble area ratio within the single detection area The calculation method is as follows: ； in, For the first in the detection area The area of ​​each bubble, , These are the width and length of the inner wall of the detection device used to acquire the RGB image and depth map, respectively.

7. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 5, characterized in that, The maximum diameter of the bubble is calculated as follows: For any planar polygon formed by the boundary points of the bubble, the centroid coordinates are calculated by decomposing the triangles. , )for: ； ； Using the centroid as the origin, calculate the centered coordinates of the bubble boundary points and construct the covariance matrix. for: ； in, Let be the dispersion of the bubble boundary points in the x-direction relative to the centroid. , Let be the dispersion of the bubble boundary points in the y-direction relative to the centroid. , The degree of correlation between the bubble boundary points and the centroid in the x and y directions. ; For the covariance matrix The decomposition yields two feature vectors. The feature vector with the larger feature value is the direction of the primary centroidal axis, and the other feature vector is the secondary centroidal axis. The distances between the primary and secondary centroidal axes and the intersection point of the polygon are calculated, and the larger value is taken as the maximum diameter of the bubble.

8. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 3, characterized in that, The depth of the bubble is calculated as follows: Establish a set of bubble interior points based on the coordinate relationship between the bubble boundary points and the reference point. , Given the total number of points inside the bubble, calculate the average depth of the reference point and its eight adjacent points as the bubble depth.

9. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 2, characterized in that, The bubble detection model includes a backbone network unit, a neck unit, a detection head unit, and a detection box optimization module. The backbone network unit is used to extract shallow feature maps from RGB images. Mid-layer feature map and deep feature map ; The neck unit is used to extract features from shallow feature maps. Mid-layer feature map and deep feature map The detection features are extracted as follows: ； ； ； in, To obtain from shallow feature maps The first detection feature extracted from it, To obtain the mid-layer feature map The second detection feature extracted from it, To extract from deep feature maps The third detection feature extracted from it. , , These are extraction and transformation modules for the first detection feature, the second detection feature, and the third detection feature, respectively. The detection head unit is used to output a set of candidate detection boxes based on the first detection feature, the second detection feature, and the third detection feature. for: ； in, , , These are three sizes of detection heads for detecting the head unit. This indicates that the candidate detection results output by the detection heads at each scale are merged; The detection box optimization module is used to optimize the candidate detection box set output by the detection head unit. The edge margin is calculated based on the width, height, and confidence of the candidate detection boxes, and adaptive box expansion is performed on the candidate detection boxes to obtain and output the final detection boxes.

10. The method for detecting air bubbles on the surface of concrete components based on machine vision and deep learning according to claim 9, characterized in that, The detection box optimization module obtains the final detection box in the following specific way: Calculate marginal remnants for: ； in, This is a truncation function. Based on the margin of the base, This is the size adjustment factor. To prevent constants with a denominator of zero, and The candidate detection box set The p-th candidate detection box Width and height, , and Candidate detection boxes The coordinates of the center point, Candidate detection boxes The prediction confidence level This is the confidence level adjustment coefficient. , These are the minimum and maximum values ​​of the margin, respectively; Based on the edge leave Obtain the final detection box for: ； ； ； ； 。

Images