A method for detecting defects of embedded parts in prefabricated buildings

Abstract

The present invention discloses a method for detecting defects of embedded parts in prefabricated buildings. First, real-time video of a mold with embedded parts on a mold platform is collected, and a target sensing area for performing online detection is set. An embedded part detection network based on the Mask R-CNN target detection algorithm and the SORT target tracking algorithm is constructed and trained. Then, based on the video frame of the target sensing area, the embedded parts are identified and tracked using the trained embedded part detection network to obtain the measured embedded part quantity information and position information of the component. Finally, the measured embedded part quantity information and position information of the component are compared with the standard embedded part quantity information and position information of the component of the type, and the detection result of the embedded parts in the component mold is obtained. The present invention can achieve the purpose of feature extraction and target tracking of frame-by-frame images, thereby more accurately obtaining the quantity information and position information of the embedded parts in each component mold.

Description

Technical Field

[0001] The invention relates to the technical field of detection of prefabricated building components, and in particular to a method for detecting defects of embedded parts in prefabricated buildings. Background Art

[0002] Prefabricated buildings are an important development direction of the construction industry. Prefabricated buildings are buildings that are pre-processed in factories and then transported to construction sites for reliable assembly. Compared with existing cast-in-place structures, prefabricated buildings have the advantages of large-scale production, fast construction speed and low construction cost.

[0003] Among them, the quality control of embedded parts in prefabricated building components is the core of ensuring the quality of prefabricated buildings. The construction quality of embedded parts directly affects the construction progress and structural safety of the construction project, so it must be firmly controlled. Any defect in an embedded part may have an inevitable impact on the final building quality, and then bring immeasurable losses to the entire construction project. Problems such as the centerline position and buried height of embedded parts such as wire boxes, wire tubes, hanging points, and embedded iron parts exceed the allowable deviation value of the specification. Embedded parts problems occur in batches more frequently during component production, resulting in rework and repair, affecting production progress, and more seriously affecting the later construction and use of the project.

[0004] At present, the position inspection of various embedded parts is carried out by workers on-site before pouring cement. However, for engineering projects with a relatively large construction area, there are many embedded parts and their positions are scattered. For example, the wiring of embedded parts such as junction boxes and pipes is complicated and difficult to inspect. Therefore, manual inspection is time-consuming and labor-intensive, and inefficient. In addition, during the inspection of finished product acceptance, the concrete has been poured. Even if problems are detected, they cannot be reworked, resulting in waste of resources, economic losses, and even possible failure to deliver on time. There is also a YOLO target detection algorithm for embedded parts detection, but since embedded parts are often covered by objects such as steel bars, this detection method has problems such as confusion in the identification of embedded parts and low recognition efficiency. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting defects of embedded parts in prefabricated buildings, so as to solve the problems of low efficiency of manual quality inspection of embedded parts in the prior art and poor detection accuracy and real-time performance of existing automated inspection methods.

[0006] The technical solution of the present invention is:

[0007] A method for detecting defects of embedded parts in prefabricated buildings comprises the following steps:

[0008] (1) Using a video acquisition device disposed above the mold platform to collect real-time video of the mold with embedded parts on the mold platform;

[0009] (2) Setting a target sensing area for online detection by a video acquisition device, the area captured by the video acquisition device at a fixed viewing angle is the defect detection area, and the shooting area of ​​the target sensing area video frame is within the range of the defect detection area;

[0010] (3) Construct and train an embedded parts detection network, which includes a target detection model based on the Mask R-CNN target detection algorithm and a target tracking model based on the SORT target tracking algorithm;

[0011] (4) Based on the video frame of the target sensing area, the embedded parts are identified and tracked using the trained embedded parts detection network to obtain the actual number and position information of the embedded parts of the component;

[0012] (5) Compare the measured embedded parts quantity information and position information of the component with the standard embedded parts quantity information and position information of the component of the same type to obtain the detection result of the embedded parts in the component mold.

[0013] The video acquisition device shoots the component mold carrying embedded parts on the mold platform in real time along a fixed downward angle of view. Along the conveying direction of the mold platform, the intersection of the lens axis of the video acquisition device and the mold platform is located behind the vertical intersection of the video acquisition device and the mold platform.

[0014] The video frame length of the target sensing area is equal to the video frame length of the mold real-time video. The pixel value Wa of the video frame width of the target sensing area is calculated using the following formula (1):

[0015]

[0016] In formula (1), Wv is the video frame width pixel value of the mold real-time video, Tmax is the maximum time that the embedded part is in the mold real-time video, Fmax is the average frame rate of the mold real-time video, and Fmin is the minimum number of frames that the center point of the embedded part stays in the target sensing area video frame.

[0017] The process of using the Mask R-CNN target detection algorithm to detect embedded parts is as follows:

[0018] a. Input the embedded parts image data into the Backbone network of Mask R-CNN. The Backbone network consists of the EfficientNet network and the FPN network.

[0019] First, the EfficientNet network uniformly scales the three parameters of network depth d, width w, and input image resolution r. The scaling formula is shown in the following formula (2):

[0020]

[0021] In formula (2), α, β, and γ are all constants, and φ is the scaling factor;

[0022] The feature map processed by the EfficientNet network is then input into the FPN network. The FPN network first uses a 1×1 convolution to perform a convolution operation on the input feature map to obtain feature maps, and then sets a ROI for each point in the feature map to obtain multiple ROI candidate boxes.

[0023] b. Multiple ROI candidate boxes are input into the RPN network and enter two different branches after a 3×3 convolutional layer. One branch performs binary classification on multiple ROI candidate boxes, and the other branch performs border regression on multiple ROI candidate boxes, thereby filtering out some ROI candidate boxes and finally generating candidate region proposals. In the RPN network, the scales of the ROI candidate boxes are set to the following six types: [32×32], [64×64], [128×128], [256×256], [512×512],

[0024] [1024×1024]; the width-to-length ratio of the ROI candidate box is one of the following four: [1:2], [1:1], [2:1], [1:3];

[0025] c. Input the candidate region proposals into the ROI Align layer. When the ROI Align layer traverses the candidate regions, the floating point boundaries are kept unquantized and the bilinear interpolation method is used to calculate the coordinate positions of the four vertices of each candidate region. The formula of the bilinear interpolation method is shown in the following formula (3):

[0026]

[0027] In formula (3), f(x,y) represents the pixel value of the center (x,y) of each candidate region, (x1,y1),

[0028] (x1, y2), (x2, y1), (x2, y2) are the coordinates of the four vertices of the candidate region, f(x1, y1) is the pixel value of the vertex (x1, y1), and f(x2, y1) is the pixel value of the vertex (x2, y1);

[0029] d. Input each ROI candidate box output by the ROI Align layer into the FCN network. The FCN network consists of three branches, which respectively realize target classification, bounding box regression and Mask generation, detect embedded parts and obtain their location information.

[0030] The loss function of the Mask R-CNN target detection algorithm training is L total , the formula is shown in the following formula (4):

[0031] L total =L cls +L box +L mask (4);

[0032] In formula (4), L cls is the target classification loss, and the calculation formula is shown in the following formula (5); L box is the bounding box regression loss, and its calculation formula is shown in the following formula (6); L mask is the mask branch loss, and the calculation formula is shown in the following formula (8);

[0033]

[0034] In formula (5), Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, and N represents the number of samples;

[0035]

[0036] In formula (6), Ti = [Tx, Ty, Tw, Th] represents the regression parameters of the i-th predicted bounding box relative to the candidate bounding box, represents the regression parameter of the i-th true bounding box relative to the candidate bounding box, and smooth represents the smooth function, which is defined as follows (7):

[0037]

[0038] In formula (7), x represents the numerical difference between the predicted bounding box and the true bounding box;

[0039]

[0040] In formula (8), α and β represent two weight coefficients, Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, N represents the number of samples, |X∩Y| represents the number of pixels of the intersection of the predicted label and the true label, |X| represents the number of pixels of the true label, and |Y| represents the number of pixels of the predicted label. Represents the proportion of predicted labels to true labels. Perform a square operation to speed up the function convergence.

[0041] The specific steps of identifying and tracking embedded parts using the trained embedded parts detection network based on the video frame of the target sensing area are as follows:

[0042] S41. Initialize the counter Counter value of the target tracking model to 0. When the embedded part is detected in the target sensing area for the first time, that is, when the target detection model based on the Mask R-CNN target detection algorithm detects the first embedded part, the target tracking model based on the SORT target tracking algorithm tracks each embedded part extracted by the target detection model in each frame, assigns a globally unique identity code to each newly added embedded part, and calculates its position information. The counter Counter value of the target tracking model based on the SORT target tracking algorithm is increased by 1. The target tracking model saves the image frame at this moment, and the target detection model identifies the position coordinates of the embedded part at this moment. When the component mold scanning is completed, the value of the counter is recorded as Count-cur, and then the position coordinates of all embedded parts recorded by the target detection model are normalized to obtain the position coordinate set Locations-cur of the embedded parts.

[0043] S42, judging whether the component mold currently carrying embedded parts is collected and inspected for the first time, that is, judging whether the component mold of this type is inspected for the first time according to the model of the component to which the mold belongs; when it is judged to be the first inspection, the embedded parts information in the component mold is verified with the embedded parts information of the standard component, and when the verified embedded parts quantity is consistent, the embedded parts set Tracks in the component mold is recognized as the standard embedded parts set of this type of component, and the embedded parts quantity and position coordinate set are saved, which are recorded as Count-tem and Locations-tem respectively;

[0044] S43, when it is determined that the component mold with embedded parts is not detected for the first time, a position distance cost matrix is ​​constructed according to the embedded parts position coordinates between the real-time embedded parts set of the component mold currently detected in real time and the corresponding standard embedded parts set. Specifically, the number of embedded parts Count-cur in the real-time embedded parts set and the number of embedded parts Count-tem in the standard embedded parts set are first compared. When the two are equal, it is determined that the embedded parts in the component mold are qualified. When the two are not equal, a position distance cost matrix CostMatrix is ​​constructed according to the position coordinate set Locations-cur in the real-time embedded parts set and the position coordinate set Locations-tem in the standard embedded parts set. Then, the obtained position distance cost matrix CostMatrix is ​​input into the Hungarian algorithm for matching to obtain a matching result. The embedded parts that fail to match are recorded as missing embedded parts. The construction of the position distance cost matrix CostMatrix is ​​shown in the following formula (9):

[0045]

[0046] In formula (9), (X i ,Yi ) is the position coordinate of the embedded parts in the real-time embedded parts set, (X j ,Y j ) are the position coordinates of the embedded parts in the standard embedded parts set.

[0047] The target tracking model based on the SORT target tracking algorithm, the SORT target tracking algorithm updates the target tracking trajectory by constructing a cost matrix COST, and the calculation formula of the cost matrix COST is shown in the following formulas (10)-(12):

[0048]

[0049]

[0050]

[0051] In formulas (10)-(12), A represents the area of ​​the prediction box, B represents the area of ​​the detection box, and C represents the area of ​​the smallest circumscribed rectangular box that can contain the prediction box and the detection box; IOU (i,j) Represents the intersection-over-union ratio of each embedded part tracking detection position, GIOU (i,j) Represents the generalized intersection-over-union ratio of each embedded part tracking detection position, GIOU min Represents the minimum GIOU value, IOU (i,j) The value range is [0,1], GIOU (i,j) The value range is [-1,1].

[0052] The video frame of the component mold in the target sensing area is specifically confirmed by the following steps: when the front edge of the mold carrying the embedded parts, that is, when the line between the two corners of the front end of the mold coincides with the front edge of the target sensing area, the embedded parts detection network identifies and tracks the embedded parts currently entering the target sensing area, otherwise it continues to wait; when the embedded parts detection network identifies and tracks the embedded parts, when the rear edge of the mold carrying the embedded parts, that is, the line between the two corners of the rear end of the mold coincides with the rear edge of the target sensing area, the embedded parts detection network ends the identification and tracking of the embedded parts in the mold, and the video frame of the component mold in the target sensing area is obtained.

[0053] The standard embedded parts quantity information and position information of the components are stored in a cloud database, that is, the standard embedded parts quantity information and position information of the components are the embedded parts feature information of all types of components to be produced extracted from the BIM model; when the production line starts production and detection, the quantity information and position information of the embedded parts of the first component produced by each type of component is obtained through the embedded parts detection network, and then compared with the embedded parts information of this type of standard component in the BIM model. When the first component meets the error requirement of the embedded parts information of this type of standard component, the embedded parts quantity information and position information obtained from the first component are used as the data of the standard embedded parts set in the Hungarian algorithm matching; when the first component produced does not meet the error requirement, the current component is scrapped, and the first component is reproducible and determined.

[0054] When the measured embedded parts quantity information and position information of the component are compared with the standard embedded parts quantity information and position information of the component of this type, if the quantity information and position information of the two are completely consistent, it is determined that the embedded parts in the current component mold are buried in accordance with the requirements; otherwise, it is determined that there are defects in the embedded parts in the component mold, the movement of the mold table is stopped, and an alarm signal is issued. After receiving the alarm signal, the quality inspector removes and recycles the corresponding component mold.

[0055] Advantages of the present invention:

[0056] (1) The present invention collects real-time video images for detection and identification. The embedded parts detection network adopts the Mask R-CNN target detection algorithm for target detection. The Mask R-CNN target detection algorithm is a deep learning model for target detection and instance segmentation. In terms of target detection, Mask R-CNN can effectively identify different embedded parts and their positions in the image, thereby improving the accuracy of embedded parts extraction and identification. The SORT network model is used as the target tracking model to achieve the purpose of frame-by-frame image feature extraction and target tracking, thereby more accurately obtaining the quantity information and position information of the embedded parts in each component mold.

[0057] (2) The present invention sets a target sensing area to locate and segment the video frame. Since the range of the real-time video shooting of the mold is large, it may simultaneously include multiple different component molds containing embedded parts on the mold platform, which brings difficulty to the processing process of the embedded part detection network, making it impossible for the embedded part detection network to accurately distinguish different component molds, or because the scale of the processed data is large, the accuracy and real-time performance of the feature extraction results are affected. In order to solve this problem, a target sensing area for online detection of a video acquisition device is set, which improves the accuracy of the present invention in the embedded part recognition process, overcomes the defect of insufficient generalization performance in high-resolution images, and reduces the calculation pressure of the embedded part detection network, thereby ensuring the real-time performance of the detection results.

[0058] (3) The present invention uses the EfficientNet network to replace the ResNet network in the Mask R-CNN backbone network. The EfficientNet network is more lightweight, has higher speed and efficiency under the same computing resources, and is more advantageous for processing real-time applications. The image acquisition of the present invention is based on video frames, and has high real-time requirements. Replacing the ResNet network with the EfficientNet network will bring faster reasoning speed.

[0059] (4) The existing RPN network (region candidate network) extracts ROI candidate frames. There are only five scales of ROI candidate frames: [32×32], [64×64], [128×128], [256×256], and [512×512]. There are only three width-to-length ratios of ROI candidate frames: [1:2], [1:1], and [2:1]. The present invention is mainly used for embedded parts detection on mold platforms in the field of prefabricated buildings, and the embedded parts candidate frames include slender candidate frames such as embedded wire tubes. Therefore, the scale of the ROI candidate frame is increased by one [1024×1024], and the width-to-length ratio is increased by one: [1:3]. This makes the candidate region provided by the RPN network more accurate, reduces the amount of calculation, and saves model training time.

[0060] (5) The loss function of the Mask R-CNN target detection algorithm training of the present invention is the sum of the multi-task loss functions L total , where the mask branch loss L mask The iterative process of calculating samples is slow and cannot be optimized to the optimum. In the detection task of the present invention, the background pixels are much more than the foreground pixels, which may cause the model to learn the characteristics of the background more easily, but the recognition of the foreground target is not accurate enough. Therefore, the present invention reduces the mask branch loss L mask A weighted improvement was made by adjusting the two weight coefficients α and β to improve the model's prediction accuracy and robustness.

[0061] (6) The present invention uses the quantity information and position information of the embedded parts of the first component as the data of the standard embedded parts set in the Hungarian algorithm matching because some factors in the actual production process may cause the actual situation to differ from the ideal situation in the BIM model. These factors include production equipment accuracy, human operation errors, environmental conditions, etc. The first component is produced in actual production, which can reflect the embedded parts situation under specific production conditions in the actual production process. By comparing the actual situation of the first component with the ideal parameters in the BIM model, the differences in the actual production process can be better understood, and the production line can be adjusted and optimized to ensure that the components finally produced meet the design requirements;

[0062] (7) Because the present invention detects and tracks embedded parts in component molds during the production process, during the detection process, problems such as jitter of the video acquisition device may cause a target to switch or confuse between different frames, resulting in no overlap between the Kalman filter prediction frame A and the target detection algorithm detection frame B, thereby causing target tracking loss. Therefore, a generalized intersection-over-union loss function is used to construct a cost matrix between the prediction frame and the target frame. This can not only measure the degree of overlap between targets when there is overlap, but also measure the distance between the two when there is no overlap. By constructing a logistic regression loss function, the generalized intersection-over-union (GIOU) value is mapped to [0,1] to screen out targets that are too far away, greatly improve the processing efficiency of the algorithm, and reduce the probability of target tracking loss.

[0063] (8) When the present invention detects defects in the embedded parts in the component mold on the mold table, it can promptly issue an alarm and stop the operation of the production line, thereby reducing the defect rate of the component products and reducing production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of the present invention.

[0065] Figure 2 It is a structural schematic diagram of the video acquisition device of the present invention being arranged above the mold platform.

[0066] Figure 3 It is a network architecture diagram of the target detection model based on the Mask R-CNN target detection algorithm of the present invention.

[0067] Figure 4 It is a target detection view of the embedded parts in the test sample 1 of the embodiment of the present invention.

[0068] Figure 5 This is a target detection view of the embedded parts in the test sample 2 according to the embodiment of the present invention.

[0069] Figure 6 This is a target detection view of the embedded parts in the test sample 3 according to the embodiment of the present invention.

[0070] Figure numerals: 01-mold platform, 02-video acquisition device, 03-component mold, 04-embedded parts. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] See Figure 1 A method for detecting defects of embedded parts in prefabricated buildings, characterized in that it specifically includes the following steps:

[0073] S1. Use a video acquisition device 02 disposed above the mold platform 01 to acquire a real-time video of the mold with embedded parts on the mold platform 01; wherein the video acquisition device 02 shoots the component mold 03 with embedded parts 04 on the mold platform 01 moving in real time along a fixed angle of view tilted downward, and along the conveying direction of the mold platform 01, the intersection point of the lens axis of the video acquisition device 02 and the mold platform 01 is located behind the vertical intersection point of the video acquisition device 02 and the mold platform 01;

[0074] S2, setting the target sensing area for the video acquisition device 02 to perform online detection, the area captured by the video acquisition device at a fixed viewing angle is the defect detection area, and the shooting area of ​​the target sensing area video frame is within the range of the defect detection area;

[0075] The video frame length of the target sensing area is equal to the video frame length of the mold real-time video. The pixel value Wa of the video frame width of the target sensing area is calculated using the following formula (1):

[0076]

[0077] In formula (1), Wv is the video frame width pixel value of the mold real-time video, Tmax is the maximum time that the embedded part is in the mold real-time video, Fmax is the average frame rate of the mold real-time video, and Fmin is the minimum number of frames that the center point of the embedded part stays in the target sensing area video frame;

[0078] S3. Build and train an embedded parts detection network, which includes a target detection model based on the Mask R-CNN target detection algorithm and a target tracking model based on the SORT target tracking algorithm;

[0079] The process of embedded parts detection by Mask R-CNN target detection algorithm is as follows:

[0080] a. Input the embedded parts image data into the Backbone network of Mask R-CNN. The Backbone network consists of the EfficientNet network and the FPN network.

[0081] First, the EfficientNet network uniformly scales the three parameters of network depth d, width w, and input image resolution r. The scaling formula is shown in the following formula (2):

[0082]

[0083] In formula (2), α, β, and γ are all constants, and φ is the scaling factor;

[0084] The feature map processed by the EfficientNet network is then input into the FPN network. The FPN network first uses a 1×1 convolution to perform a convolution operation on the input feature map to obtain feature maps, and then sets a ROI for each point in the feature map to obtain multiple ROI candidate boxes.

[0085] b. Multiple ROI candidate boxes are input into the RPN network and enter two different branches after a 3×3 convolutional layer. One branch performs binary classification on multiple ROI candidate boxes, and the other branch performs border regression on multiple ROI candidate boxes, thereby filtering out some ROI candidate boxes and finally generating candidate region proposals. In the RPN network, the scales of the ROI candidate boxes are set to the following six types: [32×32], [64×64], [128×128], [256×256], [512×512], [1024×1024]; the width-to-length ratios of the ROI candidate boxes are the following four types: [1∶2], [1∶1], [2∶1], [1∶3];

[0086] c. Input the candidate region proposals into the ROI Align layer. When the ROI Align layer traverses the candidate regions, the floating point boundaries are kept unquantized and the bilinear interpolation method is used to calculate the coordinate positions of the four vertices of each candidate region. The formula of the bilinear interpolation method is shown in the following formula (3):

[0087]

[0088] In formula (3), f(x,y) represents the pixel value of the center of each candidate region (x,y), (x1,y1), (x1,y2), (x2,y1), (x2,y2) are the coordinates of the four vertices of the candidate region, f(x1,y1) is the pixel value of the vertex (x1,y1), and f(x2,y1) is the pixel value of the vertex (x2,y1);

[0089] d. Input each ROI candidate box output by the ROI Align layer into the FCN network. The FCN network consists of three branches, which respectively realize target classification, bounding box regression and Mask generation, detect embedded parts and obtain their location information;

[0090] The loss function for Mask R-CNN target detection algorithm training is L total , the formula is shown in the following formula (4):

[0091] L total =L cls +L box +Lmask (4);

[0092] In formula (4), L cls is the target classification loss, and the calculation formula is shown in the following formula (5); L box is the bounding box regression loss, and its calculation formula is shown in the following formula (6); L mask is the mask branch loss, and the calculation formula is shown in the following formula (8);

[0093]

[0094] In formula (5), Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, and N represents the number of samples;

[0095]

[0096] In formula (6), Ti = [Tx, Ty, Tw, Th] represents the regression parameters of the i-th predicted bounding box relative to the candidate bounding box, T i * =[T x * ,T y * ,T w * ,T h * ] represents the regression parameter of the i-th true bounding box relative to the candidate bounding box, and smooth represents the smooth function, which is defined as follows (7):

[0097]

[0098] In formula (7), x represents the numerical difference between the predicted bounding box and the true bounding box;

[0099]

[0100] In formula (8), α and β represent two weight coefficients, Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, N represents the number of samples, |X∩Y| represents the number of pixels of the intersection of the predicted label and the true label, |X| represents the number of pixels of the true label, and |Y| represents the number of pixels of the predicted label. Represents the proportion of predicted labels to true labels. Perform a square operation to speed up the function convergence;

[0101] The training process of the target detection model is as follows:

[0102] S31, acquiring original images of embedded parts in molds of various types of components, and preprocessing the original images of embedded parts to obtain clear images of embedded parts to form an original data set;

[0103] S32, manually annotating the images in the original data set, where the annotated objects are embedded parts, and the annotated marking information includes the quantity information and position information of the embedded parts, and saving the images and their corresponding marking information to obtain a new data set, and then randomly dividing the new data set into a training set and a test set according to a 4:1 image quantity ratio;

[0104] S33, using the training set to perform multiple rounds of iterative training on the constructed target detection model, and using the test set to test it, the target detection model that meets the test requirements is the trained target detection network;

[0105] S4. Based on the real-time video of the mold in the target sensing area, the embedded parts are identified and tracked using the trained embedded parts detection network to obtain the measured embedded parts quantity information and location information. The specific steps are as follows:

[0106] S41, initialize the counter value of the target tracking model to 0, when the front edge of the mold carrying the embedded parts, that is, when the line between the two corners of the front end of the mold coincides with the front edge of the target sensing area, the embedded parts detection network identifies and tracks the embedded parts currently entering the target sensing area, otherwise continue to wait;

[0107] S42, when the target detection model based on the Mask R-CNN target detection algorithm detects the first embedded part, the target tracking model based on the SORT target tracking algorithm tracks the embedded parts extracted by the target detection model in each frame, assigns a globally unique identity code to each newly added embedded part, and calculates its position information, adds 1 to the counter Counter value of the target tracking model, and then returns the embedded part information with the identity code and position information to the target detection model, the target detection model identifies the position coordinates of the embedded part at this moment, when the rear edge of the mold carrying the embedded part, that is, the line between the two corners of the rear end of the mold coincides with the rear edge of the target sensing area, the embedded part detection network ends the identification and tracking of the embedded part in the mold, and the value of the counter is recorded as Count-cur, which is the number of embedded parts identified in real time, and then the position coordinates of all embedded parts recorded by the target detection model are normalized to obtain the position coordinate set Locations-cur of the embedded parts;

[0108] S43, judging whether the component mold currently carrying embedded parts is collected and inspected for the first time, that is, judging whether the component mold of this type is inspected for the first time according to the model of the component to which the mold belongs; when it is judged to be the first inspection, the embedded parts information in the component mold is verified with the embedded parts information of the standard component, and when the verified embedded parts quantity is consistent, the embedded parts set Tracks in the component mold is identified as the standard embedded parts set of this type of component, and the embedded parts quantity and position coordinate set are saved, which are recorded as Count-tem and Locations-tem respectively;

[0109] S44, when it is determined that the component mold with embedded parts is not detected for the first time, a position distance cost matrix is ​​constructed according to the embedded parts position coordinates between the real-time embedded parts set of the component mold currently detected in real time and the corresponding standard embedded parts set. Specifically, the number of embedded parts Count-cur in the real-time embedded parts set and the number of embedded parts Count-tem in the standard embedded parts set are first compared. When the two are equal, it is determined that the embedded parts in the component mold are qualified. When the two are not equal, a position distance cost matrix CostMatrix is ​​constructed according to the position coordinate set Locations-cur in the real-time embedded parts set and the position coordinate set Locations-tem in the standard embedded parts set. Then, the obtained position distance cost matrix CostMatrix is ​​input into the Hungarian algorithm for matching to obtain a matching result. The embedded parts that fail to match are recorded as missing embedded parts. The construction of the position distance cost matrix CostMatrix is ​​shown in the following formula (9):

[0110]

[0111] In formula (9), (X i ,Y i ) is the position coordinate of the embedded parts in the real-time embedded parts set, (X j ,Y j ) is the position coordinate of the embedded parts in the standard embedded parts set;

[0112] Among them, based on the target tracking model of the SORT target tracking algorithm, the SORT target tracking algorithm updates the target tracking trajectory by constructing a cost matrix COST. The calculation formula of the cost matrix COST is shown in the following equations (10)-(12):

[0113]

[0114]

[0115]

[0116] In formulas (10)-(12), A represents the area of ​​the prediction box, B represents the area of ​​the detection box, and C represents the area of ​​the smallest circumscribed rectangular box that can contain the prediction box and the detection box; IOU (i,j) Represents the intersection-over-union ratio of each embedded part tracking detection position, GIOU (i,j) Represents the generalized intersection-over-union ratio of each embedded part tracking detection position, GIOU min Represents the minimum GIOU value, IOU (i,j) The value range is [0,1], GIOU (i,j) The value range is [-1, 1]; that is, GIOU establishes a connection between the non-overlapping prediction box and the detection box by introducing the minimum circumscribed rectangular box of the two targets. It can not only measure the degree of overlap between targets when there is overlap like IOU, but also measure the distance between the two when there is no overlap. Then, the GIOU value is mapped to [0, 1] using formula (5), and the targets that are too far away are filtered out to construct the cost matrix;

[0117] S5. When comparing the measured quantity and position information of embedded parts of a component with the quantity and position information of standard embedded parts of the component of this type, if the quantity and position information of the two are completely consistent, it is determined that the embedding of embedded parts in the current component mold meets the requirements; otherwise, it is determined that there are defects in the embedding of embedded parts in the component mold, the movement of the mold table is stopped, and an alarm signal is issued. After receiving the alarm signal, the quality inspector removes and recycles the corresponding component mold.

[0118] Among them, the quantity information and position information of standard embedded parts of components are stored in the cloud database, that is, the quantity information and position information of standard embedded parts of components are the embedded parts feature information of all types of components to be produced extracted from the BIM model; when the production line starts production and inspection, the quantity information and position information of the embedded parts of the first component produced by each type of component is obtained through the embedded parts inspection network, and then compared with the embedded parts information of this type of standard component in the BIM model. When the first component meets the error requirements of the embedded parts information of this type of standard component, the embedded parts quantity information and position information obtained from the first component are used as the data of the standard embedded parts set in the Hungarian algorithm matching; when the first component produced does not meet the error requirements, the current component is scrapped, and the first component is reproducible and determined.

[0119] In order to verify the effectiveness of the method provided in this embodiment, this embodiment adopts an experimental method to verify the performance of the detection method. The specific experimental process includes four steps: data collection, data preprocessing, detection model construction, and embedded parts information detection.

[0120] 1. Data collection:

[0121] The data required for this experiment was collected by manual shooting, including a total of 1,000 original images as the original data set.

[0122] 2. Data preprocessing:

[0123] The images with blur, ghosting, and poor image quality in the original dataset were cleared, leaving 925 images. The original dataset was divided into a training set and a test set in a 4:1 ratio. The training set contains 740 images and the test set contains 185 images.

[0124] 3. Detection model construction:

[0125] In this embodiment, a Mask R-CNN target detection model is built, and its network architecture is shown in the following figure. Figure 3 shown.

[0126] 4. Embedded parts information detection:

[0127] In this embodiment, three test samples are set to test the embedded parts information. Each image test sample contains embedded parts 04, specifically embedded parts 1 and embedded parts 2. In test sample 1, the number of embedded parts 1 is 3, and the number of embedded parts 2 is also 3. In test sample 2, the number of embedded parts 1 is 6, and the number of embedded parts 2 is 10. In test sample 3, the number of embedded parts 1 is 7, and the number of embedded parts 2 is 11. The results of each test sample after being identified by the embedded parts detection network in this embodiment are as follows: Figure 4-6 shown.

[0128] After a large number of experiments, the accuracy of the information of each embedded part in the prefabricated building component in this embodiment was statistically analyzed, and the accuracy was more than 98%. Compared with manual quality inspection, it has achieved automation and high accuracy, so it has high practical value and can be promoted and applied.

[0129] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting defects of embedded parts in prefabricated buildings, characterized in that: The specific steps include: S1. Use a video acquisition device disposed above the mold platform to collect real-time video of the mold with embedded parts on the mold platform; S2, setting a target sensing area for online detection by the video acquisition device, the area captured by the video acquisition device at a fixed viewing angle is the defect detection area, and the shooting area of ​​the target sensing area video frame is within the range of the defect detection area; S3. Build and train an embedded parts detection network, which includes a target detection model based on the Mask R-CNN target detection algorithm and a target tracking model based on the SORT target tracking algorithm; The process of using the Mask R-CNN target detection algorithm to detect embedded parts is as follows: a. Input the embedded parts image data into the Backbone network of Mask R-CNN. The Backbone network consists of the EfficientNet network and the FPN network; First, the EfficientNet network uniformly scales the three parameters of network depth d, width w, and input image resolution r. The scaling formula is shown in the following formula (2): In formula (2), α, β, γ are all constants, and φ is the scaling factor; The feature map processed by the EfficientNet network is then input into the FPN network. The FPN network first uses a 1×1 convolution to perform a convolution operation on the input feature map to obtain feature maps, and then sets a ROI for each point in the feature maps to obtain multiple ROI candidate boxes. b. Multiple ROI candidate boxes are input into the RPN network and enter two different branches after a 3×3 convolutional layer. One branch performs binary classification on multiple ROI candidate boxes, and the other branch performs border regression on multiple ROI candidate boxes, thereby filtering out some ROI candidate boxes and finally generating candidate region proposals. In the RPN network, the scales of the ROI candidate boxes are set to the following six types: [32×32], [64×64], [128×128], [256×256], [512× 512], [1024×1024]; the width-to-length ratios of the ROI candidate boxes are the following four: [1∶2], [1∶1], [2∶1], [1∶3]; c. Input the candidate region proposals into the ROI Align layer. When the ROI Align layer traverses the candidate regions, the floating point boundaries are kept unquantized and the bilinear interpolation method is used to calculate the coordinate positions of the four vertices of each candidate region. The formula of the bilinear interpolation method is shown in the following formula (3): In formula (3), f(x,y) represents the pixel value of the center (x,y) of each candidate region, (x1,y1), (x1,y2), (x2,y1), (x2,y2) are the coordinates of the four vertices of the candidate region, f(x1,y1) is the pixel value of the vertex (x1,y1), and f(x2,y1) is the pixel value of the vertex (x2,y1); d. Input each ROI candidate box output by the ROI Align layer into the FCN network. The FCN network consists of three branches, which respectively realize target classification, bounding box regression and Mask generation, detect embedded parts and obtain their location information; S4, based on the video frame of the target sensing area, the embedded parts are identified and tracked using the trained embedded parts detection network to obtain the actual measured embedded parts quantity information and position information of the component; S5. Compare the actually measured embedded parts quantity information and position information of the component with the standard embedded parts quantity information and position information of the component of this type to obtain the detection result of the embedded parts in the component mold.

2. A method for detecting defects of embedded parts in prefabricated buildings according to claim 1, characterized in that: The video acquisition device shoots the component mold carrying embedded parts on the mold platform in real time along a fixed downward angle of view. Along the conveying direction of the mold platform, the intersection of the lens axis of the video acquisition device and the mold platform is located behind the vertical intersection of the video acquisition device and the mold platform.

3. The method for detecting defects of embedded parts in prefabricated buildings according to claim 1, characterized in that: The video frame length of the target sensing area is equal to the video frame length of the mold real-time video. The pixel value Wa of the video frame width of the target sensing area is calculated using the following formula (1): In formula (1), Wv is the pixel value of the video frame width of the mold real-time video, Tmax is the maximum time that the embedded part is in the mold real-time video, Fmax is the average frame rate of the mold real-time video, and Fmin is the minimum number of frames that the center point of the embedded part stays in the target sensing area video frame.

4. The method for detecting defects of embedded parts in prefabricated buildings according to claim 1, characterized in that: The loss function of the Mask R-CNN target detection algorithm training is L total , the formula is shown in the following formula (4): L total =L cls +L box +L mask (4); In formula (4), L cls is the target classification loss, and the calculation formula is shown in the following formula (5); L box is the bounding box regression loss, and its calculation formula is shown in the following formula (6); L mask is the mask branch loss, and the calculation formula is shown in the following formula (8); In formula (5), Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, and N represents the number of samples; In formula (6), Ti = [Tx, Ty, Tw, Th] represents the regression parameters of the i-th predicted bounding box relative to the candidate bounding box, represents the regression parameter of the i-th true bounding box relative to the candidate bounding box, and smooth represents the smooth function, which is defined as follows (7): In formula (7), x represents the numerical difference between the predicted bounding box and the true bounding box; In formula (8), α and β represent two weight coefficients, Xi represents the true label of the i-th sample, Yi represents the predicted label of the i-th sample, N represents the number of samples, |X∩Y| represents the number of pixels of the intersection of the predicted label and the true label, |X| represents the number of pixels of the true label, and |Y| represents the number of pixels of the predicted label. Represents the proportion of predicted labels to true labels. Perform a square operation to speed up the function convergence.

5. The method for detecting defects of embedded parts in prefabricated buildings according to claim 1, characterized in that: The specific steps of identifying and tracking embedded parts using the trained embedded parts detection network based on the video frame of the target sensing area are as follows: S41. Initialize the counter Counter value of the target tracking model to 0. When the embedded part is detected in the target sensing area for the first time, that is, when the target detection model based on the Mask R-CNN target detection algorithm detects the first embedded part, the target tracking model based on the SORT target tracking algorithm tracks each embedded part extracted by the target detection model in each frame, assigns a globally unique identity code to each newly added embedded part, and calculates its position information. The counter Counter value of the target tracking model based on the SORT target tracking algorithm is increased by 1. The target tracking model saves the image frame at this moment, and the target detection model identifies the position coordinates of the embedded part at this moment. When the component mold scanning is completed, the value of the counter is recorded as Count-cur, and then the position coordinates of all embedded parts recorded by the target detection model are normalized to obtain the position coordinate set Locations-cur of the embedded parts. S42, judging whether the component mold currently carrying embedded parts is collected and inspected for the first time, that is, judging whether the component mold of this type is inspected for the first time according to the model of the component to which the mold belongs; when it is judged to be the first inspection, the embedded parts information in the component mold is verified with the embedded parts information of the standard component, and when the verified embedded parts quantity is consistent, the embedded parts set Tracks in the component mold is recognized as the standard embedded parts set of this type of component, and the embedded parts quantity and position coordinate set are saved, which are recorded as Count-tem and Locations-tem respectively; S43, when it is determined that the component mold with embedded parts is not detected for the first time, a position distance cost matrix is ​​constructed according to the embedded parts position coordinates between the real-time embedded parts set of the component mold currently detected in real time and the corresponding standard embedded parts set. Specifically, the number of embedded parts Count-cur in the real-time embedded parts set and the number of embedded parts Count-tem in the standard embedded parts set are first compared. When the two are equal, it is determined that the embedded parts in the component mold are qualified. When the two are not equal, a position distance cost matrix CostMatrix is ​​constructed according to the position coordinate set Locations-cur in the real-time embedded parts set and the position coordinate set Locations-tem in the standard embedded parts set. Then, the obtained position distance cost matrix CostMatrix is ​​input into the Hungarian algorithm for matching to obtain a matching result. The embedded parts that fail to match are recorded as missing embedded parts. The construction of the position distance cost matrix CostMatrix is ​​shown in the following formula (9): In formula (9), (X i ,Y i ) is the position coordinate of the embedded parts in the real-time embedded parts set, (X j ,Y j ) are the position coordinates of the embedded parts in the standard embedded parts set.

6. A method for detecting defects of embedded parts in prefabricated buildings according to claim 5, characterized in that: The target tracking model based on the SORT target tracking algorithm, the SORT target tracking algorithm updates the target tracking trajectory by constructing a cost matrix COST, and the calculation formula of the cost matrix COST is shown in the following formulas (10)-(12): In formulas (10)-(12), A represents the area of ​​the prediction box, B represents the area of ​​the detection box, and C represents the area of ​​the minimum bounding rectangle that can contain the prediction box and the detection box; IOU (i,j) Represents the intersection-over-union ratio of each embedded part tracking detection position, GIOU (i,j) Represents the generalized intersection-over-union ratio of each embedded part tracking detection position, GIOU min Represents the minimum GIOU value, IOU (i,j) The value range is [0,1], GIOU (i,j) The value range is [-1,1].

7. A method for detecting defects of embedded parts in prefabricated buildings according to claim 5, characterized in that: The video frame of the component mold in the target sensing area is specifically confirmed by the following steps: when the front edge of the mold carrying the embedded parts, that is, when the line between the two corners of the front end of the mold coincides with the front edge of the target sensing area, the embedded parts detection network identifies and tracks the embedded parts currently entering the target sensing area, otherwise it continues to wait; when the embedded parts detection network identifies and tracks the embedded parts, when the rear edge of the mold carrying the embedded parts, that is, the line between the two corners of the rear end of the mold coincides with the rear edge of the target sensing area, the embedded parts detection network ends the identification and tracking of the embedded parts in the mold, and the video frame of the component mold in the target sensing area is obtained.

8. The method for detecting defects of embedded parts in prefabricated buildings according to claim 5, characterized in that: The standard embedded parts quantity information and position information of the components are stored in a cloud database, that is, the standard embedded parts quantity information and position information of the components are the embedded parts feature information of all types of components to be produced extracted from the BIM model; when the production line starts production and detection, the quantity information and position information of the embedded parts of the first component produced by each type of component is obtained through the embedded parts detection network, and then compared with the embedded parts information of this type of standard component in the BIM model. When the first component meets the error requirement of the embedded parts information of this type of standard component, the embedded parts quantity information and position information obtained from the first component are used as the data of the standard embedded parts set in the Hungarian algorithm matching; when the first component produced does not meet the error requirement, the current component is scrapped, and the first component is reproducible and determined.

9. The method for detecting defects of embedded parts in prefabricated buildings according to claim 1, characterized in that: When the measured embedded parts quantity information and position information of the component are compared with the standard embedded parts quantity information and position information of the component of this type, if the quantity information and position information of the two are completely consistent, it is determined that the embedded parts in the current component mold are buried in accordance with the requirements; otherwise, it is determined that there are defects in the embedded parts in the component mold, the movement of the mold table is stopped, and an alarm signal is issued. After receiving the alarm signal, the quality inspector removes and recycles the corresponding component mold.

Images