Laminated slab embedded structure detection method and system and readable medium

Abstract

The invention relates to the technical field of civil engineering and prefabricated part visual inspection, in particular to a laminated slab embedded structure detection method and system and a readable medium. According to the laminated slab embedded structure detection method provided by the invention, a backbone network is improved on the basis of a YOLOv8 framework, and a new laminated slab detection model is formed so as to detect the embedded structure on the unpoured laminated slab. In a new backbone network AIEB network module, on the basis of a traditional backbone network, a multi-stage auxiliary information feature extraction operation composed of a CBLINE operation and a CBFuse calling operation is added, so that target feature information of reinforcing ribs with a small number to dozens of pixel points in a photo and target feature information of a laminated slab with the size to almost a whole photo can be well extracted; therefore, the new backbone network can extract richer image features, and more information is provided for feature fusion processing of the subsequent neck network. According to the invention, the problem of lack of an effective pre-pouring prefabricated part detection technology is solved.

Description

Technical Field

[0001] The invention relates to the technical field of civil engineering and prefabricated component visual inspection, in particular to a composite plate embedded structure inspection method, system and readable medium. Background Art

[0002] Prefabricated components are increasingly being used in the construction industry. Currently, prefabricated components are mainly prefabricated in factories or off-site, and then transported to construction sites for assembly. This off-site production and assembly method brings convenience and efficiency to on-site construction; however, if the quality of prefabricated components does not meet standards, it will delay construction progress.

[0003] At present, most prefabricated components are visually inspected after pouring concrete. After pouring, the embedded components are obvious and the inspection accuracy is high. However, if quality problems are detected after pouring, the rework cost is too high, which does not conform to the production concept of high efficiency and low cost.

[0004] Before pouring concrete, the structure of prefabricated components is very complex and visual inspection is difficult. Taking the existing YOLOv detection network framework as an example, the current best performance is Figure 2 The YOLOv8 framework shown.

[0005] like Figure 2As shown in the figure, the traditional YOLOv8 architecture mainly consists of three parts: the backbone network, the neck network, and the head network. In the backbone network, the convolution operation of the input image is first performed to reduce the computational complexity of subsequent operations and adapt to the required feature map size. Then, the deep features of the image are extracted through four Conv+C2f module feature extraction operations. The number of modules of the four layers of C2f operations performed successively is different, namely 3, 6, 6, and 3 C2f operations. These multiple feature extraction operations per layer can well extract deeper image features. However, for data sets with large target size spans such as embedded parts on laminated boards, multiple C2f module processing at one layer may cause part of the feature map information to be lost between multiple C2f operations, making it impossible to extract key information. In addition, YOLOv8 improves the SPPF (spatial pyramid pooling fusion) module based on YOLOv5, which is a significant enhancement for accelerating model reasoning. The YOLOv8 model mainly uses the idea of ​​PAN-FPN (Path Aggregation Network-Feature Pyramid Network) in the neck network, and performs a bottom-up (PAN structure) upsampling operation on the feature map output by the backbone network to increase the size of the feature map, and fuses it with the shallow information of the backbone network to increase the amount of feature information. Then, through the top-down downsampling operation, it is fused with the information output by the PAN (path aggregation network) structure to increase the target context information. In this way, the entire PAN-FPN structure can more accurately fuse the features extracted by the backbone network and hand them over to the subsequent detection head for prediction. For the detection head part, YOLOv8 abandons the previously used Anchor-Based mechanism and adopts the Anchor-Free method to locate the object by referring to the center of the target and estimating the distance between the center and the bounding box. This enhancement improves the accuracy and efficiency of factory detection of prefabricated composite panel components.

[0006] When the backbone network of the YOLOv8 model performs feature extraction, information from different layers of the feature pyramid can be used to perform different detection tasks. For example, when shallow feature information is fused with deep network features, shallow features will mainly detect small objects, and the system will treat other large and small objects as background. However, this behavior will cause the deep feature pyramid to lose contextual information of the detected target. For example, when detecting reinforcing ribs, because there are fewer target pixels, it is not enough to refer to only the local features of the reinforcing ribs. It is also necessary to analyze the surrounding information of the target. At present, the accuracy of the YOLOv8 framework in large target detection can reach more than 90%, but it needs to be strengthened in small target detection. It cannot even reach an accuracy of more than 70% in the detection of some embedded components. It can be seen that prefabricated components still need more accurate visual detection methods before casting. Summary of the invention

[0007] In order to overcome the defect of the above-mentioned prior art that there is a lack of effective prefabricated component detection technology before pouring, the present invention proposes a composite slab embedded structure detection method, which can achieve high-precision detection of the embedded structure of the prefabricated component before pouring.

[0008] The present invention proposes a method for detecting embedded structures of laminated plates. First, a laminated plate detection model is constructed to mark the laminated plate contour and the embedded structure contour on the laminated plate image. The embedded component structure includes embedded components and embedded holes. The laminated plate detection model is trained on a laminated plate data set marked with the laminated plate contour and the embedded structure contour until the model converges. The laminated plate image to be detected is photographed and input into the converged laminated plate detection model. The laminated plate detection model outputs the embedded structure detection result. The overlapped plate detection model is improved on the basis of the YOLOv8 framework; the improvement method includes replacing the backbone network with the AIEB network module; the AIEB network module includes a 29-layer network, namely the dimension splitting layer of the P0 layer, the convolution layers of the P1 layer, the P2 layer, the P4 layer, the P6 layer, the P8 layer, the P15 layer, the P17 layer, the P20 layer, the P23 layer and the P26 layer; C2f network layer for P3, P5, P7, P9, P19, P22, P25 and P28 layers; CBLinear network layer for P10, P11, P12, P13 and P14 layers; CBFuse network layer for P16, P18, P21, P24 and P27 layers; The P0-P9 layers are connected sequentially, and the P15 to P28 layers are connected sequentially; the input of the P15 convolutional layer is connected to the output of the P0 dimension splitting layer; the C2f network layer of the P9 layer is connected to the CBFuse network layers of the P16, P18, P21, P24, and P27 layers through the CBLinear network layer of the P10 layer; The C2f network layer of the P7 layer is connected to the CBFuse network layers of the P16 layer, the P18 layer, the P21 layer, and the P24 layer through the CBLinear network layer of the P11 layer; the C2f network layer of the P5 layer is connected to the CBFuse network layers of the P16 layer, the P18 layer, and the P21 layer through the CBLinear network layer of the P12 layer; The C2f network layer of the P3 layer is connected to the CBFuse network layers of the P16 and P18 layers through the CBLinear network layer of the P13 layer; the convolutional layer of the P1 layer is connected to the CBFuse network layer of the P16 layer through the CBLinear network layer of the P14 layer; the output of the C2f network layer of the P22 layer, the output of the C2f network layer of the P25 layer, and the output of the C2f network layer of the P28 layer are connected to the neck network as the output of the backbone network.

[0009] Preferably, the improvement method of the YOLOv8 framework also includes: replacing the neck network with a SOBI network module; the SOBI network module includes a first convolutional network, a second convolutional network, a third convolutional network, a fourth convolutional network, and a first sampling splicing module, a second sampling splicing module, a third sampling splicing module, a first convolution splicing module, a second convolution splicing module, and a third convolution splicing module connected in sequence; The first sampling splicing module, the second sampling splicing module and the third sampling splicing module are collectively referred to as sampling splicing modules, which are used to sample the acquired data and then perform feature fusion; the first convolution splicing module, the second convolution splicing module and the third convolution splicing module are collectively referred to as convolution splicing modules, which are used to perform convolution processing on the acquired data and then perform feature fusion; The input end of the first convolutional network, the input end of the second convolutional network, the input end of the third convolutional network and the input end of the fourth convolutional network are used as the four input ends of the SOBI network module, which are used to connect the four C2f network layers of the backbone network from shallow to deep; the input of the first sampling splicing module is connected to the output of the fourth convolutional network and the output of the third convolutional network, the input of the second sampling splicing module is also connected to the output of the second convolutional network, and the input of the third sampling splicing module is also connected to the output of the first convolutional network; the input of the first convolutional splicing module is also connected to the output of the second convolutional network and the output of the second sampling splicing module, the input of the second convolutional splicing module is also connected to the output of the third convolutional network and the output of the first sampling splicing module, and the input of the third convolutional splicing module is also connected to the output of the fourth convolutional network; The output end of the third sampling splicing module, the output end of the first convolution splicing module, the output end of the second convolution splicing module and the output end of the third convolution splicing module serve as the four output ends of the SOBI network module, which are used to connect the corresponding segmentation heads of the head network.

[0010] Preferably, the sampling splicing module is composed of a sampling network, a splicing network and a C2f network connected in sequence; the convolution splicing module is composed of a convolution network, a splicing network and a C2f network connected in sequence; and in the sampling splicing module and the convolution splicing module, the back end of the splicing network is connected to a 3-layer C2f network; the input of the sampling network in the first sampling splicing module is connected to the output of the fourth convolution network; the input of the splicing network in the first sampling splicing module is also connected to the output of the third convolution network; the input of the splicing network in the second sampling splicing module is also connected to the output of the second convolution network, and the input of the splicing network in the third sampling splicing module is also connected to the output of the first convolution network; the input of the splicing network in the first convolution splicing module is also connected to the output of the second convolution network and the output of the second sampling splicing module, the input of the splicing network in the second convolution splicing module is also connected to the output of the third convolution network and the output of the first sampling splicing module, and the input of the splicing network in the third convolution splicing module is also connected to the output of the fourth convolution network.

[0011] Preferably, the input end of the second convolutional network, the input end of the third convolutional network and the input end of the fourth convolutional network are connected to the outputs of the last three C2f network layers in the backbone network in sequence, and the input end of the first convolutional network is connected to the output of the C2f network layer of the P2 layer of the backbone network or the output of the C2f network layer of the P19 layer.

[0012] Preferably, the stitching network in the third sampling stitching module, the stitching network in the first convolution stitching module, the stitching network in the second convolution stitching module and the stitching network in the third convolution stitching module all adopt SDI modules; the SDI module includes: a spatial attention unit, a channel attention unit, a sampling unit, a smoothed convolution network and a channel stitching unit; the multiple data input by the SDI module all pass through the spatial attention unit and the channel attention unit in sequence to extract spatial attention features and channel attention features; the data output by the channel attention unit is processed by the sampling unit so that the multiple data output by the sampling unit maintain the same dimension; the multiple data of the same dimension output by the sampling unit are convolved through a smoothed convolution network, and then sent to the channel stitching unit for dimensional stitching as the output data of the SDI module.

[0013] Preferably, the sampling networks in the first sampling splicing module and the second sampling splicing module are configured as Dysample networks.

[0014] Preferably, in the neck network, the sampling network adopts the Upsample network, and the splicing network adopts the Concat network.

[0015] Preferably, the method for acquiring the superimposed plate data set is: first, take an image of the superimposed plate without pouring concrete, and mark the outline of the superimposed plate and the outline of the embedded structure; then, perform data enhancement on the captured superimposed plate image, and the data enhancement method includes exposure, darkening, noise and sharpening processing; count the captured superimposed plate images, the data enhanced superimposed plate images and the marked outlines of all superimposed plate images to form a superimposed plate data set.

[0016] The present invention proposes a composite plate pre-embedded structure detection system, which includes a memory and a processor. The memory stores a computer program. The processor is connected to the memory and is used to execute the computer program to implement the composite plate pre-embedded structure detection method.

[0017] The present invention provides a readable medium storing a computer program, which is used to implement the composite plate embedded structure detection method when executed.

[0018] The advantages of the present invention are: (1) The method for detecting embedded structures of composite panels proposed in the present invention improves the backbone network based on the YOLOv8 framework to form a new composite panel detection model to detect embedded structures on uncast composite panels. The new backbone network AIEB network module in the present invention adds a multi-level auxiliary information feature extraction operation consisting of CBLinear operations and CBFuse call operations on the basis of the traditional backbone network, so that the target feature information of the composite panel in the photo, from a reinforcement rib as small as a dozen pixels to a composite panel as large as almost the entire photo, can be better extracted; thereby, the new backbone network can extract richer image features and provide more information for the subsequent feature fusion processing of the neck network.

[0019] (2) In the present invention, the model structure is further improved, the neck network is replaced by the SOBI network module, the AIEB network module and the SOBI network module together form a serial multi-level auxiliary branch structure (SMA), and a jump connection is added from the feature pyramid of the new backbone network AIEB network module to the new neck network SOBI network module downsampling structure, so as to perform a more comprehensive feature fusion of feature information and increase the proportion of small targets in the feature fusion information.

[0020] (3) In the SMA1 structure, the shallow information of the backbone network is fused into the neck network. The receptive field of the feature map of the fusion layer is very small, which reduces the information ratio of larger targets and greatly improves the detection accuracy of smaller targets. In the SMA2 structure, the multi-level auxiliary information of the deep layer of the backbone network is fused into the neck network, including large-scale target information with a larger receptive field. In this way, the information of the small target fusion layer fused into the neck network contains part of the large target information, which can effectively prevent the attention of small targets from being too much, resulting in the loss of large target information such as the composite board frame.

[0021] (4) The present invention introduces an SDI module, and the feature maps of each layer are processed by the spatial and channel attention mechanism before adding the multiple layers of feature maps. The attention mechanism improves the accuracy of the feature map information by integrating the local spatial information and the global channel information in the feature map, and solves the problem of the fusion weight of the feature map information with a large number of layers.

[0022] (5) The present invention introduces the Dysample network, which can dynamically adjust the upsampling target and area, so that more small target pixels such as the reinforcement in the composite plate image are sampled, thereby increasing the proportion of small target pixels in the feature map and reducing the problem of loss of fine-grained details. It improves the detection speed while improving the accuracy, thus better meeting the engineering needs.

[0023] (6) The method for detecting embedded structures of composite slabs proposed in the present invention uses an improved model to detect composite slab components before casting. During the model improvement process, attention is paid to the characteristic information of small target components, which provides an effective means for visual inspection of uncast composite slabs with complex structures, and is beneficial to efficiency improvement and cost control in engineering applications.

[0024] (7) The composite plate pre-embedded structure detection system and readable medium proposed in the present invention provide a carrier for the composite plate pre-embedded structure detection method proposed in the present invention, thereby facilitating the promotion and application of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a method for detecting embedded structures of composite panels; Figure 2 This is the YOLOv8 model structure diagram; Figure 3 This is the AIEB network module structure diagram; Figure 4 This is the YOLOv8_AIEB model structure diagram; Figure 5 It is the SOBI network framework; Figure 6 This is the YOLOv8_SMA1 model structure diagram; Figure 7 This is the SDI structure diagram; Figure 8(a) is a diagram of the YOLOv8_SMA1_S model structure; Figure 8(b) is a diagram of the YOLOv8_SMA1_D model structure; Figure 8(c) is a diagram of the YOLOv8_ASDS1 model structure; Fig. 9 The performance comparison of two connection modes SAM1 and SAM2 under different module combinations; SMA1 represents the YOLOv8_SMA1 framework, SMA2 represents the YOLOv8_SMA2 framework; v8 represents the YOLOv8 framework, and DS represents the Dysample network; Figure 10(a) shows the mAP comparison of the models on each target; Figure 10(b) shows the accuracy comparison of the model on each target; Figure 10(c) shows the recall comparison of the models on each target. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Reference Figure 1 , a method for detecting the embedded structure of a composite plate proposed in this embodiment: Firstly, the YOLOv8 framework is improved to build a laminate detection model. The laminate detection model is used to mark the laminate contour and embedded structure contour on the laminate image. The embedded structure includes embedded components and embedded holes. Then, the laminated plate detection model is trained on the laminated plate data set labeled with the laminated plate contour and the embedded structure contour until the model converges; Then, the image of the superimposed plate to be inspected is photographed and input into the converged superimposed plate inspection model, which outputs the inspection result of the embedded structure, i.e., the outline of the superimposed plate and the outlines of each embedded structure are marked on the image of the superimposed plate to be inspected.

[0028] Considering that there is a lack of research on instance segmentation of composite panels and a lack of public datasets, in this implementation, researchers first go to the prefabricated component factory for field photography, obtain images of composite panels containing components but without concrete as original images, and manually annotate the composite panel frame (FL) and embedded structures, including metal junction boxes (MJB), plastic junction boxes (PJB), reserved holes (RH), truss bars (TB) and lifting reinforcement bars (RR); then perform data enhancement on the original images; then combine the original images and the data enhanced images to form a composite panel dataset.

[0029] The following introduces various improved composite plate detection models proposed by the present invention.

[0030] The first composite board detection model YOLOv_AIEB Reference Figure 3 , Figure 4 , the overlapping plate detection model YOLOv_AIEB replaces the backbone network with the AIEB network module compared to the model YOLOv8.

[0031] The AIEB network module includes 29 layers of networks, specifically, the sequentially stacked dimension splitting layer, convolution layer, convolution layer, C2f network layer, convolution layer, C2f network layer, convolution layer, C2f network layer, convolution layer, C2f network layer, 5 CBLinear network layers, convolution layer, CBFuse network layer, convolution layer, CBFuse network layer, C2f network layer, convolution layer, CBFuse network layer, C2f network layer, convolution layer, CBFuse network layer, C2f network layer, convolution layer, CBFuse network layer, C2f network layer, convolution layer, CBFuse network layer, C2f network layer; According to the stacking order, the 29 layers of the network are recorded as P0-P28; that is, the P0 layer is the dimension splitting layer (Silence), the P1 layer, P2 layer, P4 layer, P6 layer, P8 layer, P15 layer, P17 layer, P20 layer, P23 layer, and P26 layer are all convolutional layers; the P3 layer, P5 layer, P7 layer, P9 layer, P19 layer, P22 layer, P25 layer, and P28 layer are all C2f network layers; P10-P14 layers are all CBLinear network layers; P16, P18, P21, P24, and P27 layers are all CBFuse network layers; In the AIEB network module, the P0-P9 layers are connected sequentially, and the P15 to P28 layers are connected sequentially; The input of the P15 convolutional layer is connected to the output of the P0 dimension splitting layer; The C2f network layer of the P9 layer connects the CBFuse network layers of the P16, P18, P21, P24, and P27 layers through the CBLinear network layer of the P10 layer; The C2f network layer of the P7 layer connects to the CBFuse network layers of the P16, P18, P21, and P24 layers through the CBLinear network layer of the P11 layer; The C2f network layer of the P5 layer connects to the CBFuse network layers of the P16 layer, the P18 layer, and the P21 layer through the CBLinear network layer of the P12 layer; The C2f network layer of the P3 layer connects the CBFuse network layers of the P16 and P18 layers through the CBLinear network layer of the P13 layer; The convolutional layer of the P1 layer is connected to the CBFuse network layer of the P16 layer through the CBLinear network layer of the P14 layer; The output of the C2f network layer of the P22 layer, the output of the C2f network layer of the P25 layer, and the output of the C2f network layer of the P28 layer are connected to the neck network as the output of the backbone network.

[0032] In this implementation, CBLinear operations and CBFuse call operations are added after the traditional backbone network to generate multi-level auxiliary information, reduce information bottlenecks, and improve feature extraction capabilities. The subsequent neck network of the main branch is then connected in series to the multi-level auxiliary information feature extraction operation, so that the AIEB network module includes a feature extraction network of P0-P9 layers and a multi-level auxiliary information feature extraction network of P10-P28 layers, thereby enabling the new backbone network to extract richer image features and provide more information for the feature fusion processing of the subsequent neck network.

[0033] In this implementation, 5 layers of CBLinear operations are designed and CBFuse call operations are performed successively, so that the information of all layers in the backbone network of the feature pyramid structure is aggregated and distributed, so that the feature map of each composite board component target has surrounding information and the feature extraction capability is maximized. The CBLinear network layer splits the feature pyramid information of the corresponding layer into multiple parts after convolution for subsequent CBFuse network layers to call, and the information called each time is only a part of the information split in the output result of the CBLinear network layer. In this way, each CBFuse operation calls the multi-layer information of the feature pyramid, which not only extracts all the information of large and small targets and contexts, but also improves the calculation speed of the model due to the splitting operation of the CBLinear network layer.

[0034] The C2f network layer in the YOLOv8 backbone network has multiple operations per layer. This module addition method can quickly convolve to a deep network in a unidirectional backbone network. After adding multiple layers of auxiliary information structures to the AIEB network module, multiple C2f operations per layer will not only greatly increase the amount of calculation, but may also cause the pyramid information gradient to be too large, so that a lot of information input to the neck network will be lost, which is not conducive to improving the network accuracy. Therefore, the AIEB network module takes all C2f network layers as 1 C2f operation. And in order to adapt to the number of channels and feature map sizes calculated in each layer of the structure, ensure that the number of feature pyramid information channels called by the CBFuse network layer is consistent each time. The last SPPF module is deleted in the AIEB network module. This processing not only unifies the number of channels and makes the network lightweight, but also prevents target information of each scale from being lost in the pyramid gradient.

[0035] The second composite plate detection model YOLOv_SMA1 In YOLOv_AIEB, the neck network uses the original PAN-FPN structure. The PAN structure extracts three layers of feature pyramid information in the backbone network for data feature fusion. The three layers of feature pyramid information transmission FPN (Feature Pyramid Network) structure will cause some information loss, especially for targets with large spans such as composite plate frames and reinforcement ribs, which require more complete information. Therefore, refer to Figure 5 ,This implementation considers further improving the neck network on the basis of YOLOv_AIEB, adding jump connections from the feature pyramid to the neck network downsampling structure, performing a more comprehensive feature fusion of the feature information, and increasing the proportion of small targets in the feature fusion information.

[0036] Reference Figure 6 The new neck network adopts the SOBI network module, which includes the first convolution network, the second convolution network, the third convolution network, the fourth convolution network, the first sampling splicing module, the second sampling splicing module, the third sampling splicing module, the first convolution splicing module, the second convolution splicing module and the third convolution splicing module; the first convolution network, the second convolution network, the third convolution network and the fourth convolution network are used to connect the backbone network.

[0037] The first sampling splicing module, the second sampling splicing module and the third sampling splicing module have the same structure, collectively referred to as sampling splicing modules, which are composed of a sampling network, a Concat network and a C2f network connected in sequence; and the back end of the Concat network is connected to a 3-layer C2f network. The sampling splicing module takes the input of the sampling network as input and the output of the C2f network as output.

[0038] The first convolutional splicing module, the second convolutional splicing module and the third convolutional splicing module have the same structure and are collectively referred to as convolutional splicing modules. The convolutional splicing module is composed of a sequentially connected convolutional network, a Concat network and a C2f network; and the back end of the Concat network is connected to a 3-layer C2f network. The convolutional splicing module takes the input of the convolutional network as input and the output of the C2f network as output.

[0039] The sampling network may specifically adopt an Upsample network; and the concatenation network may specifically adopt a Concat network.

[0040] The input of the first sampling splicing module is connected to the output of the fourth convolutional network, and the input of the splicing network in the first sampling splicing module is also connected to the output of the third convolutional network; the input of the splicing network in the second sampling splicing module is also connected to the output of the second convolutional network, and the input of the splicing network in the third sampling splicing module is also connected to the output of the first convolutional network; The input of the splicing network in the first convolutional splicing module is also connected to the output of the second convolutional network and the output of the second sampling splicing module, the input of the splicing network in the second convolutional splicing module is also connected to the output of the third convolutional network and the output of the first sampling splicing module, and the input of the splicing network in the third convolutional splicing module is also connected to the output of the fourth convolutional network.

[0041] Based on YOLOv_AIEB, the model formed by replacing the neck network with SOBI is recorded as YOLOv_SMA1. In YOLOv_SMA1, the head network contains four detection heads, and the inputs of the four detection heads are respectively connected to the output of the third sampling splicing module, the output of the first convolution splicing module, the output of the second convolution splicing module, and the output of the third convolution splicing module.

[0042] The SOBI network module first convolves the four layers of information extracted by the backbone network to adjust the feature map size, and then performs feature fusion. During the fusion process, the information is output from the shallow layer to the deep layer to the four size segmentation detection heads for target segmentation. This implementation method focuses on designing four detection heads in prediction, which is more suitable for the situation where the target size span of the superimposed plate data set is too large.

[0043] In the model YOLOv_SMA1, the input of the first convolutional network is connected to the output of the P3 layer C2f network layer in the AIEB network module, the input of the second convolutional network is connected to the output of the P22 layer C2f network layer in the AIEB network module, the input of the third convolutional network is connected to the output of the P25 layer C2f network layer in the AIEB network module, and the input of the fourth convolutional network is connected to the output of the P28 layer C2f network layer in the AIEB network module.

[0044] The P3 layer only performs one convolution operation, the feature map receptive field is very small, and the information contained basically does not contain large targets. When this layer is integrated into the neck network, the proportion of information about large targets is very small.

[0045] In this embodiment, the backbone network adopts the AIEB network module, the neck network adopts the SOBI network module, and the network structure in which the input of the first convolutional network in the SOBI network module is connected to the output of the P3 layer C2f network layer in the AIEB network module is recorded as the SMA1 structure.

[0046] The third composite plate detection model YOLOv_SMA2 Reference Figure 6Compared with YOLOv_SMA1, in YOLOv_SMA2, the input of the first convolutional network in the SOBI module is connected to the output of the P19 layer C2f network layer in the AIEB network module. The P19 layer C2f network layer in the AIEB network module belongs to the multi-level auxiliary information module. The P19 layer C2f network layer has a small receptive field, but the feature map has a part of the deep information of the feature pyramid through the aggregation of multi-level auxiliary information, that is, it includes large-scale target information with a large receptive field. In this way, the small target fusion layer has a part of the large target information in the information fused into the neck network, which can effectively prevent the small target from paying too much attention, resulting in the loss of large target information such as the composite board frame.

[0047] In this embodiment, the backbone network adopts the AIEB network module, the neck network adopts the SOBI network module, and the network structure in which the input of the first convolutional network in the SOBI network module is connected to the output of the P19 layer C2f network layer in the AIEB network module is recorded as the SMA2 structure.

[0048] The fourth composite plate detection model YOLOv_SMA1_S In the above improved model, the neck network improvement solves the multi-scale problem of the data set target by adding multiple layers of information fusion. However, adding multiple layers of information also makes it more difficult to distinguish information of different scales. In order to better distinguish information of different scales and perform targeted fusion, this implementation proposes the overlapped plate detection model YOLOv_SMA1_S and introduces a multi-scale feature fusion module SDI. The SDI structure is as follows Figure 7 As shown in the figure, before adding multiple layers of feature maps, the feature maps of each layer are processed by the spatial and channel attention mechanism. This attention mechanism improves the accuracy of feature map information by integrating local spatial information and global channel information in the feature map; for example, when fusing information in the composite plate data set, the local concentrated information of the small target of the reinforcement rib can be sorted out with the overall information around the target, and then the feature is extracted and input into the detection head for detection, making the target information clearer. The design of the SDI module solves the problem of information fusion weight of feature maps with a large span of layers.

[0049] Compared with the model YOLOv_SMA1, the model YOLOv_SMA1_S uses SDI modules for the splicing network in the third sampling splicing module, the splicing network in the first convolution splicing module, the splicing network in the second convolution splicing module, and the splicing network in the third convolution splicing module, as shown in Figure 8(a).

[0050] The SDI module includes: a spatial attention unit, a channel attention unit, a sampling unit, a smooth convolution network (SmoothConv) and a channel stitching unit; the multiple data input to the SDI module are sequentially passed through the spatial attention unit and the channel attention unit to extract spatial attention features and channel attention features; the data output by the channel attention unit is processed by the sampling unit, the sampling unit downsamples the high-dimensional data, downsamples the low-dimensional data, and outputs the intermediate-dimensional data with the original dimension, so that the multiple data output by the sampling unit maintain the same dimension; the multiple data of the same dimension output by the sampling unit are convolved through the smooth convolution network, and then sent to the channel stitching unit for dimensional stitching as the output data of the SDI module.

[0051] In the SOBI network module which is the neck network, the feature information fusion layers of the first sampling splicing module and the second sampling splicing module have a small span. YOLOv_SMA1_S only uses SDI modules in the third sampling splicing module, the first convolution splicing module, the second convolution splicing module and the third convolution splicing module, which can improve the detection accuracy while maintaining a high detection speed.

[0052] The fifth composite plate detection model YOLOv_SMA1_D Compared with the model YOLOv_SMA1, the model YOLOv_SMA1_S sets the sampling networks in the first sampling splicing module and the second sampling splicing module to the Dysample network, as shown in Figure 8(b).

[0053] The Dysample network can dynamically adjust the upsampling target and area, so that more small target pixels such as reinforcement ribs in the composite plate image are sampled, which increases the proportion of small target pixels in the feature map and reduces the problem of loss of fine-grained details. There are three upsampling processes in the SOBI network module, namely the first sampling splicing module, the second sampling splicing module and the third sampling splicing module; the results of the first two upsamplings need to go through multiple feature fusion operations, but the third upsampling result is directly output to the detection head for detection after a fusion extraction operation, and the fusion is the shallowest small target information. If the third sampling splicing module uses the Dysample network, the information of small targets such as reinforcement ribs detected by the detection head may account for too much in the feature map, and the context information will be lost, making it difficult to integrate the information of large targets such as composite plates, which will reduce the detection accuracy. Therefore, in this embodiment, only the first two upsampling modules are replaced with the Dysample network.

[0054] The sixth composite plate detection model YOLOv_ASDS1 Compared with model YOLOv_SMA1, model YOLOv_ASDS1 uses SDI modules for the splicing network in the third sampling splicing module, the splicing network in the first convolution splicing module, the splicing network in the second convolution splicing module, and the splicing network in the third convolution splicing module; and sets the sampling networks in the first sampling splicing module and the second sampling splicing module to Dysample networks, as shown in Figure 8(c).

[0055] The seventh composite plate detection model YOLOv_SMA2_S Compared with the model YOLOv_SMA2, the model YOLOv_SMA2_S uses the SDI module for the splicing network in the third sampling splicing module, the splicing network in the first convolution splicing module, the splicing network in the second convolution splicing module, and the splicing network in the third convolution splicing module.

[0056] The eighth composite plate detection model YOLOv_SMA2_D Compared with the model YOLOv_SMA2, the model YOLOv_SMA2_S sets the sampling network in the first sampling splicing module and the second sampling splicing module to the Dysample network.

[0057] The ninth composite board detection model YOLOv_ASDS2 Compared with the model YOLOv_SMA2, the model YOLOv_ASDS2 uses SDI modules for the splicing network in the third sampling splicing module, the splicing network in the first convolution splicing module, the splicing network in the second convolution splicing module, and the splicing network in the third convolution splicing module; and sets the sampling networks in the first sampling splicing module and the second sampling splicing module to Dysample networks.

[0058] The structure of YOLOv_SMA2_S is similar to that of YOLOv_SMA1_S, the structure of YOLOv_SMA2_D is similar to that of YOLOv_SMA1_D, and the structure of YOLOv_ASDS2 is similar to that of YOLOv_ASDS1, so they will not be described in detail.

[0059] The above nine model structures are verified in combination with specific embodiments below.

[0060] The experimental environment of this embodiment uses the Pytorch supercomputing platform of the Linux system, the model training environment is Python-3.9.15 torch-2.1.0, and the NVIDIA A800-SXM4-80GB computing model is used. The hyperparameters are all set to the default settings, such as the initial learning rate is 0.01, the weight decay is 0.0005, etc.

[0061] The laminated board dataset in this embodiment contains 307 original images with a pixel size of 1024×768. The original images are exposed, darkened, noised, and sharpened to expand the dataset, and finally 1535 image samples are obtained to form the laminated board dataset. The object contours annotated in the laminated board dataset include: metal junction box (MJB), plastic junction box (PJB), reserved holes (RH), truss reinforcement (TB), hoisting reinforcement (RR) and laminated slab frame (FL). The detailed information of the laminated slab dataset is shown in Table 1. The laminated slab dataset is divided into training set, validation set and test set according to the ratio of 7:2:1.

[0062] Table 1: Dataset information ; In this embodiment, the model training steps are as follows: The first step is to extract training samples from the training set and let the model learn the training samples autonomously; The second step is to extract validation samples from the validation set and input them into the model. The loss function of the model is calculated on the validation samples. Specifically, the cross entropy loss can be used. The model parameters are updated by backpropagation according to the loss function. The third step is to calculate the model prediction accuracy on the test set to determine whether the model accuracy has converged in the last 50 times, or whether the model update times have reached 300 times; if yes, fix the model parameters; if not, return to the first step.

[0063] In this embodiment, multiple comparison models are also set, including: YOLOv5, YOLOv7, YOLOv8, YOLOv9, SOLOv2 and Cascade Mask R-CNN, YOLOv8_SDI and YOLOv8_Dy; YOLOv8_SDI: replace the three Concat networks in the deep layer of the YOLOv8 neck network with the SDI module; YOLOv8_Dy: Replace the 3 Upsample networks of the YOLOv8 neck network with Dysample modules.

[0064] In this embodiment, the above nine improved models YOLOv_AIEB, YOLOv_SMA1, YOLOv_SMA2, YOLOv_SMA1_S, YOLOv_SMA1_D, YOLOv_ASDS1, YOLOv_SMA2_S, YOLOv_SMA2_D, YOLOv_ASDS2 and 8 comparison models are all trained on the same composite plate data set using the same model training method, and the model performance is tested on the test set after the training is completed.

[0065] In this embodiment, the comparison results of the nine improved models with YOLOv8, YOLOv8_SDI and YOLOv8_Dy on the test set are shown in Table 2 and Fig. 9 shown.

[0066] Table 2: Test results of improved model ; In Table 2, the S-mAP index is the mean average accuracy (mAP) of the model on small-sized targets, the B-mAP index is the mean average accuracy (mAP) of the model on large-sized targets, and the mAP index is the average accuracy of the model on all targets. The targets are the annotated objects, including metal junction boxes (MJB), plastic junction boxes (PJB), reserved holes (RH), truss bars (TB), hoisting reinforcement bars (RR), and laminated plate frames (FL); small-sized targets refer to those with actual sizes less than or equal to 300 cm 2 Large-size objects refer to objects with actual sizes larger than 300 cm. 2 The annotation object.

[0067] From Table 2, Fig. 9 It can be seen that when the AIEB network module acts alone, the segmentation accuracy on small-sized targets is improved by 2.1%, and the calculation speed is greatly improved. The YOLOv_SMA1 and YOLOv_SMA2 models, which are the combination of the AIEB network module and the SOBI network module, also greatly improve the model performance.

[0068] from Fig. 9 It can be seen that the average segmentation accuracy mAP of the model YOLOv_SMA2 on all targets is higher than that of the model YOLOv_SMA1. According to the S-mAP and B-mAP evaluation indicators in Table 2, the small target segmentation accuracy of the model YOLOv_SMA2 is much higher than that of the model YOLOv_SMA1, which is consistent with the theoretical derivation of the small target information of the P3 layer of the YOLOv_SMA2 model neck network fusion backbone network.

[0069] From the comparison of models YOLOv_SMA1_S, YOLOv_SMA1_D, YOLOv_ASDS1, YOLOv_SMA2_S, YOLOv_SMA2_D, and YOLOv_ASDS2, after adding the SDI module and Dysample network, the average segmentation accuracy mAP of YOLOv_SMA1_S, YOLOv_SMA1_D, and YOLOv_ASDS1 on all targets are higher than those of models YOLOv_SMA2_S, YOLOv_SMA2_D, and YOLOv_ASDS2, respectively.

[0070] Specifically, the mAP indicators of models YOLOv_SMA1_S, YOLOv_SMA1_D, and YOLOv_ASDS1 continue to rise, while the mAP indicators of models YOLOv_SMA2_S, YOLOv_SMA2_D, and YOLOv_ASDS2 have no upward trend.

[0071] Comparing the S-mAP and B-mAP values ​​of the relevant models in Table 2, we analyze the reasons. The attention mechanism of the SDI module before fusing information increases the weight of small target information, and the Dysmapl network dynamically increases the pixel size of small targets. For the SMA1 structure, further improving the weight of small target information can still increase the overall accuracy, but the SMA2 structure has already fused the shallowest small target information in the neck network. Continuing to increase the proportion of small target information will result in less information in the large receptive field. Not only will the detection accuracy of larger targets be reduced, but a lot of contextual information around small targets will be lost, resulting in a decrease in the segmentation accuracy of small targets.

[0072] Therefore, the SMA1 structure (i.e., the input of the first convolutional network in the SOBI network module is connected to the C2f network layer of the P3 layer in the AIEB network module) perfectly matches the SDI and Dysample modules, and the final model YOLOv_ASDS1 has an accuracy that is 1.2% higher than that of YOLOv_ASDS2, and has a faster detection speed. Compared with the original model YOLOv8, the accuracy of the model YOLOv_ASDS1 has increased by 3.5%, and the detection speed is only 13FPS slower, still retaining 108.70FPS, which meets the real-time detection requirements.

[0073] In this embodiment, the accuracy (mAP), precision and recall rate of the meta-model YOLOv8 and the improved models YOLOv_SAM1_S, YOLOv_SAM1_D, and YOLOv_ASDS1 are further compared on each labeled object, and the results are shown in Figures 10(a), 10(b), and 10(c).

[0074] It can be seen that under the SAM1 structure, the various improved combinations have increased in various evaluation indicators compared to the original YOLOv8. In addition, when the SMA1 structure is added, because the multi-level auxiliary information extracts more small target information and its contextual information, the additional small target fusion circuit and detection head of the SOBI network module integrate and detect the small target information more completely from the extracted information. Therefore, the accuracy (mAP) of smaller targets such as lifting reinforcement bars (RR), plastic junction boxes (PJB), and reserved holes (RH) has increased, especially the model YOLOv_SAM1 has increased its precision and recall rate by 4.6% and 19.4% on lifting reinforcement bars (RR). But at the same time, the detection information is overly biased towards small targets, which also causes the detection accuracy of larger targets such as truss reinforcement bars (TB) and laminated plate frames (FL) to decrease. The SDI module of multi-scale feature fusion introduced by the model YOLOv_SAM1_D integrates local information and overall information, balancing the attention of small and large target information. Although the segmentation accuracy of small targets such as lifting reinforcement ribs (RR) is slightly reduced, the segmentation accuracy of large-scale targets is greatly improved, and the average accuracy of all targets is further increased. On this basis, YOLOv_ASDS1 introduces the dynamic upsampling Dysample network, dynamically adjusts the upsampling operation according to the needs of each scale, and evenly improves the segmentation accuracy of targets of each scale without increasing the amount of calculation through point sampling. The average mAP of the final model reaches 93.9%.

[0075] In this embodiment, the model YOLOv_ASDS1 is further compared with the comparison models YOLOv5, YOLOv7, YOLOv8, YOLOv9, SOLOv2 and Cascade Mask R-CNN, and the results are shown in Table 3.

[0076] Table 3: Model performance comparison ; As can be seen from Table 3, YOLOv8 has a higher average accuracy of each target than other systems, and the basic model has reached 90.4% mAP, and it is more lightweight. It is more reasonable for the present invention to select YOLOv8 as the baseline model. Although the overall accuracy of YOLOv9 is slightly lower than that of YOLOv8, YOLOv9 only performs poorly on the hoisting reinforcement ribs (RR), and performs better than YOLOv8 on other labeled objects. YOLOv_ASDS1 not only retains and surpasses the advantage of YOLOv8 in detecting hoisting reinforcement ribs (RR) with high accuracy, but also continues the advantage of YOLOv9 in detecting other targets with high accuracy. The improved model YOLOv_ASDS1 finally obtained has an accuracy of 93.9%, which is much higher than other models, and the FPS is only 20 lower than that of YOLOv8, far exceeding other comparison models, and maintains lightweight.

[0077] Of course, it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0078] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0079] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A method for detecting a pre-embedded structure of a composite plate, characterized in that: Firstly, a superimposed plate detection model is constructed to mark the superimposed plate contours and embedded structure contours on the superimposed plate image. The embedded structure includes embedded components and embedded holes. The superimposed plate detection model is trained on a superimposed plate dataset marked with the superimposed plate contours and embedded structure contours until the model converges. The superimposed plate image to be detected is photographed and input into the converged superimposed plate detection model, and the superimposed plate detection model outputs the embedded structure detection result.

2. The method for detecting the embedded structure of a composite plate according to claim 1, characterized in that: The overlapped plate detection model is improved on the basis of the YOLOv8 framework; the improvement method includes replacing the backbone network with the AIEB network module; the AIEB network module includes 29 layers of networks, namely, the dimension splitting layer of the P0 layer, the convolutional layers of the P1 layer, the P2 layer, the P4 layer, the P6 layer, the P8 layer, the P15 layer, the P17 layer, the P20 layer, the P23 layer and the P26 layer; the C2f network layers of the P3 layer, the P5 layer, the P7 layer, the P9 layer, the P19 layer, the P22 layer, the P25 layer and the P28 layer; the CBLinear network layers of the P10 layer, the P11 layer, the P12 layer, the P13 layer and the P14 layer; the CBFuse network layers of the P16 layer, the P18 layer, the P21 layer, the P24 layer and the P27 layer; The P0-P9 layers are connected sequentially, and the P15 to P28 layers are connected sequentially; the input of the P15 convolutional layer is connected to the output of the P0 dimension splitting layer; the C2f network layer of the P9 layer is connected to the CBFuse network layers of the P16, P18, P21, P24, and P27 layers through the CBLinear network layer of the P10 layer; the C2f network layer of the P7 layer is connected to the CBFuse network layers of the P16, P18, P21, and P24 layers through the CBLinear network layer of the P11 layer; the C2f network layer of the P5 layer is connected to the CBFuse network layers of the P12 layer The CBLinear network layer of the P3 layer is connected to the CBFuse network layers of the P16 layer, the P18 layer, and the P21 layer; the C2f network layer of the P3 layer is connected to the CBFuse network layers of the P16 layer and the P18 layer through the CBLinear network layer of the P13 layer; the convolutional layer of the P1 layer is connected to the CBFuse network layer of the P16 layer through the CBLinear network layer of the P14 layer; the output of the C2f network layer of the P22 layer, the output of the C2f network layer of the P25 layer, and the output of the C2f network layer of the P28 layer are connected to the neck network as the output of the backbone network.

3. The method for detecting the embedded structure of the composite plate according to claim 2, characterized in that: The improvement method of the YOLOv8 framework also includes: replacing the neck network with a SOBI network module; the SOBI network module includes a first convolutional network, a second convolutional network, a third convolutional network, a fourth convolutional network, and a first sampling splicing module, a second sampling splicing module, a third sampling splicing module, a first convolution splicing module, a second convolution splicing module, and a third convolution splicing module connected in sequence; The first sampling splicing module, the second sampling splicing module and the third sampling splicing module are collectively referred to as sampling splicing modules, which are used to sample the acquired data and then perform feature fusion; the first convolution splicing module, the second convolution splicing module and the third convolution splicing module are collectively referred to as convolution splicing modules, which are used to perform convolution processing on the acquired data and then perform feature fusion; The input end of the first convolutional network, the input end of the second convolutional network, the input end of the third convolutional network and the input end of the fourth convolutional network are used as the four input ends of the SOBI network module, which are used to connect the four C2f network layers of the backbone network from shallow to deep; the input of the first sampling splicing module is connected to the output of the fourth convolutional network and the output of the third convolutional network, the input of the second sampling splicing module is also connected to the output of the second convolutional network, and the input of the third sampling splicing module is also connected to the output of the first convolutional network; the input of the first convolutional splicing module is also connected to the output of the second convolutional network and the output of the second sampling splicing module, the input of the second convolutional splicing module is also connected to the output of the third convolutional network and the output of the first sampling splicing module, and the input of the third convolutional splicing module is also connected to the output of the fourth convolutional network; The output end of the third sampling splicing module, the output end of the first convolution splicing module, the output end of the second convolution splicing module and the output end of the third convolution splicing module serve as the four output ends of the SOBI network module, which are used to connect the corresponding segmentation heads of the head network.

4. The method for detecting the embedded structure of the composite plate according to claim 3, characterized in that: The sampling splicing module is composed of a sampling network, a splicing network and a C2f network connected in sequence; the convolution splicing module is composed of a convolution network, a splicing network and a C2f network connected in sequence; and in the sampling splicing module and the convolution splicing module, the back end of the splicing network is connected to a 3-layer C2f network; the input of the sampling network in the first sampling splicing module is connected to the output of the fourth convolution network; the input of the splicing network in the first sampling splicing module is also connected to the output of the third convolution network; the input of the splicing network in the second sampling splicing module is also connected to the output of the second convolution network, and the input of the splicing network in the third sampling splicing module is also connected to the output of the first convolution network; the input of the splicing network in the first convolution splicing module is also connected to the output of the second convolution network and the output of the second sampling splicing module, the input of the splicing network in the second convolution splicing module is also connected to the output of the third convolution network and the output of the first sampling splicing module, and the input of the splicing network in the third convolution splicing module is also connected to the output of the fourth convolution network.

5. The method for detecting the embedded structure of the composite plate according to claim 3, characterized in that: The input ends of the second convolutional network, the third convolutional network, and the fourth convolutional network are connected to the outputs of the last three C2f network layers in the backbone network in sequence, and the input end of the first convolutional network is connected to the output of the C2f network layer of the P2 layer of the backbone network or the output of the C2f network layer of the P19 layer.

6. The method for detecting the embedded structure of the composite plate according to claim 5, characterized in that: The splicing network in the third sampling splicing module, the splicing network in the first convolution splicing module, the splicing network in the second convolution splicing module and the splicing network in the third convolution splicing module all adopt the SDI module; the SDI module includes: a spatial attention unit, a channel attention unit, a sampling unit, a smoothed convolution network and a channel splicing unit; the multiple data input by the SDI module all pass through the spatial attention unit and the channel attention unit in sequence to extract the spatial attention features and the channel attention features; the data output by the channel attention unit is processed by the sampling unit so that the multiple data output by the sampling unit maintain the same dimension; the multiple data of the same dimension output by the sampling unit are convolved through the smoothed convolution network, and then sent to the channel splicing unit for dimensional splicing as the output data of the SDI module.

7. The method for detecting the embedded structure of a composite plate according to claim 6, characterized in that: The sampling networks in the first sampling splicing module and the second sampling splicing module are set as Dysample networks; in the neck network, the sampling network adopts Upsample network, and the splicing network adopts Concat network.

8. The method for detecting the embedded structure of a composite plate according to claim 1, characterized in that: The method for acquiring the superimposed slab dataset is as follows: first, take images of the superimposed slabs without concrete poured, and mark the outlines of the superimposed slabs and the outlines of the embedded structures; then, perform data enhancement on the captured superimposed slab images, and the data enhancement methods include exposure, darkening, noise, and sharpening processing; and count the captured superimposed slab images, the data-enhanced superimposed slab images, and the annotated outlines of all superimposed slab images to form a superimposed slab dataset.

9. A composite plate embedded structure detection system, characterized in that: It includes a memory and a processor, the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to implement the method for detecting the embedded structure of the composite plate as described in any one of claims 1-8.

10. A readable medium, characterized in that: A computer program is stored, and when the computer program is executed, it is used to implement the method for detecting the embedded structure of the composite plate as described in any one of claims 1-8.

Images