A prefabricated component detection and graph attention reasoning model construction method
Patent Information
- Application Number
- CN202410872274.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2044-07-01
AI Technical Summary
[0021](1)本发明提出的用于预制构件检测的图注意力推理模型构建方法，基于YOLOv8构建了自适应特征金字塔网络(AFPN，Adaptive Feature Pyramid Network)的预埋件检测模型，实现自适应特征融合；本发明还设计了OffsetDC(Offset Dynamic Convolution，偏移动态卷积)模块，从而关注了空域、输入通道、输出通道等维度上的动态性。同时在OffsetDC模块中引入偏移参数，自适应的调整卷积核以适应目标边界的变化，能够更精细地适应输入数据的特征，并减少边界效应和目标位置的偏差，帮助网络更好建模目标的位置偏移，从而缓解拍摄所产生的位置偏差，提高特征提取的效果，提高预制构件上预埋件检测的精确度。
Smart Images
Figure CN118865052B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of prefabricated component detection, image recognition, and big data modeling, and in particular to a method for prefabricated component detection and graph attention reasoning model construction. Background Technology
[0002] Precast components (or precast parts) refer to steel, wood, or concrete components prefabricated in a factory or on-site according to design specifications. The application of precast components makes construction more efficient, safe, and convenient. The quality of precast components directly affects subsequent assembly and installation; therefore, the inspection of precast components is very rigorous. Currently, a combination of manual and machine inspection methods is mainly used.
[0003] The location and dimensions of embedded parts are the main evaluation objects of precast components. The inspection of embedded parts is very important in the production of precast components. The automation and intelligence of embedded part inspection can effectively ensure the quality of precast components leaving the factory. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies that lack high-precision intelligent detection techniques for embedded parts, this invention proposes a graph attention inference model construction method for prefabricated component detection. By combining graph attention mechanism (GAT) and offset dynamic convolution (OffsetDC module), it achieves fine attention to local features of the image, greatly improving the detection accuracy of embedded parts.
[0005] This invention proposes a graph attention reasoning model construction method for prefabricated component detection. First, a basic model and labeled samples are constructed. The labeled samples are components labeled with embedded part detection information, including the location and category of the embedded parts. Then, the basic model is subjected to machine learning on the labeled samples to obtain the converged basic model as the embedded part detection model.
[0006] The base model improves upon the YOLOv8 model by adding a neck network. The neck network of the base model consists of three branches: the first branch, the second branch, and the third branch. The first branch comprises sequentially connected sixth convolutional network, first GAT network, fifth C2f module, second GAT network, and sixth C2f module. The second branch comprises sequentially connected seventh convolutional network, third GAT network, seventh C2f module, fourth GAT network, and eighth C2f module. The third branch comprises sequentially connected eighth convolutional network, fifth GAT network, and ninth C2f module.
[0007] The output of the sixth convolutional network is also connected to the input of the third GAT network, and the output of the seventh convolutional network is also connected to the input of the first GAT network; the output of the seventh C2f module is also connected to the input of the second GAT network and the input of the fifth GAT network respectively, the output of the fifth C2f module is also connected to the input of the fourth GAT network and the input of the fifth GAT network respectively, and the output of the eighth convolutional network is also connected to the input of the second GAT network and the input of the fourth GAT network respectively.
[0008] In the sixth, seventh, and eighth convolutional networks, at least one uses an OffsetDC module. The OffsetDC module includes: a GAP network, a first FC network, a first activation layer, a dimension stacking network, and four attention branches. The GAP network, the first FC network, and the first activation layer are connected sequentially, and the input of the GAP network is the input of the ODC module.
[0009] The attention branch consists of connected activation layers and fully connected (FC) networks. The input of the activation layer serves as the input of the attention branch, and the output of the FC network serves as the output of the attention branch. The input of each attention branch is connected to the output of the first activation layer.
[0010] Each attention branch outputs feature data of dimension N; the feature data output by the four attention branches are multiplied by the set weights and offset vectors of each dimension, and then the input dimension stacking network is used to stack the data on the N data dimensions before outputting; the feature map output by the OffsetDC module is the result of the dot product between the input feature map of the OffsetDC module and the output data of the dimension stacking network.
[0011] Preferably, any two of the sixth, seventh, and eighth convolutional networks employ the OffsetDC module.
[0012] Preferably, both the seventh and eighth convolutional networks use the OffsetDC module.
[0013] Preferably, in the sixth, seventh, and eighth convolutional networks, the convolutional networks other than the OffsetDC module all use the CBS network module or the Conv convolutional network.
[0014] Preferably, the backbone network of the basic model includes an input layer, a first convolutional network, a second convolutional network, a first C2f module, a third convolutional network, a second C2f module, a fourth convolutional network, a third C2f module, a fifth convolutional network, a fourth C2f module, and an SPPF module connected in sequence; the input of the sixth convolutional network is connected to the output of the second C2f module, the input of the seventh convolutional network is connected to the output of the third C2f module, and the input of the eighth convolutional network is connected to the output of the SPPF module.
[0015] Preferably, the first, second, third, fourth, and fifth convolutional networks all employ CBS network modules or Conv convolutional networks.
[0016] Preferably, the first activation layer uses the ReLU activation function.
[0017] Preferably, the activation layer in the attention branch uses the Softmax activation function or the Sigmoid activation function.
[0018] The present invention proposes a precast component detection method using the graph attention inference model construction method for precast component detection, characterized in that: firstly, the graph attention inference model construction method for precast component detection is used to obtain an embedded component detection model; then, the precast component to be detected is input into the embedded component detection model, and the embedded component detection model outputs embedded component detection information.
[0019] The present invention proposes a precast component inspection system, which includes a memory and a processor. The memory stores a computer program and an embedded part inspection model. The processor is connected to the memory and is used to execute the computer program to realize the precast component inspection method.
[0020] The advantages of this invention are:
[0021] (1) The graph attention inference model construction method for prefabricated component detection proposed in this invention constructs an embedded component detection model based on YOLOv8 using an Adaptive Feature Pyramid Network (AFPN) to achieve adaptive feature fusion. This invention also designs an OffsetDC (Offset Dynamic Convolution) module to focus on the dynamics in dimensions such as spatial domain, input channel, and output channel. At the same time, an offset parameter is introduced into the OffsetDC module to adaptively adjust the convolution kernel to adapt to changes in the target boundary. This allows for more precise adaptation to the features of the input data and reduces boundary effects and target position deviations. It helps the network better model the target position offset, thereby mitigating position deviations caused by shooting, improving the effect of feature extraction, and increasing the accuracy of embedded component detection on prefabricated components.
[0022] (2) The OffsetDC module adopts a multidimensional dynamic attention module, which learns complementary attention of the convolution kernel in all four dimensions of the kernel space through a parallel strategy. The four different attentions are generated through four branches and normalized by the activation function. This is beneficial to improve attention to the unique shape and features of the local view.
[0023] (3) This embedded part detection model can dynamically generate feature pyramids of different scales to adapt to the detection of targets of different sizes; by learning weight parameters, it can determine the importance and effectiveness of features at different scales, thereby deciding how to perform feature fusion. This adaptive feature fusion method ensures that features at each scale can help the current model detect embedded parts of different composite plates, thereby increasing attention to the feature details of embedded parts such as metal junction boxes, plastic junction boxes and embedded holes, and improving the accuracy of embedded part detection.
[0024] (4) This invention introduces GAT (Graph Attention Networks), which combines attention mechanism and graph neural network to extract pixel-level depth information between components.
[0025] (5) This invention combines an adaptive feature pyramid network and a graph attention network to capture information in the graph structure by dynamically learning the importance weights between nodes, which has greater flexibility and adaptability.
[0026] (6) The present invention also proposes a method for detecting precast components, which detects embedded parts on precast components based on the embedded part detection model proposed in the present invention, thereby achieving a lighter model structure and higher detection efficiency and accuracy. Attached Figure Description
[0027] Figure 1 This is a structural diagram of the YOLOv8 model;
[0028] Figure 2 This is a diagram of the head network structure;
[0029] Figure 3 This is a structural diagram of the YOLOv8n model;
[0030] Figure 4 This is a structural diagram of the YOLOv8n_2_3 model;
[0031] Figure 5 Here is a structural diagram of the OffsetDC module;
[0032] Figure 6 Flowchart of a graph attention inference model construction method for prefabricated component inspection;
[0033] Figure 7 The confusion matrix of the YOLOv8n_2 model;
[0034] Figure 8 This is the confusion matrix of the YOLOv8n_3 model. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] The existing YOLOv8 model has the following structure: Figure 1 , Figure 2 As shown.
[0037] The YOLOv8 model consists of a backbone network, a neck network, and a head network.
[0038] The backbone network consists of a first convolutional network, a second convolutional network, a first C2F module, a third convolutional network, a second C2F module, a fourth convolutional network, a third C2F module, a fifth convolutional network, a fourth C2F module, and an SPPF module, which are connected in sequence.
[0039] The neck network comprises, in sequence: a first sampling layer, a first stitching module, a second sampling layer, a second stitching module, a sixth convolutional network, a third stitching module, a seventh convolutional network, and a fourth stitching module;
[0040] The head network comprises three output modules, whose inputs are connected to the outputs of the second, third, and fourth stitching modules, respectively. Each output module predicts the embedded part's bounding box and category based on the output of its corresponding stitching module. Specifically, each output module includes two parallel annotation branches, each consisting of a connected Conv convolutional network and a Conv2d network. The input of the Conv convolutional network is connected to the input of the annotation branch, and the output of the Conv2d network is the output of the annotation branch. Within the output module, one annotation branch generates the embedded part's bounding box, and the other generates the embedded part's category.
[0041] The output of the SPPF module is the output of the backbone network and is connected to the input of the neck network. The inputs of the first sampling layer and the fourth stitching module are both connected to the inputs of the neck network.
[0042] The first, second, third, and fourth splicing modules have the same structure and are collectively referred to as splicing modules. Each splicing module consists of a connected Concat splicing network and a C2f module. The input of the Concat splicing network is the input of the splicing module, and the output of the Concat splicing network is connected to the input of the C2f module. The output of the C2f module is the output of the splicing module.
[0043] The input of the first splicing module is also connected to the output of the third C2F module, the input of the second splicing module is also connected to the output of the second C2F module, the input of the third splicing module is also connected to the output of the first splicing module, and the input of the fourth splicing module is also connected to the output of the SPPF module.
[0044] This invention also proposes a pre-embedded component detection model for comparative verification, denoted as YOLOv8n, whose structure is as follows: Figure 3 As shown.
[0045] Compared to the YOLOv8 model, YOLOv8n replaces the neck network. The neck network of YOLOv8n consists of three branches: the first branch, the second branch, and the third branch. The first branch consists of the sixth convolutional network, the first GAT network, the fifth C2f module, the second GAT network, and the sixth C2f module, connected in sequence. The second branch consists of the seventh convolutional network, the third GAT network, the seventh C2f module, the fourth GAT network, and the eighth C2f module, connected in sequence. The third branch consists of the eighth convolutional network, the fifth GAT network, and the ninth C2f module, connected in sequence.
[0046] The output of the sixth convolutional network is also connected to the input of the third GAT network, and the output of the seventh convolutional network is also connected to the input of the first GAT network; the output of the seventh C2f module is also connected to the input of the second GAT network and the input of the fifth GAT network, respectively; the output of the fifth C2f module is also connected to the input of the fourth GAT network and the input of the fifth GAT network, respectively; and the output of the eighth convolutional network is also connected to the input of the second GAT network and the input of the fourth GAT network, respectively.
[0047] In the YOLOv8 model, the input of the first branch is connected to the output of the second C2F module, the input of the second branch is connected to the output of the third C2F module, and the input of the third branch is connected to the output of the SPPF module. The output of the first branch is connected to the input of the first output module of the head network, the output of the second branch is connected to the input of the second output module of the head network, and the output of the third branch is connected to the input of the third output module of the head network.
[0048] The following is a list of the embedded part detection models proposed in this invention.
[0049] Let the position of the sixth convolutional network be denoted as position 1, the position of the seventh convolutional network as position 2, and the position of the eighth convolutional network as position 3. Let YOLOv8n_i represent replacing the convolutional network at position i with the OffsetDC module in YOLOv8n; let YOLOv8n_i_j represent replacing the convolutional networks at positions i and j with the OffsetDC module in YOLOv8n. i∈{1,2,3}, j∈{1,2,3}.
[0050] The embedded component detection model proposed in this invention includes: YOLOv8n_1, YOLOv8n_2, YOLOv8n_3, YOLOv8n_1_2, YOLOv8n_1_3, and YOLOv8n_2_3. YOLOv8n_2_3 is as follows... Figure 4 As shown.
[0051] Reference Figure 5 The OffsetDC module includes: a GAP network (global average pooling layer), a first FC network (fully connected layer), a first activation layer, a dimension stacking network, and four attention branches; the GAP network, the first FC network, and the first activation layer are connected sequentially, and the input of the GAP network is the input of the ODC module;
[0052] The attention branch consists of connected activation layers and fully connected (FC) networks. The input of the activation layer serves as the input of the attention branch, and the output of the FC network serves as the output of the attention branch. The input of each attention branch is connected to the output of the first activation layer.
[0053] Each attention branch outputs feature data of dimension N; the feature data output by the four attention branches are multiplied by the set weights and offset vectors of each dimension, and then the input dimension stacking network is used to stack the data on the N data dimensions before outputting; the feature map output by the OffsetDC module is the result of the dot product between the input feature map of the OffsetDC module and the output data of the dimension stacking network.
[0054] That is, let the feature data output by the i-th attention branch be denoted as (a_i1, a_i2, ..., a_in, ..., a_iN), where a_in represents the data in the n-th dimension of the feature data output by the i-th attention branch, and N is the number of dimensions of the feature data output by the attention branches. In this embodiment, N can be set to 3. After the feature data output by the four attention branches are multiplied by the set weights and offset vectors of each dimension, the input dimension superposition network superimposes the data in each of the N data dimensions and outputs the feature map y = x * add(a_in), where x represents the input feature map of the ODC module and add(a_in) represents the output data of the dimension superposition network.
[0055] In the following embodiments, existing models YOLOv5, YOLOv7, YOLOv8, and YOLOv9 are used as comparison models. Models YOLOv8n, YOLOv8n_1, YOLOv8n_2, YOLOv8n_3, YOLOv8n_1_2, YOLOv8n_1_3, YOLOv8n_2_3 and the comparison models are trained and validated on the same labeled dataset.
[0056] In this embodiment, in models YOLOv8 and YOLOv8n, the first, second, third, fourth, fifth, sixth, seventh, and eighth convolutional networks all employ CBS network modules. YOLOv8n_1, YOLOv8n_2, YOLOv8n_3, YOLOv8n_1_2, YOLOv8n_1_3, and YOLOv8n_2_3 are based on YOLOv8n, but replace the convolutional networks at corresponding positions with OffsetDC modules.
[0057] In this embodiment, the OffsetDC module uses the ReLU activation function for the first activation layer and the Softmax activation function for the activation layer in the attention branch.
[0058] Reference Figure 6 During model training, labeled samples are first obtained. The labeled samples are images of prefabricated parts with embedded part detection information. The embedded part detection information includes the location and category of the embedded part. The location of the embedded part is highlighted by the label box. The labeled samples are divided into training set, validation set and test set in a ratio of 7:2:1. Each comparison model and models YOLOv8n, YOLOv8n_1, YOLOv8n_2, YOLOv8n_3, YOLOv8n_1_2, YOLOv8n_1_3 and YOLOv8n_2_3 are trained on the training set and validation set until convergence. Then the model performance is tested on the test set.
[0059] In this embodiment, images of the precast composite slab captured by a high-definition camera are used as the original annotation samples, which are manually annotated. Considering the influence of lighting conditions at the component fabrication site, the intensity and angle of light significantly affect the accuracy of the detection process. Therefore, to simulate images of the precast composite slab under different lighting conditions, data augmentation processing, including inversion, exposure adjustment, darkening, and overnoise, is performed on the original image samples to generate more annotation samples. Based on the composite slab's structural composition, the embedded parts are categorized into five types: metal junction boxes, plastic junction boxes, reserved holes, truss reinforcement, and hoisting reinforcement. The annotation samples are divided into training, validation, and test sets in a 7:2:1 ratio. The embedded part data for each dataset is shown in Table 1 below.
[0060] Table 1: Statistics on Dataset Types
[0061]
[0062] The experiments were conducted on a Linux operating system. The hardware environment consisted of two NVIDIA Ampere A100 GPUs (microprocessors) with 100GB of memory each, used to train and test the YOLOv8n model and various comparison models. For the software environment, PyTorch version 1.11.0 was chosen as the deep learning library, and Python version 3.8.0 was used as the programming language to develop the proposed method. During the training phase, the batch size and training epochs were set to 32 and 300, respectively. Furthermore, stochastic gradient descent (SGD) was selected as the optimizer, with a momentum parameter of 0.937 and a weight decay of 0.0005; to ensure a balance between computational speed and accuracy, a learning rate of 0.001 was used.
[0063] The steps for training the model on the training and validation sets are as follows:
[0064] S1. Select multiple training samples from the training set and let the model learn from the training samples to update the model parameters;
[0065] S2. Select multiple validation samples from the validation set, and let the model infer the embedded part detection information of the validation samples; calculate the model loss on the validation samples;
[0066] S3. Determine if the model has converged; if not, update the model parameters in reverse according to the model's loss, and then return to step S1; if yes, fix the model and complete the model training.
[0067] The test results of models YOLOv8n, YOLOv8n_1, YOLOv8n_2, YOLOv8n_3, YOLOv8n_1_2, YOLOv8n_1_3 and YOLOv8n_2_3 on the test set are shown in Table 2 below.
[0068] Table 2: Model Testing
[0069]
[0070] Table 3 shows the performance comparison between model YOLOv8n_2_3 and other comparative models.
[0071] Table 3: Performance Comparison of YOLOv8n_2_3 with Various Comparative Models
[0072]
[0073] As shown in Tables 2 and 3, model YOLOv8n_1 has the highest accuracy, exceeding all comparison models; model YOLOv8n_2_3 has the highest average accuracy, also exceeding all comparison models. Given the diverse types of embedded parts, average accuracy is the most critical indicator in practical implementation. Therefore, the model YOLOv8n_2_3 proposed in this invention is best suited for identifying embedded parts in precast composite slabs.
[0074] In this embodiment, the recognition accuracy of models YOLOv8n_2 and YOLOv8n_3 in each category is further tested, and represented by a confusion matrix as follows: Figure 7 , Figure 8 As shown, adding the OffsetDC module at positions 2 and 3 resulted in the highest accuracy of the network in identifying truss reinforcement, reaching 1.0. The accuracy for plastic junction boxes and metal junction boxes was also relatively high, at 0.94 and 0.91 respectively, demonstrating good robustness and accuracy of the model.
[0075] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0076] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0077] The technologies, shapes, and structures not described in detail in this invention are all known technologies.
Claims
1. A method for constructing a graph attention reasoning model for prefabricated component detection, characterized in that, First, a basic model and labeled samples are constructed. The labeled samples are components labeled with embedded part detection information, including the location and category of the embedded parts. Then, the basic model is run on the labeled samples to perform machine learning, and the converged basic model is used as the embedded part detection model. The base model improves upon the YOLOv8 model by adding a neck network. The neck network of the base model consists of three branches: the first branch, the second branch, and the third branch. The first branch comprises sequentially connected sixth convolutional network, first GAT network, fifth C2f module, second GAT network, and sixth C2f module. The second branch comprises sequentially connected seventh convolutional network, third GAT network, seventh C2f module, fourth GAT network, and eighth C2f module. The third branch comprises sequentially connected eighth convolutional network, fifth GAT network, and ninth C2f module. The output of the sixth convolutional network is also connected to the input of the third GAT network, and the output of the seventh convolutional network is also connected to the input of the first GAT network; the output of the seventh C2f module is also connected to the input of the second GAT network and the input of the fifth GAT network respectively, the output of the fifth C2f module is also connected to the input of the fourth GAT network and the input of the fifth GAT network respectively, and the output of the eighth convolutional network is also connected to the input of the second GAT network and the input of the fourth GAT network respectively. In the sixth, seventh, and eighth convolutional networks, at least one uses an OffsetDC module. The OffsetDC module includes: a GAP network, a first FC network, a first activation layer, a dimension stacking network, and four attention branches. The GAP network, the first FC network, and the first activation layer are connected sequentially, and the input of the GAP network is the input of the ODC module. The attention branch consists of connected activation layers and fully connected (FC) networks. The input of the activation layer serves as the input of the attention branch, and the output of the FC network serves as the output of the attention branch. The input of each attention branch is connected to the output of the first activation layer. Each attention branch outputs feature data of dimension N; the feature data output by the four attention branches are multiplied by the set weights and offset vectors of each dimension, and then the input dimension stacking network is used to stack the data on the N data dimensions before outputting; the feature map output by the OffsetDC module is the result of the dot product between the input feature map of the OffsetDC module and the output data of the dimension stacking network.
2. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, Any two of the sixth, seventh, and eighth convolutional networks use the OffsetDC module.
3. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, Both the seventh and eighth convolutional networks use the OffsetDC module.
4. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, In the sixth, seventh, and eighth convolutional networks, all convolutional networks other than the OffsetDC module use the CBS network module or the Conv convolutional network.
5. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, The backbone network of the basic model consists of a sequentially connected input layer, a first convolutional network, a second convolutional network, a first C2f module, a third convolutional network, a second C2f module, a fourth convolutional network, a third C2f module, a fifth convolutional network, a fourth C2f module, and an SPPF module; the input of the sixth convolutional network is connected to the output of the second C2f module, the input of the seventh convolutional network is connected to the output of the third C2f module, and the input of the eighth convolutional network is connected to the output of the SPPF module.
6. The graph attention reasoning model construction method for prefabricated component detection as described in claim 5, characterized in that, The first, second, third, fourth, and fifth convolutional networks all use CBS network modules or Conv convolutional networks.
7. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, The first activation layer uses the ReLU activation function.
8. The graph attention reasoning model construction method for prefabricated component detection as described in claim 1, characterized in that, The activation layers in the attention branch use either the Softmax activation function or the Sigmoid activation function.
9. A method for detecting precast components using the graph attention reasoning model construction method for precast component detection as described in any one of claims 1-8, characterized in that, First, the embedded part detection model is obtained using the graph attention reasoning model construction method for prefabricated component detection as described in any one of claims 1-8; then, the prefabricated component to be detected is input into the embedded part detection model, and the embedded part detection model outputs embedded part detection information.
10. A precast component inspection system, characterized in that, It includes a memory and a processor. The memory stores a computer program and an embedded part detection model. The processor is connected to the memory and is used to execute the computer program to implement the prefabricated component detection method as described in claim 9.