A multi-path aggregated construction waste detection method

Abstract

This invention discloses a multi-path aggregation method for detecting construction waste, comprising the following steps: Step 1: Acquiring and preprocessing sample images of construction waste using a visual sensor to create labels; Step 2: Constructing multi-scale color fusion features to fuse shallow detail information and high-level semantic information of construction waste images; Step 3: Extracting deep texture features of digital images based on local binary patterns to mine deep texture information of construction waste images; Step 4: Determining the category and location information of region proposal boxes through an RPN network, and extracting the feature information of the target and feeding it into an R-CNN sub-network; Step 5: Designing a multi-branch neural network fusion strategy to output the final category confidence of the target; Step 6: Designing a multi-task weighted loss function to achieve end-to-end training of the model; Step 7: Achieving accurate detection of various categories of construction waste.

Description

Technical Field

[0001] This invention relates to the field of construction waste detection technology, and in particular to a multi-path aggregation method for detecting construction waste. Background Technology

[0002] Construction waste sorting is a key step in the recycling of construction waste resources, and its sorting rate and accuracy significantly impact the quality of recycled materials. Traditional construction waste sorting relies primarily on manual labor, which is inefficient, costly, and exposes workers to harsh working conditions that harm their health. With the development of industrial automation technology, some scholars have proposed using methods such as crushing, screening, magnetic separation, and air separation to sort construction waste. However, these methods suffer from low sorting accuracy, hindering the utilization of recycled materials.

[0003] With the development of hardware, machine vision technology, as a non-destructive testing tool, has been successfully applied in agriculture, medicine, and industry. With increasing awareness of sustainable development, some scholars have extracted features such as color, texture, and density from construction waste to achieve the identification of single types or specific pairs of construction waste. From an industrial perspective, due to the centralized stacking of various materials for construction waste in my country, the application scenarios of the above methods are limited. From an information perspective, manually extracting single features from images easily leads to the loss of some image information, affecting the accuracy of construction waste detection. With the development of deep learning, convolutional neural networks have been widely used in image segmentation, object detection, and other fields by extracting latent features from images. Currently, object detection algorithms based on convolutional neural networks can be divided into two-stage object detection frameworks, mainly R-CNN, Fast R-CNN, and Faster R-CNN; and one-stage object detection frameworks, represented by YOLO. Two-stage detection algorithms, based on the extraction of candidate detection regions, perform object detection through classification and regression. Compared with one-stage object detection algorithms, two-stage object detection algorithms have higher detection accuracy. Although object detection algorithms based on convolutional neural networks are mature, research on construction waste detection algorithms applicable to industrial environments is currently limited due to significant differences in color, shape, and even occlusion and contamination in the waste generated from the use and demolition of the same type of building material. Against this backdrop, there is an urgent need to research accurate detection methods for various types of building materials. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a multi-path aggregation method for detecting construction waste, so as to solve the problem of low accuracy in automated sorting of construction waste, which affects the utilization of recycled construction waste materials, and to achieve accurate detection of various building materials.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for detecting construction waste using multi-path aggregation includes the following steps;

[0007] Step 1: Collect and preprocess images of five types of construction waste samples—concrete, red brick, foam, hard plastic, and wood blocks—using a visual sensor, and create labels;

[0008] Step 2: Based on the Faster RCNN network, a multi-scale feature fusion network is proposed to construct a color multi-scale fusion feature, which is used to fuse shallow detail information and high-level semantic information of construction waste images. This feature is denoted as the color multi-scale fusion feature branch, which is used to obtain the candidate region extraction layer in Step 4 and to provide features for the multi-branch neural network fusion strategy in Step 5.

[0009] Step 3: Based on the local binary pattern of the digital image, extract the deep texture features of the digital image, mine the deep texture information of the construction waste image, and use it as a deep texture feature branch to provide features for the multi-branch neural network fusion strategy in Step 5.

[0010] Step 4: Based on the color multi-scale fusion features in Step 2, determine the category and location information of the region proposal box through the RPN network, and extract the feature information of the target and feed it into the R-CNN sub-network;

[0011] Step 5: Based on the target category confidence score output by the R-CNN sub-network in Step 4, and combining the color multi-scale fusion features in Step 2 and the deep texture features in Step 3, design a multi-branch neural network fusion strategy to output the final category confidence score of the target.

[0012] Step 6: Design a multi-task weighted loss function to achieve end-to-end model training;

[0013] Step 7: Improve the sorting accuracy of construction waste by training the model and achieve accurate detection of various types of construction waste.

[0014] Furthermore, the image preprocessing method in step one is as follows:

[0015] Step 1.1: Use LabelImg software to create labels for the acquired images;

[0016] Step 1.2: Expand the image set of construction waste through rotation, flipping, and color gamut transformation operations.

[0017] Furthermore, in step two, a multi-scale feature fusion network is proposed based on the Faster R-CNN network to construct multi-scale color fusion features. The method is as follows:

[0018] Step 2.1: Extract feature maps from the third, fourth, and fifth layers of the VGG network; shallow feature maps are used to describe the geometric features of construction waste, while deep feature maps are used to describe the high-level semantic information of construction waste.

[0019] Step 2.2: The shallow feature map is fused with the deep feature map through downsampling and convolution operations to form a candidate region extraction layer. The number of channels in the candidate region extraction layer is the same as that in the traditional Faster RCNN model. Therefore, this module can fuse the shallow detail information and high-level semantic information of the construction waste image with almost no increase in algorithm complexity, so as to achieve mutual complementarity of image information and help improve the detection accuracy of construction waste.

[0020] Furthermore, the method for extracting deep texture features of the target based on the local binary pattern of the digital image in step three is as follows:

[0021] Step 3.1, extract digital image texture features: First, convert the digital image acquired in Step 1 to grayscale. Then, taking each pixel in the image as the center pixel, calculate and encode the difference between the neighboring pixels and the center pixel in the 3×3 neighborhood of the center pixel using the following formula to obtain the texture features of the target.

[0022]

[0023]

[0024] In the formula, Let be the number of neighboring pixels. The center pixel value, The pixel values ​​are those of the center point's neighboring pixels;

[0025] Step 3.2: The texture feature map of the digital image extracted in Step 3.1 is fed into the first two convolutional layers of the VGG16 network to further extract the deep texture features of the image, supplement the local detail information of the construction waste image, and be used as the input to the multi-branch neural network fusion strategy model in Step 5.

[0026] Furthermore, in step four, based on the color multi-scale fusion features from step two, the method for determining the category and location information of the region proposal box through the RPN network, extracting the deep color and texture information of the target, and feeding it into the R-CNN network is as follows:

[0027] Step 4.1: Use the feature map obtained by the color multi-scale fusion feature branch as the candidate region extraction layer, and establish the RPN network to determine the category and location information of the region proposal box;

[0028] Step 4.2: Based on the category and location information of the region proposal box, map the proposal box to the color multi-scale fusion feature branch and the deep texture feature branch to obtain the color features and deep texture features of the target.

[0029] Step 4.3: Using the color features of the target from Step 4.2, obtain the target's category confidence and precise location information through a fully connected layer. Based on the target's deep texture features from Step 4.2, obtain the target's category confidence again through a fully connected layer.

[0030] Furthermore, in step five, based on the target category confidence score output by the R-CNN network in step four, the method for the multi-branch neural network fusion strategy is as follows:

[0031] Step 5.1: Set discount fusion weights based on the target detection accuracy of each convolutional neural network, as shown in the formula.

[0032]

[0033] In the formula, M represents a specific event in the event set {red brick, concrete, wood block, hard plastic, foam, θ}; {θ} represents the combination of events: red brick, concrete, wood block, hard plastic, and foam, which represents the degree of imprecision of the network. and These represent the construction waste detection accuracy of the color and texture convolutional neural network branches, respectively. and These represent the probabilities that the color and texture convolutional neural network branches determine the bounding box belongs to class M, respectively.

[0034] Step 5.2: Based on the class confidence scores discounted from the color and texture branches, design a multi-branch neural network fusion strategy, determine the final class confidence score of DFM-Net, and obtain the target class, as shown in the formula.

[0035]

[0036] In the formula, M and N represent a specific event in the event set {red brick, concrete, wood block, hard plastic, foam, θ}; {θ} represents the combination of events: red brick, concrete, wood block, hard plastic, and foam, which represents the degree of imprecision of the network. P represents the probability that the bounding box belongs to class M based on the color multi-scale fusion feature branch. L (N) represents the probability that the deep texture feature branch determines that the bounding box belongs to class N; P(A) represents the probability that the DFM-Net model determines that the bounding box belongs to class A.

[0037] Furthermore, the method for designing a multi-task weighted loss function in step six to achieve end-to-end model training is as follows:

[0038] The loss function of the DFM-Net model is as follows, which is the region proposal network bounding box regression loss L. loc·N And classification loss L cls·N Color multi-scale convolutional neural network region of interest bounding box regression loss L loc·I And classification loss L cls·I And texture convolutional neural network region of interest classification loss L cls·L Weighted composition;

[0039] L = L loc·N +L loc·I +L cls·N +L cls·I +L cls·L

[0040]

[0041]

[0042] In the formula, y i The label of border i represents the true category label; when the border is foreground, y represents the true category label. i =1, y = 1 when the border is the background i =0; t i Represents the coordinate vector of the predicted bounding box; K represents the true coordinate vector of the border; reg Indicates the number of borders; K cls P represents the total number of samples in a batch; ic y represents the probability that the i-th bounding box is predicted to be of class c; ic It is the actual label of the border.

[0043] The beneficial effects of this invention are:

[0044] This invention utilizes machine vision technology to accurately classify various types of construction waste materials, providing a solution for diverse market demands. By constructing an efficient deep learning network model, it extracts multiple deep features of construction waste materials. Finally, through intelligent extraction and decision fusion of material features, it improves the detection accuracy of construction waste materials, realizes automated and intelligent sorting of construction materials, and reduces the cost of recycling construction waste solid waste. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the construction waste dataset of the present invention.

[0046] Figure 2 This is the DFM-Net network model of the present invention.

[0047] Figure 3 This is a diagram of the multi-scale feature fusion network structure of the present invention.

[0048] Figure 4 This is a comparison chart of the accuracy and recall of the multi-model approach in this invention.

[0049] Figure 5 This is a comparison chart of the multi-model detection results of the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings.

[0051] A multi-path aggregation method for construction waste detection involves first constructing a construction waste detection device, which comprises a transmission device, a camera device, and a data processing center. Then, a construction waste detection model is embedded into the data processing device. Finally, the camera device captures construction waste on the conveyor belt and transmits it to the data processing center. The data processing center determines the category and location of the construction waste in the image based on the construction waste detection model. The construction of the construction waste detection model mainly includes the following steps:

[0052] Step 1: Acquire and preprocess images of five types of construction waste samples—concrete, red brick, foam, hard plastic, and wood blocks—using a visual sensor, and create labels. The ratio of training set:test set:validation set is 8:1:1. Figure 1 As shown.

[0053] Step 2: Based on the Faster RCNN network, a multi-scale feature fusion network is proposed to construct multi-scale color fusion features.

[0054] Step 2.1, combined Figure 3 We extract feature maps from the third, fourth, and fifth layers of the VGG network and perform downsampling and convolution operations on the shallow features.

[0055] Step 2.2: Fuse the feature maps of each layer to form a candidate region extraction layer, so that the feature map can retain the shallow details and high-level semantic information of the image, and realize the mutual supplementation of image information.

[0056] Step 3: Extract deep texture features of the target based on the local binary pattern of the digital image.

[0057] Step 3.1: First, convert the digital image to grayscale. Then, using each pixel in the image as the center pixel, calculate and encode the difference between the neighboring pixels and the center pixel within a 3×3 neighborhood of the center pixel using the following formula to obtain the texture features of the target.

[0058]

[0059] In the formula, The present invention sets the number of neighboring pixels to be [number of pixels]. The center pixel value, The pixel value is the value of the neighborhood of the center point.

[0060] Step 3.2: Input the texture feature map of the digital image extracted in Step 3.1 into the first two convolutional layers of the VGG16 network to further extract the deep texture features of the image.

[0061] Step 4: Based on the color multi-scale fusion features in Step 2, determine the category and location information of the region proposal box through the RPN network, and send the feature information of the target into the R-CNN sub-network.

[0062] Step 4.1: Use the feature map obtained by the color multi-scale fusion feature branch as the candidate region extraction layer to establish the RPN network to determine the category and location information of the region proposal box.

[0063] Step 4.2: Based on the category and location information of the region proposal box, map the proposal box to the color multi-scale fusion feature branch and the deep texture feature branch to obtain the color features and deep texture features of the target.

[0064] Step 4.3: Using the target's color features from Step 4.2, obtain the target's category confidence and precise location information through a fully connected layer. Based on the target's deep texture features from Step 4.2, obtain the target's category confidence again through a fully connected layer.

[0065] Step 5: Based on the target class confidence score output by the R-CNN network in Step 4, design a new multi-branch neural network fusion strategy to output the final class confidence score of the target.

[0066] Step 5.1: Set discount fusion weights based on the target detection accuracy of each convolutional neural network, as shown in the formula.

[0067]

[0068] In the formula, M represents a specific event in the event set {red brick, concrete, wood block, hard plastic, foam, θ}; {θ} represents the combination of events: red brick, concrete, wood block, hard plastic, and foam, which represents the degree of imprecision of the network. and These represent the construction waste detection accuracy of the color and texture convolutional neural network branches, respectively. and These represent the probabilities of the color and texture convolutional neural network branches determining that the bounding box belongs to class M, respectively.

[0069] Step 5.2: Based on the class confidence scores discounted from the color and texture branches, design a multi-branch neural network fusion strategy, determine the final class confidence score of DFM-Net, and obtain the target class, as shown in the formula.

[0070]

[0071] In the formula, P(A) represents the probability that the DFM-Net model determines that the bounding box belongs to class A.

[0072] Step 6: Design a multi-task weighted loss function to achieve end-to-end training of the model.

[0073] Step 6.1, the multi-task weighted loss function of the DFM-Net model is as follows, derived from the region proposal network bounding box regression loss L. loc·N And classification loss L cls·N Color multi-scale convolutional neural network region of interest bounding box regression loss L loc·I And classification loss L cls·I And texture convolutional neural network region of interest classification loss L cls·L Weighted composition.

[0074] L = L loc·N +L loc·I +L cls·N +L cls·I +L cls·L

[0075]

[0076] In the formula, y i The label of border i represents the true category label; when the border is foreground, y represents the true category label. i =1, y = 1 when the border is the background i =0; t i Represents the coordinate vector of the predicted bounding box; K represents the true coordinate vector of the border; reg Indicates the number of borders; K cls P represents the total number of samples in a batch; ic y represents the probability that the i-th bounding box is predicted to be of class c; ic It is the actual label of the border.

[0077] Step 6.2, Model Training. This paper sets nine anchor types, with three sizes (8, 16, and 32) and three aspect ratios (1:2, 1:1, and 2:1). To enable faster model fitting, considering the generalizability of the backbone feature extraction network's features, a pre-trained parameter based on the ImageNet dataset is loaded during model training, based on the idea of ​​transfer learning, to initialize the network model. Then, the DFM-Net network is trained using a construction waste dataset. During the model training phase, the Adam optimizer is used, and the experiment is set to 60 training batches. The first 30 training rounds freeze the backbone feature extraction network, only fine-tuning the network to accelerate training efficiency; the last 30 rounds enter the unfreezing phase, updating all parameters of the DFM-Net network. The initial learning rates for the freeze and unfreeze phases are set to 0.0001 and 0.00001, respectively, and the learning rate decays to 0.96 times its original value after each iteration.

[0078] To assess the effectiveness of the DFM-Net model, it was compared with Faster-RCNN, SSD, and YOLO-V3 algorithms. The results of the accuracy and recall comparisons are as follows: Figure 4 As shown, the detection results of each model are as follows: Figure 5 As shown in Table 1, the detection results of DFM-Net are as follows.

[0079] Table 1 DFM-Net detection results

[0080]

Claims

1. A method for detecting construction waste using multi-path aggregation, characterized in that, Includes the following steps; Step 1: Collect and preprocess images of five types of construction waste samples—concrete, red brick, foam, hard plastic, and wood blocks—using a visual sensor, and create labels; Step 2: Based on the Faster RCNN network, a multi-scale feature fusion network is proposed to construct a color multi-scale fusion feature, which is used to fuse shallow detail information and high-level semantic information of construction waste images. This feature is denoted as the color multi-scale fusion feature branch, which is used to obtain the candidate region extraction layer in Step 4 and to provide features for the multi-branch neural network fusion strategy in Step 5. Step 3: Based on the local binary pattern of the digital image, extract the deep texture features of the digital image, mine the deep texture information of the construction waste image, and use it as a deep texture feature branch to provide features for the multi-branch neural network fusion strategy in Step 5. Step 4: Based on the color multi-scale fusion features in Step 2, determine the category and location information of the region proposal box through the RPN network, and extract the feature information of the target and feed it into the R-CNN sub-network; Step 5: Based on the target category confidence score output by the R-CNN sub-network in Step 4, and combining the color multi-scale fusion features in Step 2 and the deep texture features in Step 3, design a multi-branch neural network fusion strategy to output the final category confidence score of the target. Step 6: Design a multi-task weighted loss function to achieve end-to-end model training; Step 7: Improve the sorting accuracy of construction waste by training the model and achieve accurate detection of various types of construction waste; In step four, based on the color multi-scale fusion features from step two, the method for determining the category and location information of the region proposal box through the RPN network, extracting the deep color and texture information of the target, and feeding it into the R-CNN network is as follows: Step 4.1: Use the feature map obtained by the color multi-scale fusion feature branch as the candidate region extraction layer, and establish the RPN network to determine the category and location information of the region proposal box; Step 4.2: Based on the category and location information of the region proposal box, map the proposal box to the color multi-scale fusion feature branch and the deep texture feature branch to obtain the color features and deep texture features of the target. Step 4.3: Using the color features of the target from Step 4.2, obtain the target's category confidence and precise location information through a fully connected layer. Based on the target's deep texture features from Step 4.2, obtain the target's category confidence again through a fully connected layer. In step five, based on the target category confidence score output by the R-CNN network in step four, the method for the multi-branch neural network fusion strategy is as follows: Step 5.1: Set discount fusion weights based on the target detection accuracy of each convolutional neural network, as shown in the formula. In the formula, M represents a specific event in the event set {red brick, concrete, wood block, hard plastic, foam, θ}; {θ} represents the combination of events: red brick, concrete, wood block, hard plastic, and foam, which represents the degree of imprecision of the network. and These represent the construction waste detection accuracy of the color and texture convolutional neural network branches, respectively. and These represent the probabilities that the color and texture convolutional neural network branches determine the bounding box belongs to class M, respectively. Step 5.2: Based on the class confidence scores discounted from the color and texture branches, design a multi-branch neural network fusion strategy, determine the final class confidence score of DFM-Net, and obtain the target class, as shown in the formula. In the formula, M and N represent a specific event in the event set {red brick, concrete, wood block, hard plastic, foam, θ}; {θ} represents the combination of events: red brick, concrete, wood block, hard plastic, and foam, which represents the degree of imprecision of the network. P represents the probability that the bounding box belongs to class M based on the color multi-scale fusion feature branch. L (N) represents the probability that the deep texture feature branch determines that the bounding box belongs to class N; P(A) represents the probability that the DFM-Net model determines that the bounding box belongs to class A.

2. The method for detecting construction waste using multi-path aggregation according to claim 1, characterized in that, In step two, based on the Faster R-CNN network, a multi-scale feature fusion network is proposed to construct multi-scale color fusion features. The method is as follows: Step 2.1: Extract feature maps from the third, fourth, and fifth layers of the VGG network; shallow feature maps are used to describe the geometric features of construction waste, while deep feature maps are used to describe the high-level semantic information of construction waste. Step 2.2: The shallow feature map is fused with the deep feature map through downsampling and convolution operations to form a candidate region extraction layer, which integrates the shallow detail information and high-level semantic information of the construction waste image to achieve mutual complementarity of image information.

3. The method for detecting construction waste using multi-path aggregation according to claim 1, characterized in that, The method for extracting deep texture features of the target based on the local binary pattern of the digital image in step three is as follows: Step 3.1, extract digital image texture features: First, convert the digital image acquired in Step 1 to grayscale. Then, taking each pixel in the image as the center pixel, calculate and encode the difference between the neighboring pixels and the center pixel in the 3×3 neighborhood of the center pixel using the following formula to obtain the texture features of the target. In the formula, Let be the number of neighboring pixels. The center pixel value, The pixel values ​​are those of the center point's neighboring pixels; Step 3.2: The texture feature map of the digital image extracted in Step 3.1 is fed into the first two convolutional layers of the VGG16 network to further extract the deep texture features of the image, supplement the local detail information of the construction waste image, and be used as the input to the multi-branch neural network fusion strategy model in Step 5.

4. The method for detecting construction waste using multi-path aggregation according to claim 1, characterized in that, The image preprocessing method in step one is as follows: Step 1.1: Use LabelImg software to create labels for the acquired images; Step 1.2: Expand the image set of construction waste through rotation, flipping, and color gamut transformation operations.

5. The method for detecting construction waste using multi-path aggregation according to claim 1, characterized in that, The method for designing a multi-task weighted loss function in step six to achieve end-to-end model training is as follows: The loss function of the DFM-Net model is as follows, which is the region proposal network bounding box regression loss L. loc·N And classification loss L cls·N Color multi-scale convolutional neural network region of interest bounding box regression loss L loc·I And classification loss L cls·I And texture convolutional neural network region of interest classification loss L cls·L Weighted composition; L=L loc·N +L loc·I +L cls·N +L cls·I +L cls·L In the formula, y i This represents the true category label of the border i, and y represents the border when it is the foreground. i =1, y = 1 when the border is the background i =0; t i Represents the coordinate vector of the predicted bounding box; K represents the true coordinate vector of the border; reg Indicates the number of borders; K cls P represents the total number of samples in a batch; ic y represents the probability that the i-th bounding box is predicted to be of class c; ic It is the actual label of the border.

Images