Intelligent classification method for construction waste based on deep learning

Abstract

This invention discloses a deep learning-based intelligent classification method for construction waste, comprising: obtaining a standardized dataset of construction waste operation site images; generating a global candidate image region set based on the standardized construction waste operation site image dataset; forming a set of node feature vectors; obtaining a heterogeneous object-relationship graph structure of construction waste containing node type labels and edge type labels; generating gating coefficients and fused multi-scale feature representations; generating subgraph-level representations; generating edge consistency constraint signals based on boundary cues output from local feature extraction branches and pixel masks of the global candidate image region set; and defining construction waste category labels, instance contours, and segmentation masks according to the updated node representations, subgraph-level representations, and edge consistency constraint signals. This invention significantly improves the instance-level and material-level segmentation accuracy of construction waste targets in scenarios with stacking, occlusion, inter-class similarity, and large intra-class differences.

Description

Technical Field

[0001] This invention relates to the field of intelligent waste sorting technology, and in particular to an intelligent sorting method for construction waste based on deep learning. Background Technology

[0002] With the acceleration of urbanization and large-scale upgrading and renovation of infrastructure, the output of construction waste continues to rise, driving the demand for intelligent treatment and reuse of engineering solid waste. In order to improve the automation and intelligence level of construction waste resource treatment, construction waste identification and sorting methods based on computer vision have become a research hotspot in the industry.

[0003] Existing construction waste recognition methods based on convolutional neural networks or single-branch features are poorly adapted to complex backgrounds with mixed materials, broken and stacked targets, and adhered boundaries. Because different construction waste materials in the construction site environment are visually highly similar and often present as scattered, small pieces, piles, and occluded states, relying solely on local texture or global semantic information can easily lead to material misclassification, missed detection of instances, and limited accuracy in recognizing boundaries and small targets.

[0004] Existing technologies that apply graph neural networks to intelligent identification of construction waste typically establish static adjacency relationships only at the pixel or region level. They fail to effectively combine the material properties, geometric shapes, and spatial structural heterogeneous information of objects, and ignore the complexity of object-relationship structures in scenarios where construction waste is fragmented, adhered, and physically mixed. This results in insufficient graph structure expressive power and limited information transmission. Summary of the Invention

[0005] One objective of this invention is to propose a deep learning-based intelligent classification method for construction waste. This invention significantly improves the instance-level and material-level segmentation accuracy of construction waste targets in scenarios involving stacking, occlusion, inter-class similarity, and large intra-class differences.

[0006] A method for intelligent classification of construction waste based on deep learning according to an embodiment of the present invention includes:

[0007] Collect image data of construction waste operation sites and preprocess it to obtain a standardized image dataset of construction waste operation sites;

[0008] A global candidate image region set is generated based on a standardized dataset of construction waste operation site images.

[0009] For each candidate image region in the global candidate image region set, a set of node feature vectors is formed;

[0010] Using the global candidate image region set as nodes, a heterogeneous object-relationship graph structure of construction waste containing node type annotations and edge type annotations is obtained;

[0011] On a standardized construction waste operation site image dataset, local feature representations are generated using a local feature extraction branch, and global feature representations are generated using a global feature extraction branch. These representations are then input into a local-global gating fusion module to generate gating coefficients and fused multi-scale feature representations.

[0012] The improved graph attention network V2 model is improved by concatenating the set of node feature vectors with the fused multi-scale feature representation as the initial node state input. This model performs multi-head attention message passing and node updates to obtain updated node representations and edge attention weights. Subgraph aggregation is then performed in the heterogeneous object-relationship graph structure of construction waste to generate subgraph-level representations.

[0013] Edge consistency constraint signals are generated based on the boundary cues output by the local feature extraction branch and the pixel masks of the global candidate image region set.

[0014] Based on the updated node representation, subgraph-level representation, and edge consistency constraint signal, define construction waste category labels, instance outlines, and segmentation masks.

[0015] Optionally, generating a candidate image region set based on a standardized construction waste operation site image dataset includes:

[0016] For each standardized construction waste operation site image in the standardized construction waste operation site image dataset, a preliminary candidate region set is generated through a candidate region detection strategy.

[0017] Calculate the target confidence score for each preliminary candidate region in the preliminary candidate region set;

[0018] Based on the target confidence score of each preliminary candidate region, a candidate region filtering operation is performed. Only candidate regions with a target confidence score not lower than the confidence threshold are retained. An effective candidate region set is constructed and merged to form a global candidate image region set.

[0019] Optionally, the formation of the node feature vector set includes:

[0020] Extract the local texture feature vector of each candidate image region from the global candidate image region set;

[0021] Extract the edge and fragmentation feature vectors of each candidate image region;

[0022] Extract the shape statistical feature vector of each candidate image region;

[0023] Extract the material prior descriptor vector for each candidate image region;

[0024] Extract the near-infrared descriptor vector for each candidate image region;

[0025] For each candidate image region, the local texture feature vector, edge and fragmentation feature vector, shape statistical feature vector, material prior descriptor vector, and near-infrared descriptor vector of the candidate image region are concatenated in order of feature dimension to form the node feature vector of the candidate image region.

[0026] Optionally, obtaining the heterogeneous object-relationship graph structure of construction waste, which includes node type annotations and edge type annotations, includes:

[0027] Each candidate image region in the global candidate image region set is introduced into the node set as a graph node;

[0028] Obtain the geometric center coordinates of any two candidate image regions in the global candidate image region set, and use the geometric center coordinates as the basis to calculate the spatial Euclidean distance between the two candidate image regions in the image plane.

[0029] Obtain the binary pixel masks of any two candidate image regions in the global candidate image region set. Calculate the intersection-union ratio (IUR) based on the binary pixel masks. Determine whether the IUR is greater than the wrapping relationship threshold and whether the inclusion boundary condition is met. When the IUR is greater than the wrapping relationship threshold and the inclusion boundary condition is met, establish a wrapping relationship edge.

[0030] Calculate the correlation coefficient and determine whether the correlation coefficient is greater than the texture continuity threshold. If the correlation coefficient is greater than the texture continuity threshold, establish texture continuity relationship edges.

[0031] The cosine similarity is calculated based on the material prior descriptor vector. It is then determined whether the cosine similarity is greater than the material affinity threshold. If the cosine similarity is greater than the material affinity threshold, a material affinity relationship edge is established.

[0032] Spatial adjacency edges, wrapping edges, texture continuity edges, and material affinity edges are stored as a unified edge set;

[0033] Based on the set of nodes and the set of edges, a heterogeneous object-relationship graph structure for construction waste is constructed.

[0034] Optionally, the generation of the gating coefficients and the fused multi-scale feature representation includes:

[0035] The standardized construction waste operation site image dataset is input into the local feature extraction branch. The local feature extraction branch uses multi-scale convolution kernels to extract features from each standardized construction waste operation site image to obtain a set of local feature representations.

[0036] The standardized construction waste operation site image dataset is input into the global feature extraction branch. The global feature extraction branch uses a global context modeling structure to extract features from each standardized construction waste operation site image to obtain a global feature representation set.

[0037] The local feature representation and the global feature representation are input into the local-global gated fusion module. The local-global gated fusion module performs spatial size alignment and channel number matching on the local feature representation and the global feature representation to obtain aligned local feature representation and aligned global feature representation.

[0038] The local-global gating fusion module concatenates the aligned local feature representations and the aligned global feature representations along the channel dimension, and inputs the concatenation result into a linear mapping and a Sigmoid function to generate a gating coefficient map.

[0039] The aligned local feature representation and the aligned global feature representation are weighted and fused using a gating coefficient map. The aligned local feature representation is multiplied element-wise with the gating coefficient map, and the aligned global feature representation is multiplied element-wise with the complement of the gating coefficient map. The corresponding elements are then summed to generate the fused multi-scale feature representation.

[0040] Optionally, the generation of subgraph-level representations includes:

[0041] The node feature vector set and the fused multi-scale feature representation set are concatenated one-to-one according to the node index to form the initial node state set.

[0042] For any pair of adjacent nodes, calculate the material consistency coefficient, morphological consistency coefficient, and spatial consistency coefficient respectively, and sum them according to the set weights to obtain the consistency modulation coefficient. The material consistency coefficient is used to measure the similarity between two nodes in the material prior descriptor, the morphological consistency coefficient is used to measure the similarity between two nodes in the shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity between two nodes in the image plane.

[0043] In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded as an attention prior modulation factor into the attention score calculation between each pair of nodes to obtain the improved attention coefficient.

[0044] The improved attention coefficient is used to update the features of all nodes. Each layer of attention mechanism determines the aggregation weight of node features based on the node features between each node, the fused multi-scale features and the consistency modulation coefficient. The feature outputs of all heads are concatenated to obtain a new node feature vector.

[0045] During the subgraph aggregation phase, each node feature vector in the updated set of node feature vectors is used as the unique input feature vector for subgraph aggregation. Based on the consistency threshold, nodes are only allowed to be merged into the same subgraph-level representation when the consistency modulation coefficients among all nodes in the subgraph are not lower than the set consistency threshold.

[0046] Optionally, generating the edge consistency constraint signal includes:

[0047] For each candidate image region in the boundary cue set output by the local feature extraction branch and the global candidate image region set, the boundary intensity map of the candidate image region in the boundary cue set output by the local feature extraction branch and the binary pixel mask of the candidate image region are obtained respectively.

[0048] The boundary intensity map is thresholded to obtain the boundary pixel set of the candidate image region. The binary pixel mask is used for edge extraction to obtain the contour pixel set of the candidate image region. The number of overlapping pixels between the boundary pixel set and the contour pixel set of the candidate image region is counted and divided by the number of pixels in the contour pixel set to obtain the boundary overlap rate of the candidate image region.

[0049] The boundary overlap rate of the candidate image region is compared with the preset edge consistency threshold. When the boundary overlap rate is greater than or equal to the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to 1. When the boundary overlap rate is less than the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to the ratio of the boundary overlap rate to the edge consistency threshold.

[0050] Optionally, defining the construction waste category label, instance outline, and segmentation mask based on the updated node representation, subgraph-level representation, and edge consistency constraint signal includes:

[0051] Construction waste category label: When the concrete response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the condition that the volume is greater than the threshold and the compactness is higher than the threshold, the output category label is concrete;

[0052] When the steel reinforcement response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the slender shape and the boundary fracture density is higher than the threshold, the output category label is steel reinforcement.

[0053] When the wood response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the local texture feature vector shows fibrous or annual ring texture, the output category label is wood.

[0054] The output of other category labels is determined according to preset material properties and shape feature matching rules;

[0055] Instance contour output rules: For each candidate image region, obtain all external contour point sequences based on the connected pixel set in the segmentation mask, and output the external contour point sequences in spatial order. The instance contour represents the boundary shape of the construction waste target.

[0056] Segmentation mask output rules: For each candidate image region, generate a binary segmentation mask with the same spatial resolution as the standardized construction waste operation site image. In the binary segmentation mask, a pixel value of 1 indicates that the pixel belongs to the construction waste target, and a pixel value of 0 indicates that the pixel belongs to the background region.

[0057] The beneficial effects of this invention are:

[0058] (1) This invention uses Graph Attention Network V2 as the core of heterogeneous object-relationship modeling. It combines local feature extraction branches and global feature extraction branches. Through the local-global gating fusion module, it adaptively fuses fine-grained texture, boundary cues, scene-level spatial layout, and material co-occurrence information. Combined with the node features of the input graph structure, it achieves multi-scale semantic complementarity. The introduction of material prior, morphological statistics, and spatial consistency multi-factor modulation within the graph structure significantly improves the instance-level and material-level segmentation accuracy of construction waste targets in scenarios with stacking, occlusion, inter-class appearance similarity, and large intra-class differences. The average segmentation accuracy and stability are better than existing GNN models under the conditions of adhered fragments and new domain changes.

[0059] (2) This invention introduces material consistency coefficient, morphological consistency coefficient and spatial consistency coefficient into the graph attention network V2 as consistency modulation coefficients and dynamically embeds them into the multi-head attention weight calculation. This makes the feature transfer weight between each pair of nodes determined by the node features themselves and the actual physical and geometric relationship between objects. It can adaptively adjust the information fusion path according to the material properties, geometric compactness and spatial proximity between different construction waste fragments. It effectively avoids erroneous aggregation and feature interference when construction waste has multiple materials, structural adhesion or blurred target boundaries, improves the accuracy of boundary segmentation and the distinguishing power of category discrimination, and overcomes the problems of limited graph structure information expression ability and missegmentation in adhesion scenarios in the prior art.

[0060] (3) The present invention combines the boundary clues output by the local feature extraction branch with the candidate region pixel mask to dynamically generate the edge consistency constraint signal driven by the boundary overlap rate, and jointly modulates the feature aggregation weight in the node feature and subgraph aggregation stages, effectively suppressing the boundary contour missegmentation and omission segmentation caused by target adhesion, fragmentation and occlusion. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a flowchart of a deep learning-based intelligent classification method for construction waste proposed in this invention. Detailed Implementation

[0063] Example 1:

[0064] refer to Figure 1 A deep learning-based intelligent classification method for construction waste includes:

[0065] Collect image data of construction waste operation sites and preprocess it to obtain a standardized image dataset of construction waste operation sites;

[0066] Preprocessing includes dehazing, color constantization, dynamic range compression, and noise suppression of image data from construction waste operation sites.

[0067] A global candidate image region set is generated based on a standardized dataset of construction waste operation site images.

[0068] In this embodiment, a candidate image region set is generated based on a standardized construction waste operation site image dataset, including:

[0069] For each standardized construction waste operation site image in the standardized construction waste operation site image dataset, a preliminary candidate region set is generated through a candidate region detection strategy.

[0070] The candidate region detection strategy is instance segmentation, superpixel segmentation, or region proposal network. Each standardized construction waste operation site image generates a preliminary candidate region set through the candidate region detection strategy. Each preliminary candidate region in the preliminary candidate region set contains a bounding rectangle and a binary pixel mask. Each standardized construction waste operation site image has its corresponding preliminary candidate region set.

[0071] Calculate the target confidence score for each preliminary candidate region in the preliminary candidate region set;

[0072] The target confidence score measures the likelihood that the initial candidate region includes construction waste targets.

[0073] Based on the target confidence score of each preliminary candidate region, a candidate region filtering operation is performed. Only candidate regions with a target confidence score not lower than the confidence threshold are retained. An effective candidate region set is constructed and merged to form a global candidate image region set.

[0074] The set of valid candidate regions corresponding to each standardized construction waste operation site image consists of all preliminary candidate regions that meet the target confidence score criteria, and all preliminary candidate regions that do not meet the criteria are eliminated.

[0075] The global candidate image region set includes all valid candidate regions of standardized construction waste operation site images. Each valid candidate region includes its bounding rectangle, binary pixel mask, and corresponding target confidence score.

[0076] In Example 1, a non-maximum suppression method can be used to remove highly overlapping redundant regions with target confidence scores below a threshold from each candidate region in the global candidate image region set. The global candidate image region set after overlap suppression is used as the sole input source for extracting the node feature vector set. A unique index is established for each candidate region in the global candidate image region set after overlap suppression, and an index mapping is performed with the standardized construction waste operation site image dataset, so that each candidate region can be traced back to its original standardized construction waste operation site image and its spatial location.

[0077] For each candidate image region in the global candidate image region set, a set of node feature vectors is formed;

[0078] In this embodiment, the formation of the node feature vector set includes:

[0079] Extract the local texture feature vector of each candidate image region from the global candidate image region set;

[0080] The local texture feature vector is obtained by performing feature response on the candidate image region using a multi-scale convolutional filter. The local texture feature vector represents the surface texture details of the candidate image region.

[0081] Extract the edge and fragmentation feature vectors of each candidate image region;

[0082] In Example 1, edge detection is performed on the pixel set of the candidate image region to obtain the edge pixel set of the candidate image region. The edge gradient distribution of the candidate image region is statistically analyzed based on the edge pixel set. The edge gradient distribution is obtained by histogram statistics of the gradient magnitude and direction of all edge pixels. The connectivity analysis of the edge pixel set is performed to determine the number of boundary break points. The boundary break density is the ratio of the number of boundary break points to the total number of pixels in the outline of the candidate image region. The edge gradient distribution and the boundary break density are concatenated in the order of feature dimensions to form the edge and fragmentation feature vector of the candidate image region.

[0083] Extract the shape statistical feature vector of each candidate image region;

[0084] The shape statistical feature vector is calculated from the area, perimeter, and aspect ratio of the candidate image region. The shape statistical feature vector represents the geometric shape of the candidate image region.

[0085] Extract the material prior descriptor vector for each candidate image region;

[0086] Within the pixel range corresponding to each candidate image region, the mean and variance of multi-channel reflectance of the candidate image region are statistically analyzed. The reflectance data comes from the original pixel values ​​of the construction waste operation site image in the visible and near-infrared bands. The surface roughness of the candidate image region is calculated by statistically analyzing the standard deviation of the pixel gray-level gradient within the candidate image region. This standard deviation is used to measure the degree of surface texture undulation. The brightness pattern of the candidate image region is calculated by normalizing the histogram statistical features of the pixel brightness distribution within the candidate image region. This pattern is used to represent the light response characteristics of different materials. The feature values ​​of reflectance, surface roughness, and brightness pattern are statistically analyzed and spliced ​​in a predetermined order to form the material prior descriptor vector of the candidate image region.

[0087] Extract the near-infrared descriptor vector for each candidate image region;

[0088] In Example 1, for each candidate image region, a set of pixels for the candidate image region is extracted from the near-infrared imaging data. The mean reflectance and standard deviation of the reflectance of the pixel set are calculated. The mean reflectance represents the overall near-infrared reflectance characteristics of the candidate image region, and the standard deviation of reflectance represents the reflectance variation within the candidate image region. Based on the near-infrared pixel set, the local texture features of the candidate image region under near-infrared imaging are extracted using the gray-level co-occurrence matrix. By performing a ratio analysis on the average pixel values ​​of the candidate image region under different near-infrared band imaging, the material absorption characteristic parameters of the candidate image region under near-infrared imaging are obtained. The material absorption characteristic parameters are used to distinguish the energy absorption capacity of different construction waste materials in the near-infrared band. The mean reflectance, standard deviation of reflectance, local texture feature parameters, and material absorption characteristic parameters are sequentially concatenated to form a unique near-infrared descriptor vector corresponding to the candidate image region.

[0089] For each candidate image region, the local texture feature vector, edge and fragmentation feature vector, shape statistical feature vector, material prior descriptor vector, and near-infrared descriptor vector of the candidate image region are concatenated in order of feature dimension to form the node feature vector of the candidate image region.

[0090] For all candidate image regions in the global candidate image region set, repeat the steps to obtain a set of node feature vectors. The set of node feature vectors corresponds one-to-one with each candidate image region in the global candidate image region set.

[0091] Using the global candidate image region set as nodes, a heterogeneous object-relationship graph structure of construction waste containing node type annotations and edge type annotations is obtained;

[0092] In this embodiment, a heterogeneous object-relationship graph structure of construction waste containing node type annotations and edge type annotations is obtained, including:

[0093] Each candidate image region in the global candidate image region set is introduced into the node set as a graph node;

[0094] The node set consists of all global candidate image regions. Each graph node is associated with a unique corresponding node feature vector. The generation method of the node feature vector is determined according to the generation method of the node feature vector set. The dimension of each node feature vector corresponds one-to-one with the candidate image regions in the global candidate image region set.

[0095] Obtain the geometric center coordinates of any two candidate image regions in the global candidate image region set, and use the geometric center coordinates as the basis to calculate the spatial Euclidean distance between the two candidate image regions in the image plane.

[0096] Spatial Euclidean distance measures the spatial proximity of two graph nodes in the image plane, and the unit of spatial Euclidean distance is pixels.

[0097] Obtain the binary pixel masks of any two candidate image regions in the global candidate image region set. Calculate the intersection-union ratio (IUR) based on the binary pixel masks. Determine whether the IUR is greater than the wrapping relationship threshold and whether the inclusion boundary condition is met. When the IUR is greater than the wrapping relationship threshold and the inclusion boundary condition is met, establish a wrapping relationship edge.

[0098] The intersection-union ratio is equal to the ratio of the number of pixels in the intersection of the two binary pixel masks to the number of pixels in the union.

[0099] Calculate the correlation coefficient and determine whether the correlation coefficient is greater than the texture continuity threshold. If the correlation coefficient is greater than the texture continuity threshold, establish texture continuity relationship edges.

[0100] The correlation coefficient is equal to the sum of the products of each feature dimension after removing the mean, divided by the product of the standard deviations.

[0101] The cosine similarity is calculated based on the material prior descriptor vector. It is then determined whether the cosine similarity is greater than the material affinity threshold. If the cosine similarity is greater than the material affinity threshold, a material affinity relationship edge is established.

[0102] Cosine similarity is equal to the inner product of the two material prior descriptor vectors divided by the product of the magnitudes of the two material prior descriptor vectors.

[0103] Spatial adjacency edges, wrapping edges, texture continuity edges, and material affinity edges are stored as a unified edge set;

[0104] Based on the set of nodes and the set of edges, construct a heterogeneous object-relationship graph structure for construction waste;

[0105] The heterogeneous object-relationship graph structure of construction waste includes a set of nodes, a set of edges, a set of node type labels, and a set of edge type labels. The set of node type labels contains the target category or candidate region type of construction waste to which each node belongs, and the set of edge type labels contains spatial adjacency, enclosing, texture continuity, and material affinity relationship types.

[0106] On a standardized construction waste operation site image dataset, local feature representations are generated using a local feature extraction branch, and global feature representations are generated using a global feature extraction branch. These representations are then input into a local-global gating fusion module to generate gating coefficients and fused multi-scale feature representations.

[0107] In this embodiment, generating the gating coefficients and the fused multi-scale feature representation includes:

[0108] The standardized construction waste operation site image dataset is input into the local feature extraction branch. The local feature extraction branch uses multi-scale convolution kernels to extract features from each standardized construction waste operation site image to obtain a set of local feature representations.

[0109] Each local feature is represented as the spatial distribution of fine-grained texture, cracks, holes, and boundary cues in the corresponding standardized construction waste operation site image.

[0110] The standardized construction waste operation site image dataset is input into the global feature extraction branch. The global feature extraction branch uses a global context modeling structure to extract features from each standardized construction waste operation site image to obtain a global feature representation set.

[0111] Each global feature is represented as the overall spatial features of scene semantics, yard layout, and illumination distribution of the corresponding standardized construction waste operation site image.

[0112] The local feature representation and the global feature representation are input into the local-global gated fusion module. The local-global gated fusion module performs spatial size alignment and channel number matching on the local feature representation and the global feature representation to obtain aligned local feature representation and aligned global feature representation.

[0113] In Example 1, the local feature representation and the global feature representation are respectively input into the spatial size adjustment module and the channel number matching module of the local-global gating fusion module. The spatial size adjustment module uses bilinear interpolation to adjust the local feature representation and the global feature representation to the same spatial size. The channel number matching module uses one-dimensional convolution to adjust the local feature representation and the global feature representation to the same number of channels, thus obtaining the aligned local feature representation and the aligned global feature representation. The aligned local feature representation and the aligned global feature representation correspond one-to-one with the standardized construction waste operation site image in terms of spatial size and number of channels.

[0114] The local-global gating fusion module concatenates the aligned local feature representations and the aligned global feature representations along the channel dimension, and inputs the concatenation result into a linear mapping and a Sigmoid function to generate a gating coefficient map.

[0115]

[0116] Where [·; ·] represents feature concatenation along the channel dimension, W g To fuse the weight matrix, b g For the fusion bias term, σ(·) is the Sigmoid function, and g k Represents the gating coefficient diagram. This is a representation of the aligned local features. This is the aligned global feature representation.

[0117] The aligned local feature representation and the aligned global feature representation are weighted and fused using a gating coefficient map. The aligned local feature representation is multiplied element-wise with the gating coefficient map, and the aligned global feature representation is multiplied element-wise with the complement of the gating coefficient map. The corresponding elements are then summed to generate the fused multi-scale feature representation.

[0118]

[0119] Where ⊙ represents element-wise multiplication, f fusion,k This represents the multi-scale feature representation after fusion.

[0120] The improved graph attention network V2 model is improved by concatenating the set of node feature vectors with the fused multi-scale feature representation as the initial node state input. This model performs multi-head attention message passing and node updates to obtain updated node representations and edge attention weights. Subgraph aggregation is then performed in the heterogeneous object-relationship graph structure of construction waste to generate subgraph-level representations.

[0121] In this embodiment, generating a subgraph-level representation includes:

[0122] The node feature vector set and the fused multi-scale feature representation set are concatenated one-to-one according to the node index to form the initial node state set.

[0123] Each initial node state is composed of the node feature vector and the fused multi-scale feature representation of the node, spliced ​​together in the order of feature dimensions. The node feature vector is a representation of the candidate image region.

[0124] For any pair of adjacent nodes, calculate the material consistency coefficient, morphological consistency coefficient, and spatial consistency coefficient respectively, and sum them according to the set weights to obtain the consistency modulation coefficient. The material consistency coefficient is used to measure the similarity between two nodes in the material prior descriptor, the morphological consistency coefficient is used to measure the similarity between two nodes in the shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity between two nodes in the image plane.

[0125] In Example 1, the material consistency coefficient is obtained by taking the inner product of the material prior descriptor vectors of the two nodes and dividing it by the product of the magnitudes of the two material prior descriptor vectors. The morphological consistency coefficient is obtained by subtracting the larger of the difference in shape compactness between the two nodes. The spatial consistency coefficient is obtained by exponentially operating on the ratio of the spatial Euclidean distance between the two nodes to the spatial decay constant.

[0126] In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded as an attention prior modulation factor into the attention score calculation between each pair of nodes to obtain the improved attention coefficient.

[0127] By embedding the consistency modulation coefficient as an attention prior modulation factor into the attention score calculation between each pair of nodes, the attention score between nodes is not only affected by the node features and the fused multi-scale features, but also by the consistency of materials, morphology and space between nodes. The attention score between nodes is adaptively adjusted with the change of the consistency modulation coefficient, thereby improving the ability to distinguish between fragmented, agglomerated and mixed material phenomena of construction waste.

[0128]

[0129] in, This represents the node v at the k-th attention head of the l-th layer. i Its neighbor node v j Improved attention coefficient, ω ij Represents node v i With node v jThe consistency modulation coefficient, representing the node pair in the heterogeneous object-relationship graph structure of construction waste, comprehensively reflects the degree of consistency between two nodes in terms of material properties, geometric shape, and spatial adjacency. A larger consistency modulation coefficient indicates that the two nodes are more likely to belong to the same type or be physically connected construction waste targets in a real construction site scenario, making information transmission more reliable. (k) W (k) Let represent the linear transformation parameters corresponding to the k-th head of the graph attention network V2, which learns the importance weights of construction waste nodes in the feature dimension. Represents node v in the l-th layer i v j The multimodal feature vector, σ(·) represents the LeakyReLU activation function, introduces nonlinear expressive power to adapt to the complex morphological changes of construction waste, and N(i) represents the node v i The set of adjacent nodes represents other regions in the graph structure that have a direct physical or semantic connection with the candidate region of construction waste.

[0130] The improved attention coefficient is used to update the features of all nodes. Each layer of attention mechanism determines the aggregation weight of node features based on the node features between each node, the fused multi-scale features and the consistency modulation coefficient. The feature outputs of all heads are concatenated to obtain a new node feature vector.

[0131]

[0132] in, This indicates the updated node v i The feature vector, K represents the total number of attention heads, and || represents the multi-head feature concatenation operation.

[0133] During the subgraph aggregation phase, based on the consistency threshold, nodes are only allowed to be merged into the same subgraph-level representation when the consistency modulation coefficients among all nodes in the subgraph are not lower than the set consistency threshold.

[0134] In Example 1, each node feature vector in the updated set of node feature vectors is used as the unique input feature vector for subgraph aggregation. Combined with the edge consistency modulation coefficient, candidate subgraphs are divided for all nodes. When dividing candidate subgraphs, nodes are only allowed to be assigned to the same candidate subgraph if the edge consistency modulation coefficient of any two nodes in the candidate subgraph is not lower than the consistency threshold. For each candidate subgraph that meets the condition, all node feature vectors in the candidate subgraph are input into the feature aggregation function for feature aggregation. The feature aggregation function is a weighted average function, and the output of the feature aggregation is used as the aggregated representation vector of the candidate subgraph.

[0135] Edge consistency constraint signals are generated based on the boundary cues output by the local feature extraction branch and the pixel masks of the global candidate image region set.

[0136] In this embodiment, generating the edge consistency constraint signal includes:

[0137] For each candidate image region in the boundary cue set output by the local feature extraction branch and the global candidate image region set, the boundary intensity map of the candidate image region in the boundary cue set output by the local feature extraction branch and the binary pixel mask of the candidate image region are obtained respectively.

[0138] In Example 1, each image in the standardized construction waste operation site image dataset is input into the local feature extraction branch to obtain a boundary response map. The boundary response map is then processed by non-maximum suppression and threshold segmentation to obtain a boundary cue map.

[0139] The pixel mask information of each candidate image region in the global candidate image region set is directly extracted from the candidate region detection strategy to obtain the binary pixel mask of the candidate image region. Each candidate image region is matched one-to-one with its spatial position in the boundary cue map through node index, ensuring that each candidate image region is uniquely associated with a boundary cue map and a binary pixel mask.

[0140] The boundary intensity map is thresholded to obtain the boundary pixel set of the candidate image region. The binary pixel mask is used for edge extraction to obtain the contour pixel set of the candidate image region. The number of overlapping pixels between the boundary pixel set and the contour pixel set of the candidate image region is counted and divided by the number of pixels in the contour pixel set to obtain the boundary overlap rate of the candidate image region.

[0141] Boundary overlap rate measures the degree of spatial overlap between the predicted boundary of a candidate image region and the true contour. The boundary overlap rate ranges from 0 to 1.

[0142] The boundary overlap rate of the candidate image region is compared with the preset edge consistency threshold. When the boundary overlap rate is greater than or equal to the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to 1. When the boundary overlap rate is less than the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to the ratio of the boundary overlap rate to the edge consistency threshold.

[0143] Edge consistency constraint signals are used to measure the consistency between boundary cues and contour pixels of candidate image regions.

[0144] Based on the updated node representation, subgraph-level representation, and edge consistency constraint signal, define construction waste category labels, instance outlines, and segmentation masks.

[0145] In this embodiment, construction waste category labels, instance outlines, and segmentation masks are defined based on the updated node representation, subgraph-level representation, and edge consistency constraint signals, including:

[0146] Construction waste category labels:

[0147] When the concrete response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the condition that the volume is greater than the threshold and the compactness is higher than the threshold, the output category label is concrete.

[0148] When the steel reinforcement response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the slender shape and the boundary fracture density is higher than the threshold, the output category label is steel reinforcement.

[0149] When the wood response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the local texture feature vector shows fibrous or annual ring texture, the output category label is wood.

[0150] The output of other category labels is determined according to preset material properties and shape feature matching rules;

[0151] Instance contour output rules: For each candidate image region, obtain all external contour point sequences based on the connected pixel set in the segmentation mask, and output the external contour point sequences in spatial order. The instance contour represents the boundary shape of the construction waste target.

[0152] Segmentation mask output rules: For each candidate image region, generate a binary segmentation mask with the same spatial resolution as the standardized construction waste operation site image. In the binary segmentation mask, a pixel value of 1 indicates that the pixel belongs to the construction waste target, and a pixel value of 0 indicates that the pixel belongs to the background region.

[0153] Example 2:

[0154] In the actual application of the intelligent sorting line for construction waste, the system is deployed on an intelligent sorting production line containing multiple high-definition cameras and infrared auxiliary modules. In the scenario, construction waste passes continuously on the conveyor belt, forming a large number of messy, mixed, fragmented, and similar waste areas. The system automatically collects the original video stream and extracts several frames per second as static image samples for processing.

[0155] The system performs dehazing and color normalization on the acquired images to eliminate visual noise caused by dust, mechanical lighting, and rain reflections. In Example 2, during a night shift, a certain image frame was dark because the concrete and old wooden boards on the conveyor belt were covered with oil stains. After preprocessing, the image contrast was significantly improved, and the boundary contours were clearer. Subsequently, through the candidate region detection module, the system automatically segmented 39 candidate regions on the image, covering rusted steel bars, broken concrete blocks, some residual insulation boards, and damaged wood. Each candidate region generates a unique bounding rectangle and a pixel-level binary mask.

[0156] For each candidate region, the system further extracts multi-scale convolutional texture features, shape statistics features, material prior descriptors, and infrared reflectance features. In Example 2, candidate region C019 mainly exhibits high-frequency irregular texture features on its surface. Shape statistics show an aspect ratio of 4.3 and a compactness of 0.62. The average infrared reflectance in the material prior is 0.41. The system determines this region to be rusted steel reinforcement with a confidence level of 0.93. Region C022 has an area of ​​less than 500 pixels, uniform grayscale distribution, porous texture features, and low reflectance in the material prior. Combined with its morphological features, it is determined to be concrete fragment.

[0157] In the heterogeneous object-relationship graph construction stage, the system establishes multi-class edges among 39 candidate regions based on spatial adjacency, boundary wrapping, material affinity, and texture continuity. In Example 2, C019 and C022 have an adjacency relationship with a spatial distance of only 8 pixels, and the material prior descriptor similarity is 0.48, the morphological consistency is 0.79, and the spatial consistency coefficient is 0.91. The system inputs these parameters into the multi-head dynamic modulation module of the improved graph attention network V2.

[0158] After iterative updates using multi-head attention, the system automatically identified C019 and C022 as adhered but heterogeneous material instances to prevent missegmentation. Simultaneously, the system compared the boundary clues output by the local feature branches with the candidate region mask. The boundary overlap rate of region C022 was calculated to be 0.88, which is greater than the set threshold of 0.85. The system set the edge consistency constraint signal of this region to 1. However, region C024 had a blurred boundary contour, with an overlap rate of only 0.57. The edge consistency constraint signal was set to 0.67, which reduced the weight of this region during node updates and subgraph aggregation, thus avoiding misjudgment.

[0159] The classification and segmentation prediction module integrates the final features and constraints of each region and outputs the following detection results:

[0160] Region C019 is classified as rebar, with a location bounding box of (230,421)-(254,520), 68 instance outline points, and 542 segmentation mask pixels.

[0161] Region C022 is classified as concrete, with a location bounding box of (318,442)-(357,480), 43 instance outline points, and 335 segmentation mask pixels.

[0162] Region C027 is classified as wood, with a boundary fracture density of 0.23, 76 instance outline points, and 682 segmentation mask pixels.

[0163] The overall image segmentation average IoU was 0.91, the boundary F1 score was 0.89, and the false negative rate was 2.5%.

[0164] To verify the performance advantages of the method of this invention, the implementers selected 300 representative images of the work line from the same batch and conducted a comparative experiment between the traditional Mask R-CNN and the method of this invention. The training samples included 2638 instances of rebar, 4151 instances of concrete, 1797 instances of timber, 812 instances of insulation board, and 2695 instances of bricks. The test data contained no fewer than 350 real-world labeled instances for each type of waste. The actual comparison results are as follows:

[0165] The average IoU of the present invention is 0.91 for steel reinforcement, 0.89 for concrete, 0.87 for wood, 0.83 for insulation board, and 0.85 for brick. The average missegmentation rate is 3.4%, the average missed detection rate is 2.6%, and the average segmentation boundary F1 is 0.90.

[0166] The average IoU for traditional methods is 0.83 for steel reinforcement, 0.82 for concrete, 0.78 for timber, 0.71 for insulation board, and 0.74 for brick. The average missegmentation rate is 9.7%, the average missed detection rate is 8.2%, and the average segmentation boundary F1 is 0.82.

[0167] In Example 2, in a stacked image, the method of the present invention can correctly distinguish three closely attached concrete blocks and two partially obscured reinforcing bars, and accurately separate the small, adhered fragments at the edge of the insulation board. In contrast, the traditional method mistakenly merges a reinforcing bar with the concrete and fails to separate the small fragments of the insulation board.

[0168] In real-time deployment testing of the sorting line, the system of this invention exhibited a single-frame processing latency of less than 65 milliseconds, supported actual 60 frames / second streaming input, and achieved an accuracy rate of over 96% for automatically pushing sorting instructions. In some manual review stages, the consistency rate between the automatic segmentation suggestions of the system and the actual physical sorting results reached over 97%.

[0169] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep learning-based intelligent classification method for construction waste, characterized in that, The method comprises the following steps: Collecting construction waste operation site image data and preprocessing to obtain a standardized construction waste operation site image dataset; Generating a global candidate image region set based on the standardized construction waste operation site image dataset; Forming a node feature vector set for each candidate image region in the global candidate image region set; Obtaining a construction waste heterogeneous object-relation graph structure containing node type labels and edge type labels with the global candidate image region set as nodes; Generating local feature representations using a local feature extraction branch on the standardized construction waste operation site image dataset, generating global feature representations using a global feature extraction branch, and inputting the local-global gated fusion module to generate a gating coefficient and a fused multi-scale feature representation; Cascade the node feature vector set and the fused multi-scale feature representation as the initial node state to input the improved graph attention network V2 model to perform multi-head attention message passing and node updating to obtain updated node representations and edge attention weights, and perform subgraph aggregation in the construction waste heterogeneous object-relation graph structure to generate subgraph-level representations; Generating edge consistency constraint signals based on the boundary clues output by the local feature extraction branch and the pixel mask of the global candidate image region set; Defining construction waste class labels, instance contours and segmentation masks based on the updated node representations, subgraph-level representations and edge consistency constraint signals. 2.The deep learning-based intelligent classification method of construction waste according to claim 1, characterized in that, The method comprises the following steps: For each standardized construction waste operation site image in the standardized construction waste operation site image dataset, generate a preliminary candidate region set through a candidate region detection strategy; Calculate the target confidence score of each preliminary candidate region in the preliminary candidate region set; Perform a candidate region screening operation based on the target confidence score of each preliminary candidate region, only keep the candidate regions with a target confidence score not lower than a confidence score threshold, construct an effective candidate region set, and merge to form a global candidate image region set. 3.The deep learning-based intelligent classification method of construction waste according to claim 1, characterized in that, The formation of the node feature vector set comprises the following steps: Extract the local texture feature vector of each candidate image region in the global candidate image region set; Extract the edge and fragmentation feature vector of each candidate image region; Extract the shape statistical feature vector of each candidate image region; Extract the material prior descriptor vector of each candidate image region; Extract the near-infrared descriptor vector of each candidate image region; For each candidate image region, sequentially concatenate the local texture feature vector, edge and fragmentation feature vector, shape statistical feature vector, material prior descriptor vector and near-infrared descriptor vector of the candidate image region according to the feature dimension order to form the node feature vector of the candidate image region.

4. The deep learning-based intelligent classification method for construction waste according to claim 1, characterized in that, The method comprises the following steps: Introduce each candidate image region in the global candidate image region set as a graph node into the node set; Obtain the geometric center coordinates of any two candidate image regions in the global candidate image region set, and calculate the spatial Euclidean distance of the two candidate image regions in the image plane based on the geometric center coordinates; Obtain the binary pixel mask of any two candidate image regions in the global candidate image region set respectively, calculate the intersection over union value based on the binary pixel mask, and judge whether the intersection over union value is greater than the wrapping relationship threshold and whether the containing boundary condition is met; when the intersection over union value is greater than the wrapping relationship threshold and the containing boundary condition is met, a wrapping relationship edge is established; Calculate the correlation coefficient, and judge whether the correlation coefficient is greater than the texture continuity threshold; when the correlation coefficient is greater than the texture continuity threshold, a texture continuity relationship edge is established; Calculate the cosine similarity based on the material prior descriptor vector, and judge whether the cosine similarity is greater than the material affinity threshold; when the cosine similarity is greater than the material affinity threshold, a material affinity relationship edge is established; Store the spatial adjacency relationship edge, the wrapping relationship edge, the texture continuity relationship edge and the material affinity relationship edge as an edge set; Based on the node set and the edge set, an architectural waste heterogeneous object-relationship graph structure is constructed.

5. The deep learning-based intelligent classification method for construction waste according to claim 1, characterized in that, The generated gating coefficient and the fused multi-scale feature representation include: Input the standardized construction waste work site image dataset into the local feature extraction branch, and the local feature extraction branch uses a multi-scale convolution kernel to extract features from each standardized construction waste work site image to obtain a local feature representation set; Input the standardized construction waste work site image dataset into the global feature extraction branch, and the global feature extraction branch uses a global context modeling structure to extract features from each standardized construction waste work site image to obtain a global feature representation set; Input the local feature representation and the global feature representation into the local-global gating fusion module, and the local-global gating fusion module aligns the spatial size and matches the channel number of the local feature representation and the global feature representation to obtain an aligned local feature representation and an aligned global feature representation; The local-global gating fusion module performs channel dimension splicing on the aligned local feature representation and the aligned global feature representation, inputs the splicing result into a linear mapping and a Sigmoid function, and generates a gating coefficient map; Weighted fusion of the aligned local feature representation and the aligned global feature representation is performed using the gating coefficient map, the aligned local feature representation is multiplied element by element with the gating coefficient map, the aligned global feature representation is multiplied element by element with the complement of the gating coefficient map, and the corresponding elements are added to generate a fused multi-scale feature representation.

6. The deep learning-based intelligent classification method for construction waste according to claim 1, characterized in that, The generated subgraph-level representation includes: Perform feature splicing operation on the node feature vector set and the fused multi-scale feature representation set according to the one-to-one correspondence of the node index, as an initial node state set; For any pair of adjacent nodes, respectively calculate the material consistency coefficient, the shape consistency coefficient and the spatial consistency coefficient, and weight sum according to the set weight to obtain the consistency modulation coefficient, the material consistency coefficient is used to measure the similarity of two nodes in the material prior descriptor, the shape consistency coefficient is used to measure the similarity of two nodes in the shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity of two nodes in the image plane; In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded as an attention prior modulation factor in the attention score calculation between each pair of nodes to obtain an improved attention coefficient; The improved attention coefficient is used for feature updating of all nodes, and the aggregation weight of the node feature is determined by the node feature, the fused multi-scale feature and the consistency modulation coefficient between nodes in each layer of attention mechanism, and the feature outputs of all heads are spliced to obtain a new node feature vector; In the subgraph aggregation stage, each node feature vector in the updated node feature vector set is taken as the only input feature vector of subgraph aggregation, and based on the consistency threshold, only when the consistency modulation coefficients between all nodes in the subgraph are not lower than the set consistency threshold, the nodes are allowed to be merged into the same subgraph level representation.

7. The deep learning-based intelligent construction waste classification method according to claim 1, characterized in that, The edge consistency constraint signal is generated, including: For each candidate image region in the boundary clue set output by the local feature extraction branch and the global candidate image region set, a boundary intensity map of the candidate image region in the boundary clue set output by the local feature extraction branch and a binary pixel mask of the candidate image region are obtained respectively; The boundary intensity map is thresholded to obtain a boundary pixel set of the candidate image region, and the binary pixel mask is edge extracted to obtain a contour pixel set of the candidate image region, and the number of overlapping pixels of the boundary pixel set and the contour pixel set of the candidate image region is counted, and the number of overlapping pixels is divided by the number of pixels of the contour pixel set to obtain the boundary overlap rate of the candidate image region; The boundary overlap rate of the candidate image region is compared with the preset edge consistency threshold, and when the boundary overlap rate is greater than or equal to the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to 1, and when the boundary overlap rate is less than the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to the ratio of the boundary overlap rate to the edge consistency threshold. 8.The deep learning-based intelligent classification method of construction waste according to claim 1, characterized in that, The building waste class label, instance contour and segmentation mask are defined according to the updated node representation, subgraph level representation and edge consistency constraint signal, including: When the concrete response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the shape statistical feature vector satisfies the volume greater than the threshold and the compactness higher than the threshold, the output class label is concrete; When the steel bar response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the shape statistical feature vector satisfies the elongated shape and the boundary fracture density is higher than the threshold, the output class label is steel bar; When the wood response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the local texture feature vector shows fibrous or annual ring texture, the output category label is wood; The output of the remaining category labels is determined according to a preset material attribute and morphology feature matching rule; Instance contour output rule: for each candidate image region, all external contour point sequences are obtained according to the connected pixel set in the segmentation mask, and the external contour point sequences are arranged and output in spatial order, and the instance contour represents the boundary shape of the construction waste target; Segmentation mask output rule: for each candidate image region, a binary segmentation mask consistent with the standardized construction waste work site image spatial resolution is generated, and the pixel value of 1 in the binary segmentation mask indicates that the pixel belongs to the construction waste target, and the pixel value of 0 indicates that the pixel belongs to the background region.

Images