Construction waste intelligent classification method based on deep learning

Abstract

The invention discloses an intelligent construction waste classification method based on deep learning. The method comprises the steps of obtaining a standardized construction waste operation site image data set; generating a global candidate image region set based on the standardized construction waste operation site image data set; forming a node feature vector set; obtaining a construction waste heterogeneous object-relation graph structure containing node type labels and edge type labels; generating a gating coefficient and fused multi-scale feature representation; generating a sub-graph level representation; generating an edge consistency constraint signal based on a boundary clue output by the local feature extraction branch and a pixel mask of the global candidate image region set; and according to the updated node representation, the sub-graph-level representation and the edge consistency constraint signal, defining a construction waste category label, an instance contour and a segmentation mask. According to the method, the instance-level and material-level segmentation precision of the construction waste target under the scenes of stacking, shielding, similar inter-class appearance and large intra-class difference is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent garbage classification, and particularly relates to a building garbage intelligent classification method based on deep learning. BACKGROUND

[0002] With the acceleration of urbanization and large-scale infrastructure renovation, the production of building garbage continues to rise, driving the demand for intelligent processing and recycling of engineering solid waste. To improve the automation and intelligence level of building garbage resource processing, building garbage identification and sorting methods based on computer vision have become a research focus in the industry.

[0003] Existing building garbage identification methods based on convolutional neural networks or single branch features have poor adaptability to complex background, mixed materials, target fragmentation and stacking, and boundary adhesion scenes. Due to the high similarity in visual performance of different building garbage materials in the construction site environment, and the often scattered, small, stacked, and occluded state, relying solely on local texture or global semantic information can easily lead to material misclassification, instance missed detection, and limited boundary and small target recognition accuracy.

[0004] Some existing technologies apply graph neural networks to building garbage intelligent identification, usually only establishing static adjacency relationships at the pixel or region level, and fail to effectively combine object material properties, geometric morphology, and spatial structure heterogeneous information, ignoring the complexity of object-relationship structures in building garbage fragmentation, adhesion, and physical mixing scenes, resulting in insufficient graph structure expression ability and limited information transmission. SUMMARY

[0005] One object of the present application is to provide a building garbage intelligent classification method based on deep learning, which significantly improves the instance-level and material-level segmentation accuracy of building garbage targets in stacking, occlusion, inter-class appearance similarity, and intra-class large difference scenes.

[0006] According to the building garbage intelligent classification method based on deep learning of the present application, the method comprises the following steps:

[0007] Collecting building garbage work site image data and performing preprocessing to obtain a standardized building garbage work site image dataset;

[0008] Generating a global candidate image region set based on the standardized building garbage work site image dataset;

[0009] Forming a node feature vector set for each candidate image region in the global candidate image region set;

[0010] Taking the global candidate image region set as nodes, obtaining a building garbage heterogeneous object-relationship graph structure containing node type labels and edge type labels;

[0011] generate a local feature representation using a local feature extraction branch, generate a global feature representation using a global feature extraction branch, and input the local-global gated fusion module to generate a gating coefficient and a fused multi-scale feature representation;

[0012] concatenate the set of node feature vectors and the fused multi-scale feature representation as an initial node state to input an improved graph attention network V2 model to perform multi-head attention message passing and node updating, obtain an updated node representation and an edge attention weight, and perform subgraph aggregation in the construction waste heterogeneous object-relation graph structure to generate a subgraph-level representation;

[0013] generate an edge consistency constraint signal based on the boundary clue output by the local feature extraction branch and the pixel mask of the set of global candidate image regions;

[0014] define a construction waste class label, an instance contour, and a segmentation mask based on the updated node representation, the subgraph-level representation, and the edge consistency constraint signal.

[0015] Optionally, the set of candidate image regions is generated based on a standardized construction waste job site image dataset, and includes:

[0016] For each standardized construction waste job site image in the standardized construction waste job site image dataset, a set of preliminary candidate regions is generated through a candidate region detection strategy;

[0017] Calculate the target confidence score of each preliminary candidate region in the set of preliminary candidate regions;

[0018] Perform a candidate region screening operation based on the target confidence score of each preliminary candidate region, only keep the candidate regions with a target confidence score not lower than a confidence threshold, construct an effective candidate region set and merge to form a set of global candidate image regions.

[0019] Optionally, the set of node feature vectors is formed, and includes:

[0020] Extract a local texture feature vector of each candidate image region in the set of global candidate image regions;

[0021] Extract an edge and fragmentation feature vector of each candidate image region;

[0022] Extract a shape statistical feature vector of each candidate image region;

[0023] Extract a material prior descriptor vector of each candidate image region;

[0024] Extract a near-infrared descriptor vector of each candidate image region;

[0025] For each candidate image region, the local texture feature vector, the edge and fragmentation feature vector, the shape statistical feature vector, the material prior descriptor vector and the near-infrared descriptor vector of the candidate image region are sequentially spliced in order of feature dimension, and the spliced node feature vector of the candidate image region is formed.

[0026] Optionally, the building waste heterogeneous object-relation graph structure containing node type labels and edge type labels comprises:

[0027] Each candidate image region in the global candidate image region set is introduced into the node set as a graph node.

[0028] The geometric center coordinates of any two candidate image regions in the global candidate image region set are obtained, and the spatial Euclidean distance of the two candidate image regions in the image plane is calculated based on the geometric center coordinates.

[0029] The binary pixel masks of any two candidate image regions in the global candidate image region set are obtained respectively, the intersection-over-union value is calculated based on the binary pixel masks, and it is judged whether the intersection-over-union value is greater than the wrapping relationship threshold and whether the containing boundary condition is met. When the intersection-over-union value is greater than the wrapping relationship threshold and the containing boundary condition is met, a wrapping relationship edge is established.

[0030] The correlation coefficient is calculated, and it is judged whether the correlation coefficient is greater than the texture continuity threshold. When the correlation coefficient is greater than the texture continuity threshold, a texture continuity relationship edge is established.

[0031] The cosine similarity is calculated based on the material prior descriptor vector, and it is judged whether the cosine similarity is greater than the material affinity threshold. When the cosine similarity is greater than the material affinity threshold, a material affinity relationship edge is established.

[0032] The spatial adjacency relationship edge, the wrapping relationship edge, the texture continuity relationship edge and the material affinity relationship edge are uniformly stored as the edge set.

[0033] Based on the node set and the edge set, a building waste heterogeneous object-relation graph structure is constructed.

[0034] Optionally, the generation of the gating coefficient and the fused multi-scale feature representation comprises:

[0035] The standardized building waste construction site image dataset is input into the local feature extraction branch. The local feature extraction branch uses multi-scale convolution kernels to extract features from each standardized building waste construction site image, and obtains a local feature representation set.

[0036] The standardized construction waste operation site image dataset is input into a global feature extraction branch, and the global feature extraction branch extracts features of each standardized construction waste operation site image by using a global context modeling structure to obtain a global feature representation set;

[0037] The local feature representation and the global feature representation are input into a local-global gating fusion module, and the local-global gating fusion module performs spatial size alignment and channel number matching on the local feature representation and the global feature representation to obtain an aligned local feature representation and an aligned global feature representation;

[0038] The local-global gating fusion module performs channel dimension splicing on the aligned local feature representation and the aligned global feature representation, and inputs the splicing result into a linear mapping and a Sigmoid function to generate a gating coefficient map;

[0039] The aligned local feature representation and the aligned global feature representation are weightedly fused by using the gating coefficient map, the aligned local feature representation is multiplied element by element with the gating coefficient map, the aligned global feature representation is multiplied element by element with a complement of the gating coefficient map, and corresponding elements are added to generate a fused multi-scale feature representation.

[0040] Optionally, the generating a subgraph-level representation comprises:

[0041] The set of node feature vectors and the set of fused multi-scale feature representations are subjected to one-to-one feature splicing operation according to node indexes to serve as an initial node state set;

[0042] For any pair of adjacent nodes, a consistency modulation coefficient is obtained by calculating a material consistency coefficient, a shape consistency coefficient and a spatial consistency coefficient respectively and weightedly summing them according to a set weight, the material consistency coefficient is used to measure the similarity of two nodes in the material prior description subgraph, the shape consistency coefficient is used to measure the similarity of two nodes in the shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity of two nodes in the image plane.

[0043] In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded into the attention score calculation between each pair of nodes as an attention prior modulation factor to obtain an improved attention coefficient;

[0044] The improved attention coefficient is used to update the features of all nodes, and the aggregation weight of the node features is determined by the node features, the fused multi-scale features and the consistency modulation coefficient between nodes in each layer of attention mechanism, and the feature outputs of all heads are spliced to obtain new node feature vectors;

[0045] In the subgraph aggregation stage, each node feature vector in the updated node feature vector set is taken as the only input feature vector of subgraph aggregation, and based on a consistency threshold, only when the consistency modulation coefficients between all nodes in the subgraph are not lower than the set consistency threshold, the nodes are allowed to be merged into the same subgraph level representation.

[0046] Optionally, the generating the edge consistency constraint signal comprises:

[0047] For each candidate image region in the boundary clue set output by the local feature extraction branch and the global candidate image region set, a boundary intensity map of the candidate image region in the boundary clue set output by the local feature extraction branch and a binary pixel mask of the candidate image region are respectively obtained;

[0048] The boundary pixel set of the candidate image region is obtained by thresholding the boundary intensity map, the contour pixel set of the candidate image region is obtained by edge extraction on the binary pixel mask, and the boundary overlap rate of the candidate image region is obtained by dividing the number of overlapping pixels of the boundary pixel set and the contour pixel set by the number of pixels of the contour pixel set.

[0049] The boundary overlap rate of the candidate image region is compared with the preset edge consistency threshold, when the boundary overlap rate is greater than or equal to the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to 1, and when the boundary overlap rate is less than the edge consistency threshold, the edge consistency constraint signal of the candidate image region is set to the ratio of the boundary overlap rate and the edge consistency threshold.

[0050] Optionally, the defining the construction waste class label, instance contour and segmentation mask according to the updated node representation, subgraph level representation and edge consistency constraint signal comprises:

[0051] The construction waste class label: when the concrete response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies that the volume is greater than a threshold and the compactness is higher than a threshold, the output class label is concrete;

[0052] When the steel bar response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies that the elongated shape and the boundary fracture density are higher than a threshold, the output class label is steel bar;

[0053] When the wood response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the local texture feature vector shows fibrous or annual ring texture, the output class label is wood;

[0054] The output of the remaining class labels is determined according to the preset material attribute and morphological feature matching rules.

[0055] Instance contour output rule: for each candidate image region, all external contour point sequences are obtained according to the connected pixel set in the segmentation mask, and the external contour point sequences are arranged and output in spatial order, and the instance contour represents the boundary shape of the construction waste target;

[0056] Segmentation mask output rule: for each candidate image region, a binary segmentation mask consistent with the normalized construction waste site image spatial resolution is generated, and the pixel value of the binary segmentation mask is 1, indicating that the pixel belongs to the construction waste target, and the pixel value of 0 indicates that the pixel belongs to the background region.

[0057] The beneficial effects of the present application are:

[0058] (1) The present application adopts graph attention network V2 as the core of heterogeneous object-relation modeling, combines local feature extraction branch and global feature extraction branch, and adaptively fuses fine-grained texture, boundary clues and scene-level spatial layout, material co-occurrence information through a local-global gating fusion module, and inputs the node features of the graph structure, realizes multi-scale semantic complementation, and introduces material priori, morphological statistics and spatial consistency multi-factor modulation in the graph structure, which significantly improves the instance-level and material-level segmentation accuracy of the construction waste target in the scene of stacking, occlusion, similar appearance between classes and large intra-class difference, and the average segmentation accuracy and stability are better than the existing GNN model in the case of adhesion fragments and new domain changes.

[0059] (2) The present application introduces material consistency coefficient, morphological consistency coefficient and spatial consistency coefficient in the graph attention network V2, which are used as consistency modulation coefficients to dynamically embed multi-head attention weight calculation, so that the feature transmission weight between each pair of nodes is determined by the node features and the actual physical and geometric relationship between objects, which can adaptively adjust the information fusion path according to the material properties, geometric compactness and spatial proximity between different construction waste fragments, effectively avoid false aggregation and feature interference when the construction waste is mixed with multiple materials, the structure is adhesion or the target boundary is fuzzy, improve the accuracy of boundary segmentation and the discrimination of class discrimination, and overcome the problems of limited graph structure information expression capability and misclassification in the adhesion scene in the prior art.

[0060] (3) The present application combines the boundary clues output by the local feature extraction branch and the candidate region pixel mask, dynamically generates an edge consistency constraint signal driven by the boundary overlap rate, and jointly modulates the feature aggregation weight in the node feature and subgraph aggregation stage, effectively inhibiting the boundary contour misclassification and missing classification phenomenon caused by target adhesion, fragmentation and occlusion. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the application, and, together with the description, to specify the application without limitation. In the drawings:

[0062] Figure 1 A flow chart of an intelligent construction waste classification method based on deep learning is proposed for the present application. DETAILED DESCRIPTION

[0063] Embodiment 1

[0064] Reference Figure 1 An intelligent construction waste classification method based on deep learning, comprising:

[0065] Collecting construction waste job site image data and pre-processing to obtain a standardized construction waste job site image data set;

[0066] The preprocessing includes fog removal, color constancy, dynamic range compression and noise suppression on the construction waste job site image data.

[0067] Generating a global candidate image region set based on the standardized construction waste job site image data set;

[0068] In this embodiment, generating a candidate image region set based on the standardized construction waste job site image data set comprises:

[0069] For each standardized construction waste job site image in the standardized construction waste job site image data set, a preliminary candidate region set is generated through a candidate region detection strategy;

[0070] The candidate region detection strategy is instance segmentation, superpixel segmentation or region proposal network, and each standardized construction waste job site image generates a preliminary candidate region set through the candidate region detection strategy. Each preliminary candidate region in the preliminary candidate region set contains an enclosing rectangular bounding box and a binary pixel mask. Each standardized construction waste job site image has its corresponding preliminary candidate region set.

[0071] Calculate the target confidence score of each preliminary candidate region in the preliminary candidate region set;

[0072] The target confidence score measures the likelihood of the preliminary candidate region containing a construction waste target.

[0073] Perform a candidate region screening operation based on the target confidence score of each preliminary candidate region, and only keep the candidate regions whose target confidence score is not lower than a confidence threshold. An effective candidate region set is constructed and merged to form a global candidate image region set.

[0074] The effective candidate region set corresponding to each standardized construction waste job site image is composed of all the preliminary candidate regions whose target confidence scores satisfy the condition, and all the preliminary candidate regions that do not satisfy the condition are removed.

[0075] The global candidate image region set includes all the effective candidate regions of all the standardized construction waste job site images, and each effective candidate region includes its bounding box, a binary pixel mask and a corresponding target confidence score.

[0076] In the embodiment 1, a non-maximum suppression method can also be used to remove redundant regions that are highly overlapped and have a target confidence score lower than a threshold value for each candidate region in the global candidate image region set. The global candidate image region set after the overlap suppression processing is used as the only input source for the node feature vector set extraction, each candidate region in the global candidate image region set after the overlap suppression processing is uniquely indexed, and is indexed and mapped with the standardized construction waste job site image dataset, so that each candidate region can be traced back to its original standardized construction waste job site image and its spatial position.

[0077] For each candidate image region in the global candidate image region set, a node feature vector set is formed.

[0078] In the embodiment, the formation of the node feature vector set includes:

[0079] A local texture feature vector of each candidate image region in the global candidate image region set is extracted.

[0080] The local texture feature vector is obtained by a multi-scale convolution filter responding to the candidate image region, and the local texture feature vector represents the surface texture details of the candidate image region.

[0081] An edge and fragmentation feature vector of each candidate image region is extracted.

[0082] In the embodiment 1, an edge detection operation is performed on the pixel set of the candidate image region to obtain an edge pixel set of the candidate image region; an edge gradient distribution of the candidate image region is counted based on the edge pixel set, the edge gradient distribution is obtained by histogram counting of the gradient amplitude and direction of all edge pixels, the number of boundary break points is determined by connectivity analysis on the edge pixel set, the boundary break density is the ratio of the number of boundary break points to the total number of pixels of the candidate image region contour, and the edge gradient distribution and the boundary break density are sequentially spliced according to the feature dimension order to obtain the edge and fragmentation feature vector of the candidate image region.

[0083] A shape statistical feature vector of each candidate image region is extracted.

[0084] The shape statistical feature vector is calculated based on the area, perimeter and aspect ratio of the candidate image region, and represents the geometric shape of the candidate image region.

[0085] extracting a material prior descriptor vector of each candidate image region;

[0086] In the pixel range corresponding to each candidate image region, the average and variance of the multi-channel reflectance of the candidate image region are calculated, the reflectance data is derived from the original pixel values of the construction waste site image in the visible and near-infrared wave bands, the surface roughness of the candidate image region is calculated by statistically calculating the standard deviation of the pixel gray gradient in the candidate image region, the surface texture fluctuation degree is measured by statistically calculating the standard deviation of the pixel gray gradient in the candidate image region, and the brightness pattern of the candidate image region is calculated by normalizing the histogram statistical features of the pixel brightness distribution in the candidate image region, which is used to represent the illumination response characteristics of different materials. The feature values of reflectance, surface roughness and brightness pattern are calculated, and are spliced in a predetermined order to form a material prior descriptor vector of the candidate image region.

[0087] extracting a near-infrared descriptor vector of each candidate image region;

[0088] In Example 1, for each candidate image region, the pixel set of the candidate image region is extracted in the near-infrared imaging data, the average and standard deviation of the reflectance of the pixel set are calculated, the average reflectance represents the near-infrared reflectance characteristics of the candidate image region as a whole, and the standard deviation of the reflectance represents the reflectance variation within the candidate image region; based on the near-infrared pixel set, the local texture features of the candidate image region under near-infrared imaging are extracted using a gray level co-occurrence matrix, the material absorption characteristic parameters of the candidate image region under near-infrared imaging are obtained by ratio analysis of the average values of the pixels under different near-infrared wave band imaging, and the material absorption characteristic parameters are used to distinguish the energy absorption capacity of different construction waste materials in the near-infrared wave band. The average reflectance, the standard deviation of the reflectance, the local texture feature parameters and the material absorption characteristic parameters are spliced in sequence to form a near-infrared descriptor vector corresponding to the candidate image region.

[0089] For each candidate image region, the local texture feature vector, the edge and fragmentation feature vector, the shape statistical feature vector, the material prior descriptor vector and the near-infrared descriptor vector of the candidate image region are spliced in sequence according to the feature dimension order, and the node feature vector of the candidate image region is formed after splicing.

[0090] For all candidate image regions in the global candidate image region set, repeat the steps to obtain a node feature vector set, and the node feature vector set corresponds to each candidate image region in the global candidate image region set.

[0091] obtain a construction waste heterogeneous object-relation graph structure containing node type labels and edge type labels, taking the global candidate image region set as nodes;

[0092] In this embodiment, obtaining a construction waste heterogeneous object-relation graph structure containing node type labels and edge type labels comprises:

[0093] introducing each candidate image region in the global candidate image region set into the node set as a graph node;

[0094] The node set is composed of all global candidate image regions, each graph node is associated with a unique node feature vector, the generation mode of the node feature vector is determined according to the generation mode of the node feature vector set, and the dimension of each node feature vector corresponds to a candidate image region in the global candidate image region set.

[0095] obtaining the geometric center coordinates of any two candidate image regions in the global candidate image region set, and calculating the spatial Euclidean distance of the two candidate image regions in the image plane based on the geometric center coordinates;

[0096] The spatial Euclidean distance measures the spatial proximity of two graph nodes in the image plane, and the unit of the spatial Euclidean distance is pixel.

[0097] obtaining the binary pixel mask of any two candidate image regions in the global candidate image region set respectively, calculating the intersection-over-union value based on the binary pixel mask, and judging whether the intersection-over-union value is greater than the wrapping relationship threshold and whether the containing boundary condition is met, when the intersection-over-union value is greater than the wrapping relationship threshold and the containing boundary condition is met, establishing a wrapping relationship edge;

[0098] The intersection-over-union value is equal to the ratio of the intersection pixel number to the union pixel number of the two binary pixel masks.

[0099] calculating the correlation coefficient, judging whether the correlation coefficient is greater than the texture continuity threshold, and when the correlation coefficient is greater than the texture continuity threshold, establishing a texture continuity relationship edge;

[0100] The correlation coefficient is equal to the sum of the products after the mean value of each feature dimension is removed divided by the product of the standard deviations.

[0101] calculating the cosine similarity based on the material prior descriptor vector, judging whether the cosine similarity is greater than the material affinity threshold, and when the cosine similarity is greater than the material affinity threshold, establishing a material affinity relationship edge;

[0102] The cosine similarity is equal to the inner product of the two material prior descriptor vectors divided by the product of the lengths of the two material prior descriptor vectors.

[0103] The spatial adjacency relationship edge, the package relationship edge, the texture continuity relationship edge and the material affinity relationship edge are uniformly stored as an edge set;

[0104] Based on the node set and the edge set, a construction waste heterogeneous object-relation graph structure is constructed.

[0105] The construction waste heterogeneous object-relation graph structure includes a node set, an edge set, a node type annotation set and an edge type annotation set, the node type annotation set contains a construction waste target category or a candidate area type to which each node belongs, and the edge type annotation set contains a spatial adjacency, a package, a texture continuity and a material affinity relationship type.

[0106] On the standardized construction waste operation site image dataset, a local feature representation is generated by using a local feature extraction branch, a global feature representation is generated by using a global feature extraction branch, and the local-global gating fusion module is inputted, so as to generate a gating coefficient and a fused multi-scale feature representation.

[0107] In the embodiment, generating the gating coefficient and the fused multi-scale feature representation includes:

[0108] The standardized construction waste operation site image dataset is inputted into the local feature extraction branch, the local feature extraction branch uses a multi-scale convolution kernel to extract features of each standardized construction waste operation site image, and a local feature representation set is obtained.

[0109] Each local feature representation is a spatial distribution of fine-grained texture, crack, hole and boundary clues of the corresponding standardized construction waste operation site image.

[0110] The standardized construction waste operation site image dataset is inputted into the global feature extraction branch, the global feature extraction branch uses a global context modeling structure to extract features of each standardized construction waste operation site image, and a global feature representation set is obtained.

[0111] Each global feature representation is an overall spatial feature of scene semantics, yard layout and light distribution of the corresponding standardized construction waste operation site image.

[0112] The local feature representation and the global feature representation are inputted into the local-global gating fusion module, the local-global gating fusion module performs spatial size alignment and channel number matching on the local feature representation and the global feature representation, and obtains an aligned local feature representation and an aligned global feature representation.

[0113] In Embodiment 1, the local feature representation and the global feature representation are respectively input into a spatial size adjustment module and a channel number matching module of the local-global gated fusion module, the spatial size adjustment module adjusts the local feature representation and the global feature representation to the same spatial size by using a bilinear interpolation method, the channel number matching module adjusts the local feature representation and the global feature representation to the same channel number by using a one-dimensional convolution, and an aligned local feature representation and an aligned global feature representation are obtained, which are one-to-one corresponding to the standardized construction waste job site image in terms of spatial size and channel number.

[0114] The local-global gated fusion module performs channel dimension splicing on the aligned local feature representation and the aligned global feature representation, inputs the splicing result into a linear mapping and a Sigmoid function, and generates a gating coefficient map;

[0115]

[0116] where [·; ·] represents feature splicing in the channel dimension, W g is a fusion weight matrix, b g is a fusion bias term, and σ(·) is a Sigmoid function. k g is the aligned local feature representation, is the aligned global feature representation.

[0117] The aligned local feature representation and the aligned global feature representation are weighted fused by using the gating coefficient map, the aligned local feature representation is multiplied element by element with the gating coefficient map, the aligned global feature representation is multiplied element by element with a complement of the gating coefficient map, and corresponding elements are added to generate a fused multi-scale feature representation;

[0118]

[0119] where ⊙ represents an element-wise multiplication operation, and f fusion,k is the fused multi-scale feature representation.

[0120] The node feature vector set and the fused multi-scale feature representation are concatenated as an initial node state to input an improved graph attention network V2 model to perform multi-head attention message passing and node updating, to obtain an updated node representation and an edge attention weight, and to perform subgraph aggregation in the construction waste heterogeneous object-relation graph structure to generate a subgraph-level representation.

[0121] In the present embodiment, the generation of the subgraph-level representation includes:

[0122] The node feature vector set and the fused multi-scale feature representation set are subjected to one-to-one feature splicing operation according to the node index, as an initial node state set;

[0123] Each initial node state is composed of the node feature vector of the node and the fused multi-scale feature representation of the node spliced in order of feature dimensions, and the node feature vector is a representation of the candidate image region.

[0124] For any pair of adjacent nodes, a consistency modulation coefficient is obtained by calculating a material consistency coefficient, a shape consistency coefficient and a spatial consistency coefficient respectively and weighted sum according to a set weight, the material consistency coefficient is used to measure the similarity of two nodes in the material prior descriptor, the shape consistency coefficient is used to measure the similarity of two nodes in the shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity of two nodes in the image plane.

[0125] In embodiment 1, the material consistency coefficient is obtained by taking the inner product of the material prior descriptor vectors of the two nodes and then dividing the product of the module lengths of the two material prior descriptor vectors, the shape consistency coefficient is obtained by taking the ratio of the difference between the shape compactness of the two nodes and the larger one and then subtracting one, and the spatial consistency coefficient is obtained by taking the opposite number of the ratio of the spatial Euclidean distance of the two nodes and the spatial decay constant and then performing exponential operation.

[0126] In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded as an attention prior modulation factor in the attention score calculation between each pair of nodes to obtain an improved attention coefficient.

[0127] Embedding the consistency modulation coefficient as an attention prior modulation factor in the attention score calculation between each pair of nodes makes the attention score between nodes not only jointly affected by node features and fused multi-scale features, but also jointly affected by material consistency, shape consistency and spatial consistency between nodes, and the attention score between nodes is adaptively adjusted with the change of the consistency modulation coefficient, thereby improving the ability to distinguish the fragmentation, adhesion and mixed material phenomenon of construction waste.

[0128]

[0129] wherein, represents the improved attention coefficient of the neighbor node v i of the node v j . ij represents the improved attention coefficient of the neighbor node v i . jThe consistency modulation coefficient between the two nodes in the building waste heterogeneous object-relation graph structure comprehensively reflects the consistency degree of the two nodes in terms of material properties, geometric morphology and spatial adjacency relationship. The greater the consistency modulation coefficient is, the more likely it is that the two nodes belong to the same type or are physically connected building waste targets in the actual construction site, and the more reliable the information transmission is. (k) (k) represents the linear transformation parameter corresponding to the kth head of the graph attention network V2, which learns the importance weight of the building waste node in the feature dimension, represents the multi-modal feature vector of the node v i j in the lth layer, and σ(·) represents the LeakyReLU activation function, which introduces a nonlinear expression capability to adapt to the complex morphology change of the building waste. N(i) represents the set of adjacent nodes of the node v i , which represents other regions in the graph structure that have direct physical or semantic contact with the building waste candidate region.

[0130] The improved attention coefficient is used to update the features of all nodes. The aggregation weight of the node features is determined by the node features, the fused multi-scale features and the consistency modulation coefficient between the nodes in each layer of the attention mechanism. The feature outputs of all heads are spliced to obtain a new node feature vector.

[0131]

[0132] wherein, represents the updated node v i feature vector, K represents the total number of attention heads, and || represents the multi-head feature splicing operation.

[0133] In the subgraph aggregation stage, based on the consistency threshold, only when the consistency modulation coefficients between all nodes in the subgraph are not lower than the set consistency threshold, the nodes are allowed to be merged into the same subgraph level representation.

[0134] In Example 1, each node feature vector in the updated node feature vector set is used as the only input feature vector for subgraph aggregation, and the edge consistency modulation coefficient is used to divide the nodes into candidate subgraphs. When dividing the candidate subgraphs, only when the edge consistency modulation coefficients of any two nodes in the candidate subgraph are not lower than the consistency threshold, the nodes are allowed to be divided into the same candidate subgraph. For each candidate subgraph that meets the condition, all node feature vectors in the candidate subgraph are input into the feature aggregation function for feature aggregation. The feature aggregation function is a weighted average function, and the output of the feature aggregation is used as the aggregated representation vector of the candidate subgraph.

[0135] ​​The edge consistency constraint signal is generated based on a boundary clue output by the local feature extraction branch and a pixel mask of a global candidate image region set;

[0136] In this embodiment, the edge consistency constraint signal is generated, including:

[0137] For each candidate image region in the global candidate image region set and the boundary clue set output by the local feature extraction branch, a boundary intensity map of the candidate image region in the boundary clue set output by the local feature extraction branch and a binary pixel mask of the candidate image region are obtained respectively.

[0138] In example 1, each image in the standardized construction waste working site image dataset is input into the local feature extraction branch to obtain a boundary response map, and the boundary response map is subjected to non-maximum suppression and threshold segmentation to obtain a boundary clue map.

[0139] The pixel mask information of each candidate image region in the global candidate image region set is directly extracted from the candidate region detection strategy to obtain a binary pixel mask of the candidate image region, and each candidate image region is corresponded to its spatial position in the boundary clue map through node indexing, so that each candidate image region is uniquely associated with a boundary clue map and a binary pixel mask.

[0140] The boundary intensity map is subjected to thresholding processing to obtain a boundary pixel set of the candidate image region, and the binary pixel mask is subjected to edge extraction to obtain a contour pixel set of the candidate image region, and the number of overlapping pixels of the boundary pixel set and the contour pixel set of the candidate image region is counted, and the number of overlapping pixels is divided by the number of pixels of the contour pixel set to obtain a boundary overlap rate of the candidate image region.

[0141] The boundary overlap rate measures the spatial overlap degree of the boundary prediction of the candidate image region and the real contour, and the boundary overlap rate has a value range of 0 to 1.

[0142] The boundary overlap rate of the candidate image region is compared with a preset edge consistency threshold value, when the boundary overlap rate is greater than or equal to the edge consistency threshold value, the edge consistency constraint signal of the candidate image region is set to 1, and when the boundary overlap rate is less than the edge consistency threshold value, the edge consistency constraint signal of the candidate image region is set to the ratio of the boundary overlap rate to the edge consistency threshold value.

[0143] The edge consistency constraint signal is used to measure the consistency degree of the boundary clue of the candidate image region and the contour pixel.

[0144] The construction waste class label, the instance contour and the segmentation mask are defined according to the updated node representation, the subgraph level representation and the edge consistency constraint signal.

[0145] In this embodiment, the construction waste class label, instance contour and segmentation mask are defined according to the updated node representation, subgraph level representation and edge consistency constraint signal, including:

[0146] The construction waste class label includes:

[0147] When the concrete response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the volume greater than a threshold and the compactness higher than a threshold, the output class label is concrete;

[0148] When the steel bar response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the shape statistical feature vector satisfies the elongated shape and the boundary fracture density higher than a threshold, the output class label is steel bar;

[0149] When the wood response component in the material prior descriptor vector of the candidate image region is greater than a set threshold, and the local texture feature vector shows fibrous or annual ring texture, the output class label is wood;

[0150] The output of the remaining class labels is determined according to the preset material attribute and shape feature matching rule;

[0151] The instance contour output rule is that for each candidate image region, all external contour point sequences are obtained from the connected pixel set in the segmentation mask, and the external contour point sequences are arranged and output in spatial order, and the instance contour represents the boundary shape of the construction waste target.

[0152] The segmentation mask output rule is that for each candidate image region, a binary segmentation mask consistent with the normalized construction waste work site image spatial resolution is generated, and the pixel value of the binary segmentation mask is 1, indicating that the pixel belongs to the construction waste target, and the pixel value of 0 indicates that the pixel belongs to the background area.

[0153] Embodiment 2:

[0154] In the actual application process of the construction waste intelligent sorting line, the system is deployed in an intelligent sorting pipeline containing multiple high-definition cameras and infrared auxiliary modules. Under the scene, the construction waste continuously passes through the belt line, forming a large number of randomly stacked, mixedly shielded, fragmented damaged and similar material waste areas. The system automatically collects the original video stream, extracts several frames per second as static image samples for processing.

[0155] The system performs dehazing and color normalization on the collected images to eliminate visual noise caused by dust, mechanical lighting, and rain reflection. In Embodiment 2, during a night shift operation, a frame of image has the overall color tone darkened due to the concrete and old wooden boards on the belt being covered with oil stains. After preprocessing, the image contrast is significantly improved, and the boundary contour is clearer. Subsequently, through the candidate region detection module, the system automatically segments 39 candidate regions on the image, covering rusted steel bars, broken concrete blocks, partially remaining insulation boards, and damaged wood. Each candidate region generates a unique bounding rectangle and a pixel-level binary mask.

[0156] The system further extracts multi-scale convolutional texture features, shape statistical features, material prior descriptors, and infrared reflection features for each region. In Embodiment 2, the candidate region numbered C019 has high-frequency irregular texture as its surface texture feature, a length-width ratio of 4.3, a compactness of 0.62, and an average infrared reflectivity of 0.41 in the material prior. The system determines that this region is a rusted steel bar with a confidence of 0.93. The region numbered C022 has an area less than 500 pixels, uniform gray distribution, porous texture features, and low reflectivity in the material prior, which, combined with the morphological features, determines it as a concrete fragment.

[0157] In the heterogeneous object-relation graph construction stage, the system establishes multi-class edges between the 39 candidate regions based on spatial adjacency relationships, boundary wrapping relationships, material affinity, and texture continuity. In Embodiment 2, there is an adjacency relationship between C019 and C022 with a spatial distance of only 8 pixels, and the material prior descriptor similarity is 0.48, the morphological consistency is 0.79, and the spatial consistency coefficient is 0.91. The system inputs these parameters into the multi-head dynamic modulation module of the improved graph attention network V2.

[0158] After multiple iterations of multi-head attention updates, the system automatically identifies C019 and C022 as adherent but different material instances, preventing missegmentation. Simultaneously, the system compares the boundary clues output by the local feature branch with the candidate region mask, calculates the boundary overlap rate of the C022 region as 0.88, which is greater than the set threshold of 0.85, and sets the edge consistency constraint signal of this region to 1. The C024 region has a boundary contour that is fuzzy, with a boundary overlap rate of only 0.57, and the edge consistency constraint signal is set to 0.67, resulting in a reduced weight of this region during node updating and subgraph aggregation, thereby avoiding misjudgment.

[0159] The classification and segmentation prediction module integrates the final features and constraints of each region to output the following detection results:

[0160] The C019 region is classified as a steel bar, with a position boundary box of (230, 421)-(254, 520), an instance contour point number of 68, and a segmentation mask pixel number of 542.

[0161] C022 region category is determined as concrete, position bounding box is (318, 442)-(357, 480), instance contour point number is 43, and segmentation mask pixel number is 335.

[0162] C027 region category is determined as wood, boundary fracture density is 0.23, instance contour point number is 76, and segmentation mask pixel number is 682.

[0163] The overall image segmentation average IoU is 0.91, the boundary F1 is 0.89, and the missed detection rate is 2.5%.

[0164] To verify the performance advantage of the method, the implementer selects 300 representative working line images of the same batch to perform a comparative experiment of the traditional Mask R-CNN and the method, the number of steel bar instances in the training sample is 2638, the number of concrete instances is 4151, the number of wood instances is 1797, the number of insulation board instances is 812, and the number of brick instances is 2695. The test data contains no less than 350 real labeled instances of each type of garbage, and the actual comparison results are as follows:

[0165] The average IoU of the steel bar category of the method is 0.91, the average IoU of the concrete is 0.89, the average IoU of the wood is 0.87, the average IoU of the insulation board is 0.83, the average IoU of the brick is 0.85, the average missegmentation rate is 3.4%, the average missed detection rate is 2.6%, and the segmentation boundary F1 average is 0.90.

[0166] The average IoU of the steel bar category of the traditional method is 0.83, the average IoU of the concrete is 0.82, the average IoU of the wood is 0.78, the average IoU of the insulation board is 0.71, the average IoU of the brick is 0.74, the average missegmentation rate is 9.7%, the average missed detection rate is 8.2%, and the segmentation boundary F1 average is 0.82.

[0167] In the example 2, in one stacked image, the method can correctly distinguish 3 close concrete blocks and 2 partially occluded steel bars, and accurately segment the small adhesive fragments of the insulation board edge, while the traditional method mismerges a steel bar with concrete and misses the small fragments of the insulation board.

[0168] In the real-time deployment test of the sorting line, the single-frame processing delay of the system is less than 65 milliseconds, supports actual 60 frames / second streaming input, and the accuracy rate of automatically pushing the sorting instruction is more than 96%. In the part of the artificial review link, the artificial confirmation that the automatic segmentation suggestion of the system is consistent with the actual physical sorting result reaches more than 97%.

[0169] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for intelligent classification of construction waste based on deep learning, characterized in that: include: Collect construction waste operation site image data and preprocess it to obtain a standardized construction waste operation site image dataset; Generate a global candidate image region set based on a standardized construction waste operation site image dataset; For each candidate image region in the global candidate image region set, a node feature vector set is formed; Taking the global candidate image region set as nodes, a heterogeneous object-relationship graph structure of construction waste containing node type annotations and edge type annotations is obtained; On a standardized construction waste operation site image dataset, a local feature extraction branch is used to generate local feature representations, and a global feature extraction branch is used to generate global feature representations. These representations are then fed into a local-global gating fusion module to generate gating coefficients and fused multi-scale feature representations. The node feature vector set and the fused multi-scale feature representation are concatenated as the initial node state input to improve the Graph Attention Network V2 model to perform multi-head attention message passing and node update, obtain updated node representations and edge attention weights, and perform subgraph aggregation in the heterogeneous object-relationship graph structure of construction waste to generate subgraph-level representations; Generate edge consistency constraint signals based on the boundary clues output by the local feature extraction branch and the pixel mask of the global candidate image region set; The construction waste category labels, instance outlines and segmentation masks are defined based on the updated node representation, subgraph-level representation and edge consistency constraint signals.

2. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: The generating of a candidate image region set based on a standardized construction waste operation site image dataset includes: For each standardized construction waste operation site image in the standardized construction waste operation site image dataset, a preliminary candidate region set is generated through the candidate region detection strategy; Calculate the target confidence score of each preliminary candidate region in the preliminary candidate region set; Based on the target confidence score of each preliminary candidate region, a candidate region screening operation is performed, and only candidate regions with target confidence scores not lower than the confidence threshold are retained. A set of valid candidate regions is constructed and merged to form a global candidate image region set.

3. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: The formation of the node feature vector set includes: Extracting a local texture feature vector of each candidate image region in the global candidate image region set; Extract edge and fragmentation feature vectors of each candidate image region; Extracting the shape statistical feature vector of each candidate image region; Extract the material prior descriptor vector of each candidate image region; Extract the near-infrared descriptor vector of each candidate image region; For each candidate image region, the local texture feature vector, edge and fragmentation feature vector, shape statistics feature vector, material prior descriptor vector and near-infrared descriptor vector of the candidate image region are spliced ​​in sequence according to the feature dimension order to form the node feature vector of the candidate image region.

4. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: The obtaining of the construction waste heterogeneous object-relationship graph structure including node type annotations and edge type annotations includes: Each candidate image region in the global candidate image region set is introduced into the node set as a graph node; Obtaining the geometric center coordinates of any two candidate image regions in the global candidate image region set, and calculating the spatial Euclidean distance between the two candidate image regions in the image plane based on the geometric center coordinates; Obtain the binary pixel masks of any two candidate image regions in the global candidate image region set respectively, calculate the intersection-union ratio based on the binary pixel masks, and determine whether the intersection-union ratio is greater than the wrapping relationship threshold and whether the inclusion boundary condition is met. When the intersection-union ratio is greater than the wrapping relationship threshold and meets the inclusion boundary condition, establish a wrapping relationship edge; Calculate the correlation coefficient and determine whether the correlation coefficient is greater than the texture continuity threshold. When the correlation coefficient is greater than the texture continuity threshold, establish a texture continuity relationship edge. Calculate the cosine similarity based on the material prior descriptor vector and determine whether the cosine similarity is greater than the material affinity threshold. When the cosine similarity is greater than the material affinity threshold, establish a material affinity relationship edge. The spatial adjacency relationship edges, wrapping relationship edges, texture continuity relationship edges and material affinity relationship edges are uniformly stored as an edge set; Based on the node set and edge set, a heterogeneous object-relationship graph structure of construction waste is constructed.

5. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: The generating of the gating coefficients and the fused multi-scale feature representation includes: The standardized construction waste operation site image dataset is input into the local feature extraction branch. The local feature extraction branch uses a multi-scale convolution kernel to extract features from each standardized construction waste operation site image to obtain a local feature representation set. The standardized construction waste operation site image dataset is input into the global feature extraction branch. The global feature extraction branch uses the global context modeling structure to extract features from each standardized construction waste operation site image to obtain a global feature representation set. The local feature representation and the global feature representation are input into the local-global gated fusion module, which performs spatial size alignment and channel number matching on the local feature representation and the global feature representation to obtain the aligned local feature representation and the aligned global feature representation; The local-global gating fusion module concatenates the aligned local feature representation and the aligned global feature representation in the channel dimension, inputs the concatenation result into the linear mapping and Sigmoid function, and generates a gating coefficient map. The gated coefficient map is used to perform weighted fusion on the aligned local feature representation and the aligned global feature representation. The aligned local feature representation is multiplied element-by-element by the gated coefficient map, and the aligned global feature representation is multiplied element-by-element by the complement of the gated coefficient map. The corresponding elements are summed to generate a fused multi-scale feature representation.

6. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: Generating a subgraph-level representation includes: Perform a one-to-one feature concatenation operation on the node feature vector set and the fused multi-scale feature representation set according to the node index as the initial node state set; For any pair of adjacent nodes, the material consistency coefficient, morphological consistency coefficient and spatial consistency coefficient are calculated respectively, and weighted summed according to the set weight to obtain the consistency modulation coefficient. The material consistency coefficient is used to measure the similarity of the two nodes in the material prior descriptor, the morphological consistency coefficient is used to measure the similarity of the two nodes in shape compactness, and the spatial consistency coefficient is used to measure the spatial proximity of the two nodes in the image plane. In the multi-head attention calculation of the improved graph attention network V2 model, the consistency modulation coefficient is embedded as the attention prior modulation factor into the attention score calculation between each pair of nodes to obtain the improved attention coefficient; The improved attention coefficient is used to update the features of all nodes. Each layer of attention mechanism determines the aggregation weight of node features based on the node features between nodes, the fused multi-scale features and the consistency modulation coefficient. The feature outputs of all heads are spliced ​​to obtain a new node feature vector. In the subgraph aggregation stage, each node feature vector in the updated node feature vector set is used as the only input feature vector for subgraph aggregation. Based on the consistency threshold, nodes are only allowed to be merged into the same subgraph level representation when the consistency modulation coefficients between all nodes in the subgraph are not lower than the set consistency threshold.

7. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: The generating of the edge consistency constraint signal comprises: For each candidate image region in the boundary clue set output by the local feature extraction branch and the global candidate image region set, respectively obtain a boundary intensity map of the candidate image region in the boundary clue set output by the local feature extraction branch and a binary pixel mask of the candidate image region; The boundary intensity map is thresholded to obtain a boundary pixel set of the candidate image region, the binary pixel mask is edge extracted to obtain a contour pixel set of the candidate image region, and the number of overlapping pixels between the boundary pixel set and the contour pixel set of the candidate image region is divided by the number of pixels in the contour pixel set to obtain the boundary overlap ratio of the candidate image region; The boundary overlap rate of the candidate image area is compared with the preset edge consistency threshold. When the boundary overlap rate is greater than or equal to the edge consistency threshold, the edge consistency constraint signal of the candidate image area is set to 1. When the boundary overlap rate is less than the edge consistency threshold, the edge consistency constraint signal of the candidate image area is set to the ratio of the boundary overlap rate to the edge consistency threshold.

8. The method for intelligent classification of construction waste based on deep learning according to claim 1 is characterized in that: Defining the construction waste category label, instance outline, and segmentation mask based on the updated node representation, subgraph-level representation, and edge consistency constraint signal includes: Construction waste category label: When the concrete response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the shape statistical feature vector satisfies the conditions that the volume is greater than the threshold and the compactness is higher than the threshold, the output category label is concrete; When the steel bar response component in the material prior descriptor vector of the candidate image region is greater than the set threshold, and the shape statistical feature vector satisfies the slender form and the boundary fracture density is higher than the threshold, the output category label is steel bar; When the wood response component in the material prior descriptor vector of the candidate image region is greater than the set threshold and the local texture feature vector shows fibrous or annual ring texture, the output class label is wood; The output of the remaining category labels is judged according to the preset material attribute and morphological feature matching rules; Instance contour output rule: For each candidate image region, all external contour point sequences are obtained based on the connected pixel set in the segmentation mask, and the external contour point sequences are arranged and output in spatial order. The instance contour represents the boundary shape of the construction waste target; Segmentation mask output rule: For each candidate image area, a binary segmentation mask consistent with the spatial resolution of the standardized construction waste operation site image is generated. The pixel value in the binary segmentation mask is 1, indicating that the pixel belongs to the construction waste target, and the pixel value is 0, indicating that the pixel belongs to the background area.

Images