Semantic segmentation method and system for construction waste based on geometry and texture coupling

Abstract

The present application provides a method and system for semantic segmentation of construction waste based on geometry and texture coupling. The method includes: acquiring an image sequence when illumination pulsation and dust particle size meet preset conditions; determining a high-quality frame image based on the image sequence; performing coordinate transformation and projection processing on the high-quality frame image to determine an alignment tensor; performing a three-dimensional gradient operation on the alignment tensor to determine a dynamic convolution anchor point; determining a first pseudo-label map based on the dynamic convolution anchor point; determining a high-confidence label map based on the first pseudo-label map; determining a second pseudo-label map based on the high-confidence label map; determining a fused label map based on the first pseudo-label image and the second pseudo-label map; determining a pile support relationship map based on the fused label map; and determining a final construction waste semantic segmentation map, instance volume, and weight list based on the pile support relationship map. The above scheme can improve the accuracy of construction waste semantic segmentation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for semantic segmentation of construction waste based on geometry and texture coupling. Background Art

[0002] With the accelerated pace of urban renewal and renovation projects, large quantities of construction waste are generated in a short period of time. This not only takes up storage space but also hinders direct recycling due to the presence of heterogeneous materials such as rebar, broken bricks, slag, and glass. Traditional sorting methods, which rely on manual labor or single visual sensors, are limited by on-site factors such as high dust levels, strong light fluctuations, and significant differences in material reflectivity. This often results in low recognition accuracy, inefficiency, and high operational risks. Summary of the Invention

[0003] The embodiments of the present application hope to provide a method and system for semantic segmentation of construction waste based on geometry and texture coupling, which can improve the accuracy of semantic segmentation of construction waste.

[0004] The technical solution of the present invention is achieved as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for semantic segmentation of construction waste based on geometry and texture coupling, the method comprising:

[0006] Acquire an image sequence when the light pulsation and the dust particle size meet preset conditions; construct an environmental index model based on the image sequence; and determine a high-quality frame image using the environmental index model;

[0007] Based on the high-quality frame image, coordinate transformation and projection processing are performed to determine an alignment tensor; and based on the alignment tensor, a three-dimensional gradient operation is performed to determine a dynamic convolution anchor point; based on the dynamic convolution anchor point, a first pseudo label map is determined;

[0008] Based on the first pseudo label map, pixel processing is performed on each material cluster to determine a fine-grained segmentation map; and the fine-grained segmentation map is fused with the first pseudo label map to determine a high-confidence label map;

[0009] Based on the high-confidence label map, a steel bar broken line anchor point and a slag peak anchor point are determined; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, a second pseudo label map is determined; based on the second pseudo label map, a symmetric divergence is calculated to determine a fused label map;

[0010] Based on the fused label map, the pixel segmentation contours of adjacent materials are extracted to determine a pile support relationship map; based on the pile support relationship map, the final construction waste semantic segmentation map, instance volume and weight list are determined.

[0011] In the above scheme, when the light pulsation and dust particle size meet preset conditions, an image sequence is acquired; an environmental index model is constructed based on the image sequence; and a high-quality frame image is determined by using the environmental index model, including:

[0012] When the light pulsation and the dust particle size meet preset conditions, synchronous line scanning is performed using an RGB camera, a structured light depth camera, and a near-infrared camera to acquire the image sequence;

[0013] constructing the environmental index model based on the visible light grayscale matrix and the near-infrared reflectance matrix of the image sequence; and obtaining environmental parameters through the environmental index model;

[0014] When the environmental parameters meet the preset threshold, near-infrared images at three different angles are collected;

[0015] The high-quality frame image is determined based on the near-infrared images at different angles.

[0016] In the above solution, determining the high-quality frame image based on the near-infrared images at different angles includes:

[0017] Calculate triangular polarization differences based on the near-infrared images at the three different angles, and determine the metal highlight label, the rough diffuse reflection label, and the glass mirror label based on the triangular polarization differences and a preset threshold;

[0018] Generate a roughness prior map based on the metal highlight label, the rough diffuse reflection label, and the glass mirror label;

[0019] Based on the roughness prior map, a comprehensive quality score is calculated; and the high-quality frame image is determined by the comprehensive quality score.

[0020] In the above solution, performing a three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; and determining a first pseudo label map based on the dynamic convolution anchor point includes:

[0021] Calculating a three-dimensional gradient based on the alignment tensor; and dividing pixels in the high-quality frame image into a sharp edge region and a gentle slope region according to the three-dimensional gradient and a step threshold;

[0022] Extracting the same number of pixels in the sharp edge area and the gentle slope area respectively to determine the dynamic convolution anchor point;

[0023] Based on the dynamic convolution anchor point, common features are determined, and low-entropy aggregation is performed on the common features to determine the first pseudo label map.

[0024] In the above solution, performing pixel processing on each material cluster based on the first pseudo-label map to determine a fine-grained segmentation map includes:

[0025] Based on the first pseudo-label image, for each material cluster, calculate the surface roughness and normalized reflectance pixel by pixel;

[0026] Performing bucketing and aggregation processing on the surface roughness and the normalized reflectivity through a two-dimensional grid to obtain a target matrix corresponding to each material cluster; wherein the target matrix is ​​determined by the surface roughness and the mean and variance of the reflectance quadruple;

[0027] The roughness means in the target matrix are sorted in ascending order and modulated by the reflectivity mean intensity, the target matrix is ​​mapped to a discrete diffusion step, and a noise intensity table is generated; and the noise intensity table is embedded in a path to determine the fine-grained segmentation map.

[0028] In the above solution, determining the second pseudo label map based on the steel bar broken line anchor point, the slag peak anchor point, and the high-confidence label map includes:

[0029] Determining a sparse core mask according to the steel bar broken line anchor point and the slag peak anchor point;

[0030] Perform pixel-by-pixel comparison based on the sparse core mask, the fine-grained segmentation map, and the high-confidence label map. If there is a category shift, perform backtracking correction to determine a repaired label image.

[0031] Generate a material-sensitive soft label probability map based on the repaired label image, normalized near-infrared reflectance, and dust index;

[0032] The material-sensitive soft label probability map is input into a geometry and texture collaborative network for reasoning to determine the second pseudo label map.

[0033] In the above solution, based on the fused label map, pixel segmentation contours of adjacent materials are extracted to determine a pile support relationship map; based on the pile support relationship map, a final semantic segmentation map of construction waste, instance volume, and weight list are determined, including:

[0034] Extracting pixel segmentation contours of adjacent materials based on the fused label map; and calculating support degrees based on the pixel segmentation contours to determine the pile support relationship map;

[0035] For the image sequence, calculating spatiotemporal stability and morphological variance in two consecutive frames of images to generate a confidence index;

[0036] Based on the pile support relationship diagram and the confidence index, parameter updating and multi-scale forward reasoning are performed to determine the final construction waste semantic segmentation map, the instance volume and the weight list.

[0037] In a second aspect, an embodiment of the present application provides a construction waste semantic segmentation system based on geometry and texture coupling, comprising: an acquisition unit and a determination unit, wherein:

[0038] The acquisition unit is configured to acquire an image sequence when the light pulsation and the dust particle size meet preset conditions; construct an environmental index model based on the image sequence; and determine a high-quality frame image using the environmental index model;

[0039] The determination unit is used to perform coordinate transformation and projection processing based on the high-quality frame image to determine an alignment tensor; and perform three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; determine a first pseudo label map based on the dynamic convolution anchor point; perform pixel processing on each material cluster based on the first pseudo label map to determine a fine-grained segmentation map; and fuse the fine-grained segmentation map and the first pseudo label map to determine a high-confidence label map; determine a steel bar broken line anchor point and a slag peak anchor point based on the high-confidence label map; and determine a second pseudo label map based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map; calculate the symmetric divergence based on the second pseudo label map to determine a fused label map; extract the pixel segmentation contours of adjacent materials based on the fused label map to determine a pile support relationship map; and determine a final semantic segmentation map, instance volume and weight list of construction waste based on the pile support relationship map.

[0040] In a third aspect, an embodiment of the present application provides a construction waste semantic segmentation device based on geometry and texture coupling, the construction waste semantic segmentation device based on geometry and texture coupling includes: a processor and a memory; wherein,

[0041] The memory is used to store computer programs;

[0042] The processor is configured to call and run the computer program from the memory to execute the method according to the first aspect.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the method described in the first aspect.

[0044] The embodiment of the present application provides a method and system for semantic segmentation of construction waste based on geometry and texture coupling, the method comprising: acquiring an image sequence when light pulsation and dust particle size meet preset conditions; constructing an environmental index model based on the image sequence; and determining a high-quality frame image through the environmental index model; performing coordinate transformation and projection processing based on the high-quality frame image to determine an alignment tensor; and performing a three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; determining a first pseudo-label map based on the dynamic convolution anchor point; and performing pixel processing on each material cluster based on the first pseudo-label map. The method determines a fine-grained segmentation map; and fuses the fine-grained segmentation map with the first pseudo-label map to determine a high-confidence label map; based on the high-confidence label map, determines the steel bar broken line anchor point and the slag peak anchor point; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, determines the second pseudo-label map; based on the second pseudo-label map, calculates the symmetric divergence to determine the fused label map; based on the fused label map, extracts the pixel segmentation contours of adjacent materials and determines the pile support relationship map; based on the pile support relationship map, determines the final construction waste semantic segmentation map, instance volume and weight list. In the above scheme, with the environmental index model as the core, millisecond-level synchronous exposure and multi-angle polarization imaging of three cameras are achieved in the scene of light and dust fluctuations, avoiding timing mismatch and material information loss from the acquisition link; three-dimensional gradient-driven dynamic anchor sampling and dual-branch gated dimensionality reduction mechanism replace traditional uniform sampling and single-channel convolution feature extraction, so that fine steel bars and large areas of slag can simultaneously obtain targeted receptive fields. Based on the dynamic convolution anchor point, the first pseudo label map is determined; pixel processing is performed on each material cluster to determine a fine-grained segmentation map; the fine-grained segmentation map and the first pseudo label map are fused to determine a high-confidence label map; based on the high-confidence label map, the steel bar broken line anchor point and the slag peak anchor point are determined; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, the second pseudo label map is determined; based on the second pseudo label map, the symmetric divergence is calculated to determine the fused label map; the fused label map contour is back-projected onto the point cloud and the support is evaluated. Compared with the approach of judging only by height or distance thresholds, a mixed feature of geometry and physical properties is constructed, which shows structural advantages in terms of data alignment accuracy, material differentiation ability and support judgment dimension, and can improve the accuracy of semantic segmentation of construction waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, serve to illustrate the technical solutions of the present application. Obviously, the drawings described below are merely some embodiments of the present application. Those skilled in the art can, without inventive effort, derive other drawings from these drawings.

[0046] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0047] Figure 1 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 1 ;

[0048] Figure 2 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 2 ;

[0049] Figure 3 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 3 ;

[0050] Figure 4 A schematic diagram of the structure of a construction waste semantic segmentation system based on geometry and texture coupling provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a construction waste semantic segmentation device based on geometry and texture coupling provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] In the following description, references to “some embodiments,” “this embodiment,” “embodiments of the present application,” and examples, etc., describe a subset of all possible embodiments. However, it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0055] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0056] This application embodiment provides a construction waste semantic segmentation method based on geometry and texture coupling. Figure 1 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 1 , will combine Figure 1 The steps shown are explained.

[0057] S101. When the light pulsation and the dust particle size meet preset conditions, an image sequence is acquired; an environmental index model is constructed based on the image sequence; and a high-quality frame image is determined through the environmental index model.

[0058] In some embodiments of the present application, an image sequence refers to an ordered set of images obtained through continuous spatiotemporal acquisition, and here refers to continuous frames of building garbage images.

[0059] In some embodiments of the present application, a construction waste semantic segmentation method based on geometry and texture coupling is suitable for construction waste classification scenarios.

[0060] In some embodiments of the present application, the execution body of the construction waste semantic segmentation method based on geometry and texture coupling is a construction waste semantic segmentation device based on geometry and texture coupling.

[0061] In some embodiments of the present application, when illumination pulsation and dust particle size meet preset conditions, synchronized line scanning is performed using an RGB camera, a structured light depth camera, and a near-infrared camera to acquire an image sequence. Based on the visible light grayscale matrix and near-infrared reflectance matrix of the image sequence, an environmental index model is constructed; environmental parameters are obtained using the environmental index model. When the environmental parameters meet preset thresholds, near-infrared images are captured at three different angles. Based on the near-infrared images at different angles, a high-quality frame image is determined.

[0062] For example, in a storage yard environment with fast light pulsation and wide dust particle size distribution, a coaxially arranged RGB camera, a structured light depth camera and a near-infrared camera are used to perform synchronous line scanning at 120 frames per second, and the average brightness and scale-normalized scattering ratio of the visible light grayscale matrix and the near-infrared reflectance matrix of continuous frames are calculated to construct an environmental index model and judge the light stability and dust uniformity; when the environmental index meets the preset threshold, the 0.5s acquisition window is locked, and a unified exposure vector is generated according to the index and the camera gain table, and the three-way camera exposure is triggered simultaneously, and then the polarizer is driven to collect three near-infrared images at angles of 0°, 60° and 120° respectively, and the triangular polarization difference is calculated and the metal highlight, rough diffuse reflection and glass mirror labels are generated according to the threshold to obtain a roughness prior map; the comprehensive quality score is calculated by combining the dust scattering index, polarization angle integrity and depth echo rate, and the frames above the first threshold are written into the high-quality sample pool to obtain high-quality frame images.

[0063] S102. Based on the high-quality frame image, coordinate transformation and projection processing are performed to determine an alignment tensor; and based on the alignment tensor, a three-dimensional gradient operation is performed to determine a dynamic convolution anchor point; based on the dynamic convolution anchor point, a first pseudo-label map is determined.

[0064] In some embodiments of the present application, based on a high-quality frame image, the three-dimensional coordinates of the pixels are converted to an RGB coordinate system, and then projected to determine an alignment tensor. Based on the alignment tensor, a three-dimensional gradient is calculated; and based on the three-dimensional gradient and the step threshold, the pixels in the high-quality frame image are divided into a sharp edge region and a gentle slope region. The same number of pixels are extracted from the sharp edge region and the gentle slope region to determine a dynamic convolution anchor point. Based on the dynamic convolution anchor point, common features are determined, and low-entropy aggregation is performed on the common features to determine a first pseudo-label map.

[0065] Exemplarily, based on the output high-quality frame image, the pixel three-dimensional coordinates are converted to the RGB coordinate system according to the depth camera extrinsic parameters and re-projected to the RGB pixel plane to obtain a point-pixel-reflection alignment tensor; the three-dimensional gradient is calculated and the sharp edge area and the gentle slope area are distinguished according to the step threshold, and one-tenth of the pixel is randomly selected in each of the two types of areas as a dynamic convolution anchor point; the depth gradient vector of the anchor point is spliced ​​with the normalized polarized reflection residual, and the dimension is reduced to 128-dimensional common features through a dual-branch gating network, and positive and negative sample pairs are constructed between adjacent view frames for comparative learning, and the common features are low-entropy aggregated based on the density peak algorithm to generate a first pseudo-label map.

[0066] S103 : Based on the first pseudo label map, perform pixel processing on each material cluster to determine a fine-grained segmentation map; and fuse the fine-grained segmentation map and the first pseudo label map to determine a high-confidence label map.

[0067] In some embodiments of the present application, surface roughness and normalized reflectance are calculated pixel by pixel for each material cluster based on the first pseudo-label map. The surface roughness and normalized reflectance are bucketed and aggregated using a two-dimensional grid to obtain a target matrix corresponding to each material cluster. The roughness means in the target matrix are sorted in ascending order and modulated by the reflectance mean intensity. The target matrix is ​​mapped to a discrete diffusion step size to generate a noise intensity table. The noise intensity table is then embedded in the path to determine a fine-grained segmentation map. The fine-grained segmentation map is then fused with the first pseudo-label map to determine a high-confidence label map.

[0068] Exemplarily, the first pseudo-label image and the depth gradient amplitude and near-infrared reflectance difference of the same resolution are read, the surface roughness and normalized reflectance are calculated pixel by pixel for each material cluster, the sample statistics are accumulated on the two-dimensional roughness-reflectance grid and the roughness-reflectance mean-variance matrix is ​​output; the matrix is ​​sorted in ascending order by roughness mean and modulated by reflectance mean intensity, mapped to discrete diffusion step size, a noise intensity table is generated and embedded in the DiffusionSegV2 network time embedding path to realize material-aware pixel-by-pixel noise scheduling, and a fine-grained segmentation map is output and fused with the first pseudo-label image to form a high-confidence label map.

[0069] It should be noted that the roughness-reflection mean-variance matrix is ​​the target matrix.

[0070] S104. Based on the high-confidence label map, determine the steel bar broken line anchor point and the slag peak anchor point; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, determine a second pseudo label map; based on the second pseudo label map, calculate the symmetric divergence and determine the fused label map.

[0071] In some embodiments of the present application, based on the high-confidence label map, the steel bar broken line anchor point and the slag peak anchor point are determined. According to the steel bar broken line anchor point and the slag peak anchor point, the sparse core mask is determined. Based on the sparse core mask, the fine-grained segmentation map and the high-confidence label map, a pixel-by-pixel comparison is performed. If there is a category offset, a backtracking correction is performed to determine the repaired label image. Based on the repaired label image, the normalized near-infrared reflectivity and the dust index, a material-sensitive soft label probability map is generated. The material-sensitive soft label probability map is input into the geometry and texture collaborative network for inference to determine the second pseudo label map. Based on the second pseudo label map and the repaired label image, the symmetric divergence is calculated to determine the fused label map.

[0072] For example, based on high confidence pseudo labels and depth gradient amplitude at the same resolution . With curvature constraint Detect the shape line segment, and The pixels that meet The top pixel of the slope is marked as the soil peak anchor point.

[0073] Based on the depth gradient curvature detection, the steel bar broken line anchor points and the slag peak anchor points are formed into a sparse core mask, and the pixels with category deviation of high confidence labels are back-corrected along the local roughness gradient direction; the normalized near-infrared reflectivity and dust index are combined to generate a material-sensitive soft label probability map, and the soft label probability map is input into the geometry-texture collaborative network inference to obtain the second pseudo label map, and then the symmetric divergence between the second pseudo label map and the repaired label image is calculated to output the fused label map.

[0074] Calculate the symmetric divergence between the repaired label image and the second pseudo label image:

[0075]

[0076] in, is the second pseudo label map One-hot distribution.

[0077] when Take the average of the two to generate a fusion label ;

[0078] when The dual labels are coexisted and the confidence mark is attached to form a hierarchical fusion result For supervision in step S5.

[0079] Through pixel-by-pixel consistency measurement, we retain redundant information for difficult partitions while ensuring rapid convergence of high-confidence regions, achieving robust and error-correctable label evolution.

[0080] It should be noted that the high-confidence pseudo-label, or high-confidence pseudo-label map, is used by the system to pixel-align the fine-grained segmentation map with the first pseudo-label, calculate a consistency score, and overwrite the first pseudo-label with the high-confidence label in the overlapping area. In the conflicting area, both versions of the label are retained for supervised correction in subsequent steps. The resulting fine-grained prior significantly improves the resolution of material boundaries and leaves room for adjustment in the semi-supervised error correction process.

[0081] S105. Based on the fused label map, extract the pixel segmentation contours of adjacent materials and determine the pile support relationship map; based on the pile support relationship map, determine the final construction waste semantic segmentation map, instance volume and weight list.

[0082] In some embodiments of this application, pixel segmentation outlines of adjacent materials are extracted based on a fused label map. Support is calculated based on these pixel segmentation outlines to determine a material support relationship map. For an image sequence, spatiotemporal stability and morphological variance are calculated within two consecutive frames to generate a confidence index. Based on the material support relationship map and confidence index, parameter updates and multi-scale forward reasoning are performed to determine the final semantic segmentation map of construction waste, along with instance volumes and weight lists.

[0083] For example, the pixel segmentation outlines of adjacent materials are extracted from the fused label map. The outline pixel set is projected back into the 3D point cloud. The contact area and contact normal are calculated using Delaunay triangulation, and the angle is obtained by comparing with the world gravity direction. The number of suspended voxels within 20mm of the upper region and the number of supported voxels within 20mm of the lower region are counted and normalized into the suspended rate and supported ratio, respectively. The support degree is calculated according to the piecewise function based on the density difference between the two regions, and a support relationship diagram is output. Based on the support relationship diagram, the spatiotemporal stability and morphological variance of each instance are calculated within two consecutive frames to generate a confidence index. Strongly confident instances are selected to update the rough-reflection statistical cluster center, and the DiffusionSegV2 noise step size and geometric branch gating weights are dynamically adjusted according to the drift amount. After the parameter update, multi-scale forward inference is performed on the current frame and its temporal neighboring frames. The decoding layer weights are iteratively fine-tuned based on the consistency loss until the maximum drift rate of the cluster center and the consistency loss reduction rate meet the termination conditions. The final construction waste semantic segmentation map, instance volume, and weight list are output.

[0084] It is understandable that with the environmental index model as the core, millisecond-level synchronous exposure and multi-angle polarization imaging of three cameras are achieved in scenes with light and dust fluctuations, avoiding timing mismatch and loss of material information in the acquisition link; three-dimensional gradient-driven dynamic anchor point sampling and dual-branch gated dimensionality reduction mechanism replace traditional uniform sampling and single-channel convolution feature extraction, so that thin steel bars and large areas of slag can obtain targeted receptive fields at the same time. Based on the dynamic convolution anchor point, the first pseudo label map is determined; pixel processing is performed on each material cluster to determine a fine-grained segmentation map; the fine-grained segmentation map and the first pseudo label map are fused to determine a high-confidence label map; based on the high-confidence label map, the steel bar broken line anchor point and the slag peak anchor point are determined; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, the second pseudo label map is determined; based on the second pseudo label map, the symmetric divergence is calculated to determine the fused label map; the fused label map contour is back-projected onto the point cloud and the support is evaluated. Compared with the approach of judging only by height or distance thresholds, a mixed feature of geometry and physical properties is constructed, which shows structural advantages in terms of data alignment accuracy, material differentiation ability and support judgment dimension, and can improve the accuracy of semantic segmentation of construction waste.

[0085] In some embodiments of the present application, S101 may be implemented through S201-S204 as follows:

[0086] S201. When the light pulsation and the dust particle size meet preset conditions, synchronous line scanning is performed through the RGB camera, the structured light depth camera and the near-infrared camera to obtain an image sequence.

[0087] S202: Construct an environmental index model based on the visible light grayscale matrix and the near-infrared reflectance matrix of the image sequence; and obtain environmental parameters through the environmental index model.

[0088] For example, in a scene where the light intensity fluctuates rapidly and the dust particle size distribution is wide in an abandoned building yard, a coaxial RGB-NIR top-down camera is used to perform line scanning on the pile surface at 120fps. Frame visible light grayscale matrix Synchronous near-infrared reflectance matrix Perform online statistics to obtain the average brightness of each frame and the normalized scattering ratio at the current frame scale .

[0089] Then, a dedicated environmental index model for the storage yard was constructed:

[0090]

[0091] Determine the light stability and dust uniformity, where and They are and The sliding mean of Offline calibration weight.

[0092] when Immediately lock a 0.5s acquisition window and trigger the flag Set to 1, otherwise And re-evaluate after 50ms.

[0093] Based on this, instantaneous gray fog and car light glare are excluded from the formal collection to ensure that subsequent feature learning does not mistake aerosols and other materials as material textures.

[0094] when When the scheduler obtains and Calculate the unified exposure vector by combining the camera gain table .

[0095] The vector is sent to the RGB camera, structured light depth camera and NIR camera at the same time through the hardware trigger bus. Each channel takes an exposure at the same time and sends back the original data frame. The time stamp difference between the three channels is limited to To ensure that the depth edge is aligned with the visible texture at the pixel level. Apply local contrast constraints to prevent overexposure of steel bar highlights and retain texture on the dark side of the slag.

[0096] S203: When the environmental parameters meet the preset threshold, collect near-infrared images at three different angles.

[0097] For example, after the synchronous exposure is completed, the servo motor is The angular velocity drives the polarizer to rotate to Stay at each angle for 25ms and trigger NIR imaging once, and obtain .

[0098] It should be noted that the three different angles are .

[0099] S204: Determine a high-quality frame image based on the near-infrared images at different angles.

[0100] In some embodiments of the present application, triangular polarization differences are calculated based on near-infrared images taken at three different angles. Based on these triangular polarization differences and preset thresholds, labels for metallic highlights, rough diffuse reflections, and glass mirrors are determined. A roughness prior map is generated based on these labels. Based on this roughness prior map, a comprehensive quality score is calculated, and this score is used to determine a high-quality frame image.

[0101] In some embodiments of the present application, S204 may be implemented through S2041-S2043 as follows:

[0102] S2041. Calculate triangular polarization differences based on near-infrared images at three different angles, and determine metal highlight labels, rough diffuse reflection labels, and glass mirror labels based on the triangular polarization differences and a preset threshold.

[0103] S2042. Generate a roughness prior map based on the metal highlight label, the rough diffuse reflection label, and the glass mirror label.

[0104] S2043. Calculate a comprehensive quality score based on the roughness prior map; and determine a high-quality frame image based on the comprehensive quality score.

[0105] For example, in obtaining After that, the control module calculates the difference on the fly , and label the three types of responses with thresholds: Recorded as metallic high-gloss label, It is recorded as a rough diffuse reflection label, and the rest are classified as glass mirror labels.

[0106] Based on this label, a roughness prior map is generated and archived with the frame, providing physical constraints for the material adaptive step size mapping of the diffusion chain, so that the sharp edges of broken glass and rusted steel bars have different iterative rhythms during the generation stage.

[0107] For single-frame data that has completed polarization wheel acquisition and contains roughness prior, according to the dust scattering index , polarization angle integrity (Effective pixel ratio of the three polarization images) and depth echo rate Calculate the overall quality score:

[0108]

[0109] in, is the constant obtained by offline parameter adjustment for the construction waste yard.

[0110] like , the frame together with the unified exposure parameters, triangular polarization response and quality metadata are written into the high-quality sample pool to obtain a high-quality frame image; otherwise, it is written into the incremental adaptive reference area.

[0111] Through hierarchical caching, subsequent unsupervised clustering only receives frames with stable lighting, low dust interference, and complete depth, preventing low-confidence samples from causing drift in cluster center convergence.

[0112] It should be noted that the three different angles are .

[0113] In some embodiments of the present application, performing a three-dimensional gradient operation based on the aligned tensor to determine a dynamic convolution anchor point in S102; determining the first pseudo label map based on the dynamic convolution anchor point can be implemented through S301-S303 as follows:

[0114] S301 , calculating a three-dimensional gradient based on the alignment tensor; and dividing pixels in the high-quality frame image into a sharp edge region and a gentle slope region according to the three-dimensional gradient and the step threshold.

[0115] For example, based on the output RGB image , purification depth and triangular polarization difference , based on the depth camera intrinsic parameters Counting pixels Three-dimensional coordinates in the depth coordinate system Through external reference Transform the coordinates to the RGB camera coordinate system and project them to the RGB pixel plane again to obtain aligned pixels , and further splice the multimodal features on the right to generate a point-pixel-reflection alignment tensor :

[0116]

[0117]

[0118] Where, is the focal length of the depth camera and the coordinates of the principal point, is the RGB internal parameter matrix, Depth to RGB rotation and translation.

[0119] After processing using the above formula, the tips of steel bars and the corners of broken bricks occupy the same pixel index in all modes, laying a spatially consistent foundation for cross-modal collaboration.

[0120] In tensor Internal calculation of 3D gradient . Based on the step threshold and Divide pixels into sharp edge regions and gentle slope areas .

[0121] S302: Extract the same number of pixels in the sharp edge area and the gentle slope area respectively to determine the dynamic convolution anchor point.

[0122] In some embodiments of the present application, the sampler only extracts one-tenth of a pixel in each of the two types of areas as a dynamic convolution anchor point, physically ensuring that both the thin steel bar lines and the large surface of slag receive specific receptive field attention at the same time, avoiding computational redundancy caused by oversampling of flat areas.

[0123] S303: Determine common features based on the dynamic convolution anchor points, perform low-entropy aggregation on the common features, and determine a first pseudo label map.

[0124] For example, polarization difference conduct After normalization, the reflection residual is obtained .Will High frequency texture with RGB particles Perform channel-level residual superposition and mark the highlight threshold in the feature map to enhance the spectral difference between metal and glass and suppress the pseudo-similar phenomenon of the same grayscale between concrete and broken bricks.

[0125] For each sampled anchor point , connect its depth gradient vector With the aforementioned reflection residual , performs dimensionality reduction through the gating network, and outputs a common feature with a dimension of one hundred and twenty-eight .

[0126] The dimensionality reduction process is based on the following dual-branch gating mechanism:

[0127]

[0128] in, and is an independent weight matrix, is Sigmoid activation, represents the Hadamard product.

[0129] In the formula, material nonlinear mapping is introduced in the gating branch to retain local roughness and compress redundant color information, outputting a low-noise and discriminable joint representation.

[0130] Select the anchor point set of two adjacent frames , based on ray consistency detection, mutually occluded pixels are removed to form positive sample pairs , and the negative sample pairs are formed based on the anchor points whose gravity direction difference in the yard is greater than 20 degrees .

[0131] The network minimizes the cosine distance of positive samples in the common feature space and maximizes the cosine distance of negative samples, so that the center of the feature cluster evolves automatically.

[0132] After completing the contrast training, the density peak algorithm is used to Perform low entropy aggregation and output the first pseudo-label map The pseudo labels are stored with the frames, directly providing category priors for the next stage of material adaptive diffusion.

[0133] In some embodiments of the present application, performing pixel processing on each material cluster based on the first pseudo-label image in S103 to determine a fine-grained segmentation image can be implemented through S401-S403 as follows:

[0134] S401 : Based on the first pseudo-label image, for each material cluster, calculate the surface roughness and normalized reflectivity pixel by pixel.

[0135] Exemplarily, based on the first pseudo label and the depth gradient magnitude aligned with the pixel resolution Difference with near-infrared reflectance , for each material cluster Calculate surface roughness pixel by pixel and normalized reflectance .

[0136] It should be noted that the first pseudo label is the first pseudo label map.

[0137] S402. Using a two-dimensional grid, perform bucketing and aggregation processing on the surface roughness and normalized reflectivity to obtain a target matrix corresponding to each material cluster; wherein the target matrix is ​​determined by the surface roughness and the mean and variance of the reflectance quadruple.

[0138] For example, The two-dimensional grid of Perform bucketing and accumulate the number of samples and statistics in each grid cell;

[0139] After the grid aggregation is completed, the roughness-reflection quadruple mean-variance matrix is ​​output for each material cluster. :

[0140]

[0141] in, and Respectively The mean and variance of the roughness of the material class, and is the corresponding reflectivity statistic.

[0142] matrix Along with the grid count, it is written to the scene property index table, providing searchable coordinates for the step map.

[0143] It should be noted that the mean-variance matrix of the rough-reflection quadruple , which is the target matrix.

[0144] S403 , sorting the roughness mean values ​​in the target matrix in ascending order and modulating the reflectivity mean intensity, mapping the target matrix to a discrete diffusion step, generating a noise intensity table; and embedding the noise intensity table into the path to determine a fine-grained segmentation map.

[0145] For example, based on the rough-reflection quadruple mean-variance matrix , the controller first follows the roughness mean Sort in ascending order to get the sequence Then, the reflectivity mean Implement intensity modulation on the sorting results to map the physical property statistics to discrete diffusion steps The step length is calculated using a linear-exponential hybrid formula:

[0146]

[0147] in, To preset the shortest step length, Controls the roughness weight, Control reflectivity modulation intensity; is the global extreme value across clusters. and At the same time, in the upper bound, you will get the maximum , slag is assigned the smallest amount because both items are low. , thus forming a convergence rhythm with obvious material distinction in the diffusion chain.

[0148] Diffusion scheduler based on noise intensity table The table is embedded into the DiffusionSegV2 temporal embedding path along with the pixel label index. At each iteration, the network first queries the material index of the current pixel. , and then from Read the corresponding step length This replaces the default random noise amplitude to achieve material-aware per-pixel noise scheduling. This mechanism ensures that high-gloss glass receives more noise early on to preserve sharp contours, while dull soil converges early on, improving the overall segmentation's ability to distinguish fine-grained textures across various materials.

[0149] To further emphasize the tiny particles and sharp edges, based on the depth gradient Polarization contrast Estimated particle size index For fine particles with low particle size index and sharp edge pixels with high gradient, the attention weight is shrunk to the center of the channel during diffusion backpropagation, which promotes the priority reconstruction of glass edges and steel bar highlights and suppresses the spread of false edges caused by dust.

[0150] When the diffusion chain is fully traced back according to the material step length, the network outputs a fine-grained segmentation map .

[0151] In some embodiments of the present application, determining the second pseudo label image based on the steel bar broken line anchor point, the slag peak anchor point, and the high-confidence label image in S104 can be implemented through S501-S504 as follows:

[0152] S501: Determine a sparse core mask based on the steel bar broken line anchor points and the slag peak anchor points.

[0153] S502: Perform pixel-by-pixel comparison based on the sparse core mask, the fine-grained segmentation map, and the high-confidence label map. If there is a category shift, perform backtracking correction to determine the repaired label image.

[0154] S503: Generate a material-sensitive soft label probability map based on the repaired label image, normalized near-infrared reflectivity, and dust index.

[0155] S504: Input the material-sensitive soft label probability map into the geometry and texture collaborative network for reasoning to determine a second pseudo label map.

[0156] For example, based on high confidence pseudo labels and depth gradient amplitude at the same resolution . With curvature constraint Detect the shape line segment, and The pixels that meet The top pixel of the slope is marked as the soil peak anchor point.

[0157] exist Intra-pixel comparison of fine-grained priors With high confidence labels If a category shift occurs, then along the local roughness gradient Backtrack until you find the Consistent adjacent pixels

[0158] The correction rule can be specifically expressed as:

[0159]

[0160] Where, is the unit vector of the backtracking direction, is the Heaviside step function.

[0161] Based on the above formula, it is ensured that the repair is only performed when the roughness gradient projection exceeds the threshold Triggered when the smooth area is over-modified.

[0162] For repaired tags , the system combines step S1 to normalize the near-infrared reflectance Dust index Generate material sensitive soft label probability map .

[0163] The probability mapping is based on the exponential temperature model:

[0164]

[0165] in, is the material class specific reflection magnification factor, is the dust attenuation coefficient. High, get hardening probability; slag due to and The multiplication is softened, producing a continuous distribution.

[0166] Material sensitive soft label probability map Back-injected into the geometry-texture collaborative network as a dynamic attention graph. The network improves the geometry branch Corresponding to the convolution channel weights, the weights of the grayscale homogeneous regions of the texture branch are reduced, and then a forward inference is performed to output the second version of the pseudo label. The process strengthens the expression of the sharp edges of steel bars and glass ridges, while filling in the missed pixels inside the broken glass.

[0167] It should be noted that the repaired label is the repaired label image.

[0168] In some embodiments of the present application, in S105, based on the fused label map, pixel segmentation contours of adjacent materials are extracted to determine the pile support relationship map; based on the pile support relationship map, the final semantic segmentation map of construction waste, instance volume, and weight list are determined by S601-S603 as follows:

[0169] S601. Extract pixel segmentation contours of adjacent materials based on the fused label map; and calculate support degrees based on the pixel segmentation contours to determine a material pile support relationship map.

[0170] Exemplary, based on fusion tags Extract pixel-level segmentation contours and use a size of A two-dimensional window is combined with a three-frame time stack to form a three-dimensional sliding body. The depth mean difference is calculated on both sides of the window Difference from the mean near-infrared reflectance When satisfied and When creating a new image, only the real material boundaries such as steel bar-slag, glass-concrete, etc. will be retained.

[0171] in accordance with The set of touching pixels in the adjacent areas in the pixel domain Project back to the 3D point cloud coordinate system and use Delaunay triangulation to calculate the contact area , and calculate the normal vector for each triangle face and then take the area weighted average to get the overall contact normal Calculate the normal and the direction of world gravity Angle .

[0172] like In the area chart Add inner edges , and record it in the edge attributes The graph structure accurately maps the adjacency relationship in the pixel domain to the physical contact surface of the point cloud.

[0173] For each edge , first compare the height of the mass center of the two regions and determine the upper region index and the area index below Then calculate the number of suspended voxels in the upper area within 20 mm above the boundary projection , and count the number of supporting voxels within 20 mm directly below the projection of the lower area .

[0174] Normalize the two items to the hanging rate Compared with support . Combined with the material density table, the density difference between the two areas is read .

[0175] Final support Determined by the following piecewise function:

[0176]

[0177] in, is the hanging rate threshold, is the preset maximum density difference constant. The first part of the formula measures the lightweight cover case with high overhang and small density difference, the second part reduces the support degree in the heavy cover case with high overhang and large density difference, and the third part directly determines whether the support is sufficient.

[0178] It should be noted that the fusion label is the fusion label map.

[0179] S602: For the image sequence, calculate the spatiotemporal stability and morphological variance in two consecutive frames of images to generate a confidence index.

[0180] Exemplary, instance mask-based and point cloud , for each instance in two consecutive frames Internal computational spatiotemporal stability and morphological variance .

[0181] The two items are linearly normalized and combined into a confidence index .when When , the instance is marked as strong confidence and participates in cluster center update later; the remaining instances are retained but do not enter the model weight adjustment.

[0182] S603: Based on the pile support relationship diagram and confidence index, parameter update and multi-scale forward reasoning are performed to determine the final construction waste semantic segmentation map, instance volume and weight list.

[0183] For example, for instances with strong confidence indicators, based on the stacking support relationship graph, the mask is mapped back to the pixel domain and filled with the value as the category unique hot vector to generate an enhanced pseudo label map. The system is based on Recalculate the roughness-reflectance statistics matrix And update the original cluster center in step S3 using exponential sliding average :

[0184]

[0185] This enables adaptive clustering center migration in geometry-texture space and improves the ability of subsequent segmentation to quickly respond to newly appearing materials.

[0186] According to the cluster center drift Dynamically adjust DiffusionSegV2 noise step size With geometric branch gating weight , iteratively updated based on the following formula:

[0187]

[0188] in, is the relative drift rate, is the adjustment coefficient; are the upper and lower stability thresholds respectively.

[0189] When the difference between the new and old cluster centers is significant (the first segment), the noise step size is increased and the geometric gating is contracted to prevent premature convergence. When the difference is minimal (the third segment), the noise step size is reduced and the geometric gating is amplified to promote high-precision segmentation. The parameters are kept constant in the intermediate interval.

[0190] After updating the parameters, the network performs multi-scale forward reasoning on the current frame and its bidirectional temporal neighboring frames, and outputs the probability map in Calculate consistency loss at three resolutions . use Backpropagation only fine-tunes the weights of the decoding layer to avoid destroying the converged geometric-texture feature expression in the deep layer.

[0191] Continue to execute until the global cluster center maximum drift rate and Consistency loss reduction rate When the iteration termination condition is met, the loop is stopped and the final semantic segmentation map of construction waste is output. And the corresponding instance volume and weight list provide high-precision perception basis for automatic sorting, crushing and screening, and resource processing.

[0192] It can be understood that compared with the existing multimodal segmentation methods based on fixed exposure or sequential acquisition, the solution proposed in this application takes the environmental index model as the core, and realizes millisecond-level synchronous exposure and multi-angle polarization imaging of three cameras in scenes with light and dust fluctuations, avoiding timing mismatch and material information loss from the acquisition link; three-dimensional gradient-driven dynamic anchor point sampling and dual-branch gated dimensionality reduction mechanism replace traditional uniform sampling and single-channel convolution feature extraction, so that thin steel bars and large-scale slag can obtain targeted receptive fields at the same time; by mapping the roughness-reflection statistics to the diffusion model time path, the solution completes material adaptive noise scheduling within the network. , rather than relying on fixed weights or simple feature splicing, it can separate high-reflective glass and low-reflective slag in the same parameter space; the label evolution process composed of curvature anchor point backtracking and soft label collaborative reasoning breaks through the limitations of static pseudo-labels and continuously corrects boundary errors under unsupervised conditions; finally, the fused label contour is projected back to the point cloud and the contact normal is calculated using Delaunay decomposition, and the support is evaluated by combining the overhang rate and density difference. Compared with the approach of judging only by height or distance thresholds, this method constructs mixed geometric and physical features, which shows structural advantages in data alignment accuracy, material differentiation ability and support judgment dimension.

[0193] In some embodiments of the present application, Figure 2 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 2 ,like Figure 2 As shown in Figure 1, the semantic segmentation method of construction waste based on geometry and texture coupling includes the following steps:

[0194] 1. Start.

[0195] In some embodiments of the present application, the system is powered on and the cache is initialized.

[0196] 2. Hardware synchronization and environmental index evaluation.

[0197] In some embodiments of the present application, 120fps line scanning, coaxial RGB-NIR-Depth-polarization statistics average brightness and scattering ratio are used to judge light pulsation and dust fluctuation, and a 0.5s acquisition window is set when the index meets the threshold.

[0198] 3. Do the light and dust meet the collection threshold?

[0199] In some embodiments of the present application, if yes, then continue to perform multi-camera unified exposure; if no, then perform hardware synchronization and environmental index evaluation.

[0200] 4. Unified exposure for multiple cameras.

[0201] In some embodiments of the present application, the scheduler queries the gain table, hard triggers RGB / Depth / NIR, the timestamp difference is <1ms, the highlights of the steel bars are not overexposed, and the texture of the dark side of the slag is preserved.

[0202] 5. Polarization wheel sequence imaging and partitioning.

[0203] In some embodiments of the present application, a servo motor rotates the polarizer and stops for 25ms x3, calculates the polarization difference, marks the metal highlights, rough diffuse reflections, and mirror glass, and generates a roughness prior map.

[0204] It should be noted that metallic highlights are metallic highlight labels; rough diffuse reflections are rough diffuse reflection labels; and mirror glass is mirror glass labels.

[0205] 6. Quality scoring and hierarchical caching.

[0206] In some embodiments of the present application, if the dust scattering, polarization angle integrity, depth echo rate, and comprehensive quality score ≥ threshold value, then the high-quality sample pool is selected; otherwise, the reference area is selected. The reference area frame is used as an auxiliary alignment.

[0207] 7. Multimodal alignment and contrastive learning.

[0208] In some embodiments of the present application, the depth coordinates are reprojected to RGB pixels, 10% anchor points are extracted for sharp edges and gentle slopes, the depth gradient and reflection residual are spliced, and cosine loss training is performed to obtain preliminary pseudo labels.

[0209] It should be noted that the initial label is the first pseudo label.

[0210] 8. Roughness-reflectance mesh statistics and step size mapping.

[0211] In some embodiments of the present application, the grid bucket statistics mean and variance are calculated, sorted by roughness, reflectivity modulated, linear-exponential mixing is used to assign diffusion steps, and a material noise intensity table is output.

[0212] 9. Geometric correction and support analysis.

[0213] In some embodiments of the present application, conflicting pseudo-labels are corrected by backtracking along the roughness gradient, and Delaunay triangulation is used to calculate the contact surface, overhang rate + support ratio + density difference, and output the fused label and instance edge graph.

[0214] 10. Spatiotemporal confidence and closed-loop updates.

[0215] In some embodiments of the present application, stability and morphological variance are calculated for consecutive frames, cluster centers are updated using strong confidence instances, noise step size and gating weights are dynamically adjusted, and multi-scale consistency loss is used to fine-tune the decoding layer.

[0216] 11. Output.

[0217] In some embodiments of the present application, fine-grained semantic segmentation maps + instance mentions / weights are used for lifting, crushing, screening, and resource processing.

[0218] It can be understood that this application solves the problem of how to obtain multimodal data with spatial alignment and complete physical features in real time in a construction waste yard with drastic fluctuations in dust concentration, obvious light pulsation and mixed materials, and on this basis perform fine-grained, material-aware semantic and instance segmentation, while reliably inferring the support relationship between the piles, providing accurate visual and geometric information for subsequent automated material retrieval and resource processing.

[0219] Figure 3 An optional process diagram of a construction waste semantic segmentation method based on geometry and texture coupling provided in an embodiment of the present application Figure 3 ,like Figure 3 As shown, the fine-grained segmentation process logic is as follows:

[0220] 21. Load multimodal pseudo-labels + depth gradient amplitude + NIR reflectance difference to introduce preliminary material priors for each pixel.

[0221] 22. Calculate the mean and standard deviation of surface roughness for each material cluster, and calculate the normalized reflectivity.

[0222] 23. Two-dimensional grid bucket aggregation, accumulate sample counts on the plane, and output the roughness-reflection mean-variance matrix.

[0223] In some embodiments of the present application, after the two-dimensional grid is bucketed and aggregated, the grid statistics are used for sorting. The roughness-reflectance distribution guides the estimation of the particle size index.

[0224] 24. Sort the sequence R in ascending order of roughness, and use the reflectivity intensity vector ρ to modulate the linear-exponential mixture to generate the discrete step length table L.

[0225] It should be noted that step scheduling affects the convergence rhythm.

[0226] 25. Embed the noise intensity table L into DiffusionSegV2, and replace the default random noise amplitude with a pixel-by-pixel query of the material index during inference.

[0227] 26. Combine depth gradient and polarization contrast to estimate particle size index, shrink the attention weight of fine particles and sharp edge pixels to the channel center.

[0228] 27. Diffusion chain reverse tracing is used to reconstruct the glass edges and steel bar highlights, converge the dark slag in advance, and output a fine-grained semantic probability map.

[0229] 28. Align with the initial pseudo-label pixels, calculate the consistency score, and generate high-confidence fine-grained priors.

[0230] The technical solution of this application is based on a synchronous multimodal acquisition link driven by an environmental index. Through real-time judgment of the average brightness of visible light and the near-infrared scattering ratio, the RGB, structured light depth and near-infrared polarization three-way cameras are triggered at one time only within the stable range of light and dust, and multi-angle polarization imaging is completed within the same frame. The roughness prior map is generated in real time and low-quality samples are filtered out, ensuring the spatiotemporal alignment of multimodal data and the integrity of material information from the source.

[0231] In the feature learning stage, the depth points are mapped back to the RGB plane through external parameters and combined with the polarization residual and depth gradient to form an alignment tensor. The system extracts anchor points only in the sharp edge area and gentle slope area based on the three-dimensional gradient intensity, and obtains 128-dimensional common features through a dual-branch gating network. High-resolution pseudo labels are generated through contrastive learning and density peak clustering, avoiding reliance on large-scale manual labeling.

[0232] For fine-grained segmentation, the scheme maps the roughness and reflectivity statistics of each material cluster into discrete diffusion steps, replaces the default noise amplitude in the time path of the diffusion model, and realizes material-adaptive pixel-by-pixel noise scheduling, thereby distinguishing multiple material details such as steel bar highlights, broken glass, and dark surfaces of slag under the same network weights.

[0233] In the geometry correction and label fusion stage, the system uses the sparse anchor points obtained by deep curvature detection to back-pattern along the roughness gradient, and then combines near-infrared reflectance and dust index to construct material-sensitive soft labels. It completes dynamic weight adjustment through the geometry-texture collaborative network and drives label iteration with symmetric divergence to improve boundary accuracy and fault tolerance.

[0234] The support relationship determination part projects the fused label contour back into the point cloud, obtains the contact area and normal based on Delaunay segmentation, and calculates the support degree by combining the 20mm overhang ratio, support ratio, and material density difference to achieve a quantitative assessment of the support status of the pile. The online adaptive closed loop driven by strong confidence instances uses cluster center drift to adjust the diffusion step size and gating weights to ensure that the model continues to adapt to new material and morphological changes on site.

[0235] Based on the above embodiment of the construction waste semantic segmentation method based on geometry and texture coupling, the embodiment of the present application also provides a construction waste semantic segmentation system based on geometry and texture coupling, such as Figure 4 As shown, Figure 4 This is a structural diagram of a construction waste semantic segmentation system based on geometry and texture coupling provided in an embodiment of the present application. The construction waste semantic segmentation system 4 based on geometry and texture coupling includes: an acquisition unit 401 and a determination unit 402, wherein:

[0236] The acquisition unit 401 is configured to acquire an image sequence when the light pulsation and the dust particle size meet preset conditions; construct an environmental index model based on the image sequence; and determine a high-quality frame image using the environmental index model;

[0237] The determination unit 402 is used to perform coordinate transformation and projection processing based on the high-quality frame image to determine an alignment tensor; and perform three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; determine a first pseudo label map based on the dynamic convolution anchor point; perform pixel processing on each material cluster based on the first pseudo label map to determine a fine-grained segmentation map; and fuse the fine-grained segmentation map and the first pseudo label map to determine a high-confidence label map; determine a steel bar broken line anchor point and a slag peak anchor point based on the high-confidence label map; and determine a second pseudo label map based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map; calculate the symmetric divergence based on the second pseudo label map to determine a fused label map; extract the pixel segmentation contours of adjacent materials based on the fused label map to determine a pile support relationship map; and determine a final semantic segmentation map, instance volume and weight list of construction waste based on the pile support relationship map.

[0238] Based on the above embodiment of the construction waste semantic segmentation method based on geometry and texture coupling, the embodiment of the present application also provides a construction waste semantic segmentation device based on geometry and texture coupling, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a device for semantic segmentation of construction waste based on geometry and texture coupling provided in an embodiment of the present application. The device 5 for semantic segmentation of construction waste based on geometry and texture coupling includes: a processor 501 and a memory 502. The memory 502 is used to store a computer program; the processor 501 is used to load and execute the computer program from the memory to implement the method for semantic segmentation of construction waste based on geometry and texture coupling as described in the above embodiment.

[0239] In the embodiments of the present application, the processor 501 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0240] An embodiment of the present application provides a computer-readable storage medium storing a computer program for implementing the construction waste semantic segmentation method based on geometry and texture coupling as described in any of the above embodiments when executed by a processor.

[0241] Exemplarily, the program instructions corresponding to a method for semantic segmentation of construction waste based on geometry and texture coupling in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to a method for semantic segmentation of construction waste based on geometry and texture coupling in the storage medium are read or executed by an electronic device, the method for semantic segmentation of construction waste based on geometry and texture coupling as described in any of the above embodiments can be implemented.

[0242] In addition, the functional modules in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional modules.

[0243] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0244] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0245] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0246] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0247] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0248] The methods disclosed in the several method embodiments provided in the embodiments of this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0249] The features disclosed in several product embodiments provided in the embodiments of this application can be arbitrarily combined to obtain new product embodiments without conflict.

[0250] The features disclosed in several method or device embodiments provided in the embodiments of this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0251] The above is merely an implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A semantic segmentation method for construction waste based on geometry and texture coupling, characterized in that: The method comprises: Acquire an image sequence when the light pulsation and the dust particle size meet preset conditions; construct an environmental index model based on the image sequence; and determine a high-quality frame image using the environmental index model; Based on the high-quality frame image, coordinate transformation and projection processing are performed to determine an alignment tensor; and based on the alignment tensor, a three-dimensional gradient operation is performed to determine a dynamic convolution anchor point; based on the dynamic convolution anchor point, a first pseudo label map is determined; Based on the first pseudo label map, pixel processing is performed on each material cluster to determine a fine-grained segmentation map; and the fine-grained segmentation map is fused with the first pseudo label map to determine a high-confidence label map; Based on the high-confidence label map, a steel bar broken line anchor point and a slag peak anchor point are determined; and based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map, a second pseudo label map is determined; based on the second pseudo label map, a symmetric divergence is calculated to determine a fused label map; Based on the fused label map, the pixel segmentation contours of adjacent materials are extracted to determine a pile support relationship map; based on the pile support relationship map, the final construction waste semantic segmentation map, instance volume and weight list are determined.

2. The method according to claim 1, characterized in that The method comprises acquiring an image sequence when the light pulsation and the dust particle size meet preset conditions; and constructing an environmental index model based on the image sequence; And determining a high-quality frame image through the environmental index model includes: When the light pulsation and the dust particle size meet preset conditions, synchronous line scanning is performed using an RGB camera, a structured light depth camera, and a near-infrared camera to acquire the image sequence; constructing the environmental index model based on the visible light grayscale matrix and the near-infrared reflectance matrix of the image sequence; and obtaining environmental parameters through the environmental index model; When the environmental parameters meet the preset threshold, near-infrared images at three different angles are collected; The high-quality frame image is determined based on the near-infrared images at different angles.

3. The method according to claim 2, characterized in that The determining the high-quality frame image based on the near-infrared images at different angles includes: Calculate triangular polarization differences based on the near-infrared images at the three different angles, and determine the metal highlight label, the rough diffuse reflection label, and the glass mirror label based on the triangular polarization differences and a preset threshold; Generate a roughness prior map based on the metal highlight label, the rough diffuse reflection label, and the glass mirror label; Based on the roughness prior map, a comprehensive quality score is calculated; and the high-quality frame image is determined by the comprehensive quality score.

4. The method according to claim 1, wherein performing a three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; Determining a first pseudo label map based on the dynamic convolution anchor point includes: Calculating a three-dimensional gradient based on the alignment tensor; and dividing pixels in the high-quality frame image into a sharp edge region and a gentle slope region according to the three-dimensional gradient and a step threshold; Extracting the same number of pixels in the sharp edge area and the gentle slope area respectively to determine the dynamic convolution anchor point; Based on the dynamic convolution anchor point, common features are determined, and low-entropy aggregation is performed on the common features to determine the first pseudo label map.

5. The method according to claim 1, wherein The step of performing pixel processing on each material cluster based on the first pseudo-label image to determine a fine-grained segmentation image includes: Based on the first pseudo-label image, for each material cluster, calculate the surface roughness and normalized reflectance pixel by pixel; Performing bucketing and aggregation processing on the surface roughness and the normalized reflectivity through a two-dimensional grid to obtain a target matrix corresponding to each material cluster; wherein the target matrix is ​​determined by the surface roughness and the mean and variance of the reflectance quadruple; The roughness means in the target matrix are sorted in ascending order and modulated by the reflectivity mean intensity, the target matrix is ​​mapped to a discrete diffusion step, and a noise intensity table is generated; and the noise intensity table is embedded in a path to determine the fine-grained segmentation map.

6. The method according to claim 1, characterized in that The determining of a second pseudo label map based on the steel bar broken line anchor point, the slag peak anchor point, and the high-confidence label map includes: Determining a sparse core mask according to the steel bar broken line anchor point and the slag peak anchor point; Perform pixel-by-pixel comparison based on the sparse core mask, the fine-grained segmentation map, and the high-confidence label map. If there is a category shift, perform backtracking correction to determine a repaired label image. Generate a material-sensitive soft label probability map based on the repaired label image, normalized near-infrared reflectance, and dust index; The material-sensitive soft label probability map is input into a geometry and texture collaborative network for reasoning to determine the second pseudo label map.

7. The method according to claim 1, characterized in that extracting pixel segmentation contours of adjacent materials based on the fused label graph and determining a pile support relationship graph; Based on the pile support relationship diagram, the final construction waste semantic segmentation map, instance volume and weight list are determined, including: Extracting pixel segmentation contours of adjacent materials based on the fused label map; and calculating support degrees based on the pixel segmentation contours to determine the pile support relationship map; For the image sequence, calculating spatiotemporal stability and morphological variance in two consecutive frames of images to generate a confidence index; Based on the pile support relationship diagram and the confidence index, parameter updating and multi-scale forward reasoning are performed to determine the final construction waste semantic segmentation map, the instance volume and the weight list.

8. A construction waste semantic segmentation system based on geometry and texture coupling, characterized by: include: Get unit and determine unit, where The acquisition unit is configured to acquire an image sequence when the light pulsation and the dust particle size meet preset conditions; construct an environmental index model based on the image sequence; and determine a high-quality frame image using the environmental index model; The determination unit is used to perform coordinate transformation and projection processing based on the high-quality frame image to determine an alignment tensor; and perform three-dimensional gradient operation based on the alignment tensor to determine a dynamic convolution anchor point; determine a first pseudo label map based on the dynamic convolution anchor point; perform pixel processing on each material cluster based on the first pseudo label map to determine a fine-grained segmentation map; and fuse the fine-grained segmentation map and the first pseudo label map to determine a high-confidence label map; determine a steel bar broken line anchor point and a slag peak anchor point based on the high-confidence label map; and determine a second pseudo label map based on the steel bar broken line anchor point, the slag peak anchor point and the high-confidence label map; calculate the symmetric divergence based on the second pseudo label map to determine a fused label map; extract the pixel segmentation contours of adjacent materials based on the fused label map to determine a pile support relationship map; and determine a final semantic segmentation map, instance volume and weight list of construction waste based on the pile support relationship map.

9. A construction waste semantic segmentation device based on geometry and texture coupling, characterized in that: include: processor and memory, wherein The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute and implement the method according to any one of claims 1 to 7.

Images