Construction scene bulk material volume estimation method and device based on multi-task joint prediction

Abstract

The application provides a construction scene bulk volume estimation method and device based on multi-task joint prediction, and relates to the technical fields of non-contact volume estimation and deep learning. The application collects images by using a monocular camera and performs preprocessing; extracts a binary mask through an image semantic segmentation network, constructs a structured feature vector representing bulk distribution and a multi-task deep learning model, deeply couples visual features and structured features through a gating fusion module of the multi-task deep learning model, jointly predicts volume, loading rate and category probability, corrects the volume prediction value based on the category probability weighting, and calculates the final loading rate in combination with the effective volume of a storage and transportation container. The application can avoid a complex three-dimensional reconstruction process, reduce the calculation cost and hardware cost, solve the monocular scale ambiguity and weak boundary segmentation problem, and significantly improve the accuracy, real-time performance and robustness of bulk measurement at a construction site.

Description

Technical Field

[0001] This invention relates to the fields of image processing, non-contact volume estimation, and deep learning technology. Specifically, it relates to a method and apparatus for estimating the volume of bulk materials in construction scenarios based on multi-task joint prediction. Background Technology

[0002] In the field of civil engineering, accurate volume measurement of bulk building materials is a core aspect of construction scheduling and cost accounting, and non-contact visual estimation has become a mainstream research direction. Existing laser scanning and multi-view vision solutions are easily affected by dust, vibration, and complex lighting conditions at construction sites, and their deployment costs are high. While monocular vision solutions offer flexibility, they currently rely heavily on complex 3D reconstruction techniques, requiring significant computational resources. When dealing with irregular shapes and weak boundary areas of bulk materials, measurement results are easily affected by environmental shadows. Furthermore, the surface textures and natural packing characteristics of different bulk materials vary significantly, and reconstruction-based methods are significantly constrained by material characteristics, leaving room for improvement in stability under diverse scenarios. To meet the requirements of rapid measurement and flexible deployment at construction sites, there is an urgent need to develop an estimation method that balances high accuracy with lightweight deployment to improve the intelligence level of bulk material measurement.

[0003] Deep learning models have significant advantages in image processing. Existing semantic segmentation algorithms face challenges in balancing model lightweighting and segmentation accuracy. Furthermore, the volume estimation process often extracts single features without fully integrating key information such as the spatial distribution of bulk materials, contour morphology, and their fit with container boundaries, leaving room for improvement in feature representation capabilities.

[0004] Currently, there is no method that deeply integrates improved lightweight segmentation networks with image-level multidimensional structured features for joint prediction of bulk material volume, loading rate regression, and physical correction in construction scenarios.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] The present invention aims to provide a method and device for estimating the volume of bulk materials in construction scenarios based on multi-task joint prediction, so as to solve the shortcomings of existing methods such as monocular vision scheme relying on complex 3D reconstruction, large consumption of computing resources, severe interference from environmental shadows, and insufficient measurement stability caused by different bulk material texture characteristics.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A method for estimating bulk material volume in construction scenarios based on multi-task joint prediction includes: S1, acquire and preprocess the bulk material image captured by the monocular RGB camera to obtain the preprocessed image; S2, input the preprocessed image into the lightweight image semantic segmentation network and output a binarized mask; S3, Based on the binarized mask and the preprocessed image, construct a structured feature vector representing the distribution of bulk materials; the structured feature vector includes geometric features, boundary fitting features, and texture grayscale features; S4, the binarized mask, the preprocessed image, and the structured feature vector are input into a multi-task deep learning model, which outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. S5, perform linear weighted correction on the volume prediction value using the bulk material category probability to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

[0009] Preferably, the preprocessing includes distortion correction, size normalization, brightness normalization, and noise suppression; wherein the expression for brightness normalization is: ; in, This indicates the corrected image at the pixel level. The brightness value at that location; and These represent the mean and standard deviation of the image, respectively. To prevent extremely small constants with a denominator of zero; For the brightness-normalized image at the pixel level Pixel value at; The image is preprocessed by applying a Gaussian filter or a median filter to the brightness-normalized image to remove random noise, local shadow interference, and high-frequency disturbances caused by reflections.

[0010] Preferably, the lightweight image semantic segmentation network is an improved MobileUNet segmentation network, including an encoder and a decoder; In the encoder, MobileNetV3 is used as the backbone network to extract multi-scale coding features layer by layer. Feature downsampling is performed through the inverse residual structure of MobileNetV3. Each layer of the backbone network contains depthwise separable convolution, batch normalization layer and activation function. Convolutional block attention module CBAM is introduced at the deep coding feature output of the encoder to jointly recalibrate the channel dimension and spatial dimension of the deep coding features. In the decoder, the highest layer features output by the Convolutional Block Attention (CBAM) module are upsampled by bilinear interpolation, and then channel-wise concatenated with the shallow encoded features from the encoding stage using skip connections. The current layer's decoded features are obtained through feature fusion. Meanwhile, the decoder sets a boundary auxiliary branch, uses the shallow coding features output by the encoder to generate a boundary probability map, uses the shallow decoding features output by the decoder to generate a pixel-level category probability map, and combines the joint loss function to constrain the extraction of the bulk material area contour, and generates a binary mask through thresholding.

[0011] Preferably, the convolutional block attention module (CBAM) includes a channel attention branch and a spatial attention branch; In the channel attention branch, global statistical information of the input features is extracted through global average pooling and global max pooling, respectively, and then channel attention weights are generated by a shared multilayer perceptron. The expression for these weights is as follows: ; in, The encoded features are the input. Channel attention weights; Indicates global average pooling; Indicates global max pooling; Indicates a shared multilayer perceptron; This represents the Sigmoid activation function; The input features are multiplied element-wise based on channel attention weights to complete the channel-dimensional feature recalibration and obtain the channel-enhanced features. In the spatial attention branch, the channel-enhanced features are subjected to average pooling and max pooling along the channel dimension, respectively. The features obtained from average pooling and max pooling are then concatenated and input into the convolutional layer to obtain the spatial attention weights; their expression is: ; in, Features after channel enhancement; Spatial attention weights; Indicates average pooling along the channel dimension; This indicates max pooling along the channel dimension; This represents a two-dimensional convolution operation with a kernel size of 7×7; Indicates feature concatenation operation; The enhanced channel features are multiplied element-wise with the spatial attention weights to complete the spatial dimension feature recalibration and obtain the CBAM output features.

[0012] Preferably, the expression for the decoding feature is: ; in, For the first Layer decoding features; For the first Layer decoding features, i.e., the decoding features of the previous layer; This represents a feature fusion module consisting of convolution, normalization, and activation functions; Indicates an upsampling operation; The first output of the encoder Layer coding features.

[0013] Preferably, the expression for the joint loss function is: ; in, For the joint loss function; Cross-entropy loss is used to constrain pixel-level classification accuracy. This is the Dice loss, used to alleviate the problem of unbalanced foreground and background pixel ratios; This is a boundary loss used to enhance the learning of the contour region; , These are the weighting coefficients for the corresponding loss function; The expression for the binarization mask is: ; in, Represents pixels The binary mask at the location has a value of 1 indicating a bulk material area and a value of 0 indicating a non-bulk material area; Represents pixels Pixel-level category probability map at the location; This is a preset threshold for judgment.

[0014] Preferably, the geometric features include the area ratio of the bulk material region, the compactness of the bulk material region, the aspect ratio of the bulk material region, and the solidity of the bulk material region; The boundary fitting feature includes the fitting rate between the loose material area and each boundary of the image; The texture grayscale features include the average grayscale of the bulk material area, the standard deviation of the grayscale of the bulk material area, and the average gradient of the bulk material area. The structured feature vector is represented as follows: ; ; ; ; ; ; ; ; ; ; ; ; ; ; in, These are structured feature vectors; This represents the percentage of the area occupied by bulk materials. For the compactness of the bulk material area; The aspect ratio of the bulk material area; The solidity of the bulk material area; The adhesion ratio between the bulk material area and the bottom edge of the image; The fit rate between the bulk material area and the left boundary of the image; The fit rate between the bulk material area and the right boundary of the image; The average gray level of the bulk material area; The standard deviation of grayscale in the bulk material area; The average gradient of the bulk material region; This represents the total number of pixels in the bulk material area. The effective imaging area; for The total number of pixels; , The height and width of the image; , These are the x and y coordinates of the pixel. Represents pixels Binarization mask at the location; , , Representing pixels , , Binarization mask at the location; Perimeter of the bulk material area; , Let the length and width be the minimum bounding rectangle of the bulk material area; The area of ​​the convex bulge in the bulk material region; To preprocess the image in pixel coordinates The grayscale value at that location; To preprocess the image at the pixel level Gradient magnitude at; , Indicates the preprocessed image at the pixel level. The gradient in the horizontal and vertical directions.

[0015] Preferably, S4 specifically comprises: The input binarized mask and preprocessed image are concatenated at the channel level and then used as the input tensor to input the visual feature extraction branch. The visual feature extraction branch uses ResNet18 as the backbone network. ResNet18 encodes the input tensor layer by layer through multiple residual modules to obtain multi-layer visual feature maps. Then, the feature pyramid fusion structure is used to upsample and laterally connect the multi-layer visual feature maps at different scales to fuse shallow detail information with deep semantic information to obtain visual features. The structured feature mapping branch includes two fully connected mapping layers. The first fully connected mapping layer maps the input structured feature vector to an intermediate dimension, and the second fully connected mapping layer further compresses and enhances the feature representation capability to obtain a structured representation. The fusion feature is obtained by fusing the visual features and the structured representation through the gated fusion mechanism of the gated fusion module, and the expression is: ; in, The fusion feature; Visual features; For structured representation; Indicates feature concatenation operation; , These are the gating weight and the gating bias term, respectively. Use the Sigmoid activation function; This represents element-wise multiplication; The volume regression branch consists of two fully connected layers and one linear output layer, used to output the predicted volume of bulk materials, expressed as: ; in, This is the predicted volume value; , These represent the weights and biases of the first fully connected layer in the volume regression branch; , These represent the weights and biases of the second fully connected layer in the volume regression branch; It is the ReLU activation function, used to introduce nonlinearity; The load factor regression branch consists of two fully connected layers and one linear output layer, used to output the predicted load factor value, expressed as: ; in, This is the predicted load factor. , These represent the weights and biases of the first fully connected layer in the load rate regression branch; , These represent the weights and biases of the second fully connected layer in the load rate regression branch; The category classification branch consists of two fully connected layers and a Softmax output layer, used to output the probability of bulk material category, expressed as: ; in, Probability of bulk material category; , These represent the weights and biases of the first fully connected layer in the category classification branch; , These represent the weights and biases of the second fully connected layer in the category classification branch; This is the normalized activation function.

[0016] Preferably, the formula for the final corrected volume is: ; in, For the final calibration volume; This is the predicted volume value; For category The probability of; , Categories The slope correction parameters and bias correction parameters are obtained by fitting the calibration data using least squares regression. Total number of categories; The formula for the final load factor is: ; in, For the final load factor; The effective volume of the storage and transportation container; These are the weighting coefficients; This is the predicted load factor.

[0017] This invention also provides a device for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction, comprising: The preprocessing unit is used to acquire and preprocess the loose material images captured by the monocular RGB camera to obtain a preprocessed image. The binarization unit is used to input the preprocessed image into the lightweight image semantic segmentation network and output a binarization mask; The bulk material distribution construction unit is used to construct a structured feature vector representing the bulk material distribution based on the binarized mask and the preprocessed image; the structured feature vector includes geometric features, boundary fitting features, and texture grayscale features; A multi-task prediction unit is used to input the binarized mask, the preprocessed image, and the structured feature vector into a multi-task deep learning model, and output volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The correction unit is used to perform linear weighted correction on the volume prediction value using the probability of the bulk material category to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

[0018] The present invention also provides a construction scenario bulk material volume estimation device based on multi-task joint prediction, including a processor and a memory. The memory stores a computer program, which can be executed by the processor to realize the construction scenario bulk material volume estimation method based on multi-task joint prediction as described above.

[0019] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement the above-described method for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction.

[0020] In summary, compared with the prior art, the present invention has the following beneficial effects: This invention applies an improved lightweight semantic segmentation network and a multi-task deep learning model to estimate the volume of bulk materials in construction scenarios. By combining the MobileNetV3 backbone network with convolutional block attention, the network's feature extraction capability for weakly boundaryed bulk material regions is enhanced while maintaining a lightweight model and reducing computational resource consumption. By introducing a boundary auxiliary branch and using a joint loss function for supervision, the problem of blurred edges of bulk materials caused by complex lighting and shadows at construction sites is effectively solved, resulting in a binarized mask with higher contour fitting accuracy.

[0021] This invention provides physical-level prior information for volume prediction by extracting image-level multidimensional structured features (geometry, boundary fitting, and texture). Dynamic weight allocation of visual and structured features through gated fusion achieves complementarity between high-dimensional semantic information and low-dimensional geometric information. The parallel design of multiple tasks—volume regression, load rate regression, and category classification—allows the model to share feature representations during learning, improving the robustness of volume prediction.

[0022] This invention introduces a category probability-based physical constraint correction in the output stage. By performing linear compensation on the packing characteristics of different materials (such as sand, stone, and soil), it eliminates systematic estimation biases caused by differences in material types. Combined with physical limit constraints on container volume, it further filters out anomalous jump values ​​that the algorithm may generate. This invention not only achieves non-contact, high-precision volume estimation but also exhibits strong adaptability to dust and lighting environments at construction sites, providing an efficient technical means for the intelligent and lightweight deployment of bulk material metering.

[0023] This invention constructs a lightweight computing architecture that eliminates the need for 3D reconstruction. By avoiding computationally complex point cloud modeling, multi-view matching, and scale recovery processes, and employing an end-to-end path of "semantic segmentation + feature fusion regression," the algorithm's computational resource consumption is significantly reduced. This design enables the algorithm to be deployed on resource-constrained edge computing devices or mobile terminals, meeting the real-time measurement needs of construction sites and improving response speed. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a method for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction, as provided in Example 1.

[0026] Figure 2 This is a schematic diagram of a construction scenario bulk material volume estimation device based on multi-task joint prediction, provided in Example 2.

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] Example 1 Embodiment 1 of the present invention provides a method for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction. This method can be implemented by a construction scenario bulk material volume estimation device based on multi-task joint prediction (hereinafter referred to as estimation device), specifically, it can be executed by one or more processors within the estimation device.

[0030] In this embodiment, the estimation device may be an electronic device equipped with a processor. The processor carries a computer program for the construction scenario bulk material volume estimation method based on multi-task joint prediction, and the computer program can be executed. For example, a computer, smartphone, smart tablet, workstation, etc. are not limited here.

[0031] A method for estimating bulk material volume in construction scenarios based on multi-task joint prediction is proposed, primarily applicable to civil engineering construction sites, building material storage and transportation centers, and sand and gravel yards. This method replaces traditional manual measurement with automated visual inspection. Through a monocular vision system combined with a multi-task deep learning model, the entire process from image acquisition, region segmentation, feature extraction to volume prediction and correction is executed. Figure 1 As shown, it includes steps S1 to S5.

[0032] S1: Acquire the bulk material image captured by the monocular RGB camera and preprocess it to obtain the preprocessed image.

[0033] In the material loading area of ​​the construction site, a monocular RGB camera is fixedly mounted on a bracket directly above or to the side of the storage and transportation container. The storage and transportation container is typically a truck container, hopper, or fixed storage tank. Before the monocular RGB camera is put into formal use, the extrinsic parameter matrix from the camera coordinate system to the world coordinate system is determined by using the intrinsic parameter matrix K and combining it with the on-site installation height and angle.

[0034] Intrinsic parameter matrix Represented as: ; in, and These represent the focal lengths in the horizontal and vertical directions, respectively. and These are the horizontal and vertical pixel coordinates of the principal point of the image, respectively.

[0035] The extrinsic parameters from the camera coordinate system to the world coordinate system can be expressed as: ,in, For rotation matrix, It is a translation vector.

[0036] For any point in three-dimensional space Its projection point on the image plane Satisfies the pinhole imaging model: ; in, Scale factor; It is the transpose symbol; Points 3D coordinates; Projection points The horizontal and vertical coordinates.

[0037] In actual operation scenarios, after the storage and transportation container enters the acquisition station and completes the bulk loading, the monocular RGB camera acquires the original bulk material image containing the bulk material accumulation pattern.

[0038] To improve the stability of subsequent semantic segmentation and volume estimation, the loose image is preprocessed. The preprocessing includes distortion correction, size normalization, brightness normalization, and noise suppression.

[0039] Distortion correction is performed on the bulk image to eliminate the influence of lens distortion on the image geometry; then the image is adjusted to a uniform resolution to ensure consistent input size for subsequent networks; and finally, the image brightness is normalized to reduce the impact of differences in image grayscale distribution under different lighting conditions.

[0040] The expression for the brightness normalization is as follows: ; in, This indicates the corrected image at the pixel level. The brightness value at that location; and These represent the mean and standard deviation of the image, respectively. To prevent extremely small constants with a denominator of zero; For the brightness-normalized image at the pixel level The pixel value at that location.

[0041] Subsequently, random noise, local shadow interference, and high-frequency disturbances caused by reflections in the image are suppressed by filtering. For example, the brightness-normalized image is processed by Gaussian filtering or median filtering operators to obtain a preprocessed image.

[0042] S2, input the preprocessed image into the lightweight image semantic segmentation network and output a binarized mask.

[0043] Considering the irregular target boundaries in bulk material areas, significant local shadows and reflective interference, and the high computational resource requirements of on-site deployment, this invention constructs a lightweight image semantic segmentation network suitable for fast segmentation of single images based on a lightweight encoder-decoder structure.

[0044] The lightweight image semantic segmentation network is an improved MobileUNet segmentation network, which includes an encoder and a decoder.

[0045] The encoder uses MobileNetV3 as the backbone network to extract multi-scale coding features layer by layer. Feature downsampling is performed through the inverse residual structure of MobileNetV3. Each layer of the backbone network contains depthwise separable convolutions, batch normalization layers and activation functions, thereby reducing the number of parameters and computation.

[0046] The encoder uses the lightweight MobileNetV3 as the backbone network and replaces traditional convolution with depthwise separable convolution. While maintaining feature extraction capabilities, it significantly reduces the number of parameters and computation time, thereby meeting the real-time processing needs of edge computing devices at construction sites.

[0047] Shallow coding features mainly preserve fine-grained information such as edges, textures, and colors, while deep coding features encode the overall semantic and spatial layout information of the bulk material pile.

[0048] To address issues such as shadows, reflections, and localized overexposure on the surface of bulk materials inside containers, a Convolutional Block Attention Module (CBAM) is introduced at the deep coding feature output of the encoder to jointly recalibrate the channel and spatial dimensions of the deep coding features.

[0049] Specifically, the Convolutional Block Attention (CBAM) module includes a channel attention branch and a spatial attention branch. The channel attention mechanism assigns weights to different feature channels, automatically enhancing channel responses related to the loose material properties and suppressing background redundancy. Simultaneously, a spatial attention mechanism generates a weight map to pinpoint the salient region where the loose material is located in the pixel space dimension.

[0050] In the channel attention branch, global statistical information of the input features is extracted through global average pooling and global max pooling, respectively, and then channel attention weights are generated by a shared multilayer perceptron. The expression for these weights is as follows: ; in, The encoded features are the input. Channel attention weights; Indicates global average pooling; Indicates global max pooling; Indicates a shared multilayer perceptron; This represents the Sigmoid activation function.

[0051] Then, based on the channel attention weights, the input features are multiplied element-wise to complete the channel-dimensional feature recalibration, resulting in the channel-enhanced features, represented as: ; in, Features after channel enhancement; This indicates element-wise multiplication.

[0052] In the spatial attention branch, the channel-enhanced features are subjected to average pooling and max pooling along the channel dimension, respectively. The features obtained from average pooling and max pooling are then concatenated and input into the convolutional layer to obtain the spatial attention weights, which highlight the spatial salience of the loose material region; its expression is: ; in, Features after channel enhancement; Spatial attention weights; Indicates average pooling along the channel dimension; This indicates max pooling along the channel dimension; This represents a two-dimensional convolution operation with a kernel size of 7×7; This indicates a feature splicing operation.

[0053] The enhanced channel features are multiplied element-wise with the spatial attention weights to complete the spatial dimension feature recalibration, resulting in the CBAM output features, represented as: ; in, This is the output feature of CBAM.

[0054] By jointly modeling channel attention and spatial attention, CBAM can simultaneously enhance the response of the bulk material area in both the feature dimension and the spatial dimension, thereby improving the network's robustness to local shadows, reflections, and box edge interference.

[0055] The recalibrated features are then input into the decoder.

[0056] In the decoder section, the decoder includes an upsampling module, skip connections, a feature fusion module, and a boundary auxiliary branch.

[0057] The highest-level deep features enhanced by CBAM attention serve as the decoding starting point. Through bilinear interpolation upsampling, the spatial size of these features is enlarged to match the size of the corresponding shallow-level encoded features in the encoder. Then, the upsampled high-level features are concatenated with the corresponding shallow-level encoded features in the encoder along the channel dimension before being input into the feature fusion module.

[0058] The feature fusion module consists of a convolutional layer, a batch normalization layer, and an activation function. It fuses and refines the concatenated features to generate the decoded features of the current layer.

[0059] The expression for the decoding feature is: ; in, For the first Layer decoding features; For the first Layer decoding features, i.e., the decoding features of the previous layer; This represents a feature fusion module consisting of convolution, normalization, and activation functions; Indicates an upsampling operation; The first output of the encoder Layer coding features.

[0060] This structure can gradually restore the target boundary and spatial details while maintaining low computational overhead, making it suitable for segmentation tasks such as bulk material stacks with complex contours and obvious local undulations.

[0061] Considering that there are usually contact, obstruction or weak boundary phenomena between the bulk material pile and the box boundary, this embodiment further sets boundary auxiliary branches to explicitly learn the contour information of the bulk material area.

[0062] In the boundary auxiliary branch, the shallow encoded features output by the encoder (such as...) are utilized respectively. ), shallow decoding features of the decoder output (such as The system generates boundary probability maps and pixel-level category probability maps, and combines them with joint loss functions to constrain the extraction of bulk material area contours. Through thresholding, a binary mask is generated, providing a reliable input basis for subsequent calculation of bulk material area proportion, contour feature extraction, and volume estimation.

[0063] For example, boundary probability diagrams The expression can be: ; in, Represents the boundary prediction function; This represents the shallow coding features of layers 1 and 2.

[0064] Boundary labels can be obtained from the real segmentation mask through morphological gradient operations. Boundary probability maps are used to characterize the boundary positions between bulk materials and boxes, and between bulk materials and the background, and form contour constraints on the main segmentation branches during training.

[0065] Pixel-level category probability map The expression can be: ; in, and This represents the weights and bias parameters of the output layer; These are the features of the first layer of shallow decoding; This is the Sigmoid activation function.

[0066] The expression for the joint loss function is: ; in, For the joint loss function; Cross-entropy loss is used to constrain pixel-level classification accuracy. This is the Dice loss, used to alleviate the problem of unbalanced foreground and background pixel ratios; This is a boundary loss used to enhance the learning of the contour region; , These are the weight coefficients corresponding to the loss function.

[0067] This embodiment uses joint loss to supervise the network, so as to balance the overall region segmentation accuracy and boundary recovery capability.

[0068] The expression for the binarization mask is: ; in, Represents pixels The binary mask at the location has a value of 1 to represent a loose material area and a value of 0 to represent a non-loose material area, such as the background / box area; Represents pixels Pixel-level category probability map at the location; This is a preset threshold for judgment.

[0069] To train the lightweight image semantic segmentation network, a container bulk material image dataset needs to be constructed. The dataset is obtained by capturing images of the interior of containers under different loading states, lighting conditions, and bulk material stacking patterns using a fixed-mounted camera. The captured objects include typical construction bulk material scenarios such as sand, gravel, and sand-gravel mixtures. Subsequently, the acquired raw images are filtered to remove blurry and severely distorted images, and the bulk material regions are manually labeled at the pixel level to form corresponding binary mask labels, where the bulk material regions are labeled as foreground, and the container body and background regions are labeled as non-foreground. To improve the model's generalization ability, the dataset undergoes enhancement processing such as random flipping, random cropping, brightness perturbation, and contrast perturbation before training. Furthermore, the dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used for model parameter learning, the validation set is used for model tuning and hyperparameter selection during training, and the test set is used to evaluate the final segmentation performance of the model, thereby ensuring that the constructed model has good robustness and stability under different loading scenarios.

[0070] S3. Based on the binarized mask and the preprocessed image, construct a structured feature vector representing the distribution of bulk materials; the structured feature vector includes geometric features, boundary fitting features and texture grayscale features.

[0071] After obtaining the semantic segmentation result of the bulk material region—the binarized mask—this embodiment further performs image-level feature modeling on the segmentation result to extract multi-dimensional features that can characterize the spatial distribution, contour morphology, boundary contact relationships, and surface texture differences of the bulk material. The purpose of this step is to transform the pixel-level bulk material region information obtained from semantic segmentation into a compact feature representation that can be used by the volume regression model, thereby providing a more stable and discriminative input for subsequent volume estimation.

[0072] Considering that image resolution, material distance, and imaging scale may vary under different shooting conditions, this embodiment prioritizes the use of normalized statistics during feature construction to reduce the impact of changes in external shooting parameters on feature stability.

[0073] Specifically, the geometric features include the area ratio of the loose material region, the compactness of the loose material region, the aspect ratio of the loose material region, and the solidity of the loose material region; the boundary fitting features include the fitting rate between the loose material region and each boundary of the image; the texture grayscale features include the average grayscale of the loose material region, the standard deviation of the grayscale of the loose material region, and the average gradient of the loose material region.

[0074] The structured feature vector is represented as follows: ; ; ; ; ; ; ; ; ; ; ; ; ; ; in, These are structured feature vectors; This represents the percentage of the area occupied by bulk materials. For the compactness of the bulk material area; The aspect ratio of the bulk material area; The solidity of the bulk material area; The adhesion ratio between the bulk material area and the bottom edge of the image; The fit rate between the bulk material area and the left boundary of the image; The fit rate between the bulk material area and the right boundary of the image; The average gray level of the bulk material area; The standard deviation of grayscale in the bulk material area; The average gradient of the bulk material region; This represents the total number of pixels in the bulk material area. The effective imaging area can be obtained by pre-setting a mask under a fixed camera pose; for The total number of pixels; , The height and width of the image; , These are the x and y coordinates of the pixel. Represents pixels Binarization mask at the location; , , Representing pixels , , Binarization mask at the location; Perimeter of the bulk material area; , Let the length and width be the minimum bounding rectangle of the bulk material area; The area of ​​the convex bulge in the bulk material region; To preprocess the image in pixel coordinates The grayscale value at that location; To preprocess the image at the pixel level Gradient magnitude at; , Indicates the preprocessed image at the pixel level. The gradient in the horizontal and vertical directions.

[0075] In this embodiment, the perimeter of the bulk material area A contour extraction algorithm can be used to extract the outer contour of the bulk material area from the binary mask, and the total perimeter can be obtained by accumulating the Euclidean distances between adjacent contour points.

[0076] The minimum bounding rectangle can be calculated using the minimum bounding rectangle algorithm. Specifically, first, principal component analysis (PCA) or a rotated bounding box is performed on the outline points of the bulk material region; then, the minimum area rectangle that can completely enclose the bulk material region is obtained; the longer side of this rectangle is the length. The shorter side is the width .

[0077] Bulk material area convex bulge area A convex hull algorithm can be used to fit the convex hull of the point set of the bulk material area to obtain the smallest convex polygon that can enclose the bulk material area, and its area can be calculated using the polygon area formula.

[0078] The area ratio of bulk material reflects the overall extent to which bulk material occupies the image plane. For container loading scenes with a fixed camera, as the volume of bulk material increases, the bulk material area typically covers a larger area in the image. Therefore, The larger the value, the larger the area occupied by the bulk material in the image, and the higher the corresponding loading capacity.

[0079] Compactness describes the regularity of the boundaries of bulk material areas. When the boundaries of bulk material areas are relatively smooth and the shape is close to a regular pattern, the compactness is usually high. However, when the edges of the bulk material are tortuous, uneven, or have obvious local undulations, the compactness will decrease accordingly.

[0080] Aspect ratio is used to characterize the direction of expansion and morphological offset of bulk material areas on the image plane. When the bulk material stack expands significantly along the length or width of the container, the aspect ratio will show a strong difference.

[0081] Solidity measures the degree of filling and boundary depression of a bulk material area. Solidity decreases when there are many depressions and voids at the edges of the bulk material pile; it increases when the overall continuity of the bulk material area is strong and the boundaries are relatively intact. Compared to compactness, solidity focuses more on reflecting the internal voids and boundary depressions of the area.

[0082] The fit rate between the bulk material area and the left edge of the image. Fit to the right boundary This describes the degree of contact between the bulk material area and the container boundary in the lateral direction. Since container images taken by fixed cameras usually have a stable geometry, whether the bulk material is close to the bottom edge is often closely related to the loading height and loading degree, while whether the bulk material touches the side edge reflects its lateral expansion range.

[0083] Average gray level primarily reflects the overall brightness level of the bulk material area, while the standard deviation of gray level reflects the range of brightness variation within the area. For sandy materials that are less affected by light and have a more uniform surface, their gray level distribution is usually more gradual; while for crushed stone materials with more distinct particle boundaries and greater surface roughness, their gray level changes are often more drastic.

[0084] The average gradient reflects the edge strength and texture complexity within the bulk material zone. Generally, crushed stone and sand-gravel mixtures have stronger gradient responses due to their larger particle size and irregular surfaces; while fine-grained sand has a relatively flat surface and a gentler gradient change.

[0085] After semantic segmentation, this embodiment comprehensively extracts various image-level information such as the area ratio, contour complexity, aspect ratio, solidity, boundary fit rate, grayscale statistics, gradient features, and texture features of the loose material region to construct a feature space with strong expressive power. This step uniformly encodes the visible geometric shape and surface appearance features of the loose material in a single image, providing input conditions for volume regression modeling in step S4.

[0086] S4, the binarized mask, the preprocessed image, and the structured feature vector are input into a multi-task deep learning model, which outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively.

[0087] After completing the image-level feature extraction of bulk materials, the preprocessed image, binarized mask, and structured feature vector are input into a multi-task deep learning model to jointly predict the bulk material volume, loading rate, and category.

[0088] Visual features retain information such as the shape of bulk material edges, changes in the pile slope, and the distribution of regional textures, and can reflect the geometric differences under different loading conditions.

[0089] Structured representation enables more stable representations of area percentage, convex hull area, perimeter, compactness, boundary fit, and texture statistics in the deep feature space.

[0090] Specifically: The input binarized mask and preprocessed image are concatenated at the channel level and then used as the input tensor to input the visual feature extraction branch. The visual feature extraction branch uses ResNet18 as the backbone network. ResNet18 encodes the input tensor layer by layer through multiple residual modules to obtain multi-layer visual feature maps. Then, the feature pyramid fusion structure is used to upsample and laterally connect the multi-layer visual feature maps at different scales to fuse shallow detail information with deep semantic information to obtain visual features. The structured feature mapping branch includes two fully connected mapping layers. The first fully connected mapping layer maps the input structured feature vector to an intermediate dimension, and the second fully connected mapping layer further compresses and enhances the feature representation capability to obtain a structured representation. The fusion feature is obtained by fusing the visual features and the structured representation through the gated fusion mechanism of the gated fusion module, and the expression is: ; in, The fusion feature; Visual features; For structured representation; Indicates feature concatenation operation; , These are the gating weight and the gating bias term, respectively. Use the Sigmoid activation function; This indicates element-wise multiplication.

[0091] This fusion method automatically adjusts the contribution ratio of deep image features to explicit image-level features based on the visual complexity of the input samples, ensuring stable output even in scenarios with significant lighting variations, complex material textures, and irregular boundary shapes. In well-lit and clear-textured scenarios, the gated fusion module automatically increases the proportion of visual features; in extreme scenarios with severe occlusion or drastic lighting fluctuations, it increases the proportion of structured features with robust statistical properties, thereby ensuring the fused features are balanced. It maintains extremely strong characterization capabilities under various working conditions.

[0092] In terms of output structure, the model sets up three parallel task heads: volume regression branch, loading rate regression branch, and category classification branch.

[0093] The volume regression branch consists of two fully connected layers and one linear output layer, used to output the predicted volume of bulk materials, expressed as: ; in, This is the predicted volume value; , These represent the weights and biases of the first fully connected layer in the volume regression branch; , These represent the weights and biases of the second fully connected layer in the volume regression branch; It is the ReLU activation function, used to introduce nonlinearity; The load factor regression branch consists of two fully connected layers and one linear output layer, used to output the predicted load factor value, expressed as: ; in, This is the predicted load factor. , These represent the weights and biases of the first fully connected layer in the load rate regression branch; , These represent the weights and biases of the second fully connected layer in the load rate regression branch; The category classification branch consists of two fully connected layers and a Softmax output layer, used to output the probability of bulk material category, expressed as: ; in, Probability of bulk material category; , These represent the weights and biases of the first fully connected layer in the category classification branch; , These represent the weights and biases of the second fully connected layer in the category classification branch; This is the normalized activation function.

[0094] Furthermore, the multi-task deep learning model training employs a multi-task joint optimization strategy. Volume labels are obtained through standardized volume calibration, while loading rate labels are calculated as the ratio of bulk material volume to the effective container volume. Category labels are directly obtained by labeling the bulk material type corresponding to the collected samples.

[0095] The training sample set was acquired by a fixed camera under the same shooting geometry. Each sample contains the original image, pixel-level segmentation mask, image-level feature vector, volume label, loading rate label, and category label. To ensure label accuracy, the volume data was constructed using a step-by-step quantitative loading method, that is, using a standard measuring tool with known volume to load the bulk material in layers, and recording the corresponding image and cumulative volume at each loading stage.

[0096] The dataset covers various loading states, including empty, low-load, medium-load, high-load, and near-full-load, and encompasses three types of bulk materials: sand, gravel, and sand-gravel mixtures. It also includes image samples with different particle sizes, different accumulation slope angles, and different lighting conditions. Before training, the original images underwent size unification, brightness normalization, noise suppression, and color standardization. The training set was then subjected to random brightness perturbation, random contrast perturbation, random rotation, random translation, and random scaling to enhance the model's adaptability to complex on-site environments. The dataset is divided into training, validation, and test sets in an 8:1:1 ratio.

[0097] After the model training is complete, a single image to be tested and its corresponding mask are input during the inference phase. and structured feature vectors The model can output the bulk material volume in one go. Loading rate and bulk material category probability .

[0098] The advantage of multi-task learning is that, by sharing a backbone network, the model can learn the intrinsic relationship between material type and volume distribution during the optimization process. For example, gravel has a larger porosity, and its visual features and volume mapping logic are systematically different from those of fine sand. The multi-task architecture forces the model to distinguish these physical characteristics in its internal representation.

[0099] S5, perform linear weighted correction on the volume prediction value using the bulk material category probability to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

[0100] Considering the differences in packing density, porosity and surface morphology among different bulk material categories, the volume prediction results output in step S4 may deviate between different categories. Therefore, this embodiment corrects the volume prediction values ​​based on the category information.

[0101] The formula for the final corrected volume is: ; in, For the final calibration volume; This is the predicted volume value; For category The probability of; , Categories The slope correction parameters and bias correction parameters are obtained by fitting the data onto the calibration data using least squares regression. For example, for different types of bulk materials (such as sand / gravel / mixtures), no fewer than 500 sets of images under different loading conditions are collected, their actual volumes are recorded, and the volume prediction values ​​are fitted using the least squares method. The linear regression equation with respect to the actual volume. The total number of categories, such as 3 categories (sand / gravel / mixed materials).

[0102] The formula for the final load factor is: ; in, For the final load factor; The effective volume of the storage and transportation container; These are weighting coefficients used to balance the results of direct regression predictions and geometric calculations; This is the predicted load factor.

[0103] It can also be used for Perform a range truncation operation to force the range to a reasonable interval between 0 and 1. If If the calculation result exceeds this range, outlier removal or smoothing will be performed according to the preset confidence level logic.

[0104] In a preferred embodiment, under the conditions of the same shooting distance, the same effective container volume, and the same bulk material projection area, the volume of sand and gravel were estimated and corrected respectively, and the comparison results are as follows: (1) Difference in actual volume under the same projected area: Sand: It has high fluidity, small angle of repose, and high porosity, resulting in a smaller actual volume for the same projected area; Crushed stone: large particles, large angle of repose, low porosity, resulting in a significantly larger actual volume for the same projected area.

[0105] (2) Before correction (direct model prediction), since the image areas are consistent, the predicted volumes are close: Sand prediction value: =11.8m³, predicted value of crushed stone: =11.8m³.

[0106] (3) After correction (category-weighted correction): Sand correction factor: =0.85, =0.03, =0.85×11.8+0.03=10.06 m³; Crushed stone correction factor: =1.18, =0.04, =1.18×11.8+0.04=13.96 m³.

[0107] , Indicates the correction parameters for the sand; , This indicates the calibration parameters for crushed stone.

[0108] (4) Comparison conclusion: Same area ≠ same volume. This embodiment can automatically compensate for the systematic deviation caused by different bulk material repacking angles and porosities through category probability weighting correction, so that the estimation results are consistent with the real physical laws.

[0109] In practical applications, such as an automated construction waste collection point, a monocular RGB camera is mounted on a gantry at the entrance. When a truck loaded with earth or gravel passes by, the system automatically triggers the data acquisition process. Step S1 performs image distortion correction and normalization. Step S2 accurately segments the earth area inside the container, shielding the truck front and road background. Step S3 calculates the earth coverage ratio and edge fit within the container. Step S4, the multi-task model identifies the material type as "earth" and provides a preliminary volume. Step S5 performs compensation based on the earth's compaction correction parameters, outputting the final cubic volume and loading percentage. The entire process is completed within seconds, without requiring the truck to stop or requiring manual measurement by climbing the vehicle with a tape measure.

[0110] For example, in the aggregate metering process at a concrete mixing plant, a monocular RGB camera is fixed to the top of the silo. When the conveyor belt stops and the material surface is relatively still, the system automatically triggers the acquisition process. The processor calculates the current volume of sand and gravel in the silo in real time and transmits the data to the monitoring terminal in the central control room via the CAN bus. Thanks to the use of a lightweight MobileNetV3 backbone and an efficient feature fusion mechanism, the entire estimation process takes less than 200ms on the embedded processor, meeting the real-time metering requirements of construction sites. Furthermore, due to the introduction of boundary auxiliary branches and physical constraints, the system maintains high measurement accuracy even in harsh environments with insufficient lighting at night or high dust levels, effectively solving the technical challenge of poor stability of traditional monocular vision solutions in complex construction environments.

[0111] In summary, compared with the prior art, the present invention has the following beneficial effects: This invention employs an improved lightweight image semantic segmentation network to perform high-precision pixel-level region locking. It utilizes a CBAM attention module to handle complex lighting interference and combines a boundary-assisted branch to address the challenge of weak boundary segmentation, ensuring the purity of the input data. By introducing image-level multidimensional structured feature vectors, it compensates for the scale recovery limitations of monocular vision systems when depth information is lacking. A gated fusion module achieves complementary advantages between visual semantics and geometric statistical information. Finally, through physical correction based on category probability, it eliminates estimation errors caused by differences in material physical properties.

[0112] The entire process does not require complex 3D reconstruction or point cloud processing, which greatly reduces computational overhead and enables high-precision bulk material volume measurement to run stably on a low-cost hardware platform, providing an efficient and non-contact digital measurement solution for construction sites.

[0113] Example 2 like Figure 2 As shown, the second embodiment of the present invention also provides a device for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction, comprising: The preprocessing unit is used to acquire and preprocess the loose material images captured by the monocular RGB camera to obtain a preprocessed image. The binarization unit is used to input the preprocessed image into the lightweight image semantic segmentation network and output a binarization mask; The bulk material distribution construction unit is used to construct a structured feature vector representing the bulk material distribution based on the binarized mask and the preprocessed image; the structured feature vector includes geometric features, boundary fitting features, and texture grayscale features; A multi-task prediction unit is used to input the binarized mask, the preprocessed image, and the structured feature vector into a multi-task deep learning model, and output volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The correction unit is used to perform linear weighted correction on the volume prediction value using the probability of the bulk material category to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

[0114] Through the coordinated operation of the aforementioned units and steps, this invention achieves accurate non-contact measurement of bulk material volume. A monocular RGB camera serves as the perception layer, acquiring rich visual information; an improved MobileUNet segmentation network serves as the segmentation layer, achieving pixel-level target extraction; multi-dimensional feature modeling and a multi-task prediction model serve as the decision layer, deeply integrating semantic information and geometric priors; and physical constraint correction serves as the optimization layer, ensuring that the results conform to objective physical laws. This progressive technical architecture enables this invention to significantly improve the intelligence level of bulk material volume estimation in construction scenarios based on multi-task joint prediction without increasing hardware costs.

[0115] Example 3 The third embodiment of the present invention also provides a construction scenario bulk material volume estimation device based on multi-task joint prediction, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the construction scenario bulk material volume estimation method based on multi-task joint prediction as described above.

[0116] Example 4 The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, the computer-readable instructions implement the above-described method for estimating the volume of bulk materials in a construction scenario based on multi-task joint prediction.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating bulk material volume in construction scenarios based on multi-task joint prediction, characterized in that, include: S1, acquire and preprocess the bulk material image captured by the monocular RGB camera to obtain the preprocessed image; S2, input the preprocessed image into the lightweight image semantic segmentation network and output a binarized mask; S3, Based on the binarized mask and the preprocessed image, construct a structured feature vector representing the distribution of bulk materials; the structured feature vector includes geometric features, boundary fitting features, and texture grayscale features; S4, the binarized mask, the preprocessed image, and the structured feature vector are input into a multi-task deep learning model, which outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. S5, perform linear weighted correction on the volume prediction value using the bulk material category probability to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

2. The method for estimating bulk material volume in construction scenarios based on multi-task joint prediction as described in claim 1, characterized in that... The preprocessing includes distortion correction, size normalization, brightness normalization, and noise suppression; wherein, the expression for brightness normalization is: ； in, This indicates the corrected image at the pixel level. The brightness value at that location; and These represent the mean and standard deviation of the image, respectively. To prevent extremely small constants with a denominator of zero; For the brightness-normalized image at the pixel level Pixel value at; The image is preprocessed by applying a Gaussian filter or a median filter to the brightness-normalized image to remove random noise, local shadow interference, and high-frequency disturbances caused by reflections.

3. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 1, characterized in that... The lightweight image semantic segmentation network is an improved MobileUNet segmentation network, which includes an encoder and a decoder; In the encoder, MobileNetV3 is used as the backbone network to extract multi-scale coding features layer by layer. Feature downsampling is performed through the inverse residual structure of MobileNetV3. Each layer of the backbone network contains depthwise separable convolution, batch normalization layer and activation function. Convolutional block attention module CBAM is introduced at the deep coding feature output of the encoder to jointly recalibrate the channel dimension and spatial dimension of the deep coding features. In the decoder, the highest layer features output by the Convolutional Block Attention (CBAM) module are upsampled by bilinear interpolation, and then channel-wise concatenated with the shallow encoded features from the encoding stage using skip connections. The current layer's decoded features are obtained through feature fusion. Meanwhile, the decoder sets a boundary auxiliary branch, uses the shallow coding features output by the encoder to generate a boundary probability map, uses the shallow decoding features output by the decoder to generate a pixel-level category probability map, and combines the joint loss function to constrain the extraction of the bulk material area contour, and generates a binary mask through thresholding.

4. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 3, characterized in that... The Convolutional Block Attention (CBAM) module includes a channel attention branch and a spatial attention branch. In the channel attention branch, global statistical information of the input features is extracted through global average pooling and global max pooling, respectively, and then channel attention weights are generated by a shared multilayer perceptron. The expression for these weights is as follows: ； in, The encoded features are the input. Channel attention weights; Indicates global average pooling; Indicates global max pooling; Indicates a shared multilayer perceptron; This represents the Sigmoid activation function; The input features are multiplied element-wise based on channel attention weights to complete the channel-dimensional feature recalibration and obtain the channel-enhanced features. In the spatial attention branch, the channel-enhanced features are subjected to average pooling and max pooling along the channel dimension, respectively. The features obtained from average pooling and max pooling are then concatenated and input into the convolutional layer to obtain the spatial attention weights; their expression is: ； in, Features after channel enhancement; Spatial attention weights; Indicates average pooling along the channel dimension; This indicates max pooling along the channel dimension; This represents a two-dimensional convolution operation with a kernel size of 7×7; Indicates feature concatenation operation; The enhanced channel features are multiplied element-wise with the spatial attention weights to complete the spatial dimension feature recalibration and obtain the CBAM output features.

5. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 3, characterized in that... The expression for the decoding feature is: ； in, For the first Layer decoding features; For the first Layer decoding features, i.e., the decoding features of the previous layer; This represents a feature fusion module consisting of convolution, normalization, and activation functions; Indicates an upsampling operation; The first output of the encoder Layer coding features.

6. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 3, characterized in that... The expression for the joint loss function is: ； in, For the joint loss function; Cross-entropy loss is used to constrain pixel-level classification accuracy. This is the Dice loss, used to alleviate the problem of unbalanced foreground and background pixel ratios; This is a boundary loss used to enhance the learning of the contour region; , These are the weighting coefficients for the corresponding loss function; The expression for the binarization mask is: ； in, Represents pixels The binary mask at the location has a value of 1 indicating a bulk material area and a value of 0 indicating a non-bulk material area; Represents pixels Pixel-level category probability map at the location; This is a preset threshold for judgment.

7. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 6, characterized in that... The geometric features include the area ratio of the bulk material region, the compactness of the bulk material region, the aspect ratio of the bulk material region, and the solidity of the bulk material region. The boundary fitting feature includes the fitting rate between the loose material area and each boundary of the image; The texture grayscale features include the average grayscale of the bulk material area, the standard deviation of the grayscale of the bulk material area, and the average gradient of the bulk material area. The structured feature vector is represented as follows: ； ； ； ； ； ； ； ； ； ； ； ； ； ； in, These are structured feature vectors; This represents the percentage of the area occupied by bulk materials. For the compactness of the bulk material area; The aspect ratio of the bulk material area; The solidity of the bulk material area; The adhesion ratio between the bulk material area and the bottom edge of the image; The fit rate between the bulk material area and the left boundary of the image; The fit rate between the bulk material area and the right boundary of the image; The average gray level of the bulk material area; The standard deviation of grayscale in the bulk material area; The average gradient of the bulk material region; This represents the total number of pixels in the bulk material area. The effective imaging area; for The total number of pixels; , The height and width of the image; , These are the x and y coordinates of the pixel. Represents pixels Binarization mask at the location; , , Representing pixels , , Binarization mask at the location; Perimeter of the bulk material area; , Let the length and width be the minimum bounding rectangle of the bulk material area; The area of ​​the convex bulge in the bulk material region; To preprocess the image in pixel coordinates The grayscale value at that location; To preprocess the image at the pixel level Gradient magnitude at; , Indicates the preprocessed image at the pixel level. The gradient in the horizontal and vertical directions.

8. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 7, characterized in that... S4 specifically refers to: The input binarized mask and the preprocessed image are concatenated at the channel level and then used as the input tensor to input the visual feature extraction branch. The visual feature extraction branch uses ResNet18 as the backbone network. ResNet18 encodes the input tensor layer by layer through multiple residual modules to obtain multi-layer visual feature maps. Then, the feature pyramid fusion structure is used to upsample and laterally connect the multi-layer visual feature maps at different scales to fuse shallow detail information with deep semantic information to obtain visual features. The structured feature mapping branch includes two fully connected mapping layers. The first fully connected mapping layer maps the input structured feature vector to an intermediate dimension, and the second fully connected mapping layer further compresses and enhances the feature representation capability to obtain a structured representation. The fusion feature is obtained by fusing the visual features and the structured representation through the gated fusion mechanism of the gated fusion module, and the expression is: ； in, The fusion feature; Visual features; For structured representation; Indicates feature concatenation operation; , These are the gating weight and the gating bias term, respectively. Use the Sigmoid activation function; This represents element-wise multiplication; The volume regression branch consists of two fully connected layers and one linear output layer, used to output the predicted volume of bulk materials, expressed as: ； in, This is the predicted volume value; , These represent the weights and biases of the first fully connected layer in the volume regression branch; , These represent the weights and biases of the second fully connected layer in the volume regression branch; It is the ReLU activation function, used to introduce nonlinearity; The load factor regression branch consists of two fully connected layers and one linear output layer, used to output the predicted load factor value, expressed as: ； in, This is the predicted load factor. , These represent the weights and biases of the first fully connected layer in the load rate regression branch; , These represent the weights and biases of the second fully connected layer in the load rate regression branch; The category classification branch consists of two fully connected layers and a Softmax output layer, used to output the probability of bulk material category, expressed as: ； in, Probability of bulk material category; , These represent the weights and biases of the first fully connected layer in the category classification branch; , These represent the weights and biases of the second fully connected layer in the category classification branch; This is the normalized activation function.

9. The method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in claim 7, characterized in that... The formula for the final corrected volume is: ； in, For the final calibration volume; This is the predicted volume value; For category The probability of; , Categories The slope correction parameters and bias correction parameters are obtained by fitting the calibration data using least squares regression. Total number of categories; The formula for the final load factor is: ； in, For the final load factor; The effective volume of the storage and transportation container; These are the weighting coefficients; This is the predicted load factor.

10. A device for estimating bulk material volume in a construction scenario based on multi-task joint prediction, used to implement the method for estimating bulk material volume in a construction scenario based on multi-task joint prediction as described in any one of claims 1-9, characterized in that, include: The preprocessing unit is used to acquire and preprocess the loose material images captured by the monocular RGB camera to obtain a preprocessed image. The binarization unit is used to input the preprocessed image into the lightweight image semantic segmentation network and output a binarization mask; The bulk material distribution construction unit is used to construct a structured feature vector representing the bulk material distribution based on the binarized mask and the preprocessed image; the structured feature vector includes geometric features, boundary fitting features, and texture grayscale features; A multi-task prediction unit is used to input the binarized mask, the preprocessed image, and the structured feature vector into a multi-task deep learning model, and output volume prediction, loading rate prediction, and bulk material category probability, respectively. The multi-task deep learning model includes a visual feature extraction branch, a structured feature mapping branch, a gated fusion module, and parallel volume regression, loading rate regression, and category classification branches. The preprocessed image and the binarized mask are used to extract visual features through the visual feature extraction branch. The structured feature vector is input into the structured feature mapping branch to extract a structured representation. The gated fusion module fuses the visual features and the structured representation to obtain a fused feature. Each parallel branch uses the fused feature as input and outputs volume prediction, loading rate prediction, and bulk material category probability, respectively. The correction unit is used to perform linear weighted correction on the volume prediction value using the probability of the bulk material category to obtain the final corrected volume, and calculate the final loading rate by combining the effective volume of the storage and transportation container.

Images