Method and system for detecting bucket fullness rate and material weight of electric shovel based on deep learning

Abstract

The invention relates to the technical field of shovel loading robots, in particular to an electric shovel bucket fullness rate and material weight detection method and system based on deep learning. Comprising the following steps: S100, acquiring point cloud data of the surface of a material pile before and after an electric shovel excavates materials in real time; image information of the bucket and the materials in the bucket after the materials are excavated is obtained in real time; s200, preprocessing the collected point cloud data and image information to improve the data quality; s300, calculating the volume of the excavated material according to the point cloud data before and after the material is excavated; s400, training a multi-task neural network model, wherein the multi-task neural network model is used for outputting an accurate material volume predicted value and a density predicted value; s500, according to the obtained predicted value of the material volume and the rated volume of the bucket, the bucket fullness rate of the bucket is calculated, a large amount of bucket fullness rate data is analyzed, and the stability of the bucket in the material excavating process is detected; and S600, the weight of the materials in the bucket is calculated according to the material volume predicted value and the density predicted value.

Description

Technical Field

[0001] The present invention relates to the technical field of shovel loading robots, and in particular to a method and system for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning. Background Art

[0002] The bucket fill rate refers to the ratio of the volume of material shoveled into the bucket to the rated capacity of the bucket. The material weight refers to the product of the volume of material shoveled into the bucket and the density of the corresponding material. The bucket fill rate and material weight are important parameters for determining the mechanical performance and productivity of the shovel loading robot. The bucket fill rate information can be used to adjust the autonomous excavation strategy to optimize the operation effect.

[0003] In recent years, the demand for efficient and precise material excavation and loading has increased significantly in engineering fields such as mining, open-pit coal mining, and large-scale infrastructure construction. As core equipment for material excavation and loading in coal mining, the efficiency of shovel-loading robots directly impacts the cost and output of the entire production process. Common methods for determining bucket fill rate have typically used machine vision and bucket position information to calculate the fill rate. This method combines the empty and full bucket position information acquired by displacement sensors with the material's external contour information captured by machine vision to calculate the volume of the loaded material. Alternatively, neural network-based image material detection algorithms can be used to directly acquire depth information to calculate the material volume and fill rate. However, when calculating material weight, the pile may contain multiple materials, such as rock, gravel, and sand. This results in non-constant material density, making it difficult to accurately determine the material weight. Furthermore, existing methods require a complete image of the bucket contents, which imposes strict limitations on sensor placement. Furthermore, for shovel-loading robots, the sensor equipment is susceptible to vibration. Furthermore, vision-based methods cannot effectively handle conditions at night or in poor lighting. Under adverse operating conditions, the loader robot may be unable to capture images or may produce poor image quality, making it impossible to complete the bucket fill rate measurement. Furthermore, existing methods rely on a single measurement method, which cannot guarantee full-range measurement accuracy. In the event of data failure, the measurement task cannot be completed. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method and system for calculating the fullness rate and material weight of an electric shovel bucket based on deep learning.

[0005] The present invention adopts the following technical solution: a method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning, comprising: S100: Real-time acquisition of point cloud data of the pile surface before and after the shovel excavates the material; Real-time acquisition of image information of the bucket and the material inside the bucket after excavation; S200: pre-processing the collected point cloud data and image information to improve data quality; S300: Calculating the volume of the excavated material based on the point cloud data before and after excavation; S400: Train a multi-task neural network model, which is used to output accurate material volume and density prediction values; S500: Calculating the bucket fullness rate based on the obtained material volume prediction value and the rated capacity of the bucket, analyzing a large amount of bucket fullness rate data, and detecting the stability of the bucket excavating the material; S600: Calculate the weight of the material in the bucket according to the predicted value of the material volume and the predicted value of the density.

[0006] In some embodiments, S300 includes: S301: Registering the point cloud of the material pile before the electric shovel excavates the material with the point cloud of the material pile after the electric shovel excavates the material based on a point cloud registration algorithm; S302: Converting the registered point cloud data into the same coordinate system; S303: Calculate the volume change between the material pile surface before and after excavation based on the position information of the two point clouds in the same coordinate system, and thus preliminarily calculate the volume of the material in the bucket. .

[0007] In some embodiments, S400 includes: S401: Acquire multi-source data related to materials, including preliminary material volume data, material true density, image data, material type, and ratio information of different materials; S402: Preprocessing the acquired data, including image denoising, filtering, and rectification, normalizing volume and density data, and encoding material type and proportion information; S403: Divide the preprocessed data into a training set, a validation set, and a test set according to a ratio of 7:2:1, and start training the neural network using the training set; S404: Build a Transformer-based multi-task neural network model and train it using the data preprocessed in S402, including: A shared feature extraction layer, which uses convolutional layers and pooling layers to extract features from image data and fuses non-image data with image features through a fully connected layer; A task-specific layer, wherein the task-specific layer is divided into a volume prediction branch and a density prediction branch; S405: Define loss function and optimizer; S406: Perform multiple forward propagations to calculate the mean and variance of the volume prediction value and the density prediction value; S407: Filter high-confidence data based on the set confidence threshold, and iteratively expand the training set to optimize the model.

[0008] In some embodiments, S406 includes: Traverse the model's prediction results for all data and calculate the volume prediction confidence separately and density prediction confidence ,Will and volume confidence threshold Compare, and density confidence threshold , only when ≥ and ≥ When the data point is selected, the selected data includes image data, material volume prediction value, material density prediction value, and material type and proportion; The selected data is quality checked and added to the original dataset to retrain the model.

[0009] In some embodiments, S500 includes: The volume of material in the bucket obtained by prediction and rated capacity of the bucket Calculate the full bucket rate, that is , by detecting and analyzing the full bucket rate data, the stability of the bucket in excavating materials is detected.

[0010] Analyze a large amount of bucket fill rate data to detect the stability of the bucket excavation process, including: S501: Obtaining full bucket rate data according to the multi-task neural network model; S502: Analyze whether the data is normally distributed; S503: Select the mean-range control chart, divide all data into several subgroups, and calculate the mean of each subgroup and range , and then calculate the mean value of all subgroups and the average of all subgroup ranges , and then calculate the control limits using the control chart coefficients based on the subgroup size n, where The center line of the mean chart is: , the upper control limit is: , the lower control limit is: , is the control chart coefficient; The center line of the standard deviation chart is: , the upper control limit is: , the lower control limit is: , 、 is the control chart coefficient; S504: Draw a control chart, plot the calculated mean, range data, and corresponding control lines on the control chart, with each subgroup's data corresponding to a position on the chart; S505: Analyze the control chart to determine whether the existing data process is stable. When the data is unstable, try to find the cause of the test instability by reproducing or backtracking the scenarios and processes that produced the abnormal data, and then improve the test. Regularly evaluate the improved test methods and results to ensure the effectiveness of the improvements.

[0011] In some embodiments, S600 includes: According to the predicted volume With predicted density Calculate the weight of the material in the bucket, i.e. .

[0012] Accurate material weight detection: By building a multi-task neural network model based on Transformer, material density can be accurately predicted. In complex actual working conditions, the material composition of the pile is diverse. Traditional methods have difficulty accurately measuring material weight due to the non-constant material density. This method can effectively handle a variety of material conditions, providing reliable data support for material weight calculation, significantly improving the accuracy of material weight measurement, and meeting the requirements for accurate material weight measurement in mining and open-pit coal mining. High-precision bucket fill rate prediction: The system uses point cloud data from the stockpile surface before and after excavation to preliminarily calculate the material volume. Using a semi-supervised learning algorithm, this volume data serves as input for a neural network model to obtain more accurate material volumes, significantly improving the accuracy of bucket fill rate calculations. Furthermore, SPC is used to monitor the stability of the excavation process, helping to promptly identify anomalies during excavation operations and enable adjustments and improvements.

[0013] Adaptable to Complex Environments: Using a LiDAR sensor to collect point cloud data, combined with a visible light and infrared camera to capture image information, the system can collect data even at night or in harsh, low-light environments. Compared to traditional vision-based methods, this effectively overcomes lighting limitations, ensuring stable and accurate material weight measurement in a variety of complex operating environments. This enhances the detection system's environmental adaptability and reliability, ensuring the continuity and stability of shoveling operations.

[0014] A method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning is provided, and includes: A data acquisition unit, which is used to collect point cloud data and image information: A data preprocessing unit, configured to perform denoising, filtering, and geometric correction on the point cloud data and image information; A central processing unit, wherein the central processing unit is configured as: The point cloud of the material pile before and after excavation is registered to the same coordinate system through the point cloud registration algorithm; The registered point cloud data is meshed, and the volume change between the two surfaces is calculated to obtain the preliminary material volume; A model building and training unit, wherein the model building and training unit is used to build and train a multi-task neural network model, and use the trained multi-task neural network model to output a volume prediction value and a density prediction value; Auxiliary unit, the auxiliary unit is used for data backup and equipment status monitoring; A power supply unit, which provides power support for each module.

[0015] The data acquisition unit includes: The LiDAR sensor, installed in the upper middle portion of the shovel's boom, collects real-time point cloud data of the pile surface before and after excavation. A visible light-infrared camera is installed in the same housing as the lidar sensor and is used to collect real-time image information of the bucket and the material after excavation; The IMU inertial measurement unit is installed on the guardrail above the shovel robot body and is used to locate the robot's position before and after excavation.

[0016] The model building and training unit includes: A multi-task neural network model whose structure includes a shared feature extraction layer, a volume prediction branch, and a density prediction branch; The confidence evaluation module is used to forward propagate the test data multiple times and calculate the mean and variance of the predicted value; The dataset expansion module filters high-confidence data based on the set confidence threshold and iteratively expands the training set to optimize the model; The bucket fill rate calculation module calculates the bucket fill rate based on the predicted volume and the rated capacity of the bucket, and generates an SPC control chart to analyze operation stability; Material weight calculation module calculates material weight based on predicted volume and density.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This system integrates lidar, a visible-infrared camera, and an IMU (Inertial Measurement Unit) with deep learning algorithms to achieve high-precision detection of the shovel bucket fill rate and material weight. This system accurately calculates material volume and density in real time, significantly improving the excavation and loading efficiency of shovel loaders in large-scale infrastructure projects such as mining and open-pit coal mining. By precisely controlling the bucket fill rate, unnecessary material waste and duplicate excavation are reduced, further improving operational efficiency.

[0018] Accurate Material Weight Detection: By building a Transformer-based multi-task neural network model, material density can be accurately predicted. In complex real-world working conditions, the composition of stockpiles varies. Traditional methods struggle to accurately measure material weight due to the non-constant density of materials. This method effectively handles a wide range of material conditions, providing reliable data support for material weight calculations and significantly improving the accuracy of material weight measurement, meeting the requirements for precise material weight measurement in mining and open-pit coal mining.

[0019] High-precision bucket fill rate prediction: The system uses point cloud data from the stockpile surface before and after excavation to preliminarily calculate the material volume. Using a semi-supervised learning algorithm, this volume data serves as input for a neural network model to obtain more accurate material volumes, significantly improving the accuracy of bucket fill rate calculations. Furthermore, SPC is used to monitor the stability of the excavation process, helping to promptly identify anomalies during excavation operations and enable adjustments and improvements.

[0020] Adaptable to Complex Environments: Using a LiDAR sensor to collect point cloud data, combined with a visible light and infrared camera to capture image information, the system can collect data even at night or in harsh, low-light environments. Compared to traditional vision-based methods, this effectively overcomes lighting limitations, ensuring stable and accurate material weight measurement in a variety of complex operating environments. This enhances the detection system's environmental adaptability and reliability, ensuring the continuity and stability of shoveling operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Provide a schematic diagram for sensor equipment installation; Figure 2 It is the block diagram of each functional module; Figure 3 This is the workflow diagram of the multi-task neural network model; Figure 4 This is the SPC analysis flow chart for full bucket rate; In the figure, 100-data acquisition unit; 200-data preprocessing unit; 300-central processing unit; 400-model building and training unit; 500-auxiliary unit; 600-power supply unit. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] The present invention provides a system for calculating the fullness rate and material weight of an electric shovel bucket based on deep learning, comprising: The data acquisition unit 100 is used to collect point cloud data and image information: A data preprocessing unit 200, configured to perform denoising, filtering, and geometric correction on the point cloud data and image information; The central processing unit 300 is configured as follows: The point cloud of the material pile before and after excavation is registered to the same coordinate system through the point cloud registration algorithm; The registered point cloud data is meshed, and the volume change between the two surfaces is calculated to obtain the preliminary material volume; A model building and training unit 400 is used to build and train a Transformer-based multi-task neural network model, and use the trained multi-task neural network model to output volume prediction values and density prediction values; Auxiliary unit 500, the auxiliary unit 500 is used for data backup and device status monitoring; The power supply unit 600 provides power support for each module.

[0024] The data acquisition unit 100 includes: The LiDAR sensor, installed in the upper middle portion of the shovel's boom, collects real-time point cloud data of the pile surface before and after excavation. A visible light-infrared camera is installed in the same housing as the lidar sensor and is used to collect real-time image information of the bucket and the material after excavation; The IMU inertial measurement unit is installed on the guardrail above the shovel robot body and is used to locate the robot's position before and after excavation.

[0025] The model building and training unit 400 includes: A multi-task neural network model whose structure includes a shared feature extraction layer, a volume prediction branch, and a density prediction branch; The confidence evaluation module is used to forward propagate the test data multiple times and calculate the mean and variance of the predicted value; The dataset expansion module filters high-confidence data based on the set confidence threshold and iteratively expands the training set to optimize the model; The bucket fill rate calculation module calculates the bucket fill rate based on the predicted volume and the rated capacity of the bucket, and generates an SPC control chart to analyze operation stability; Material weight calculation module calculates material weight based on predicted volume and density.

[0026] The sensor device of the data acquisition unit 100 in the bucket fullness and material weight detection system is installed on the existing shovel loading robot. Specifically: The LiDAR sensor is installed in the upper middle part of the boom and is used to scan the surface of the pile before and after the shovel digs the material. The visible light-infrared camera is installed in the upper middle part of the boom and is installed in the same housing as the laser radar. It is used to collect image information of the bucket and the material in the bucket. The IMU inertial measurement unit is installed on the guardrail above the shovel loader body and is used to determine the position of the shovel loader before and after digging materials; The data pre-processing unit 200 performs denoising, filtering and other processing on the collected data; The central processing unit 300 is arranged in the cab and is used to process and calculate the data; The model building and training unit 400 obtains accurate volume prediction values and density prediction values through neural network model training based on the large amount of collected data; The auxiliary unit 500 is used to back up the data collected by the sensor and detect the operating status of the device.

[0027] The power supply unit 600 is provided in the cab and is used to supply power to various sensing devices and the central processing unit 300.

[0028] A method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning, comprising: S100: Real-time acquisition of point cloud data of the pile surface before and after the electric shovel excavates the material; real-time acquisition of image information of the bucket and the material in the bucket after excavation; S200: pre-processing the collected point cloud data and image information to improve data quality; S300: Calculating the volume of the excavated material based on the point cloud data before and after excavation; S400: Train a multi-task neural network model, which is used to output accurate material volume and density prediction values; S500: Calculating the bucket fullness rate based on the obtained material volume prediction value and the rated capacity of the bucket, analyzing a large amount of bucket fullness rate data, and detecting the stability of the bucket excavating the material; S600: Calculate the weight of the material in the bucket according to the predicted value of the material volume and the predicted value of the density.

[0029] Specifically, S300 includes: S301: Registering the point cloud of the material pile before the electric shovel excavates the material with the point cloud of the material pile after the electric shovel excavates the material based on a point cloud registration algorithm; The central processing unit 300 uses a point cloud registration algorithm to register the point cloud of the material pile before and after the shovel excavates the material. Because the majority of the scene remains the same in both point clouds, with only the excavated part of the material pile changing, the ICP algorithm is used for registration.

[0030] S302: Converting the registered point cloud data into the same coordinate system; The central processing unit 300 converts the registered point cloud data into the same coordinate system, and uses the IMU (inertial measurement unit) installed on the body guardrail of the shovel loader robot for positioning to record the coordinates of the shovel loader robot when collecting the two point clouds. It then rotates and translates the material pile point cloud to unify their coordinate systems.

[0031] S303: Calculate the volume change between the material pile surface before and after excavation based on the position information of the two point clouds in the same coordinate system, and thus preliminarily calculate the volume of the material in the bucket. ; The central processing unit 300 can grid the two pile surfaces using the position information of the two point clouds in the same coordinate system, calculate the volume of the cylinder corresponding to the grid between the two surfaces, and then add them together to obtain the volume change between the pile before and after excavation, thereby preliminarily calculating the volume of the material in the bucket. .

[0032] Specifically, S400 includes: S401: Acquire multi-source data related to materials, including preliminary material volume data, material true density, image data, material type, and ratio information of different materials.

[0033] S402: Preprocessing the acquired data, including image denoising, filtering, and correction, normalizing volume and density data, and encoding material type and proportion information.

[0034] 1) For image data, Gaussian filtering is used to remove noise from the image, histogram equalization is used to enhance the image contrast, and geometric correction is performed on the image to eliminate the influence of factors such as shooting angle; 2) For volume data and density data, the minimum-maximum normalization method is used to map them to the [0, 1] interval to improve the convergence speed and accuracy of the algorithm. The formula is: , where x is the original data, and are the minimum and maximum values of the data respectively; 3) For material types, one-hot encoding is used. Suppose there are three materials A, B, and C, which are encoded as [1, 0, 0], [0, 1, 0], and [0, 0, 1] respectively; 4) For ratio information, convert it to decimal form and perform appropriate quantization.

[0035] S403: Divide the preprocessed data into a training set, a validation set, and a test set according to a ratio of 7:2:1, and use the training set to start training the neural network.

[0036] S404: Build a Transformer-based multi-task neural network model and train it using the data preprocessed in S402, including: A shared feature extraction layer, which uses convolutional layers and pooling layers to extract features from image data and fuses non-image data with image features through a fully connected layer; The task-specific layer is divided into a volume prediction branch and a density prediction branch.

[0037] 1) Shared feature extraction layer: Image feature extraction: Use MobileNet for preliminary feature extraction, and gradually convert the image into a feature map of size H×W×C (H is height, W is width, C is the number of channels) through convolutional layers and pooling layers. Extract low-level features such as edges and textures in the image, and divide the feature map into H×W small cells, each cell corresponding to a position, and then flatten these feature maps to obtain a one-dimensional vector , Each element in the feature map corresponds to the feature information of a specific position on the feature map and is mapped to a vector space with the same dimension as the Transformer input (model dimension) through a fully connected layer. ), get the image feature vector after feature transformation , where the weight matrix of the fully connected layer is , the bias is .

[0038] Non-image feature processing: The normalized preliminary material volume data, the material real density data, the one-hot encoded material type information and the material ratio information are spliced to obtain the non-image feature vector , where the preliminary material volume data, true density data, and material ratio information each occupy one position, and the material type information after unique hot encoding occupies 3 positions (assuming there are three types of materials). Then, a fully connected layer is used to transform the non-image feature vector so that it matches the image feature vector in terms of feature dimension and representation ability. The weight matrix of the fully connected layer is , the bias is , then the non-image feature vector after feature transformation is ; Position encoding and feature fusion: image feature vector and non-image feature vectors Add position codes respectively. Position codes are generated using sine and cosine functions. The formula is as follows: For even positions: ; For odd positions: , Among them, pos is the position index, the value range is [0, n-1], n is the number of positions; i is the dimension index, the value range is ; is the model dimension.

[0039] For each position, calculate the image feature vector after adding position encoding and non-image feature vectors , and concatenate them to get the fusion feature vector .

[0040] Transformer encoder: The Transformer encoder has 3 Transformer blocks, each of which contains a multi-head self-attention mechanism (the number of heads is 8, and the dimension of each head is ) and a feedforward neural network to process and transform the input fusion feature vector E, thereby extracting a more advanced feature representation. The calculation process is as follows: First, the input fusion feature vector E is transformed into query, key, and value vectors through linear transformation, and the query is obtained. ,key ,value ,in 、 、 are learnable weight matrices of dimension 512×64 respectively; then, for each head, the attention score is calculated: ,in 、 、 are the query, key, and value vectors of the i-th head, respectively. The purpose is to perform scaling operations to prevent the softmax function gradient from disappearing and make the model training more stable. Finally, the output results of the 8 heads are pieced together and then linearly transformed through a 64×512 weight matrix to obtain the final output of the multi-head self-attention mechanism. .

[0041] The feedforward neural network consists of two fully connected layers, with the ReLU activation function used in the middle to introduce nonlinearity. The weight matrix of the first fully connected layer is , bias is , the weight matrix of the second fully connected layer is , bias is , then the output of the feedforward neural network is: .

[0042] 2) Task-specific layer: Volume prediction branch: extracting feature vectors from the output of the Transformer encoder , input to a series of fully connected layers, the number of neurons in the fully connected layer gradually decreases, achieving feature dimensionality reduction and focusing. The number of neurons in the first fully connected layer is 512, and the weight matrix is , bias is ; The second fully connected layer is 256 and the weight matrix is , bias is ; The output dimension of the last fully connected layer is 1 and the weight matrix is , bias is Finally, the material volume prediction value is obtained. The ReLU activation function is used after each fully connected layer to introduce nonlinearity and enhance the expressive power of the model.

[0043] After the first fully connected layer: ; After the second fully connected layer: ; Finally, the volume prediction value is obtained: ; Density prediction branch: extracts feature vectors from the output of the Transformer encoder , processed by an independent fully connected layer structure. The number of neurons in the first fully connected layer is 512 and the weight matrix is , bias is ; The second fully connected layer is 256 and the weight matrix is , bias is ; The output dimension of the last fully connected layer is 1 and the weight matrix is , bias is , and finally output the material density prediction value, and the ReLU activation function is used between each fully connected layer.

[0044] After the first fully connected layer: ; After the second fully connected layer: ; Finally, the density prediction value is obtained: ; S405: Define the loss function and optimizer. The loss function is , try different and Combine, calculate the sum of the volume prediction mean square error and density prediction mean square error of the model on the validation set under different combinations, and select the one that minimizes the sum and As the final weight, 、 is the weight coefficient, and , is the mean squared error loss of volume prediction, is the mean square error loss function of density prediction, and the mean square error loss function formula is: ; Select Adam optimizer.

[0045] S406: Perform multiple forward propagations to calculate the mean and variance of the volume prediction value and the density prediction value.

[0046] After the training is completed, the Monte Carlo dropout method is used to perform n forward propagations on the same data in the test phase to obtain the volume prediction values , ... and density predictions , ... Calculate the mean and variance respectively: Volume mean: , volume variance: ; Density mean: , density variance: .

[0047] S407: Filter high-confidence data based on the set confidence threshold, and iteratively expand the training set to optimize the model.

[0048] Set the confidence threshold: Experiment on the validation set and try different variance thresholds and ,By calculating the mean square error of volume prediction and the mean absolute error of density prediction, the threshold that enables the model to achieve the best overall performance on the validation set is selected as the final confidence threshold.

[0049] Traverse the model's prediction results for all data and calculate the volume prediction confidence separately and density prediction confidence ,Will and volume confidence threshold Compare, and density confidence threshold , only when ≥ and ≥ When the data point is selected, the selected data includes image data, material volume prediction value, material density prediction value, and material type and proportion; The selected data is quality checked and added to the original dataset to retrain the model.

[0050] S500 includes: The volume of material in the bucket obtained by prediction and rated capacity of the bucket Calculate the full bucket rate, that is , by detecting and analyzing the full bucket rate data, the stability of the bucket in excavating materials is detected.

[0051] Analyze a large amount of bucket fill rate data to detect the stability of the bucket excavation process, including: S501: Obtaining full bucket rate data according to the multi-task neural network model; S502: Analyze whether the data is normally distributed; S503: Select the mean-range control chart, divide all data into several subgroups, and calculate the mean of each subgroup and range , and then calculate the mean value of all subgroups and the average of all subgroup ranges , and then calculate the control limits using the control chart coefficients based on the subgroup size n, where The center line of the mean chart is: , the upper control limit is: , the lower control limit is: , is the control chart coefficient; The center line of the standard deviation chart is: , the upper control limit is: , the lower control limit is: , 、 is the control chart coefficient; S503: Draw a control chart, plot the calculated mean, range data, and corresponding control lines on the control chart, with each subgroup's data corresponding to a position on the chart; S504: Analyze the control chart to determine whether the existing data process is stable. If the data is unstable, find the cause of the test instability by reproducing or backtracking the scenario and process that caused the abnormal data, and then improve the test.

[0052] S600 includes: According to the predicted volume With predicted density Calculate the weight of the material in the bucket, i.e. .

[0053] Accurate Material Weight Detection: By building a Transformer-based multi-task neural network model, material density can be accurately predicted. In complex real-world working conditions, the composition of stockpiles varies. Traditional methods struggle to accurately measure material weight due to the non-constant density of materials. This method effectively handles a wide range of material conditions, providing reliable data support for material weight calculations and significantly improving the accuracy of material weight measurement, meeting the requirements for precise material weight measurement in mining and open-pit coal mining.

[0054] High-precision bucket fill rate prediction: The system uses point cloud data from the stockpile surface before and after excavation to preliminarily calculate the material volume. Using a semi-supervised learning algorithm, this volume data serves as input for a neural network model to obtain more accurate material volumes, significantly improving the accuracy of bucket fill rate calculations. Furthermore, SPC is used to monitor the stability of the excavation process, helping to promptly identify anomalies during excavation operations and enable adjustments and improvements.

[0055] Adaptable to Complex Environments: Using a LiDAR sensor to collect point cloud data, combined with a visible light and infrared camera to capture image information, the system can collect data even at night or in harsh, low-light environments. Compared to traditional vision-based methods, this effectively overcomes lighting limitations, ensuring stable and accurate material weight measurement in a variety of complex operating environments. This enhances the detection system's environmental adaptability and reliability, ensuring the continuity and stability of shoveling operations.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning, characterized in that: include: S100: Real-time acquisition of point cloud data of the pile surface before and after the shovel excavates the material; Real-time acquisition of image information of the bucket and the material inside the bucket after excavation; S200: pre-processing the collected point cloud data and image information to improve data quality; S300: Calculating the volume of the excavated material based on the point cloud data before and after excavation; S400: Train a multi-task neural network model, which is used to output accurate material volume and density prediction values; S500: Calculating the bucket fullness rate based on the obtained material volume prediction value and the rated capacity of the bucket, analyzing a large amount of bucket fullness rate data, and detecting the stability of the bucket excavating the material; S600: Calculate the weight of the material in the bucket according to the predicted value of the material volume and the predicted value of the density.

2. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 1 is characterized in that: The S300 includes: S301: Registering the point cloud of the material pile before the electric shovel excavates the material with the point cloud of the material pile after the electric shovel excavates the material based on a point cloud registration algorithm; S302: Converting the registered point cloud data into the same coordinate system; S303: Calculate the volume change between the material pile surface before and after excavation based on the position information of the two point clouds in the same coordinate system, and thus preliminarily calculate the volume of the material in the bucket. .

3. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 1 is characterized in that: The S400 includes: S401: Acquire multi-source data related to materials, including preliminary material volume data, material true density, image data, material type, and ratio information of different materials; S402: Preprocessing the acquired data, including image denoising, filtering, and rectification, normalizing volume and density data, and encoding material type and proportion information; S403: Divide the preprocessed data into a training set, a validation set, and a test set according to a ratio of 7:2:1, and start training the neural network using the training set; S404: Build a Transformer-based multi-task neural network model and train it using the data preprocessed in S402, including: A shared feature extraction layer, which uses convolutional layers and pooling layers to extract features from image data and fuses non-image data with image features through a fully connected layer; A task-specific layer, wherein the task-specific layer is divided into a volume prediction branch and a density prediction branch; S405: Define loss function and optimizer; S406: Perform multiple forward propagations to calculate the mean and variance of the volume prediction value and the density prediction value; S407: Filter high-confidence data based on the set confidence threshold, and iteratively expand the training set to optimize the model.

4. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 3 is characterized in that: The S406 includes: Traverse the model's prediction results for all data and calculate the volume prediction confidence separately and density prediction confidence ,Will and volume confidence threshold Compare, and density confidence threshold , only when ≥ and ≥ When the data point is selected, the selected data includes image data, material volume prediction value, material density prediction value, and material type and proportion; The selected data is quality checked and added to the original dataset to retrain the model.

5. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 1 or 3, characterized in that: The S500 includes: The volume of material in the bucket obtained by prediction and rated capacity of the bucket Calculate the full bucket rate, that is , by detecting and analyzing the full bucket rate data, the stability of the bucket in excavating materials is detected.

6. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 5 is characterized in that: The analysis of a large amount of bucket fill rate data and the detection of the stability of the bucket excavation process include: S501: Obtaining full bucket rate data according to the multi-task neural network model; S502: Analyze whether the data is normally distributed; S503: Select the mean-range control chart, divide all data into several subgroups, and calculate the mean of each subgroup and range , and then calculate the mean value of all subgroups and the average of all subgroup ranges , and then calculate the control limits using the control chart coefficients based on the subgroup size n, where The center line of the mean chart is: , the upper control limit is: , the lower control limit is: , is the control chart coefficient; The center line of the standard deviation chart is: , the upper control limit is: , the lower control limit is: , 、 is the control chart coefficient; S503: Draw a control chart, plot the calculated mean, range data, and corresponding control lines on the control chart, with each subgroup's data corresponding to a position on the chart; S504: Analyze the control chart to determine whether the existing data process is stable. If the data is unstable, find the cause of the test instability by reproducing or backtracking the scenario and process that caused the abnormal data, and then improve the test.

7. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 1 or 3, characterized in that: The S600 includes: According to the predicted volume With predicted density Calculate the weight of the material in the bucket, i.e. .

8. A method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning, characterized in that: Used to implement the method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning as described in any one of claims 1 to 7, include: A data acquisition unit (100), wherein the data acquisition unit (100) is used to acquire point cloud data and image information: A data pre-processing unit (200), the data pre-processing unit (200) is used to perform denoising, filtering and geometric correction on the point cloud data and image information; A central processing unit (300), wherein the central processing unit (300) is configured as follows: The point cloud of the material pile before and after excavation is registered to the same coordinate system through the point cloud registration algorithm; The registered point cloud data is meshed, and the volume change between the two surfaces is calculated to obtain the preliminary material volume; A model construction and training unit (400), wherein the model construction and training unit (400) is used to construct and train a multi-task neural network model, and output a volume prediction value and a density prediction value using the trained multi-task neural network model; An auxiliary unit (500), the auxiliary unit (500) is used for data backup and device status monitoring; A power supply unit (600), wherein the power supply unit (600) provides power support for each module.

9. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 8, characterized in that: The data acquisition unit (100) comprises: The LiDAR sensor, installed in the upper middle portion of the shovel's boom, collects real-time point cloud data of the pile surface before and after excavation. A visible light-infrared camera is installed in the same housing as the lidar sensor and is used to collect real-time image information of the bucket and the material after excavation; The IMU inertial measurement unit is installed on the guardrail above the shovel robot body and is used to locate the robot's position before and after excavation.

10. The method for detecting the fullness rate and material weight of an electric shovel bucket based on deep learning according to claim 8, characterized in that: The model building and training unit (400) includes: A multi-task neural network model whose structure includes a shared feature extraction layer, a volume prediction branch, and a density prediction branch; The confidence evaluation module is used to forward propagate the test data multiple times and calculate the mean and variance of the predicted value; The dataset expansion module filters high-confidence data based on the set confidence threshold and iteratively expands the training set to optimize the model; The bucket fill rate calculation module calculates the bucket fill rate based on the predicted volume and the rated capacity of the bucket, and generates an SPC control chart to analyze operation stability; Material weight calculation module calculates material weight based on predicted volume and density.

Images