Method and camera for weighing material loaded by a loading machine

Abstract

This invention relates to a method for weighing materials loaded by loading machinery. The method includes: acquiring image data during the loading process where the bucket of the loading machinery loads materials onto a transport vehicle; determining the weight of the loaded materials using a model based on the acquired image data, wherein the model is designed to identify targets from the acquired image data and predict target positions and bucket fill rates, and to determine the number of buckets and the weight of the loaded materials based on the predicted target positions and bucket fill rates using key motion checks, wherein the target is the bucket or the bucket and the materials loaded in the bucket, the target position is the position of the target itself or the position of the anchor frame surrounding the target, and the bucket fill rate is the proportion of the volume of the materials in the bucket to the bucket capacity during each loading process; and outputting the determined weight, the determined weight including the weight of the materials loaded each time and / or the accumulated total weight.

Description

Technical Field

[0001] This invention relates to a method for weighing materials loaded by loading machinery and a camera. Background Technology

[0002] To better understand the daily loading efficiency of excavators and calculate cost per ton, customers need to install a weighing system on their excavators to monitor the cumulative weighing of the entire machine and its working efficiency. Existing physical weighing systems are complex in design, containing multiple components, such as bucket cylinder position sensors and boom cylinder pressure / angle sensors for collecting data related to load weight, controllers for receiving sensor data and combining it with relevant parameters for calculations, displays for showing the final data, and so on. Furthermore, existing physical weighing systems require the installation of multiple IMU sensors in the cylinders; IMU sensors are expensive, making the system not only complex to install but also costly. Existing physical weighing systems are also cumbersome to use, frequently requiring calibration, such as during initial use, changes in operating environment, regular maintenance, or after replacing components. Therefore, there is a need for an improved excavator weighing system. Summary of the Invention

[0003] Therefore, the object of the present invention is to provide a method and camera for weighing materials loaded by loading machinery, such as excavators, loaders, electric shovels, hydraulic shovels, etc., which can overcome at least some of the above-mentioned defects of the prior art.

[0004] This objective is achieved through a method, a computer program product, and a camera having the following characteristics.

[0005] According to one aspect of the present invention, a method for weighing materials loaded by loading machinery, such as excavators, loaders, electric shovels, hydraulic shovels, etc., is proposed. The method includes: acquiring image data of the loading process in which the bucket of the loading machinery loads materials onto a transport vehicle, such as a mining truck; determining the weight of the loaded materials using a model based on the acquired image data, wherein the model is designed to identify targets from the acquired image data and predict target positions and bucket fill rates, and to determine the number of buckets and the weight of the loaded materials based on the predicted target positions and bucket fill rates using key action checks, wherein the identified targets are either buckets or buckets and materials contained in buckets, the target position is the position of the target itself or the position of the anchor frame surrounding the target, especially in the vertical direction, and the bucket fill rate is the proportion of the volume of materials in the bucket to the bucket capacity during each loading process; and outputting the determined weight, which includes, for example, the weight of materials loaded each time and / or the total weight accumulated, and in particular, also outputting the determined bucket fill rate and / or the number of buckets.

[0006] This invention proposes a digital image-based weighing method to address the problems of high cost, difficult installation, and the need for periodic calibration in traditional weighing systems. Utilizing image recognition algorithms and an AI camera, through data collection, image annotation, machine learning, and AI judgment, it achieves the identification of the number of buckets and the fullness rate using only a single camera, ultimately realizing the weighing function. Traditional weighing systems are expensive, contain many parts, are complex to install, time-consuming, and require complex calibration, generally exhibiting an accuracy error of ≥±5%, and requiring periodic recalibration. In contrast, the weighing system according to this invention is inexpensive, contains only one camera, is simple to install, requires no calibration, continuously optimizes the algorithm through machine learning, automatically calibrates, and optimizes weighing accuracy through image comparison and background data compensation.

[0007] According to one embodiment, image data is detected, particularly continuously, using an image sensor, such as a camera, which is mounted on loading machinery in a fixed position.

[0008] According to another embodiment, the step of determining the number of buckets and the weight of the loaded material by means of a critical action check includes: checking whether a critical action has occurred based on the predicted target position; if a critical action has occurred, the number of buckets is incremented by 1; preferably, the critical action is the target crossing a sensitive line in a specified direction; checking whether the target crosses the sensitive line in a specified direction includes the following steps:

[0009] - Based on the target position predicted by the model, check whether the target is moving from bottom to top. For example, compare the height coordinate of a certain point on the target in the previous frame image data with the height coordinate of the corresponding point on the target in the current frame image data. If the height coordinate of the certain point on the target in the previous frame image data is less than the height coordinate of the corresponding point on the target in the current frame image data, then the target is considered to be moving from bottom to top.

[0010] - Based on the model's predicted bucket full rate, check whether there is material in the bucket. For example, compare the model's predicted bucket full rate with the value representing an empty bucket. If the bucket full rate is not equal to the value representing an empty bucket, then it is considered that there is material in the bucket.

[0011] - Based on the target position predicted by the model, check whether the target is crossing the sensitive line. For example, compare the height coordinates of the upper boundary and the lower boundary of the target with the height coordinates of the pre-determined sensitive line. If the sensitive line is located between the upper and lower boundaries of the target, it is considered that the target is crossing the sensitive line. The pre-determined height coordinates of the sensitive line are, for example, the median height of the imaging frame of the image sensor used to detect image data.

[0012] - If all three checks above are "yes", then increment the bucket count by 1 when the target completely crosses the sensitive line. The advantage here is that the range of a valid loading process can be easily determined based on the output of the image recognition algorithm.

[0013] When the bucket count is incremented by 1, the model-predicted full-bucket rate that occurs most frequently within a time period is selected as the effective full-bucket rate for this loading process. Alternatively, the average of full-bucket rates whose differences do not exceed a predetermined threshold within a time period can be used as the effective full-bucket rate. Or, the model-predicted maximum full-bucket rate within a time period can be used as the effective full-bucket rate. Preferably, this time period is the period from when the target's upper and lower boundaries are symmetrically located on both sides of the sensitive line until the target's lower boundary just crosses the sensitive line. The advantage here is that, after determining the range of the effective loading process, the full-bucket rate for each effective loading process can be accurately determined.

[0014] According to another embodiment, the step of determining the weight of the loaded material includes: multiplying the effective bucket fill rate of each loading process by the bucket capacity and the material density of the material in the bucket as the weight of the material loaded each time; or determining the weight of the material based on the effective bucket fill rate using a pre-determinable lookup table, wherein the pre-determinable lookup table describes the correspondence between the bucket fill rate and the weight for a given bucket model and a given material type. The advantage here is that, after determining the bucket fill rate, the weight of the material can be determined by simple calculation or table lookup.

[0015] Preferably, the material density is determined by the material type in the bucket, such as sand, ore, or slag. Preferably, the material type is identified from the acquired image data by the model, or the material type or the material density is provided to the model as model parameters or input data. The advantage here is that the material type and material density can be automatically identified and determined by the model.

[0016] According to another embodiment, the bucket capacity is determined by the bucket model. Preferably, the bucket model is identified from the acquired image data by the model, or the bucket model or the bucket capacity is provided to the model as model parameters or input data. The advantage here is that the bucket model can be automatically identified and the bucket capacity determined by the model.

[0017] According to another embodiment, the model is pre-trained, and the training of the model includes the following steps:

[0018] - Collect image data during the loading process where the bucket loads materials onto the transport vehicle;

[0019] - The collected image data is manually labeled to indicate the fullness rate of the buckets containing materials contained in the image data. In particular, the material type and / or bucket model of the material in the bucket are also labeled.

[0020] - The model is trained based on the collected image data and the labeled bucket full rate, especially the labeled material type and / or bucket model, so that the trained model can determine the number of buckets, the bucket full rate and thus the weight of the loaded material from the image data.

[0021] - Compare the number of hoppers determined by the model and the weight obtained from the full hopper rate with the corresponding data from the physical weighing system to optimize the model, for example by optimizing the objective function or reward / penalty algorithm.

[0022] According to another embodiment, the determined weight is output to a display, preferably mounted on or behind the loading machinery. Specifically, the display also shows image data and marks the targets included in the image data, as well as the target location and hopper fullness. The advantage here is that data such as the weight of the loaded material, hopper fullness, and number of hoppers can be displayed to the customer in real time and intuitively.

[0023] According to another aspect of the invention, a computer program product is provided, comprising instructions that are implemented to perform the method of the invention.

[0024] According to another aspect of the present invention, a camera is provided for weighing materials loaded by loading machinery, such as excavators, loaders, electric shovels, hydraulic shovels, etc. The camera includes an image sensor and a processor. The image sensor is used to detect image data during the loading process in which the bucket of the loading machinery loads materials onto a transport vehicle, such as a mining truck. The processor is designed to execute the method of the present invention.

[0025] This invention uses an AI camera to replace the corresponding sensors and controllers, enabling real-time perception of location information. It then uses deep learning algorithms (such as YOLOv5) to calculate and output the number of buckets, the fullness rate, and the weight. Specifically, the AI ​​camera captures sufficient image information for collection and annotation. Simultaneously, existing backend data, namely the measurement data from the physical weighing system, is used for accuracy compensation, establishing a corresponding mathematical compensation model. Finally, through image capture and comparison, as well as backend data compensation, high-precision weighing information is obtained.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the scope of protection of the present invention. Attached Figure Description

[0027] The invention will now be described in detail with reference to the accompanying drawings. In the drawings,

[0028] Figure 1 A flowchart of a method for weighing materials loaded by loading machinery according to the present invention is shown.

[0029] Figure 2 A counting logic block diagram of the counting algorithm according to the present invention is shown.

[0030] Figure 3 This illustrates an example of labeling the fullness rate.

[0031] Figure 4 A flowchart is shown for training the weighing model of the present invention. Detailed Implementation

[0032] Figure 1 A flowchart illustrating a method for weighing materials loaded by loading machinery, such as excavators, loaders, electric shovels, hydraulic shovels, etc., according to the present invention is shown. In step S101, image data is acquired during the loading process where the bucket of the loading machinery loads the material onto a transport vehicle, such as a mining truck. During each loading process, the bucket is in a series of complex states, such as the bucket inserting into the material, being lifted when full, moving above the transport vehicle, unloading the material onto the transport vehicle, moving away from the transport vehicle, and lowering the empty bucket. Therefore, the image data are images of the loading machinery's working site detected at different times and stages of the loading process. The image data is detected using an image sensor, such as a camera, especially continuously, or at large time intervals such as 0.5 seconds, 1 second, 2 seconds, etc. The camera can be fixedly mounted on the loading machinery (e.g., fixed to the excavator body in the case of an excavator) and captures images of the loading machinery's working site at a fixed angle horizontally. Since cameras can only capture a certain height range, the installation location and angle of the camera must be carefully selected so that the bucket and the material inside it can be observed most clearly and advantageously within the height range captured by the camera.

[0033] In step S102, the weight of the loaded material is determined using a weighing model based on the acquired image data. The weighing model of this invention is designed to identify targets from the acquired image data and predict target location and bucket fill rate. Based on the predicted target location and bucket fill rate, it determines the number of effective loading processes (i.e., the number of buckets), the weight of each loading, and the total accumulated weight using key action checks. Here, the identified target is the bucket, or the bucket and the material loaded in the bucket. The target location can be the location of the target itself or the location of an imaginary anchor frame closely surrounding the target. The anchor frame can take various shapes, such as square, circular, elliptical, or polygonal, but is preferably square. Here, the bucket fill rate, also called the filling coefficient, refers to the ratio between the bucket loading capacity used each time a loading machine performs digging or loading operations and the actual bucket capacity. The actual bucket capacity, also called bucket volume, is the volume of the bucket, usually expressed in cubic meters (m³). 3 The unit is ( ). In other words, the size of the bucket capacity directly affects the amount of soil and rock that loading machinery can excavate or load in a single operation. The bucket fill rate measures the actual utilization of the bucket capacity. Therefore, the bucket fill rate can also be defined as the proportion of the volume of material in the bucket to the bucket capacity during each loading process.

[0034] Weighing models can be implemented using image recognition algorithms. Image recognition algorithms are used for object detection, that is, locating and identifying different objects contained in an image or video. Commonly used image recognition algorithms include:

[0035] (1) R-CNN series of algorithms:

[0036] • R-CNN (Region-based Convolutional Neural Networks): Extracts features from candidate regions and then uses a classifier to classify them.

[0037] Fast R-CNN: Improves processing speed based on R-CNN.

[0038] • Faster R-CNN: Introduces Region Proposal Network (RPN) to achieve end-to-end object detection.

[0039] (2) Mask R-CNN: Based on Faster R-CNN, it adds the function of instance segmentation, which can detect targets and generate target segmentation masks at the same time.

[0040] (3) Cascade R-CNN: It uses a cascaded structure for target detection to gradually improve the accuracy of the detector.

[0041] (4) YOLO series algorithms:

[0042] YOLO (You Only Look Once): Treats object detection as a regression problem, achieving fast and accurate object detection.

[0043] YOLOv2, YOLOv3, YOLOv4, YOLO5: Continuously optimized and improved versions that enhance detection accuracy and speed.

[0044] (5) SSD (Single Shot MultiBox Detector): A single-stage target detector that performs target detection through multi-scale feature maps, which is more efficient.

[0045] (6) RetinaNet: Improves detection performance by addressing the problem of single-stage detectors in small target detection.

[0046] (7) EfficientDet: Combines EfficientNet and object detector to achieve efficient object detection.

[0047] The following example uses one algorithm to explain the method of the present invention in detail. For example, the weighing model of the present invention uses the YOLO5 algorithm as the basic model framework. YOLO5 adopts a Convolutional Neural Network (CNN) based architecture, mainly composed of an input, a backbone, a neck, and a head. The input uses techniques such as Mosaic data augmentation; the backbone (such as CSPDarknet) is used to extract image features; the neck (such as FPN+PAN structure) is responsible for fusing and transmitting features at different scales; and the head is used to predict the target's category, location, and confidence level. Therefore, the model input of YOLO5 is the acquired image data related to the bucket loading process. The model output of YOLO5 is multidimensional parameters [x] o y o [, h, w, C…], where x o y oThese are the x and y coordinates of the center point of the identified target (in this example, the x and y coordinates of the center point of the square anchor frame surrounding the bucket and the material within it, close to the ground frame). The y-direction is the height direction, and the x-direction is the horizontal direction perpendicular to the height direction within the vertical imaging plane perpendicular to the camera's horizontal shooting direction. h is the total height of the identified target, w is the width of the identified target, and C is the category, which reflects the bucket fill rate q. Here, the volume of material loaded into the bucket varies greatly during each loading process; therefore, the bucket fill rate value may be as low as 40% or 50%, or as high as 100% or 110%.

[0048] Therefore, for each image data, the YOLO5 algorithm can be used to obtain the anchor frame position data of the bucket and the material within it, as well as the category reflecting the bucket fullness q information [x]. o y o [h, w, C…]. During a loading process, if image data is detected intensively over time, for example, 25 frames per second, multiple image data points may be detected during the lifting of the bucket loaded with material. These image data points can be used by the YOLO5 algorithm to obtain multiple bucket full rate values, which may be different. However, during a loading process, the bucket full rate predicted by the YOLO5 algorithm should have a unique value and should be accumulated only once. To determine the effective bucket full rate during this loading process, that is, the bucket full rate that should be accumulated into the total weight of the corresponding material, the weighing model also accumulates the number of effective loading processes, that is, counts the buckets, so as to determine the range of an effective loading process by counting the buckets. Here, to count the buckets, it is checked whether a critical action has occurred based on the target position predicted by the YOLO5 algorithm. If a critical action has occurred, the bucket count is incremented by 1. Preferably, the critical action is that the target crosses the sensitive line in a specified direction, as described below. Figure 2 Explain the specific inspection process.

[0049] Figure 2 The diagram shows the counting logic block diagram of the counting algorithm of the present invention. The multidimensional parameters [x] output by the YOLO5 algorithm are... o y o Starting with h, w, C…], in step S201, based on the target position predicted using the YOLO5 algorithm (here, the position of the center point of the anchor frame of the bucket and the material within it, and the total height of the anchor frame), a orientation check is performed on the target to determine whether it is moving in the specified direction, here from bottom to top. To do this, the y-coordinate of the anchor frame center point in the previous frame image data is checked. o,i-1 Is the y-coordinate of the anchor box center point in the image data of the current frame smaller than the y-coordinate of the anchor box center point? o,i If y o,i-1 <y o,iThis indicates that the anchor frame is rising, meaning the orientation check has been successfully passed, and proceed to step S202. Otherwise, return to the beginning of the number-buzz algorithm and perform the orientation check again.

[0050] In step S202, based on the bucket fill rate predicted using the YOLOv5 algorithm, it is checked whether there is material in the bucket. Let bucket0 represent an empty bucket (no material in the bucket), and bucket100 represent a bucket with 100% material. If the bucket fill rate q is not equal to bucket0 (q≠bucket0), proceed to step S203. Otherwise, return to the beginning of the bucket counting algorithm and re-perform the orientation detection.

[0051] In step S203, based on the target position predicted using the YOLO5 algorithm, it is checked whether the target is crossing the sensitive line. To do this, the y-coordinate of the upper boundary of the anchor frame (i.e., y...) is... o +h / 2) and the y-coordinate of the lower boundary of the anchor frame (i.e., y o -h / 2) and the height coordinate y of the sensitive line line Compare, if y o -h / 2 <y line <y o If the value is +h / 2, it indicates that the anchor frame is crossing the sensitive line. Otherwise, return to the beginning of the algorithm and re-perform orientation detection. Here, the height coordinate of the sensitive line is y. line For example, you can choose the middle value of the height range that the camera can capture, or the middle value of the height of the camera's imaging frame.

[0052] If it is determined that the target is crossing the sensitive line, then continue until the anchor frame has completely crossed it, i.e., y o -h / 2≥y line At this point, increment the number of dou (step S204).

[0053] Increasing the bucket count by 1 signifies that this loading process is a valid loading process. Therefore, the effective full bucket rate for this loading process is determined only when the bucket count is increased by 1. Here, the full bucket rate predicted most frequently by the YOLO5 algorithm within a certain time period can be taken as the effective full bucket rate. For example, this time period could be located symmetrically on both sides of the sensitivity line from the upper and lower boundaries of the anchor frame (i.e., the y-coordinate of the anchor frame center point). oThe effective load rate for this loading process is calculated from the moment the anchor frame is positioned directly on the sensitive line until the lower boundary of the anchor frame crosses the sensitive line. The load rates predicted by the YOLO5 algorithm within this time period are analyzed, and the most frequently occurring load rate is selected as the effective load rate. Alternatively, other time periods can be used, such as the period from when the lower boundary of the anchor frame is located at the lower boundary of the camera's imaging frame to when the upper boundary of the anchor frame is located at the upper boundary of the imaging frame. Alternatively, the average load rate within a time period whose difference does not exceed a predetermined threshold can be used as the effective load rate. That is, the average of load rates with small differences is used as the effective load rate. Alternatively, the maximum load rate predicted by the YOLO5 algorithm within a time period can also be used as the effective load rate. The number of loads and the effective load rate can be stored in a matrix, for example. For example, the matrix [1,80; 2,70; 3,100…] indicates that the first load is 80% full, the second load is 70% full, and the third load is 100% full.

[0054] After determining the effective bucket fill rate, multiplying the effective bucket fill rate by the bucket capacity yields the volume of material loaded in the bucket. Then, multiplying the volume of material in the bucket by the material density calculates the weight of the material loaded each time. That is, the weight of material loaded each time = bucket fill rate × bucket capacity × material density. Here, the bucket capacity can be determined by the bucket model. The bucket model can be identified from the acquired image data by the weighing model. Then, using a pre-stored lookup table describing the correspondence between bucket models and bucket capacities, the bucket capacity value can be obtained from the identified bucket model. Alternatively, the operator can directly fix the bucket model or bucket capacity as model parameters in the weighing model, or provide it as input data to the weighing model.

[0055] Similarly, material density can be determined by material type (e.g., sand, ore, slag, etc.). Material type can be identified from the acquired image data by the weighing model, and then the material density value can be obtained from the identified material type using a pre-stored lookup table describing the correspondence between material type and material density. Alternatively, the operator can directly fix the material type or material density as a model parameter in the weighing model, or provide it as input data to the weighing model.

[0056] As an alternative to the aforementioned process of determining weight through calculation, the weight of the material can also be determined based on a predetermined lookup table, such as a map table, describing the correspondence between the bucket fill rate q and the material weight for a given bucket model and a given material type. Thus, after determining the effective bucket fill rate and identifying the bucket model and material type using an image recognition algorithm, the corresponding weight value can be directly determined using the lookup table.

[0057] After the weight is determined, in step S103, the determined weight is output. The determined weight includes, for example, the weight of the material loaded each time and / or the total accumulated weight. Preferably, the predicted hopper fullness rate and / or the cumulative number of hoppers are also output. The output weight and / or hopper fullness rate and / or number of hoppers are output to a display on the loading machinery or a remote server, such as a backend display. Additionally, in addition to the final result predicted by the model, the display can also show in real time each frame of image data and the anchor frame of the target in the bounding image, as well as the anchor frame's position data and category (or hopper fullness rate) as a result of the image recognition algorithm.

[0058] The weighing model is trained in advance. When training the weighing model, it is essential to first collect different image data during the loading process of the bucket of the loading machinery loading material onto the transport vehicle (step S401). This image data should cover the bucket's state at different loading stages, such as the bucket inserting material, lifting when full, moving above the transport vehicle, unloading material onto the transport vehicle, moving away from the transport vehicle, and lowering the empty bucket, etc., ensuring data diversity. Furthermore, image data of the loading machinery in operation should be collected from different angles, under different lighting conditions, and under different working conditions.

[0059] Then, in step S402, the collected image data is manually labeled to indicate the fullness rate of the buckets containing materials contained in the image data. Figure 3 This illustration shows one example of labeling bucket fullness. Here, the percentage of material volume in the bucket to its capacity is divided into multiple ranges in 10% increments, for example, range A is 110%, range B is 100%, range C is 90%, and so on. Where material density is determined by identifying material type and / or bucket capacity is determined by identifying bucket model, different material types and / or different bucket models also need to be labeled in the collected image data.

[0060] In step S403, a weighing model is trained based on the collected image data, the labeled bucket full rate, and optionally the labeled material type and / or bucket model. This trained model can identify the bucket and the material within it from the image data, determine the position data of the anchor frame surrounding the bucket and the material within it, and the bucket full rate. Optionally, it can also identify the material type and / or bucket model, thereby obtaining the material density of the corresponding material according to a lookup table of material type and material density, and / or obtaining the bucket capacity of the corresponding bucket according to a lookup table of bucket model and bucket capacity. Based on the comparison between the position data of the identified target's anchor frame and the height coordinates of the sensitive line, the range of an effective loading process can be determined, and the number of buckets and the effective bucket full rate of this loading process can be determined. Then, based on the model-predicted effective bucket full rate, material density, and bucket capacity, the weight of the material loaded this time, and the accumulated total weight, are determined.

[0061] In step S404, the determined number of buckets and weight are compared with the corresponding data from the physical weighing system to optimize the weighing model. That is, the weighing model is compensated for inaccuracies using actual weighing data. The data from the physical weighing system is recorded at regular time intervals, such as every 0.5 seconds, every minute, and every 8 minutes, and then transmitted back to the backend. Based on the results from the physical weighing system, the weighing model can be optimized by optimizing the objective function (e.g., using the minimum mean square error as the loss function) or a reward / penalty algorithm.

[0062] The trained weighing model can be integrated into an AI camera. Here, an AI camera refers to a camera with a certain computing power, capable not only of capturing images or videos but also of intelligent analysis, recognition, and processing. Therefore, by combining camera technology with the weighing model algorithm of this invention, an AI camera can achieve the following intelligent functions and applications: capturing image data during each loading process; identifying the bucket and the material within it; identifying the bucket model and material type; predicting and displaying the position data and fullness of the identified bucket and its material in real time; determining whether the current loading process is valid based on the predicted position data; and further determining the effective fullness of the bucket that should be added to the total weight during this loading process. Specifically, through a lookup table embedded in the AI ​​camera, the weight of the material loaded each time and the total weight accumulated are determined. Therefore, through an AI camera with an embedded lookup table and model algorithm, relevant, accuracy-optimized weighing data can be displayed to the customer in real time and intuitively via a backend system or monitor.

[0063] While the above explanation of the method of the present invention is based on the YOLOv5 algorithm, other suitable image recognition algorithms for target detection can also be used, as long as the image recognition algorithm can identify the target from the image data and determine the target position, especially the height position and the full coverage area. For example, the height position may include the height coordinates of the target's center point and the target's total height and / or maximum height, or it may include the height coordinates of the target's upper edge and lower edge.

[0064] The present invention also relates to a computer program product comprising instructions which are implemented to perform the methods described above.

[0065] The present invention also relates to an AI camera, the camera including an image sensor and a processor, the image sensor being used to detect image data during the loading process of a bucket of loading machinery loading materials onto a transport vehicle, such as a mining truck, and the processor being designed to perform the above-described method of the present invention.

[0066] In one advantageous design, an implementation of a weighing model is embedded in the AI ​​camera, which can identify the material type, bucket model, target location, and bucket fill rate from image data, as well as a map table describing the correspondence between the bucket fill rate and the corresponding material weight, so that the bucket fill rate, number of buckets, and material weight can be read in real time using only one camera.

[0067] Industrial applicability

[0068] The image weighing method and camera of this invention can be used in loading machinery in mining areas, such as excavators, loaders, electric shovels, and hydraulic shovels. A model trained using data from manual annotation and physical weighing systems can be deployed to a camera installed on the loading machinery to capture real-time image data of the machinery's working site. The model then identifies the bucket and its contents from the image data, determining the bucket's position and the bucket's fullness. Based on this, the effective loading process and effective fullness are determined, and the weight of the loaded material is calculated.

[0069] The foregoing description is merely an exemplary embodiment relating to the spirit and principles of the present invention. Those skilled in the art will understand that various changes can be made to the described examples without departing from the spirit and principles, and such changes and their various equivalents are contemplated by the inventors and fall within the scope defined by the claims of the present invention.

Claims

1. A method of weighing material loaded by a loading machine, such as an excavator, a loader, an electric shovel, a hydraulic shovel, etc., characterized by, The method includes: Acquire image data of the loading process in which the bucket of loading machinery loads materials onto transport vehicles, such as mining trucks; Based on the acquired image data, a model is used to determine the weight of the loaded material. The model is designed to identify targets from the acquired image data and predict their location and bucket fill rate. Based on the predicted target location and bucket fill rate, key motion checks are used to determine the number of buckets and the weight of the loaded material. The target is a bucket or a bucket and the material contained within it. The target location is the location of the target itself or the location of the anchor frame surrounding the target, particularly in the vertical direction. The bucket fill rate is the proportion of the material volume in the bucket to its capacity during each loading process. The output includes the determined weight, which may include, for example, the weight of the material loaded each time and / or the total weight accumulated, and in particular, the determined number of hoppers and / or the predicted hopper fullness.

2. The method of claim 1, wherein, The image data is detected, especially continuously, using an image sensor, such as a camera, which is mounted on the loading machinery in a fixed position.

3. The method according to claim 1 or 2, characterized in that, The steps for determining the number of buckets and the weight of the loaded material using critical motion checks include: based on the predicted target location, checking if a critical motion has occurred; if a critical motion has occurred, incrementing the bucket count by 1. Preferably, the key action is that the target crosses the sensitive line in a specified direction, and checking whether the target crosses the sensitive line in a specified direction includes the following steps: Based on the target position predicted by the model, check whether the target is moving from bottom to top. For example, compare the height coordinate of a certain point on the target in the previous frame image data with the height coordinate of the corresponding point on the target in the current frame image data. If the height coordinate of the certain point on the target in the previous frame image data is less than the height coordinate of the corresponding point on the target in the current frame image data, then the target is considered to be moving from bottom to top. Based on the model's predicted bucket full rate, check whether there is material in the bucket. For example, compare the model's predicted bucket full rate with the value representing an empty bucket. If the bucket full rate is not equal to the value representing an empty bucket, then it is considered that there is material in the bucket. Based on the target position predicted by the model, check whether the target is crossing the sensitive line. For example, compare the height coordinates of the upper boundary and the lower boundary of the target with the height coordinates of the pre-determined sensitive line. If the sensitive line is located between the upper and lower boundaries of the target, it is considered that the target is crossing the sensitive line. The pre-determined height coordinates of the sensitive line are, for example, the median height of the imaging frame of the image sensor used to detect image data. If the results of the above three checks are all "yes", then increment the number of beacons by 1 when the target completely crosses the sensitive line.

4. The method of claim 3, wherein, When the count of buckets is incremented by 1, the full bucket rate predicted by the model that occurs most frequently within a time period is selected as the effective full bucket rate for this loading process, or the average of the full bucket rates whose differences do not exceed a predetermined threshold within a time period is used as the effective full bucket rate, or the maximum full bucket rate predicted by the model within a time period is used as the effective full bucket rate. Preferably, this time period is the period from when the upper and lower boundaries of the target are symmetrically located on both sides of the sensitive line until the lower boundary of the target just crosses the sensitive line.

5. The method of claim 4, wherein, The steps for determining the weight of the loaded material include: The product of the effective bucket fullness multiplied by the bucket capacity for each loading process and the material density of the material in the bucket is used as the weight of the material loaded each time; or The weight of the material is determined based on the effective bucket fill rate using a pre-determinable lookup table, which describes the correspondence between the bucket fill rate and the weight for a given bucket model and a given material type.

6. The method of claim 5, wherein, The material density is determined by the material type in the bucket, which may include, for example, sand, ore, and slag. Preferably, the material type is identified from the acquired image data by the model, or the material type or the material density is provided to the model as model parameters or input data.

7. The method according to any one of claims 1 to 6, characterized in that, The bucket capacity is determined by the bucket model. Preferably, the bucket model is identified from the acquired image data by the model, or the bucket model or the bucket capacity is provided to the model as model parameters or input data.

8. The method according to any one of claims 1 to 7, characterized in that, The model is pre-trained, and the training of the model includes the following steps: Collect image data of the loading process where the bucket loads materials onto the transport vehicle; The collected image data is manually labeled to indicate the fullness rate of the buckets containing materials contained in the image data. In particular, the material type and / or bucket model of the material in the bucket are also labeled. The model is trained based on the collected image data and the labeled bucket fullness, especially the labeled material type and / or bucket model, so that the trained model can determine the number of buckets, the bucket fullness, and thus the weight of the loaded material from the image data. The model's determined number of hoppers and the weight obtained from the full-hopper rate are compared with the corresponding data from the physical weighing system to optimize the model, for example by optimizing the objective function or reward / penalty algorithm.

9. The method according to any one of claims 1 to 8, characterized in that, The determined weight is output to a display, preferably mounted on or behind the loading machinery. In particular, the display also shows image data and marks the target contained in the image data, as well as the target position and bucket fullness.

10. A computer program product comprising instructions that are implemented to perform the method according to any one of claims 1 to 9.

11. A camera for weighing material loaded by loading machinery, such as excavators, loaders, electric shovels, hydraulic shovels, etc., the camera including an image sensor and a processor, the image sensor being used to detect image data during the loading process of the bucket of the loading machinery loading material onto a transport vehicle, such as a mining truck, the processor being designed to perform the method according to any one of claims 1 to 9.

Images