Truck loading quantity automatic calculation system based on visual identification of mine card position and excavator

Abstract

The invention provides an automatic loading quantity calculation system and excavator based on visual identification of mine card positions, and the system comprises a visual detection module which is used for detecting a bounding box of a mine card in real time through a target detection algorithm and extracting coordinates of a center point of the mine card; the state judgment module is used for judging whether the mine card is in a static state or a moving state according to the continuous frame variation of the center point coordinates of the mine card; the counting triggering module is used for triggering a loading quantity counter to accumulate when the mine card is switched from the static state to the moving state and the moving distance is greater than or equal to a preset minimum distance threshold value Dmin; and the data output module is used for outputting an accumulated value of the loading quantity counter. The dependence of the limited visual angle of the camera on the direct observation of the bucket action or the material state is eliminated; full-automatic statistics of the loading times is achieved, additional sensors or manual intervention is not needed, and the deployment cost is remarkably reduced; the system adaptability is improved, and different mine truck models and complex working conditions are supported.

Description

Technical Field

[0001] The present invention relates to the technical field of mine truck detection, and in particular to an automatic calculation system for the number of loaded vehicles based on visual recognition of the position of a mine truck, and an excavator. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In construction machinery operations, accurately counting the number of times mining trucks are loaded is an important basis for measuring operational efficiency and optimizing resource allocation. However, existing technologies have the following limitations:

[0004] (1) Manual counting and sensor dependence

[0005] Traditional methods rely on manual recording or specialized sensors (such as weighing systems and infrared sensors). Manual counting is inefficient and prone to errors, while sensors require high-precision hardware, are expensive to deploy, and are susceptible to environmental interference (such as dust and vibration).

[0006] (2) Limitations of visual solutions

[0007] Visual technology is used to detect the movement of the excavator bucket or the status of the material in the mine truck compartment for counting. However, this type of solution requires accurate capture of the bucket's motion trajectory or the status of the material in the mine truck compartment. Direct observation of the bucket movement or the status of the material in the mine truck compartment is easily affected by viewing angle limitations and occlusion interference, which in turn affects the recognition results. Summary of the Invention

[0008] In order to solve any one or more technical problems existing in the prior art, the present invention provides an automatic calculation system for the number of loading trucks based on visual recognition of the position of mining trucks and an excavator, which eliminates the reliance on direct observation of bucket movement or material status due to the limited camera viewing angle; realizes fully automated statistics of the number of loading times without the need for additional sensors or manual intervention, significantly reducing deployment costs; and improves system adaptability to support different mining truck models and complex working conditions.

[0009] A first aspect of the present invention provides a system for automatically calculating the number of trucks loaded based on visual recognition of the location of mining trucks, comprising:

[0010] Image acquisition module, used to obtain mining truck operation video stream;

[0011] The visual inspection module is used to detect the bounding box of the mining truck in real time and extract the coordinates of the center point of the mining truck through the target detection algorithm;

[0012] The state determination module is used to determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck;

[0013] The counting trigger module is used when the mining truck switches from a stationary state to a moving state and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate;

[0014] The data output module is used to output the accumulated value of the loading quantity counter.

[0015] Furthermore, the state determination module includes:

[0016] The stationary state determination unit is configured to determine that the mining truck is in a stationary state if the change in the coordinates of the center point of the mining truck in N consecutive frames is less than or equal to the pixel threshold ΔT;

[0017] The moving state determination unit is configured to determine that the mining truck is in a moving state if the change in the coordinates of the center point of the mining truck in N consecutive frames is greater than the pixel threshold ΔT.

[0018] Furthermore, in the counting trigger module, the moving distance is obtained by a mapping relationship calculation formula between pixels and actual distance, and the mapping relationship calculation formula is:

[0019] D 实际 =D 像素 k;

[0020] Among them, D 实际 D is the actual moving distance of the mining card. 像素 is the pixel value of the mining card movement in the camera, and k is the camera calibration parameter;

[0021] The calculation formula for camera calibration parameters is:

[0022]

[0023] Where H is the camera installation height, the vertical distance from the camera to the bottom of the excavator; f is the focal length; and the unit of k is meter / pixel.

[0024] Furthermore, the target detection algorithm adopts any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single Shot MultiBox Detector, Faster R-CNN and Mask R-CNN.

[0025] Furthermore, the system also includes an adaptive threshold calibration module for dynamically adjusting the minimum distance threshold D according to the actual size of the mining truck and the camera viewing angle parameters. min , the adjustment formula is as follows:

[0026] D min =α·L 实际 ·cosθ;

[0027] Among them, α is a fixed threshold with a value range of (0,1), which means that the mining truck needs to move more than or equal to α times its own length before it is judged to have completed loading; θ is the camera pitch angle, which is used to compensate for the projection error caused by the tilt of the viewing angle; L 实际 is the length of the truck compartment.

[0028] Furthermore, the data output module transmits the accumulated value of the loading quantity counter to the cab display screen via the CAN bus, and uploads it to the cloud via TBOX.

[0029] A second aspect of the present invention provides a method for automatically calculating the number of trucks loaded based on visual recognition of the location of a mining truck, comprising the following steps:

[0030] S1: Detect the bounding box of the mining truck in real time through the target detection algorithm and extract the coordinates of the center point of the mining truck;

[0031] S2: Determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck;

[0032] S3: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate;

[0033] S4: Output the accumulated value of the loading quantity counter.

[0034] Furthermore, the calculation method of the moving distance includes:

[0035] S41: Calculate the pixel movement D of the mining card center point based on the coordinates of the mining card center point 像素 ,The pixel movement is calculated by the coordinate difference of the mining card center point between consecutive frames;

[0036] S42: Obtain camera calibration parameter k, and calculate the calibration parameter according to the camera installation height H and focal length f:

[0037] The calibration parameter k represents the actual distance corresponding to the unit pixel.

[0038] S43: Calculate the actual moving distance D of the mining truck 实际 , move the pixel by D 像素 Multiply it by the calibration parameter k to get the actual moving distance;

[0039] S44: Trigger technical condition determination, if D 实际 Greater than or equal to the minimum distance threshold D min , then the loading quantity counter is triggered to accumulate.

[0040] A third aspect of the present invention provides an excavator, wherein the excavator is integrated with the above-mentioned automatic calculation system for the number of loaded vehicles.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium storing program instructions, which implement the steps of the above method when executed by a processor.

[0042] Compared with the prior art, the present invention provides an automatic calculation system for the number of loaded trucks based on visual recognition of the location of mining trucks and an excavator, which have the following beneficial effects:

[0043] (1) Fully automated counting eliminates manual intervention and hardware dependence. The visual inspection module uses a pure visual solution (camera + target detection algorithm), eliminating the need for hardware such as weighing sensors and infrared devices. The state determination module automatically identifies the state of the mining truck (stationary / moving) through the change in continuous frame coordinates, and the counting trigger module automatically accumulates counts based on the state switching logic. No manual intervention is required throughout the process, reducing deployment costs by more than 70%.

[0044] (2) Breaking through viewing angle limitations and improving robustness in complex environments. The core logic shifts from "detecting the loading process (bucket movement)" to "detecting the loading results (mining truck movement)." Counting is triggered by switching the mining truck's "stationary → moving" state, without the need to observe the bucket or material. The dual-state machine model (stationary state judgment unit + moving state judgment unit) combined with continuous frame detection filters out short-term mining truck jitter (such as engine idle vibration) and small displacement interference, resulting in a misjudgment rate of less than 2%.

[0045] (3) Dynamic calibration and generalization capabilities, adapting to multiple vehicle models and working conditions. The adaptive threshold calibration module dynamically adjusts the minimum distance threshold D based on the actual size of the mining truck (vehicle length, wheelbase) and camera parameters (installation height, focal length). min . BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0047] Figure 1 This is a framework diagram of the system for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks provided by the present invention;

[0048] Figure 2 This is a flowchart of the steps of the method for automatically calculating the loading quantity based on visual recognition of the mining truck position provided by the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0052] All data in this embodiment is obtained in compliance with laws and regulations and based on the consent of the user, and is used legally.

[0053] Example 1

[0054] This embodiment provides a system for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks. Figure 1 ,include:

[0055] Image acquisition module, used to obtain mining truck operation video stream;

[0056] The visual detection module is used to detect the bounding box of the mining truck in real time through the target detection algorithm and extract the coordinates of the center point of the mining truck (x c ,y c );

[0057] The state determination module is used to determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck;

[0058] The counting trigger module is used when the mining truck switches from a stationary state to a moving state and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate;

[0059] The data output module is used to output the accumulated value of the loading quantity counter.

[0060] Specifically, the state determination module includes:

[0061] The stationary state determination unit is configured to determine that the mining truck is in a stationary state if the change in the coordinates of the center point of the mining truck in N consecutive frames is less than or equal to the pixel threshold ΔT;

[0062] The moving state determination unit is configured to determine that the mining truck is in a moving state if the change in the coordinates of the center point of the mining truck in N consecutive frames is greater than the pixel threshold ΔT;

[0063] In a specific embodiment, in a stationary state, if the coordinate change of the center point of the mining card for N consecutive frames (such as N=10, the frame rate is 30fps) is less than a threshold value ΔT=5 pixels, it is determined to be stationary; in a moving state, if the coordinate change exceeds ΔT, it is determined to be moving.

[0064] Specifically, in the counting trigger module, the moving distance is obtained by a mapping relationship calculation formula between pixels and actual distance, and the mapping relationship calculation formula is:

[0065] D 实际 =D 像素 k;

[0066] Among them, D 实际 D is the actual moving distance of the mining card. 像素 is the pixel value of the mining card movement in the camera, and k is the camera calibration parameter;

[0067] The calculation formula for camera calibration parameters is:

[0068]

[0069] Where H is the camera installation height, the vertical distance from the camera to the bottom of the excavator; f is the focal length; and the unit of k is meter / pixel.

[0070] In a specific embodiment, when the mining truck enters the moving state from the stationary state, and the moving distance D≥D min (D min =2 meters, calculated by pixel-actual distance mapping), it is determined that one loading is completed;

[0071] Counter update: cumulative loading quantity C = C + 1.

[0072] Specifically, the target detection algorithm adopts any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single Shot MultiBox Detector, Faster R-CNN and Mask R-CNN.

[0073] In one specific embodiment, the YOLOv7 algorithm is used to detect the bounding boxes of mining trucks in real time. YOLOv7 is a single-stage object detection algorithm that implements multi-scale detection through an anchor box mechanism and a feature pyramid network (FPN). Its core approach is to divide the input image into grid cells, directly predicting the bounding box coordinates, confidence score, and class probability for each cell, achieving end-to-end real-time detection.

[0074] YOLOv7 optimizes detection accuracy through a composite scaling strategy and dynamic label allocation while maintaining real-time performance (≥30 FPS). It utilizes a Path Aggregation Network (PANet) to enhance feature fusion and adapt to the detection needs of mining trucks of varying sizes. It can be deployed on embedded devices (such as Jetson Xavier), making it suitable for the real-time operation environments of construction machinery.

[0075] Specifically, the system also includes an adaptive threshold calibration module for dynamically adjusting the minimum distance threshold D according to the actual size of the mining truck and the camera viewing angle parameters. min , to adapt to different mining truck models and complex working environments.

[0076] 1. Input parameters and prerequisites:

[0077] (1) Mining truck size

[0078] Direct detection: Real-time measurement of the truck compartment length L using target detection algorithms 实际 (Unit: meter) and width W 实际 ;

[0079] Implementation method: Based on the pixel size of the YOLO detection frame, the conversion is performed in combination with the camera calibration parameter k (meters / pixel);

[0080] Database matching: If target detection is unreliable (e.g., severe occlusion), the preset mining truck model database is called to match the size based on the license plate or appearance features.

[0081] (2) Camera parameters

[0082] Installation height H: the vertical distance from the camera to the bottom of the excavator (unit: meter);

[0083] Field of view (FOV): The horizontal and vertical field of view of the camera.

[0084] 2. Moving distance threshold (D min ) dynamic adjustment

[0085] Target: According to the length L of the mining card 实际 and the camera angle of view to determine the minimum moving distance threshold for triggering loading; adjust the formula:

[0086] Dmin =α·L 实际 cosθ

[0087] Among them, α: 0.6, means that the mining truck needs to move 60% of its own length before it is considered to have completed loading;

[0088] θ: Camera pitch angle (unit: radians), used to compensate for projection errors caused by viewing angle tilt.

[0089] Dynamic calibration mechanism:

[0090] Real-time feedback: If the loading trigger fails multiple times due to insufficient moving distance of the mining truck, α will be automatically increased (by 0.05 each time);

[0091] Vehicle type adaptation: α values are preset for different vehicle types (α = 0.5 for small mining trucks and α = 0.7 for large mining trucks).

[0092] 3. Dynamic Adjustment of the Stillness Judgment Threshold (ΔT)

[0093] Target: According to the width W of the mining card 实际 and the camera resolution, determine the maximum allowed pixel offset in the static state.

[0094] Adjustment formula:

[0095] ΔT=β·W 实际 / k (unit: pixel)

[0096] Where, β is the tolerance factor (β=0.02), which means that the maximum deviation of the mining truck when it is stationary is 2% of its width;

[0097] Dynamic calibration mechanism:

[0098] Environmental adaptation: Dynamically adjust β according to image clarity (β increases in dusty environments);

[0099] Jitter filtering: If the mining card is briefly offset due to vibration (such as ΔT < 24 pixels), it is judged to be in a stationary state.

[0100] 4. Camera Angle Compensation

[0101] Camera pitch angle θ and roll angle This will cause the projection of the mining card position to be distorted.

[0102] Compensation method:

[0103] (1) Coordinate correction formula:

[0104] x 校准 =x 原始 / cosθ,

[0105] Among them, the roll angle Usually can be ignored.

[0106] (2) Viewing angle adaptive threshold: D min The calculations of ΔT are based on the corrected coordinates.

[0107] 5. Exception handling and fault tolerance mechanism

[0108] (1) Mining card size failure:

[0109] If visual inspection fails and there is no database match, the default threshold (D min = 2 meters, ΔT = 5 pixels);

[0110] (2) Manual intervention interface:

[0111] Allows the operator to manually enter card dimensions or adjust threshold parameters.

[0112] The data output module also transmits the accumulated value of the loading quantity counter to the cab display screen via the CAN bus and uploads it to the cloud via TBOX.

[0113] Example 2

[0114] This embodiment provides a method for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks. Figure 2 , including the following steps:

[0115] S1: Get the mining truck operation video stream;

[0116] S2: Detect the bounding box of the mining truck in real time through the target detection algorithm and extract the coordinates of the center point of the mining truck;

[0117] S3: Determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck;

[0118] S4: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate;

[0119] S5: Output the accumulated value of the loading quantity counter.

[0120] Specifically, the calculation method of the moving distance includes:

[0121] S41: Calculate the pixel movement D of the mining card center point based on the coordinates of the mining card center point 像素 , the pixel movement is calculated by the coordinate difference of the mining card center point between consecutive frames:

[0122]

[0123] Among them, (xt ,y t ) and (x t-1 ,y t-1 ) are the coordinates of the mining card center point of the current frame and the previous frame respectively;

[0124] S42: Obtain camera calibration parameter k, and calculate the calibration parameter based on the camera installation height H (unit: meter) and focal length f (unit: pixel):

[0125]

[0126] The calibration parameter k represents the actual distance corresponding to the unit pixel (meter / pixel).

[0127] S43: Calculate the actual moving distance D of the mining truck 实际 , move the pixel by D 像素 Multiplying it with the calibration parameter k, we get the actual moving distance:

[0128] D 实际 =D 像素 ·k

[0129] S44: Trigger technical condition determination, if D 实际 Greater than or equal to the minimum distance threshold D min , then the loading quantity counter is triggered to accumulate.

[0130] Example 3

[0131] This embodiment provides an excavator equipped with the automatic calculation system for the number of loaded vehicles based on visual recognition of the position of mining trucks as provided in the first embodiment.

[0132] Example 4

[0133] This embodiment provides a computer-readable storage medium storing program instructions. When the program instructions are executed by a processor, the steps of the method for automatically calculating the number of loaded vehicles based on visual recognition of the position of a mining truck provided in the second embodiment are implemented.

[0134] The computer-readable storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. The stored program instructions include a data acquisition program, a data preprocessing program, a multi-source data fusion program, a dynamic weight adjustment program, a calibration result calculation program, and a feedback execution program. When these program instructions are executed by a processor, all steps of the method for automatically calculating the number of loaded vehicles based on visual recognition of the location of mining trucks described in the present invention are implemented.

[0135] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A system for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks, characterized in that: include: Image acquisition module, used to obtain mining truck operation video stream; The visual inspection module is used to detect the bounding box of the mining truck in real time and extract the coordinates of the center point of the mining truck through the target detection algorithm; The state determination module is used to determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck; The counting trigger module is used when the mining truck switches from a stationary state to a moving state and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate; The data output module is used to output the accumulated value of the loading quantity counter.

2. The automatic calculation system for the number of loaded vehicles based on visual recognition of the position of mining trucks according to claim 1, characterized in that: The state determination module includes: The stationary state determination unit is configured to determine that the mining truck is in a stationary state if the change in the coordinates of the center point of the mining truck in N consecutive frames is less than or equal to the pixel threshold ΔT; The moving state determination unit is configured to determine that the mining truck is in a moving state if the change in the coordinates of the center point of the mining truck in N consecutive frames is greater than the pixel threshold ΔT.

3. The automatic calculation system for the number of loaded vehicles based on visual recognition of the position of mining trucks according to claim 1, characterized in that: In the counting trigger module, the moving distance is obtained by a mapping relationship calculation formula between pixels and actual distance, and the mapping relationship calculation formula is: D 实际 =D 像素 ·k; Among them, D 实际 D is the actual moving distance of the mining card. 像素 is the pixel value of the mining card movement in the camera, and k is the camera calibration parameter; The calculation formula for camera calibration parameters is: Where H is the camera installation height, the vertical distance from the camera to the bottom of the excavator; f is the focal length; and the unit of k is meter / pixel.

4. The automatic calculation system for the number of loaded vehicles based on visual recognition of the position of mining trucks according to claim 1, characterized in that: The target detection algorithm adopts any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single Shot MultiBox Detector, Faster R-CNN and Mask R-CNN.

5. The system for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks according to claim 1, characterized in that: The system also includes an adaptive threshold calibration module for dynamically adjusting the minimum distance threshold D according to the actual size of the mining truck and the camera viewing angle parameters. min , the adjustment formula is as follows: D min =α·L 实际 ·cosθ; Among them, α is a fixed threshold with a value range of (0,1), which means that the mining truck needs to move more than or equal to α times its own length before it is judged to have completed loading; θ is the camera pitch angle, which is used to compensate for the projection error caused by the tilt of the viewing angle; L 实际 is the length of the truck compartment.

6. The system for automatically calculating the number of vehicles loaded based on visual recognition of the location of mining trucks according to claim 1, characterized in that: The data output module also transmits the accumulated value of the loading quantity counter to the cab display screen via the CAN bus, and uploads it to the cloud via TBOX.

7. A method for automatically calculating the loading quantity based on visual recognition of the location of mining trucks, characterized in that: The following steps are involved: S1: Get the mining truck operation video stream; S2: Detect the bounding box of the mining truck in real time through the target detection algorithm and extract the coordinates of the center point of the mining truck; S3: Determine whether the mining truck is in a stationary state or a moving state based on the continuous frame change of the center point coordinates of the mining truck; S4: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D min When , the loading quantity counter is triggered to accumulate; S5: Output the accumulated value of the loading quantity counter.

8. The method for automatically calculating the loading quantity based on visual recognition of the mining truck position according to claim 7, characterized in that: The calculation method of the moving distance includes: S41: Calculate the pixel movement D of the mining card center point based on the coordinates of the mining card center point 像素 ,The pixel movement is calculated by the coordinate difference of the mining card center point between consecutive frames; S42: Obtain camera calibration parameter k, and calculate the calibration parameter according to the camera installation height H and focal length f: The calibration parameter k represents the actual distance corresponding to the unit pixel. S43: Calculate the actual moving distance D of the mining truck 实际 , move the pixel by D 像素 Multiply it by the calibration parameter k to get the actual moving distance; S44: Trigger technical condition determination, if D 实际 Greater than or equal to the minimum distance threshold D min , then the loading quantity counter is triggered to accumulate.

9. An excavator, characterized in that: Integrate the automatic calculation system for the number of loaded vehicles based on visual recognition of the position of mining trucks as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing program instructions, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 7 to 8 are implemented.

Images