A loading quantity automatic calculation system based on visual identification of mine card position and excavator

Abstract

The application provides a loading quantity automatic calculation system and a excavator based on visual identification of a mine card position, comprising: a visual detection module, which is used for detecting a boundary box of the mine card in real time through a target detection algorithm and extracting a mine card center point coordinate; a state determination module, which is used for determining whether the mine card is in a static state or a moving state according to a continuous frame variation of the mine card center point coordinate; a counting trigger module, which is used for triggering a loading quantity counter to accumulate when the mine card switches from the static state to the moving state and a moving distance is greater than or equal to a preset minimum distance threshold D min ; a data output module, which is used for outputting an accumulated value of the loading quantity counter. The dependence of direct observation of the bucket action or the material state on the limited camera view angle is eliminated; the automatic statistics of the loading times are realized, no additional sensors or manual intervention are needed, the deployment cost is significantly reduced, and the system adaptability is improved to support different mine truck types and complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of mining truck detection technology, specifically to an automatic calculation system for the loading quantity of mining trucks based on visual recognition of the truck's location, and an excavator. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In construction machinery operations, accurately counting the number of times mining trucks are loaded is a crucial basis for measuring operational efficiency and optimizing resource allocation. However, existing technologies have the following limitations: (1) Manual counting and sensor dependence Traditional methods rely on manual recording or specialized sensors (such as weighing systems and infrared sensors). Manual counting is inefficient and prone to errors, while sensors require high-precision hardware support, are costly to deploy, and are susceptible to environmental interference (such as dust and vibration).

[0004] (2) Limitations of visual solutions Counting is achieved by detecting the movement of the excavator bucket or the state of the material inside the mining truck using visual technology. However, such solutions require accurate capture of the bucket's movement trajectory or the state of the material inside the mining truck. Direct observation of the bucket's movement or the state of the material inside the mining truck is easily affected by viewing angle limitations and occlusion interference, which in turn affects the recognition results. Summary of the Invention

[0005] To address any one or more technical problems existing in the prior art, this invention provides an automatic loading quantity calculation system and excavator based on visual recognition of mining truck location, eliminating the reliance on direct observation of bucket movements or material status due to the limited field of view of the camera; achieving fully automated statistics of loading times without the need for additional sensors or manual intervention, significantly reducing deployment costs; and improving system adaptability, supporting different mining truck models and complex working conditions.

[0006] The first aspect of the present invention provides an automatic calculation system for loading quantity based on visual recognition of mining truck location, comprising: The image acquisition module is used to acquire video streams of mining truck operations; The visual inspection module is used to detect the bounding box of the mining card in real time and extract the coordinates of the center point of the mining card through the target detection algorithm; The status determination module is used to determine whether the mining truck is in a stationary or moving state based on the continuous frame change of the center point coordinates of the mining truck. The counting trigger module is used when the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to a preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; The data output module is used to output the accumulated value of the loading quantity counter.

[0007] Furthermore, the state determination module includes: The stationary state determination unit is configured to determine that the mining card is stationary if the change in the coordinates of the center point of the mining card within N consecutive frames is less than or equal to the pixel threshold ΔT. The motion state determination unit is configured to determine that the mining card is in motion state if the change in the center point coordinates of the mining card within N consecutive frames is greater than the pixel threshold ΔT.

[0008] Furthermore, in the counting trigger module, the moving distance is obtained through a calculation formula based on the mapping relationship between pixels and actual distance. The mapping relationship calculation formula is as follows: ; in, This represents the actual distance the mining truck traveled. These are the pixel values ​​representing the movement of the mining card within the camera. Calibrate the camera parameters; The formula for calculating the camera calibration parameters is: ; in, H Determine the camera's installation height and the vertical distance from the camera to the bottom of the excavator. f Focal length; k The unit is meters per pixel.

[0009] Furthermore, the target detection algorithm employs any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single Shot MultiBox Detector, Faster R-CNN, and Mask R-CNN.

[0010] Furthermore, the system also includes an adaptive threshold calibration module, used to dynamically adjust the minimum distance threshold D based on the actual size of the mining truck and the camera's viewing angle parameters. min The formula is adjusted as follows: D min =α·L 实际 ·cosθ; Where α is a fixed threshold value ranging from (0,1), indicating that the loading is considered complete only when the mining truck has moved at least α times its own length; θ is the camera's tilt angle, used to compensate for projection errors caused by viewing angle tilt; L 实际 This refers to the length of the mining truck's cargo compartment.

[0011] Furthermore, the data output module also transmits the accumulated value of the vehicle quantity counter to the cab display screen via the CAN bus, and uploads it to the cloud via TBOX.

[0012] A second aspect of the present invention provides an automatic calculation method for the loading quantity based on visual recognition of the location of mining trucks, comprising the following steps: S1: Real-time detection of the bounding box of the mining card and extraction of the coordinates of the center point of the mining card through the target detection algorithm; S2: Determine whether the mining card is stationary or moving based on the continuous frame changes in the center point coordinates of the mining card. S3: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; S4: Output the accumulated value of the vehicle loading quantity counter.

[0013] Furthermore, the method for calculating the movement distance includes: S41: Calculate the pixel movement of the mining card's center point based on the coordinates of the mining card's center point. The pixel movement is calculated by the difference in coordinates of the center point of the mining card between consecutive frames; S42: Obtain camera calibration parameters k According to the camera installation height H With focal length f Calculate the calibration parameters: ; Calibration parameters k This represents the actual distance corresponding to a unit pixel.

[0014] S43: Calculate the actual distance the mining truck has moved. , shift pixel amount With calibration parameters k Multiply them to get the actual distance traveled; S44: Triggering technical condition determination, if Greater than or equal to the minimum distance threshold D min This will trigger the loading quantity counter to increment.

[0015] A third aspect of the present invention provides an excavator that integrates the above-described automatic loading quantity calculation system.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] Compared with existing technologies, the automatic loading quantity calculation system and excavator based on visual recognition of mining truck location provided by this invention have the following advantages: (1) Fully automated counting, eliminating manual intervention and hardware dependence. The visual inspection module adopts a pure vision solution (camera + target detection algorithm), which does not require the installation of weighing sensors, infrared devices and other hardware; the status determination module automatically identifies the status of the mining truck (stationary / moving) through the continuous frame coordinate change, and the counting trigger module automatically accumulates the count based on the status switching logic. No manual intervention is required throughout the process, reducing deployment costs by more than 70%.

[0018] (2) Breaking through perspective limitations and improving robustness in complex environments. The core logic has been transformed from "detecting the loading process (bucket movement)" to "detecting the loading result (mining truck movement)". The counting is triggered by the "stationary → moving" state switch of the mining truck, without the need to observe the bucket or material. The dual-state machine model (stationary state judgment unit + moving state judgment unit) combined with continuous frame detection filters out brief shaking of the mining truck (such as engine idling vibration) and small displacement interference, with a false judgment rate of less than 2%.

[0019] (3) Dynamic calibration and generalization capabilities, adaptable to multiple vehicle models and operating conditions. The adaptive threshold calibration module dynamically adjusts the minimum distance threshold D based on the actual dimensions of the mining truck (vehicle length, wheelbase) and camera parameters (installation height, focal length). min . Attached Figure Description

[0020] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0021] Figure 1 A framework diagram of the automatic loading quantity calculation system based on visual recognition of mining truck location provided by the present invention; Figure 2 The flowchart illustrates the steps of the automatic calculation method for loading quantity based on visual recognition of mining truck location provided by this invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0026] Example 1 This embodiment provides an automatic calculation system for the loading quantity based on visual recognition of the location of mining trucks, such as... Figure 1 ,include: The image acquisition module is used to acquire video streams of mining truck operations; The visual inspection module is used to detect the bounding box of the mining card in real time using an object detection algorithm and extract the coordinates of the center point of the mining card (x). c y c ); The status determination module is used to determine whether the mining truck is in a stationary or moving state based on the continuous frame change of the center point coordinates of the mining truck. The counting trigger module is used when the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to a preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; The data output module is used to output the accumulated value of the loading quantity counter.

[0027] Specifically, the state determination module includes: The stationary state determination unit is configured to determine that the mining card is stationary if the change in the coordinates of the center point of the mining card within N consecutive frames is less than or equal to the pixel threshold ΔT. The motion state determination unit is configured to determine that the mining card is in motion state if the change in the coordinates of the center point of the mining card within N consecutive frames is greater than the pixel threshold ΔT. In one specific embodiment, the stationary state is determined as follows: if the change in the coordinates of the center point of the mining card is less than the threshold ΔT = 5 pixels for N consecutive frames (e.g., N=10, frame rate 30fps), it is determined as stationary; the moving state is determined as follows: if the change in coordinates exceeds ΔT, it is determined as moving.

[0028] Specifically, in the counting trigger module, the moving distance is obtained through a calculation formula based on the mapping relationship between pixels and actual distance. The mapping relationship calculation formula is as follows: ; in, This represents the actual distance the mining truck traveled. These are the pixel values ​​representing the movement of the mining card within the camera. Calibrate the camera parameters; The formula for calculating the camera calibration parameters is: ; in, H Determine the camera's installation height and the vertical distance from the camera to the bottom of the excavator. f Focal length; k The unit is meters per pixel.

[0029] In one specific embodiment, when the mining truck moves from a stationary state to a moving state, and the moving distance D ≥ D min (D) min =2 meters (calculated through pixel-actual distance mapping) is considered a completed loading operation; Counter update: Cumulative number of vehicles loaded C = C + 1.

[0030] Specifically, the target detection algorithm employs any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single Shot MultiBox Detector, Faster R-CNN, and Mask R-CNN.

[0031] In one specific embodiment, the YOLOv7 algorithm is used to detect the bounding boxes of mining cards in real time. YOLOv7 is a single-stage object detection algorithm that achieves multi-scale detection through anchor boxes and a feature pyramid network (FPN). Its core is to divide the input image into grid cells, and each cell directly predicts the bounding box coordinates, confidence score, and class probability, achieving end-to-end real-time detection.

[0032] YOLOv7 maintains real-time performance (≥30 FPS) while optimizing detection accuracy through a composite scaling strategy and dynamic label allocation. It utilizes a path aggregation network (PANet) to enhance feature fusion, adapting to the detection needs of mining trucks of different sizes. It can be deployed on embedded devices (such as Jetson Xavier), making it suitable for the real-time operating environment of construction machinery.

[0033] Specifically, the system also includes an adaptive threshold calibration module, used to dynamically adjust the minimum distance threshold D based on the actual size of the mining truck and the camera's viewing angle parameters.min To adapt to different mining truck models and complex working conditions.

[0034] I. Input Parameters and Prerequisites: (1) Mining card size Direct detection: Real-time measurement of the length L of the mining truck bed using a target detection algorithm. 实际 (Unit: meters) and width W 实际 ; Implementation method: Based on the pixel size of the YOLO detection bounding box, combined with camera calibration parameters. k Convert (meters / pixels); Database matching: If target detection is unreliable (e.g., severe occlusion), a preset mining truck model database is called to match the size based on the license plate or appearance features.

[0035] (2) Camera parameters Installation height H: Vertical distance from the camera to the bottom of the excavator (unit: meters); Field of view (FOV): The horizontal and vertical angles of view of the camera.

[0036] II. Movement Distance Threshold (D) min Dynamic adjustment Objective: Based on the length L of the mining card 实际 Based on the camera's perspective, determine the minimum movement distance threshold that triggers loading; adjust the formula: D min =α·L 实际 ·cosθ Where α: 0.6 means that the mining truck needs to move 60% of its own length before it is considered to have completed loading; θ: Camera pitch angle (unit: radians), used to compensate for projection errors caused by viewing angle tilt.

[0037] Dynamic calibration mechanism: Real-time feedback: If loading fails multiple times due to insufficient distance traveled by the mining truck, α will be automatically increased (by 0.05 each time). Vehicle model compatibility: Preset α values ​​for different vehicle models (α=0.5 for small mining trucks, α=0.7 for large mining trucks).

[0038] III. Dynamic Adjustment of the Static Judgment Threshold (ΔT) Objective: Based on the width W of the mining card 实际 Based on the camera resolution, determine the maximum allowable pixel offset in a stationary state.

[0039] Adjust the formula: ΔT=β·W 实际 / (Unit: pixels) Wherein, β: fault tolerance coefficient (β=0.02), representing the maximum allowable deviation of the mining truck from its width by 2% when stationary; Dynamic calibration mechanism: Environment Adaptive: β is dynamically adjusted based on image sharpness (β increases in dusty environments); Jitter Filtering: If the mining card is briefly shifted due to vibration (e.g., ΔT < 24 pixels), it is judged as a stationary state.

[0040] IV. Camera View Compensation The camera's pitch angle θ and tilt angle φ can cause distortion in the projection of the mining truck's position.

[0041] Compensation methods: (1) Coordinate correction formula: x 校准 =x 原始 / cosθ,y 校准 =y 原始 / cosφ; The roll angle φ is usually negligible.

[0042] (2) Viewpoint adaptive threshold: D min The calculation of ΔT is based on the corrected coordinates.

[0043] V. Exception Handling and Fault Tolerance Mechanism (1) Mining card size failure: If visual detection fails and there is no database match, the default threshold (D) will be used. min =2 meters, ΔT=5 pixels); (2) Manual intervention interface: Operators are allowed to manually enter the mining card size or adjust threshold parameters.

[0044] The data output module also transmits the accumulated value of the vehicle quantity counter to the cab display screen via the CAN bus, and uploads it to the cloud via TBOX.

[0045] Example 2 This embodiment provides a method for automatically calculating the loading quantity based on visual recognition of the mining truck's location, such as... Figure 2 This includes the following steps: S1: Obtain the video stream of the mining operation; S2: Real-time detection of the bounding box of the mining card and extraction of the coordinates of the center point of the mining card through the target detection algorithm; S3: Determine whether the mining card is stationary or moving based on the continuous frame changes in the center point coordinates of the mining card. S4: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; S5: Output the accumulated value of the vehicle loading quantity counter.

[0046] Specifically, the methods for calculating the distance traveled include: S41: Calculate the pixel movement of the mining card's center point based on the coordinates of the mining card's center point. The pixel movement is calculated by the difference in coordinates of the center point of the mining card between consecutive frames:

[0047] in, and These are the coordinates of the center point of the mining card in the current frame and the previous frame, respectively. S42: Obtain camera calibration parameters k According to the camera installation height H (Unit: meters) and focal length f (Unit: pixels), calculate calibration parameters: ; Calibration parameters k This represents the actual distance (meters per pixel) corresponding to a unit pixel.

[0048] S43: Calculate the actual distance the mining truck moves. , shift pixel amount With calibration parameters k Multiply by each other to get the actual distance traveled:

[0049] S44: Triggering technical condition determination, if Greater than or equal to the minimum distance threshold D min This will trigger the loading quantity counter to increment.

[0050] Example 3 This embodiment provides an excavator equipped with an automatic loading quantity calculation system based on visual recognition of mining truck location, as provided in Embodiment 1.

[0051] Example 4 This embodiment provides a computer-readable storage medium storing program instructions, which, when executed by a processor, implement the steps of the automatic calculation method for loading quantity based on visual recognition of mining truck location provided in Embodiment 2.

[0052] The computer-readable storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. The stored program instructions include data acquisition programs, data preprocessing programs, multi-source data fusion programs, dynamic weight adjustment programs, calibration result calculation programs, and feedback execution programs. When these program instructions are executed by the processor, all steps of the automatic calculation method for the loading quantity based on visual recognition of the mining truck location described in this invention are implemented.

[0053] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. An automatic calculation system for loading quantity based on visual recognition of mining truck location, characterized in that, include: The image acquisition module is used to acquire video streams of mining truck operations; The visual inspection module is used to detect the bounding box of the mining card in real time and extract the coordinates of the center point of the mining card through the target detection algorithm; The status determination module is used to determine whether the mining truck is in a stationary or moving state based on the continuous frame change of the center point coordinates of the mining truck. The counting trigger module is used when the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to a preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; The data output module is used to output the accumulated value of the loading quantity counter; The system also includes an adaptive threshold calibration module, used to dynamically adjust the minimum distance threshold D based on the actual size of the mining truck and the camera's viewing angle parameters. min The formula is adjusted as follows: D min =α·L 实际 ·cosθ; Where α is a fixed threshold value ranging from (0,1), indicating that the loading is considered complete only when the mining truck has moved at least α times its own length; θ is the camera's tilt angle, used to compensate for projection errors caused by viewing angle tilt; L 实际 The length of the mining truck compartment.

2. The automatic loading quantity calculation system based on visual recognition of mining truck location as described in claim 1, characterized in that, The status determination module includes: The stationary state determination unit is configured to determine that the mining card is stationary if the change in the coordinates of the center point of the mining card within N consecutive frames is less than or equal to the pixel threshold ΔT. The motion state determination unit is configured to determine that the mining card is in motion state if the change in the center point coordinates of the mining card within N consecutive frames is greater than the pixel threshold ΔT.

3. The automatic loading quantity calculation system based on visual recognition of mining truck location as described in claim 1, characterized in that, In the counting trigger module, the moving distance is obtained through a formula for calculating the mapping relationship between pixels and actual distance. The formula for calculating the mapping relationship is as follows: ； in, This represents the actual distance the mining truck traveled. These are the pixel values ​​representing the movement of the mining card within the camera. Calibrate the camera parameters; The formula for calculating the camera calibration parameters is: ； in, H Determine the camera's installation height and the vertical distance from the camera to the bottom of the excavator. f Focal length; k The unit is meters per pixel.

4. The automatic loading quantity calculation system based on visual recognition of mining truck location as described in claim 1, characterized in that, The target detection algorithm adopts any one or more of the following algorithms: YOLOv5, YOLOv7, YOLOv8, Single ShotMultiBox Detector, Faster R-CNN and Mask R-CNN.

5. The automatic loading quantity calculation system based on visual recognition of mining truck location as described in claim 1, characterized in that, The data output module also transmits the accumulated value of the vehicle quantity counter to the cab display screen via the CAN bus, and uploads it to the cloud via TBOX.

6. A method for automatically calculating the loading quantity of mining trucks based on visual recognition of the location of mining trucks, characterized in that, The automatic loading quantity calculation system based on visual recognition of mining truck location as described in any one of claims 1-5 includes the following steps: S1: Obtain the video stream of the mining operation; S2: Real-time detection of the bounding box of the mining card and extraction of the coordinates of the center point of the mining card through the target detection algorithm; S3: Determine whether the mining card is stationary or moving based on the continuous frame changes in the center point coordinates of the mining card. S4: When the mining truck switches from a stationary state to a moving state, and the moving distance is greater than or equal to the preset minimum distance threshold D. min At that time, the loading quantity counter is triggered to increment; S5: Output the accumulated value of the vehicle loading quantity counter.

7. The automatic calculation method for loading quantity based on visual recognition of mining truck location as described in claim 6, characterized in that, The method for calculating the distance traveled includes: S41: Calculate the pixel movement of the mining card's center point based on the coordinates of the mining card's center point. The pixel movement is calculated by the difference in coordinates of the center point of the mining card between consecutive frames; S42: Obtain camera calibration parameters k According to the camera installation height H With focal length f Calculate the calibration parameters: ; Calibration parameters k This represents the actual distance corresponding to a unit pixel. S43: Calculate the actual distance the mining truck has moved. , shift pixel amount With calibration parameters k Multiply them to get the actual distance traveled; S44: Triggering technical condition determination, if Greater than or equal to the minimum distance threshold D min This will trigger the loading quantity counter to increment.

8. An excavator, characterized in that, The system integrates the automatic calculation system for loading quantity based on visual recognition of mining truck location as described in any one of claims 1-5.

9. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed by the processor, they implement the steps of the method as described in any one of claims 6-7.

Images