Intelligent excavator loading efficiency analysis system based on visual inspection and time sequence modeling

Abstract

The invention discloses an excavator loading efficiency intelligent analysis system based on visual inspection and time sequence modeling, which comprises a camera and an edge computing device, and is characterized in that the camera is used for collecting a real-time video stream of an excavator operation site; a target detection model, an idle time detection module, a key frame queue generation module, a time sequence classification model and a data recording and analyzing module are arranged in the edge computing device, and the target detection model obtains multi-frame image data marked with bounding box positions of a mining truck and an excavator bucket and excavator bucket state information; the idle time detection module obtains idle starting time and idle ending time; the key frame queue generation module constructs a time sequence key frame queue; the time sequence classification model obtains a loading efficiency index; and the data recording and analyzing module generates loading efficiency analysis data according to the loading efficiency index. According to the method, key efficiency data such as the number of loading buckets, loading duration and idle time can be automatically counted, and real-time and accurate data support is provided for efficiency optimization of the excavator.

Description

Technical Field

[0001] This invention belongs to the field of intelligent monitoring technology for construction machinery operations, specifically involving an intelligent analysis system for excavator loading efficiency based on visual detection and time-series modeling. Background Technology

[0002] In open-pit mining, excavators, as core loading equipment, directly impact the overall project progress and resource utilization. Data shows that excavator loading accounts for 35%–50% of the total time spent transporting minerals in open-pit mines; a 10% increase in efficiency at this stage can boost overall mine capacity by 8%–12%. Key indicators for evaluating excavator loading efficiency include loading trips, loading time per trip, number of buckets loaded, and equipment idle time. Traditional manual statistical methods are susceptible to subjective factors, resulting in data lag, significant errors, and difficulty in meeting the real-time management needs of mines. Accurately identifying these indicators provides data support for equipment scheduling optimization and reduces ineffective energy consumption by analyzing the causes of idle time, thus possessing significant practical value for improving overall project production efficiency.

[0003] While current solutions for excavator efficiency calculation and identification exist, they generally suffer from significant shortcomings. Some solutions rely on pre-installed specialized equipment, resulting in poor compatibility with leased or multi-brand equipment. Others calculate efficiency based solely on work cycle time and basic equipment parameters, neglecting common idle periods and abnormal delays encountered in actual operations. Furthermore, some solutions require extensive labeled data to build models and have relatively fixed applicable scenarios, making them difficult to adapt to complex and ever-changing construction site environments.

[0004] Image-based excavator efficiency recognition solutions have become a research focus due to their low cost and lack of invasive equipment modifications. However, early solutions required manual intervention for key action annotation, resulting in low efficiency. Subsequent deep learning-based solutions sometimes rely on synthetically generated data for model training, and most depend on fixed-viewpoint image acquisition. Monitoring is interrupted when the excavator moves out of the field of view or is obstructed by other equipment. Furthermore, these solutions suffer from high model complexity and demanding hardware computing power, making them unsuitable for the low-computing edge devices commonly found on construction sites and unable to meet the needs of real-time on-site analysis. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides an intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling. This system solves the problems of high cost, complex installation, low recognition accuracy, low detection reliability, inability to acquire data in a timely manner, and inability to comprehensively and automatically collect loading efficiency data in existing technologies. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides an intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling, comprising a camera and an edge computing device, wherein... The camera is used to collect real-time video streams from the excavator's work site and send the real-time video streams to the edge computing device; The edge computing device includes a trained target detection model, an idle time detection module, a keyframe queue generation module, a trained temporal classification model, and a data recording and analysis module, wherein... The trained target detection model is used to obtain multi-frame image data, labeled with the bounding box positions of the mining truck and bucket, and the bucket status information, based on the real-time video stream from the excavator's operating site. The idle time detection module is used to obtain the idle start time and idle end time during the excavator's operation based on the multi-frame image data output by the target detection model. The keyframe queue generation module is used to construct a temporal keyframe queue based on the multi-frame image data output by the target detection model, the temporal keyframe queue including frames arranged in temporal order. M The image at the moment of unloading; the trained temporal classification model is used to obtain loading efficiency indicators based on the temporal keyframe queue, the loading efficiency indicators including the start and end time of a single loading cycle and the number of loading buckets; the data recording and analysis module is used to perform loading efficiency analysis using the start and end time of a single loading cycle, the number of loading buckets, the start and end time of idle time, and the idle duration, and generate loading efficiency analysis data.

[0006] In one embodiment of the present invention, the object detection model employs a YOLOv8 network, and the training process of the object detection model includes: Collect a large amount of video data from excavator operation sites, including video data under different lighting conditions, weather conditions, mining truck types and operation scenarios. Extract the video data into multi-frame images, and label the bounding boxes of the mining truck and bucket and the state of the bucket in each of the multi-frame images to form a training dataset for the target detection model. Using a YOLOv8 network pre-trained on the COCO dataset as the initial model, the initial model is trained using the object detection model training dataset. During the training process, a transfer learning strategy is adopted to obtain a trained object detection model.

[0007] In one embodiment of the present invention, the idle time detection module is specifically used for: Receive multi-frame image data from the target detection model, labeled with the bounding box positions of the mining truck and bucket, and bucket status information. Calculate the movement distance of the bucket bounding box center point between adjacent frames to determine if the bucket has moved. If continuous... N If the center point of the bucket bounding box in the frame image moves less than the set movement threshold, the bucket is determined to be stationary, that is, the excavator is in an idle state, and the time corresponding to the current frame image is recorded as the idle start time. After determining the idle start time, the movement distance of the center point of the bucket bounding box between subsequent adjacent frames is calculated. If the movement distance of the center point of the bucket bounding box in a certain frame exceeds the set movement threshold, it is determined whether the time interval between the current time and the idle start time exceeds the set minimum time threshold. If not, the movement distance of the center point of the bucket bounding box is calculated. If so, the current time is recorded as the idle end time.

[0008] In one embodiment of the present invention, the keyframe queue generation module is used to receive multi-frame image data from the target detection model, which is labeled with the bounding box positions of the mining truck and the bucket, and the bucket status information. The multi-frame images are judged sequentially according to the time sequence to determine whether the bucket is above the vehicle in the current image. If not, the current image is removed. If so, the bucket status of the current image is recorded, and the bucket status in the current frame image is determined according to the bucket status in the previous and next frames to determine whether the bucket in the current frame image has switched from a full bucket to an empty bucket. If so, the current image is saved to the time sequence keyframe queue in sequence. If not, only the current bucket status is recorded.

[0009] In one embodiment of the present invention, the number of images that can be accommodated in the time-series keyframe queue is greater than the number of loading buckets in a single loading cycle.

[0010] In one embodiment of the present invention, the temporal classification model includes a feature extraction module, a Transformer temporal encoder, and a fully connected layer classifier, wherein, The feature extraction module is used to extract features from each keyframe image in the temporal keyframe queue to obtain the corresponding visual feature vector. The Transformer temporal encoder is used to receive the visual feature vector, model the inter-frame dependency using a self-attention mechanism, and output a fixed-dimensional sequence representation vector, which contains the spatiotemporal information of the entire unloading sequence. The fully connected layer classifier is used to obtain the loading start time, loading end time, and number of loading buckets in a single loading cycle based on the features output by the Transformer time encoder.

[0011] In one embodiment of the present invention, the training process of the time-series classification model includes: Collect a large amount of video data from excavator operation sites and extract multiple complete loading cycle sequences; Manually mark the start time, end time, and number of buckets loaded for each loading cycle; Based on the labeled loading cycle sequence, a sliding window strategy is used to augment the data to increase the amount of data and form a training dataset for the time series classification model. The time-series classification model is trained using the training dataset, with the loss function being a weighted combination of cross-entropy loss and conditional random field loss.

[0012] In one embodiment of the present invention, the intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling further includes an upload module and a display terminal interface, wherein, The upload module is connected to the data recording and analysis module and the dispatch center, and is used to transmit loading times, start and end times of a single loading cycle, number of loading buckets in a single loading cycle, idle start and end times and idle duration, and loading efficiency analysis data to the dispatch center in real time. The display terminal interface is connected to the data recording and analysis module and is used to generate an audio-visual or status signal when a loading completion event or a system status change is detected, and to send the audio-visual or status signal to the cab display.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: The intelligent excavator loading efficiency analysis system based on visual detection and temporal modeling provided by this invention first uses a target detection model to identify the positions of the mining truck and the bucket. When the bucket is detected above the mining truck and its state changes from full to empty, it is determined that loading of that bucket is complete, and the image at this moment is acquired and saved into a temporal keyframe queue. Then, multiple temporal images are used as input to a temporal classification model to identify the start and end times of loading and the number of buckets loaded. The target detection model is used to determine whether the excavator is working by checking whether the bucket position has moved, thus completing the excavator idle time statistics and obtaining the excavator idle start and end times and idle duration. The data recording and analysis module saves the above data to generate loading efficiency analysis data. In this way, based on real-time video streams captured by cameras at the excavator's work site, intelligent monitoring of the excavator's loading status and work efficiency is achieved using target detection and time-series analysis. Loading counts and efficiency statistics can be completed without the need for additional hardware such as weight sensors and position sensors, significantly reducing installation complexity and system costs. By integrating AI algorithms for target detection and time-series classification, key action events (such as unloading) can be accurately identified and the continuous working process can be analyzed, reducing false positives and false negatives, resulting in higher recognition accuracy and improved detection reliability. The system can comprehensively and automatically collect key efficiency data such as the number of buckets loaded, loading time, work cycle, and idle time. The entire system runs in real time on the excavator's local edge computing device, with low network dependence, and can promptly acquire data and provide feedback to the dispatch center or on-site operators.

[0014] This invention does not require complex 3D modeling or a large amount of labeled data. The model can be deployed on edge devices and can provide real-time and accurate data support for optimizing excavator efficiency.

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of an intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling, provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the process of obtaining the idle start time and idle end time of an excavator, as provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a timing keyframe queue provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the identification results of loading start and end times and the number of loading buckets provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of another intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling provided in an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a detailed explanation of an intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling proposed in accordance with the present invention.

[0018] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0019] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of an intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling, provided by an embodiment of the present invention. The analysis system includes a camera and an edge computing device. The camera is used to collect real-time video streams from the excavator's operating site and send these streams to the edge computing device. The edge computing device is equipped with a trained target detection model, an idle time detection module, a keyframe queue generation module, a trained temporal classification model, and a data recording and analysis module.

[0021] In this embodiment, the camera and edge computing device can be mounted on the boom or top of the cab of an excavator. The camera is aimed at the excavator's working area (such as the area where the excavator's bucket interacts with the mining truck), and is used to acquire a high-frame-rate real-time video stream of the entire loading process of the mining truck. The high-frame-rate real-time video stream acquired by the camera is transmitted to the edge computing device for real-time processing via wired or wireless means. The aforementioned mining truck refers to a vehicle used for transportation at the mining site.

[0022] The trained object detection model is used to obtain multi-frame image data labeled with the bounding box positions, category confidence scores, and bucket status information of mining trucks and buckets based on real-time video streams from excavator operation sites. The object detection model in this embodiment uses a YOLOv8 network. Before practical object detection applications, the model needs to be trained. The specific training process includes: collecting a large amount of video data from excavator operation sites, including video data under different lighting conditions, weather conditions, mining truck types, and operation scenarios; extracting multi-frame images from the video data; and labeling the bounding boxes of the mining trucks and buckets, as well as the bucket status, in each frame to form a training dataset for the object detection model. The bucket status includes whether the bucket is full or empty.

[0023] A YOLOv8 network pre-trained on the COCO dataset was used as the initial model. The initial model was then trained using a training dataset for object detection models, employing a transfer learning strategy to obtain the trained object detection model. It should be noted that the labeled data must cover various typical working conditions, including: the state of mining trucks in different locations (in position, moving, leaving); the state of the bucket in different operational stages (digging, lifting, rotating, unloading); and changes in the bucket state under different material types.

[0024] Subsequently, a YOLOv8 network pre-trained on the COCO dataset was used as the initial model. The initial model was then trained using a training dataset for object detection models, employing a transfer learning strategy to obtain a trained object detection model. Training parameter settings: initial learning rate of 0.001, cosine annealing scheduler, batch size of 8-16, and training epochs of 100-200.

[0025] After training, the object detection model in PyTorch format is exported to ONNX format, and then optimized and quantized using inference engines such as TensorRT or RKNN. Quantization strategies can include: FP16 half-precision quantization: reducing model size and inference time while maintaining accuracy; INT8 integer quantization: determining the activation value distribution of each layer through a calibration dataset to achieve maximum acceleration. In practice, different quantization methods can be adopted depending on the deployment device and accuracy requirements. When deploying the trained object detection model on edge devices, a pipelined parallel processing strategy is adopted, overlapping the steps of video decoding, image preprocessing, model inference, and post-processing to ensure real-time performance.

[0026] The idle time detection module in this embodiment is used to obtain the idle start time and idle end time during the excavator's operation based on multi-frame image data output by the target detection model. For details, please refer to... Figure 2 , Figure 2This is a flowchart illustrating the acquisition of the idle start time and idle end time of an excavator according to an embodiment of the present invention. The idle time detection module receives multi-frame image data from the target detection model, which is labeled with the bounding box positions of the mining truck and the bucket, the category confidence level, and the bucket status information. It calculates the movement distance of the center point of the bucket bounding box between adjacent frames to determine whether the bucket has moved. If the movement distance of the center point of the bucket bounding box in N consecutive frames is less than a set movement threshold, the bucket is determined to be stationary, i.e., the excavator is in an idle state. The time corresponding to the current frame image is recorded as the idle start time. The value of N is related to the camera frame rate, and in this embodiment, it can be set to 10~30.

[0027] After determining the idle start time, the movement distance of the center point of the bucket bounding box between subsequent adjacent frames is calculated. If the movement distance of the center point of the bucket bounding box in a certain frame exceeds the set movement threshold, the bucket is determined to start moving again. Then, it is determined whether the time interval between the current time and the idle start time exceeds the set minimum time threshold. If not, the time interval is not considered to be the real idle time, and the movement distance of the center point of the bucket bounding box is calculated again. If so, the current time is recorded as the idle end time.

[0028] The keyframe queue generation module in this embodiment is used to construct a temporal keyframe queue based on multi-frame image data output by the target detection model. The temporal keyframe queue includes frames arranged in temporal order. M The image at the moment of unloading. Specifically, the keyframe queue generation module receives multi-frame image data from the target detection model, which is labeled with the bounding box positions, category confidence, and bucket status information of the mining truck and bucket. The multi-frame images are judged sequentially according to the time sequence to determine whether the bucket is above the vehicle in the current image. If not, the image is removed. If so, the current bucket status is recorded as full or empty. Based on the bucket status in the previous and next frames, it is determined whether the bucket in the current frame has switched from full to empty. If so, it is determined as a valid unloading, and the current image is saved to the temporal keyframe queue in sequence. If not, only the current bucket status is recorded and the image is removed.

[0029] In other words, the temporal keyframe queue stores the most recent M Images at each unloading moment, each image containing the location information of the mining truck and bucket output by the object detection model, such as... Figure 3 As shown. It should be noted that the number of images that can be accommodated in the temporal keyframe queue is greater than the number of buckets loaded in a single loading cycle. For example, if a mining truck needs the capacity of 10 buckets to fill one load, then the number of images that can be accommodated in the temporal keyframe queue should be greater than 10, and can be set to 12.

[0030] Furthermore, the trained temporal classification model is used to obtain loading efficiency indicators based on the temporal keyframe queue. These indicators include the start and end times of a single loading cycle and the number of loading buckets in a single loading cycle. The temporal classification model in this embodiment includes a feature extraction module, a Transformer temporal encoder, and a fully connected layer classifier. The feature extraction module extracts features from each keyframe image in the temporal keyframe queue to obtain a corresponding visual feature vector. In this embodiment, the feature extraction module is a lightweight CNN network. The Transformer temporal encoder receives the visual feature vector, models inter-frame dependencies using a self-attention mechanism, and outputs a fixed-dimensional sequence representation vector containing the spatiotemporal information of the entire unloading sequence. The fully connected layer classifier obtains the loading efficiency indicators, including the start and end times of a single loading cycle and the number of loading buckets in a single loading cycle, based on the features output by the Transformer temporal encoder. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating the recognition results of loading start and end times and the number of loading buckets provided in an embodiment of the present invention. The time-series keyframe queue input to the trained time-series classification model includes 12 keyframe images. By recognition, the loading start time and loading end time within the current loading cycle (i.e., the cycle from the start to the end of loading for a mining truck) and the number of loading buckets in a single loading cycle can be obtained. Figure 5 The number of buckets shown is 10.

[0031] Before applying the model, it is necessary to train the time-series classification model. The specific training process includes: A large number of complete loading cycle sequences were extracted from actual operation videos. The start and end times and the number of loading bins for each cycle were manually labeled. A sliding window strategy was used to enhance the training data, thus forming the training dataset for the time-series classification model. Subsequently, the time-series classification model was trained using this training dataset. The total loss function consisted of a weighted combination of cross-entropy loss (BCE) and conditional random field loss (CRF) to improve the generalization performance of the time-series classification model. The expression for the total loss function is: , in, Represents cross-entropy loss, This represents the loss in a conditional random field. Indicates the first t The weight coefficients in the next iteration are adjusted according to the number of training iterations. Gradually decay. In the early stages of training, BCE loss is the primary method, while CRF loss is gradually increased in the later stages. The expression is: , in, and These represent the initial weights and the final weights, respectively. This represents the total number of training iterations. In the early stages of training, Larger, the model mainly depends on Develop the ability to distinguish keyframe categories; as training progresses, Gradually decrease, The weights are increased accordingly, thereby guiding the model to learn the reasonable transition rules between states.

[0032] After training, the PyTorch-formatted temporal classification model is exported to ONNX format and quantized to ensure inference efficiency on edge computing devices. Frame caching and batch inference strategies are employed to optimize performance, taking into account the sequence processing characteristics of the temporal classification model.

[0033] The data recording and analysis module in this embodiment is used to analyze loading efficiency by utilizing the start and end times of a single loading cycle, the number of loading buckets in a single loading cycle, the start and end times of idle time, and the idle duration, to obtain the total number of loading times within a preset working time and generate loading efficiency analysis data.

[0034] Specifically, assuming that the working time and number of mining trucks in a shift are fixed within a day, the total number of loading times within the preset working time can be obtained by using the start and end times of a single loading cycle, the number of loading buckets in a single loading cycle, the start and end times of idle time, and the idle duration. Thus, the loading efficiency within the preset working time can be analyzed.

[0035] In this embodiment, the intelligent analysis system for excavator loading efficiency based on visual detection and time-series modeling further includes: an upload module and a display terminal interface. The upload module is connected to the data recording and analysis module and the dispatch center, and is used to transmit loading times, start and end times of a single loading cycle, number of buckets loaded in a single loading cycle, idle start and end times and idle duration, and loading efficiency analysis data to the dispatch center in real time. The display terminal interface is connected to the data recording and analysis module, and is used to generate audible and visual or status signals when a loading completion event or a system status change (a change in the excavator's working status) is detected, and send the audible and visual or status signals to the cab display to alert the operator.

[0036] Specifically, the data recording and analysis module saves data entries for each complete loading cycle, including loading sequence number, loading start time, loading end time, number of loading buckets, and idle time periods. This data can be transmitted in real-time to the dispatch center or cloud platform, or ultimately exported to generate efficiency analysis reports. For example, managers can obtain information from the provided efficiency analysis report such as: how many loading operations a particular excavator completed in a shift, the average loading time per truck, the distribution of loading buckets, and the percentage of idle time for the equipment. Based on this data analysis, construction units can better assess the work efficiency of excavators and drivers, optimize vehicle scheduling and loading coordination, and improve overall production efficiency.

[0037] In this embodiment, the edge computing device is an industrial computer, an embedded AI acceleration terminal, or an integrated hardware and software device. The edge computing device includes an artificial intelligence hardware acceleration unit, which is a graphics processing unit (GPU) or a neural network processing unit (NPU). GPUs and NPUs can accelerate the inference computation of object detection models and temporal classification models to ensure real-time processing and analysis of real-time video streams.

[0038] This invention provides an intelligent excavator loading efficiency analysis system based on visual detection and temporal modeling. First, a target detection model identifies the positions of the mining truck and the bucket. When the bucket is detected above the mining truck and its state changes from full to empty, loading for that bucket is considered complete, and an image of this moment is acquired and saved to a temporal keyframe queue. Then, multiple temporal images are used as input to a temporal classification model, which identifies the number of loading attempts, loading start and end times, and the number of buckets loaded. Finally, the target detection model is used to determine whether the excavator is working based on whether the bucket position has moved, completing the excavator idle time statistics and obtaining the excavator idle start and end times and idle duration. A data recording and analysis module is used to save the above data to generate loading efficiency analysis data. In this way, based on real-time video streams captured by cameras at the excavator's work site, intelligent monitoring of the excavator's loading status and work efficiency is achieved using target detection and time-series analysis. Loading counts and efficiency statistics can be completed without the need for additional hardware such as weight sensors and position sensors, significantly reducing installation complexity and system costs. Through an AI algorithm that integrates target detection and time-series classification, key action events (such as unloading) can be accurately identified and the continuous working process analyzed, reducing false positives and false negatives, resulting in higher accuracy and reliability of detection. The AI ​​efficiency recognition algorithm can comprehensively and automatically calculate key efficiency data such as the number of buckets loaded, loading time, work cycle, and idle time. The entire system runs in real-time on the excavator's local edge computing device, with low network dependence, and can promptly acquire data and provide feedback to the dispatch center or on-site operators. This invention does not require complex 3D modeling or large amounts of labeled data; the model can be deployed on edge devices, providing real-time and accurate data support for excavator efficiency optimization.

[0039] In summary, this invention uses a purely visual approach to acquire loading efficiency data, offering advantages such as convenient installation and strong adaptability. The camera can be installed at the excavator's optimal viewing angle as needed, and is equipped with dustproof, vibration-proof, and night vision devices to adapt to open-air environments. The edge computing device uses industrial-grade hardware, enabling stable operation in high-temperature, low-temperature, dusty, and vibration-prone environments. Since all intelligent analysis is performed locally, the excavator loading efficiency intelligent analysis system based on visual detection and temporal modeling minimizes its dependence on communication networks, thus enabling it to function even in construction environments with poor network connectivity. Furthermore, the software modules of this invention are scalable. For example, the target detection model and temporal classification model can be updated to adapt to different operating equipment and environmental conditions. Therefore, the excavator loading efficiency intelligent analysis system based on visual detection and temporal modeling provided by this invention can be flexibly applied to various excavator loading scenarios, providing an intelligent and efficient auxiliary tool for engineering construction and mining operations; it can automatically complete the statistical analysis of key loading efficiency indicators, offering significant advantages in improving operational safety and efficiency.

[0040] In the several embodiments provided by this invention, it should be understood that the apparatus and methods disclosed in this invention can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0041] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0042] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling, characterized in that, Including cameras and edge computing devices, among which, The camera is used to collect real-time video streams from the excavator's work site and send the real-time video streams to the edge computing device; The edge computing device includes a trained target detection model, an idle time detection module, a keyframe queue generation module, a trained temporal classification model, and a data recording and analysis module, wherein... The trained target detection model is used to obtain multi-frame image data, labeled with the bounding box positions of the mining truck and bucket, and the bucket status information, based on the real-time video stream from the excavator's operating site. The idle time detection module is used to obtain the idle start time and idle end time during the excavator's operation based on the multi-frame image data output by the target detection model. The keyframe queue generation module is used to construct a temporal keyframe queue based on the multi-frame image data output by the target detection model, the temporal keyframe queue including frames arranged in temporal order. M The image at the moment of unloading; the trained temporal classification model is used to obtain loading efficiency indicators based on the temporal keyframe queue, the loading efficiency indicators including the start and end time of a single loading cycle and the number of loading buckets; the data recording and analysis module is used to perform loading efficiency analysis using the start and end time of a single loading cycle, the number of loading buckets, the start and end time of idle time, and the idle duration, and generate loading efficiency analysis data.

2. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling as described in claim 1, characterized in that, The object detection model uses a YOLOv8 network, and the training process of the object detection model includes: Collect a large amount of video data from excavator operation sites, including video data under different lighting conditions, weather conditions, mining truck types and operation scenarios. Extract the video data into multi-frame images, and label the bounding boxes of the mining truck and bucket and the state of the bucket in each of the multi-frame images to form a training dataset for the target detection model. Using a YOLOv8 network pre-trained on the COCO dataset as the initial model, the initial model is trained using the object detection model training dataset. During the training process, a transfer learning strategy is adopted to obtain a trained object detection model.

3. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to claim 1, characterized in that, The idle time detection module is specifically used for: Receive multi-frame image data from the target detection model, labeled with the bounding box positions of the mining truck and bucket, and bucket status information. Calculate the movement distance of the bucket bounding box center point between adjacent frames to determine if the bucket has moved. If continuous... N If the center point of the bucket bounding box in the frame image moves less than the set movement threshold, the bucket is determined to be stationary, that is, the excavator is in an idle state, and the time corresponding to the current frame image is recorded as the idle start time. After determining the idle start time, the movement distance of the center point of the bucket bounding box between subsequent adjacent frames is calculated. If the movement distance of the center point of the bucket bounding box in a certain frame exceeds the set movement threshold, it is determined whether the time interval between the current time and the idle start time exceeds the set minimum time threshold. If not, the movement distance of the center point of the bucket bounding box is calculated. If so, the current time is recorded as the idle end time.

4. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to claim 1, characterized in that, The keyframe queue generation module receives multi-frame image data from the target detection model, which is labeled with the bounding box positions of the mining truck and the bucket, as well as the bucket status information. The module then sequentially judges the multi-frame images according to the time sequence, determining whether the bucket is above the vehicle in the current image. If not, the current image is removed. If so, the bucket status of the current image is recorded, and the module determines whether the bucket in the current frame has switched from a full bucket to an empty bucket based on the bucket status in the preceding and following frames. If so, the current image is saved sequentially to the time-series keyframe queue. If not, only the current bucket status is recorded.

5. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to claim 4, characterized in that, The number of images that can be accommodated in the time-series keyframe queue is greater than the number of truckloads in a single loading cycle.

6. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to claim 1, characterized in that, The temporal classification model includes a feature extraction module, a Transformer temporal encoder, and a fully connected layer classifier, wherein... The feature extraction module is used to extract features from each keyframe image in the temporal keyframe queue to obtain the corresponding visual feature vector. The Transformer temporal encoder is used to receive the visual feature vector, model the inter-frame dependency using a self-attention mechanism, and output a fixed-dimensional sequence representation vector, which contains the spatiotemporal information of the entire unloading sequence. The fully connected layer classifier is used to obtain the loading start time, loading end time, and number of loading buckets in a single loading cycle based on the features output by the Transformer time encoder.

7. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to claim 1, characterized in that, The training process of the time-series classification model includes: Collect a large amount of video data from excavator operation sites and extract multiple complete loading cycle sequences; Manually mark the start time, end time, and number of buckets loaded for each loading cycle; Based on the labeled loading cycle sequence, a sliding window strategy is used to augment the data to increase the amount of data and form a training dataset for the time series classification model. The time-series classification model is trained using the training dataset, with the loss function being a weighted combination of cross-entropy loss and conditional random field loss.

8. The intelligent analysis system for excavator loading efficiency based on visual detection and temporal modeling according to any one of claims 1 to 7, characterized in that, It also includes an upload module and a display terminal interface, among which, The upload module is connected to the data recording and analysis module and the dispatch center, and is used to transmit loading times, start and end times of a single loading cycle, number of loading buckets in a single loading cycle, idle start and end times and idle duration, and loading efficiency analysis data to the dispatch center in real time. The display terminal interface is connected to the data recording and analysis module and is used to generate an audio-visual or status signal when a loading completion event or a system status change is detected, and to send the audio-visual or status signal to the cab display.

Images