Method and device for monitoring excavation efficiency of rockfill material

Abstract

The invention provides a rockfill material excavation efficiency monitoring method and device. The method can significantly improve the recognition precision of the interaction action of the excavator and the truck in a complex shielding scene, and improves the judgment accuracy of the working state of the equipment, thereby effectively calculating the utilization rate of the construction equipment and the collaborative operation efficiency, and optimizing the earthwork scheduling management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water conservancy construction monitoring, and in particular to a rockfill excavation efficiency monitoring method and device. BACKGROUND

[0002] In the field of earthwork engineering, rockfill excavation, as a key link in water conservancy construction, directly affects the overall project progress and resource scheduling. With the development of intelligent construction technology, construction monitoring systems based on deep vision have gradually become an important means to improve the efficiency of earthwork operations. In related technologies, a dynamic monitoring system for earthmoving equipment is constructed through the coordinated operation of target detection, multi-target tracking and action recognition. Specifically, this system covers the whole process from equipment state perception to operation cycle modeling, including key links such as excavator action recognition, truck trajectory tracking, and equipment coordination efficiency calculation. Among them, advanced algorithms such as YOLOv8 and TimeSformer are widely used in target detection and behavior recognition, while tracking algorithms such as DeepSORT are used to maintain the continuity of the association of equipment in video sequences.

[0003] However, in existing visual monitoring methods, directly using the traditional YOLOv8 model for target detection often fails to allocate sufficient weight to discriminative regions, resulting in increased recognition errors in occlusion scenarios. Meanwhile, TimeSformer may ignore key local information due to overly simple convolution operations during image feature processing, affecting the accuracy of action recognition. In addition, multi-target tracking algorithms are prone to trajectory breaks or mismatching problems in dense equipment and frequent occlusion construction environments, leading to equipment state determination bias. Specifically, occlusion between trucks, interactive occlusion between excavators and trucks, and interference from other objects can all cause target recognition failure or unstable tracking, ultimately affecting the calculation accuracy of equipment utilization and coordination efficiency, limiting the practical application effect of the system in complex construction scenarios. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, the first object of the present application is to propose a rockfill excavation efficiency monitoring method based on deep vision multi-mechanical interactive action recognition.

[0006] The second object of the present application is to propose a rockfill excavation efficiency monitoring device based on deep vision multi-mechanical interactive action recognition.

[0007] The third object of the present application is to propose an electronic device.

[0008] The fourth object of the present application is to propose a computer-readable storage medium.

[0009] To achieve the above purpose, the first aspect of the present application provides a rock excavation efficiency monitoring method based on deep vision multi-machine interactive action recognition, comprising: S1, target detection is performed on excavators and trucks in the construction site by an improved YOLOv8 model integrated with a lightweight depth separable convolution module to enhance local discriminative feature extraction and improve detection stability in occlusion scenarios; S2, based on the detection results of the improved YOLOv8, a weighted matching strategy that fuses appearance features and kinematic distances is adopted, and the DeepSORT algorithm is used to realize continuous trajectory tracking of multiple trucks to maintain target correlation in occlusion states; S3, a TimeSformer model with a multi-scale spatial attention mechanism is used to recognize the actions of the excavator, and differential convolution kernels are used to extract local details and global context features, and action timing logic modeling is combined to distinguish different interactive actions of the excavator; S4, according to the tracking trajectories of the excavator and the truck and the action recognition results, a dynamic state judgment rule base is constructed, and based on the motion state, relative distance and action cycle logic of the equipment, the working state and non-working state of the equipment are dynamically judged.

[0010] In an embodiment of the present application, the S1 comprises: S11, a combination structure of pointwise convolution and depth convolution is adopted, wherein the depth convolution is used to generate feature maps corresponding to the number of convolution kernels to capture multi-scale local features, and the pointwise convolution is used to assign different weights according to the importance of each feature map to retain key local features of the truck; S12, the calculation complexity is reduced by independently setting the convolution kernel for each channel, and the convolution kernel weight is dynamically adjusted during the detection process to adapt to the feature extraction requirements under different occlusion degrees.

[0011] In an embodiment of the present application, the S3 comprises: S31, a 3x3 small convolution kernel is used to extract local detail features such as bucket posture to enhance the recognition ability of the excavator's fine actions; S32, a 7x7 large convolution kernel is used to model global context features such as swing range, and a timing attention mechanism is combined to model continuous action segments to recognize composite action sequences.

[0012] In an embodiment of the present application, it further comprises: S5, according to the tracking trajectories of the excavator and the truck and the action recognition results, the cooperative work cycle between the equipment is calculated, wherein the cooperative work cycle is defined as the total time for the excavator to complete one loading-unloading action and the truck to be in the effective loading range, which is used to evaluate the cooperative efficiency of the equipment.

[0013] To achieve the above object, the second aspect of the present application proposes a rock excavation efficiency monitoring device based on deep vision multi-machine interactive action recognition, comprising: A target detection module is configured to detect the excavator and truck in the construction site through an improved YOLOv8 model integrated with a lightweight depth separable convolution module to enhance local discriminative feature extraction and improve detection stability in occlusion scenarios. A trajectory tracking module is configured to track the continuous trajectories of multiple trucks based on the detection results of the improved YOLOv8, adopt a weighted matching strategy that fuses appearance features and kinematic distances, and combine the DeepSORT algorithm to maintain target correlation in occlusion scenarios. An action recognition module is configured to recognize the actions of the excavator using a TimeSformer model with a multi-scale spatial attention mechanism, extract local details and global context features through differentiated convolution kernels, and combine action timing logic modeling to distinguish different interactive actions of the excavator. A state determination module is configured to construct a dynamic state determination rule library based on the tracking trajectories of the excavator and truck and the action recognition results, dynamically determine the working state and non-working state of the equipment based on the motion state, relative distance, and action cycle logic of the equipment.

[0014] To achieve the above object, the third aspect of the present application proposes an electronic device, comprising: A processor; A memory storing executable instructions; the processor executes the instructions to implement the above method.

[0015] To achieve the above object, the fourth aspect of the present application proposes a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above method.

[0016] The method, device, electronic device, and computer-readable storage medium of the present application significantly improve the recognition accuracy of the interactive actions of the excavator and truck in complex occlusion scenarios, dynamically determine the working state of the equipment, and accurately calculate the efficiency of collaborative work, thereby optimizing the scheduling of earthwork construction and resource utilization.

[0017] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which: Figure 1A flowchart illustrating a method for monitoring the efficiency of rockfill excavation based on depth vision multi-mechanical interaction action recognition, provided for an embodiment of this application; Figure 2 A schematic diagram of intelligent unmanned compactor group scheduling based on deep reinforcement learning algorithm provided in an embodiment of this application; Figure 3 An example diagram illustrating the determination of the working status of trucks and excavators based on a partial occlusion consideration framework provided in this application embodiment; Figure 4 This is a schematic diagram illustrating the action state recognition of an excavator within a work cycle, provided in an embodiment of this application. Figure 5 This is a structural diagram of a rockfill excavation efficiency monitoring device based on depth vision multi-mechanical interaction action recognition, provided in an embodiment of this application. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] The following description, with reference to the accompanying drawings, describes a method, apparatus, electronic device, and computer-readable storage medium for monitoring the efficiency of rockfill excavation based on depth vision multi-mechanical interaction action recognition, according to embodiments of the present invention.

[0022] Example 1 Figure 1 This is a flowchart of a method for monitoring the efficiency of rockfill excavation based on depth vision and multi-mechanical interaction action recognition according to an embodiment of the present invention, as follows: Figure 1 As shown, it includes: S1 uses an improved YOLOv8 model that integrates a lightweight, depthwise separable convolution module to perform target detection on excavators and trucks at construction sites, thereby enhancing local discriminative feature extraction and improving detection stability in occluded scenarios.

[0023] Specifically, the present step uses an improved YOLOv8 model integrated with a light-weight depth separable convolution module (LightConv) to detect excavators and trucks in the construction site, aiming to enhance the extraction ability of local discriminative features and improve the detection stability in occlusion scenarios. YOLOv8 is a single-stage object detection model that achieves a good balance between speed and accuracy, but its sensitivity to local features in complex occlusion scenarios is insufficient, which can lead to target recognition failure or false detection. Therefore, the present invention embeds a LightConv module in the backbone network of YOLOv8, which is composed of point-wise convolution and depth-wise convolution. Through independent processing of each channel, the module significantly reduces the computational complexity while preserving key local features.

[0024] In specific implementation, the LightConv module replaces part of the standard convolution layers in the feature extraction stage of YOLOv8, and its input is a multi-channel feature map Xi,j,c, where c represents the number of channels, and i,j are the spatial coordinates of the feature map. The depth-wise convolution operation uses a k×k size convolution kernel (such as 3×3 or 5×5), and each channel is independently convolved, avoiding information confusion between channels and enhancing the extraction ability of local prominent features such as truck outlines, tires, and cabs. The subsequent point-wise convolution then weights and fuses each channel feature through a 1×1 convolution kernel, strengthening the expression of discriminative regions. The module optimizes the weights Wi,j,c of each channel through backpropagation during training, so that the model can maintain high recall rate and detection accuracy when facing partial occlusion (such as trucks being occluded by other equipment or soil piles).

[0025] The YOLOv8 model uses an input image size of 640×640 pixels, a batch size of 16, 120 training rounds, an AdamW optimizer, an initial learning rate of 0.001, and a cosine annealing strategy for dynamic adjustment. The loss function combines CIoU loss and Focal loss to improve the boundary box regression accuracy and the ability to handle class imbalance problems. Through the above improvements, the mAP (mean average precision) of YOLOv8 in complex construction scenarios can be improved to more than 0.82, especially in scenarios with an occlusion rate of more than 30%, the detection success rate is improved by more than 15% compared to the original model.

[0026] This step serves as the foundation module of the entire system, directly affecting the subsequent multi-target tracking and action recognition effect. In practical applications, construction site cameras are usually deployed at high altitudes or fixed positions, resulting in frequent occlusion and deformation of equipment at different angles. Through the LightConv-enhanced YOLOv8 model, the key parts of the truck and excavator can be stably identified, providing high-quality detection input for the DeepSORT tracking algorithm, thereby improving the accuracy of trajectory continuity and equipment state determination, laying a solid foundation for real-time monitoring of rock excavation efficiency.

[0027] Further, S1 comprises: S11, adopting a combination structure of pointwise convolution and depthwise convolution, wherein the depthwise convolution is used to generate feature maps corresponding to the number of convolution kernels to capture multi-scale local features, and the pointwise convolution is used to assign different weights according to the importance of each feature map to retain key local features of the truck.

[0028] Specifically, this step adopts a combination structure of pointwise convolution (Pointwise Convolution) and depthwise convolution (Depthwise Convolution) to improve the feature extraction capability and recognition robustness of earthmoving equipment (such as trucks) in complex construction scenes. In some implementations, the depthwise convolution first generates feature maps corresponding to the number of convolution kernels through independent channel convolution operations, thereby capturing multi-scale local features. For example, using multiple depthwise convolution kernels of different sizes (such as 3x3, 5x5), the edge profile, hub structure, and body texture of the truck can be extracted to enhance the model's ability to perceive occluded areas. Further, the pointwise convolution is used to linearly combine the channels of these feature maps, assigning different weights according to the discriminative contribution of each feature map in the current task, thereby retaining feature information highly relevant to truck state recognition.

[0029] In terms of parameter settings, the number of channels of the depthwise convolution is consistent with the number of convolution kernels, usually set to 32 or 64 channels, and each channel uses an independent convolution kernel for feature extraction. The convolution kernel size can be configured as 3x3 or 5x5, with a step size of 1 and a padding mode of "same" to maintain the spatial resolution of the feature map. The pointwise convolution uses a 1x1 convolution kernel, with the number of channels consistent with the depthwise convolution output, and the feature maps are weighted and fused through a fully connected manner. During training, a channel attention mechanism (such as SE Block) can be introduced to dynamically adjust the weights of the pointwise convolution, improving the model's sensitivity to key features.

[0030] The structure is mainly used in the application to improve the feature extraction capability of the YOLOv8 target detection model, especially in scenes where trucks are partially occluded or in complex backgrounds. Through multi-scale deep convolution, the model can identify multiple local regions of the truck, while point-by-point convolution ensures that the most discriminative features in these regions are retained and strengthened, thereby improving the accuracy and recall rate of target detection. In practical applications, this module can be deployed in a video monitoring system in a construction site to stably identify and track trucks under multiple angles and lighting conditions, providing high-quality input features for subsequent multi-target tracking and action recognition. The technical implementation of this step effectively solves the problem of feature loss in occluded scenes in traditional convolution structures, providing key support for improving the recognition accuracy and operation efficiency monitoring of earthmoving equipment.

[0031] S12, by independently setting the convolution kernel of each channel, the computational complexity is reduced, and the convolution kernel weight is dynamically adjusted during the detection process to adapt to the feature extraction demand under different occlusion degrees.

[0032] Specifically, this step reduces the computational complexity by independently setting the convolution kernel of each channel, and dynamically adjusts the convolution kernel weight during the detection process to adapt to the feature extraction demand under different occlusion degrees. This technical solution is improved based on the YOLOv8 architecture, and a light depth separable convolution (LightConv) module is used. The core idea is to split the standard convolution operation into point-by-point convolution (Pointwise Convolution) and depth convolution (Depthwise Convolution), thereby effectively reducing the model parameter quantity and computational quantity without significantly reducing the feature expression capability. Specifically, the original multi-channel shared convolution kernel structure is used in YOLOv8, and in the present application, the convolution kernel is independently set for each channel, i.e., in the depth convolution stage, an independent k×k convolution kernel (k is usually 3 or 7) is used for feature extraction for each channel, avoiding feature confusion between channels and improving the expression capability of local discriminative features.

[0033] The convolution kernel size can be dynamically adjusted according to the saliency of the target region, for example, a small 3×3 convolution kernel is used in the tail of the truck and other easily occluded regions to capture detailed features, and a large 7×7 convolution kernel is used in the body and other large-area regions to model global semantics. The dynamic adjustment mechanism of the convolution kernel weight is based on the attention mechanism and feature saliency evaluation. By introducing a channel attention module (such as an SE module or a CBAM), the weights of each channel are adaptively adjusted after the generation of the feature map, so that in the occluded region, the model can automatically enhance the feature extraction capability of the unoccluded part.

[0034] This step has significant value in complex construction scenarios, especially in environments with multiple devices and frequent occlusions, effectively improving the robustness and computational efficiency of target detection. By reducing the computational complexity of the model, GPU resource usage is reduced, improving real-time processing capabilities, while the dynamic weight adjustment mechanism enhances the model's adaptability to occluded targets, providing more stable and accurate input features for subsequent multi-target tracking and action recognition.

[0035] S2, based on the detection results of the improved YOLOv8, a weighted matching strategy is adopted to fuse appearance features and kinematic distance, and a DeepSORT algorithm is used to realize continuous trajectory tracking of multiple trucks to maintain target relevance in the occluded state.

[0036] Specifically, based on the detection results of the improved YOLOv8, a weighted matching strategy is adopted to fuse appearance features and kinematic distance, and a DeepSORT algorithm is used to realize continuous trajectory tracking of multiple trucks to maintain target relevance in the occluded state. In complex construction scenarios, due to the dense equipment and frequent occlusions, traditional tracking methods are prone to target loss or mismatching problems, therefore, the present application improves the tracking robustness by improving the fusion mechanism of YOLOv8 and DeepSORT.

[0037] In terms of technical implementation, YOLOv8 serves as the basic detector, providing bounding boxes and class information for DeepSORT. To further enhance the detection capabilities of YOLOv8 in occluded scenarios, the present application embeds a LightConv module in YOLOv8, which consists of pointwise convolution and depthwise convolution. By using independent channel convolution kernels to extract local discriminative features of trucks, the module effectively preserves key region information, improving detection accuracy and recall rate. The detection results are predicted by Kalman filtering for the next frame of target position, and are matched with appearance features such as ReID feature vectors.

[0038] This step is mainly used for continuous tracking of multiple trucks in earthwork engineering, especially during the interaction between excavators and trucks, trucks may temporarily disappear due to occlusion by other equipment or the environment. By fusing appearance and motion information, DeepSORT can re-match after the target is temporarily lost, maintaining the continuity of the trajectory, and providing reliable data support for subsequent action recognition and productivity calculation.

[0039] The technical effect of this step is to significantly improve the robustness and continuity of multi-target tracking in occlusion scenarios, ensuring that the truck's trajectory is not interrupted in complex construction environments, thereby providing stable basic data for the interaction action recognition, work cycle division, and productivity calculation of the excavator and truck, enhancing the overall monitoring accuracy and scheduling decision-making ability of the system.

[0040] S3, a TimeSformer model with a multi-scale spatial attention mechanism is used to identify the action of the excavator, local details and global context features are extracted through differential convolution kernels, and action timing logic modeling is combined to distinguish different interactive actions of the excavator.

[0041] Specifically, this step involves using a TimeSformer model with a multi-scale spatial attention mechanism (MSSAM) to identify the action of the excavator, the core of which is to extract local details and global context features through differential convolution kernels, and combine action timing logic modeling to distinguish different interactive actions of the excavator. This technical solution has significant practical value in earthwork engineering, especially in complex construction scenarios, and can effectively deal with challenges such as occlusion, lighting changes, and dynamic backgrounds.

[0042] In terms of technical implementation, the TimeSformer model is based on the Transformer architecture, modeling the spatio-temporal dependencies in video sequences through self-attention mechanisms. To enhance its ability to distinguish excavator actions, the invention introduces multi-scale convolution operations in the spatial attention module. Specifically, a 3x3 small convolution kernel is used to extract local detail features (such as bucket posture, boom angle changes, etc.), while a 7x7 large convolution kernel is used to model global context information (such as rotation range, whole machine motion trajectory, etc.). The feature maps output by the two convolution kernels are concatenated and input into the attention mechanism to achieve differential weight allocation for key regions, thereby improving the robustness and accuracy of action recognition.

[0043] The input image resolution of the TimeSformer model is set to 256x256 pixels, which is adjusted to 224x224 pixels to adapt to the model structure. During training, pre-trained weight files are used for transfer learning, with 15 training rounds and a batch size of 32. In the multi-scale spatial attention module, the 3x3 and 7x7 convolution kernels correspond to different receptive fields, with the 3x3 convolution kernel capturing local significant features and the 7x7 convolution kernel modeling more extensive context information. Through dynamic adjustment of attention weights, the model can adaptively focus on key operation areas of the excavator, such as the boom, bucket, and rotation platform.

[0044] This step is mainly used to identify the continuous actions of the excavator during the rock pile excavation process, such as the typical operation cycle of "loading-rotation-unloading". By combining the action timing logic modeling, the model can identify the causal relationship between actions, thereby distinguishing between effective operation actions and invalid states (such as empty movement, equipment failure, etc.). In the multi-machine collaborative operation scene, this method can provide accurate action recognition results for the on-site management system, assist in determining whether the equipment is in working condition, and provide a reliable basis for subsequent productivity calculation.

[0045] The introduction of the multi-scale spatial attention mechanism significantly improves the action recognition accuracy of the TimeSformer model in complex construction environments. Through differential convolution kernel design, the model balances between local details and global semantics, effectively reducing recognition errors caused by occlusion or background interference. In addition, combined with timing modeling, this method can capture the evolution law of excavator actions, accurately divide continuous operation cycles, and provide key support for equipment utilization and collaborative efficiency calculation. This step plays a bridging role in the entire system and is the core link of intelligent monitoring of construction efficiency.

[0046] Further, S3 comprises: S31, a 3x3 small convolution kernel is used to extract local detail features such as bucket posture, to enhance the recognition ability of fine actions of the excavator.

[0047] Specifically, this step uses a 3x3 small convolution kernel to extract local detail features such as bucket posture, aiming to enhance the recognition ability of fine actions of the excavator. In terms of technical implementation, this method is based on the improved TimeSformer architecture, combined with the multi-scale spatial attention mechanism (MSSAM), and through the introduction of convolution kernels of different sizes, it realizes the enhanced extraction of key local features. Among them, the 3x3 small convolution kernel is mainly used to capture the edge, angle, and motion trajectory of key components such as the bucket and the boom, etc. detail information, with high spatial resolution and local perception ability. In the feature extraction process, the small convolution kernel gradually extracts high-order semantic features through layer-by-layer stacking, while preserving the key local structure in the original image, thereby improving the discrimination accuracy of continuous actions such as "loading", "rotation", and "unloading".

[0048] At the parameter level, the number of channels in the 3×3 convolutional kernel is dynamically adjusted according to the depth of the input feature map, typically set to 64~256 channels in the MSSAM module to adapt to feature extraction needs at different scales. The convolution operation uses a combination of ReLU activation and Batch Normalization (BN) to enhance the model's non-linear expressiveness and training stability. Furthermore, the stride of the convolutional layer is set to 1 to ensure the spatial resolution of the feature map and avoid loss of detail information due to downsampling. In the attention mechanism, local features extracted by the small convolutional kernel are fused with global semantic features extracted by the 7×7 large convolutional kernel, and weighted and integrated through a multi-head attention mechanism to enhance the model's robustness to occlusion and complex backgrounds.

[0049] This step is primarily applied to the monitoring of collaborative operations between excavators and trucks in earthmoving projects. At construction sites, due to the dense concentration of equipment and frequent obstructions, traditional methods struggle to accurately identify changes in bucket posture. By extracting local detail features using 3×3 small convolutional kernels, minute bucket movements, such as bucket opening angle and insertion depth, can be effectively identified, thus more accurately determining whether the excavator is in a "loading" or "unloading" state. This method is suitable for video monitoring systems under multi-angle and multi-light conditions, maintaining high accuracy even at night or in rainy or foggy weather.

[0050] This step significantly improves the accuracy and stability of excavator motion recognition. Experiments show that after introducing a 3×3 small convolution kernel, the average accuracy (mAP) of bucket posture recognition is improved by approximately 12%, and the false detection rate is reduced by 8%. Furthermore, this method enhances TimeSformer's ability to perceive local discriminative features, helping to more accurately model the temporal evolution of excavator movements and providing a reliable basis for subsequent equipment status determination and productivity calculation.

[0051] S32 uses a 7×7 large convolutional kernel to model global context features such as rotation range, and combines a temporal attention mechanism to model continuous action segments in order to identify compound action sequences.

[0052] Specifically, this step employs a 7×7 large convolutional kernel to model global contextual features such as the rotation range, and combines a temporal attention mechanism to model continuous action segments to identify composite action sequences such as "loading-rotation-unloading". This technical solution is based on the TimeSformer architecture and aims to improve the accuracy and robustness of excavator action recognition in complex construction scenarios.

[0053] In terms of technical implementation, a 7x7 large convolution kernel is used to extract the global semantic features of the excavator during the execution of the turning action. Since the excavator's turning action involves large-scale changes in mechanical structure, such as the coordinated movement of the boom, stick, and bucket, a convolution operation with a large receptive field is needed to capture the overall posture changes. This convolution kernel is used in parallel with a 3x3 small convolution kernel during the feature extraction stage. The former focuses on global structural information, while the latter focuses on local details (such as bucket posture, hydraulic rod movement, etc.), thereby achieving multi-scale feature fusion. In terms of temporal modeling, TimeSformer introduces a self-attention mechanism in the temporal dimension to model the action segments in consecutive video frames, enabling the model to perceive the evolution of actions, such as the temporal logic from "loading" to "turning" to "unloading".

[0054] In terms of parameter indicators, the size of the convolution kernel in this step is set to 7x7, and the number of channels is adapted according to the dimension of the previous feature map, typically ranging from 64 to 256 channels. The input image resolution is set to 256x256 pixels, which is adjusted to 224x224 pixels to adapt to the input requirements of TimeSformer. During training, a multi-scale spatial attention mechanism (MSSAM) is used to differentially weight features in different regions through a learnable weight matrix, enhancing the recognition ability of key action regions. In addition, the window size of the temporal attention mechanism is set to 16 frames to ensure the continuity modeling of action segments.

[0055] In application scenarios, this step is mainly used to identify the complex action sequences of the excavator in earth excavation operations, such as loading, turning, and unloading. By combining global and local features, the model can effectively address the action recognition difficulties caused by occlusion, lighting changes, and mechanical overlap in the construction site. Especially in the typical operation cycle of "loading-turning-unloading", the temporal modeling can accurately capture the dependency between actions, providing a reliable basis for subsequent equipment utilization calculation.

[0056] The technical effect of this step lies in significantly improving the temporal consistency and spatial accuracy of excavator action recognition. By introducing a 7x7 large convolution kernel and a multi-scale spatial attention mechanism, the model can more comprehensively model the mechanical structure changes and action evolution process, thereby enhancing the recognition ability of complex action sequences. Further, combined with the temporal attention mechanism, it can effectively distinguish between effective working time and non-working time, providing key support for construction efficiency monitoring.

[0057] S4, according to the tracking trajectories of the excavator and the truck and the action recognition results, a dynamic state judgment rule library is constructed, based on the motion state, relative distance, and action cycle logic of the equipment, the working state and non-working state of the equipment are dynamically judged.

[0058] Specifically, this step aims to construct a dynamic state judgment rule base based on the tracking trajectories of excavators and trucks and the action recognition results, to achieve real-time and accurate judgment of the working and non-working states of the equipment. This step is the core logic module in the entire rock pile excavation efficiency monitoring system, and its technical implementation is based on the motion state, relative distance, and action cycle logic of the equipment, which improves the accuracy and robustness of equipment state recognition by fusing spatio-temporal information and behavior semantics.

[0059] In terms of technical implementation, this step first relies on the target detection results based on the improved YOLOv8 and LightConv module in the previous step, and combines the multi-truck tracking trajectory based on the DeepSORT algorithm. At the same time, TimeSformer and multi-scale spatial attention mechanism (MSSAM) are used to identify continuous action sequences of excavators, such as "loading-rotation-unloading", etc. By spatio-temporally aligning the tracking trajectory and the action recognition result, the system can construct a dynamic judgment rule base for the state of the equipment. This rule base not only contains static rules (such as the equipment being in a static state and being outside the effective working range, which is judged as a non-working state), but also supports dynamic rule updating, such as adjusting the judgment threshold according to the historical action pattern.

[0060] The system sets the effective working distance range of the excavator and the truck to be 0.5 to 2 times the length of the excavator arm, which is based on the physical constraints of equipment interaction in actual construction scenes. At the same time, the Hungarian algorithm is used for trajectory matching, with kinematic distance weight coefficient a = 0.7, appearance feature distance weight coefficient b = 0.3, threshold t1 = 0.5, t2 = 0.3, to balance the matching accuracy of motion prediction and appearance similarity. In the action recognition module, TimeSformer uses 16-frame image blocks for time series modeling, combined with 3x3 and 7x7 dual-scale convolution kernels for local detail and global semantic feature extraction, respectively, to improve the context awareness ability of action recognition.

[0061] This step is suitable for scenarios of dense equipment collaborative work in earthwork engineering, especially in complex construction sites with frequent occlusion and multi-device parallel work. By dynamically determining the equipment state, the system can effectively distinguish non-working time caused by waiting, failure, or improper scheduling, thereby providing accurate equipment utilization data for the on-site scheduling system.

[0062] By fusing multi-source information and dynamic rule base, the accuracy and real-time performance of equipment state judgment are significantly improved, providing reliable data support for subsequent productivity calculation and scheduling optimization. Compared with traditional methods based on fixed rules or single sensors, this scheme has stronger environmental adaptability and intelligence level, which helps to realize fine management of the construction process.

[0063] The multi-scale space attention mechanism-based multi-mechanical interaction action recognition method of the embodiment of the application improves the accuracy and robustness of excavator and truck interaction action recognition in complex construction scenes, effectively solves the tracking and state determination problems caused by equipment occlusion, and improves the operation efficiency calculation precision of earthmoving equipment.

[0064] Further, it also includes: S5, calculating the cooperative operation period between the equipment according to the tracking trajectories and action recognition results of the excavator and the truck, wherein the cooperative operation period is defined as the total time during which the excavator completes one loading-unloading action and the truck is in the effective loading range, and is used for evaluating the cooperative efficiency of the equipment.

[0065] Specifically, this step aims to calculate the cooperative operation period between the excavator and the truck according to the tracking trajectories and action recognition results of the excavator and the truck, so as to evaluate the cooperative efficiency of the equipment. The cooperative operation period is defined as the total time during which the truck is in the effective loading range during the cycle of the excavator completing one “loading-unloading” action. This index can reflect the tightness of the cooperation between the equipment, and is a key parameter for measuring the overall efficiency of the earthmoving system.

[0066] In terms of technical implementation, this step first relies on the target detection results based on the improved YOLOv8 and LightConv module in the previous step, and the action recognition output combining TimeSformer and multi-scale space attention mechanism (MSSAM). The continuous trajectory data of the excavator and the truck is obtained through a multi-target tracking algorithm (such as the improved DeepSORT), including its position coordinates in the video frame, timestamp and action state label. The “loading-unloading” action recognition of the excavator is based on the timing logic of its bucket posture change, rotation angle and unloading action, while the effective loading range of the truck is dynamically calculated according to the rotation radius of the excavator's working arm, usually set as a conical region of 0.5 to 2 times the arm length.

[0067] In terms of parameter index, the calculation of the cooperative operation period needs to set an action recognition confidence threshold (such as ≥0.85) to ensure the accuracy of the action label. At the same time, the spatial distance between the truck and the excavator needs to be calculated in real time through the Euclidean distance formula, and if the distance is less than a set threshold (such as ≤3 meters), it is determined that the truck is in the effective loading range. The time window is usually set to 10 seconds for detecting whether a complete “loading-unloading” action cycle is formed.

[0068] In application scenarios, this step is suitable for earthwork engineering scenes such as rock excavation, especially in complex construction sites with multi-device cooperative operation and frequent occlusion. By quantifying the cooperative operation period, the system can identify bottlenecks in equipment cooperation, such as long waiting time of the truck or frequent idling of the excavator, thereby providing data support for dispatch optimization.

[0069] Technically, this step realizes the objective quantitative evaluation of the cooperative efficiency of the equipment, avoids the subjectivity and hysteresis of traditional manual statistics. By combining the space-time trajectory and action recognition results, the system can accurately identify the effective work period, exclude invalid time caused by occlusion or misidentification, significantly improve the accuracy and real-time performance of productivity analysis, and provide reliable basis for intelligent construction management.

[0070] The multi-machine interactive action recognition method based on the multi-scale space attention mechanism of the embodiment of the application further quantifies the cooperation efficiency of the equipment in the "loading-unloading" process by calculating the cooperative work cycle between the excavator and the truck, realizes the fine evaluation of the whole process of the rock pile excavation work, and optimizes the construction scheduling and improves the comprehensive operation efficiency of the earthwork transportation system.

[0071] Embodiment 2 The purpose of the application is to solve the problem of inaccurate recognition of excavator and truck interactive action and difficult calculation of work efficiency caused by equipment occlusion in complex construction scenes. The traditional method relies on manual statistics or sensor monitoring, which has the limitations of low efficiency, high cost and weak anti-occlusion ability, especially in dense occlusion environment, which easily leads to misjudgment of equipment state and affects scheduling decision. Therefore, deep learning technology is introduced, and a solution combining improved DeepSORT and TimeSformer is proposed: the continuous tracking ability of DeepSORT to occluded trucks is enhanced by integrating a lightweight convolution module (LCYL), and the space-time feature extraction of TimeSformer to continuous actions of excavators is optimized by combining a multi-scale space attention mechanism (MSSAM). At the same time, a device activity perception module is designed, which dynamically determines the working and non-working time of the equipment based on the target motion state and relative distance, and constructs a cooperative efficiency calculation model to provide data support for optimizing scheduling and reducing equipment idle.

[0072] To achieve the above object, the application proposes a method for monitoring the efficiency of rock excavation based on deep vision multi-mechanical interaction action recognition. First, the target detection model is improved, the local feature extraction is strengthened by integrating a lightweight deep separable convolution module, the detection stability of the occluded target is improved by combining multi-scale feature fusion, and the continuous trajectory association of the equipment is realized by using a detection-tracking joint framework. Second, the spatio-temporal behavior recognition model is optimized, a multi-scale spatial attention mechanism is designed, local details and global context features are extracted by using differentiated convolution kernels, and the action evolution law is modeled by using a phased spatio-temporal attention mechanism. Then, a dynamic state judgment rule base is constructed, the working and non-working states are distinguished based on the equipment motion trajectory and relative position, and the equipment utilization rate is quantified by defining a collaborative work cycle model. Finally, the detection, tracking and behavior recognition modules are integrated to form a cascade pipeline, the video stream is analyzed in real time, and the equipment state, collaborative efficiency indicators and visual reports are output to assist the construction scheduling optimization. The scheme can significantly improve the monitoring accuracy in the dense occlusion scene by enhancing the robustness of feature extraction, fine spatio-temporal modeling and dynamic rule judgment, and provide efficient technical support for intelligent construction management.

[0073] The embodiment of the application first data acquisition and processing; track the truck based on the improved DeepSORT tracking algorithm, and identify the action of the excavator by combining the TimeSformer algorithm. The LightConv module is introduced to capture the local discriminative features of the truck, and a multi-scale spatial attention module is designed to assign different weights to different regions of the excavator to distinguish its action. The action perception module of the earthmoving equipment is used to determine whether the truck and the excavator are in a working state according to their motion state. The transportation productivity is calculated by integrating the tracking results of the truck and the action recognition results of the excavator.

[0074] In an embodiment of the application, data acquisition and processing mainly include truck dataset preparation, including video file processing and dataset image labeling. The truck dataset preparation includes data acquisition, data labeling, data enhancement, quality verification and other steps to ensure that the collected earthwork construction site monitoring videos cover different lighting conditions (day / night), weather (sunny / rainy) and multi-angle views. In the quality verification part, the consistency of the labeling is checked by cross-validation, a pre-trained model is used for preliminary reasoning, and the false positive / missed frames are manually reviewed. For the action recognition algorithm, first, a video record of the earthwork construction site monitoring is collected. Then, by extracting a video every few seconds, the records are divided into several short video segments. The excavator detection dataset is created by extracting the excavator target from the video frames. Each folder containing extracted excavator images is named in Arabic mathematical increasing order and assigned with corresponding action labels. The truck in the video is labeled frame by frame using the data labeling software DarkLabel, and the labeled file can be used as a reference for evaluating the accuracy of the tracking results.

[0075] In an embodiment of the present application, training the YOLOv8 target detection model mainly includes two steps: pre-training weight file preparation, training set test set division, and training parameter setting.

[0076] After the above preparation work is completed, the obtained video file, extracted frame file, label file and weight file are input into the corresponding algorithm for calculation in the training stage.

[0077] It can be understood that in earthwork engineering, the interaction action recognition of excavators and trucks is crucial for computing efficiency and optimizing scheduling. Traditional methods rely on manual statistics or sensor-based positioning technology, but have defects such as high cost, complex maintenance, and unstable signal transmission. Although existing computer vision methods can partially replace sensors, in complex construction sites, the equipment is dense, the target is frequently blocked, and the target changes dynamically, which leads to insufficient detection and tracking accuracy. For example, mutual occlusion between trucks and excavators can cause target loss or misjudgment, thereby affecting the accuracy of equipment utilization analysis. In addition, existing action recognition algorithms have limited ability to extract local features, making it difficult to adapt to the multi-scale spatial variation requirements of construction scenes. Therefore, there is an urgent need for a productivity analysis system that can effectively deal with occlusion and accurately recognize multi-device interaction actions to improve construction efficiency and optimize resource scheduling.

[0078] To solve the above problems, the present application proposes a method for monitoring the efficiency of rock excavation based on deep vision multi-machine interaction action recognition. The core of the method achieves efficient collaborative work monitoring through three technical breakthroughs: first, a lightweight convolution module is embedded in YOLOv8, which enhances the extraction ability of local discriminative features of trucks by combining point-wise convolution and deep convolution, and solves the problem of track loss in occlusion scenarios by improving the multi-target tracking algorithm, significantly improving the robustness of target association; second, a multi-scale spatial attention mechanism is introduced into TimeSformer, which captures the spatio-temporal correlation features of excavator actions through a differentiated weight allocation strategy, and uses double-scale convolution kernels to focus on local details and global context information respectively, effectively identifying the temporal logic of continuous actions such as rotation and unloading; further, a soil working equipment collaborative perception module is designed, which dynamically determines the working state of the equipment based on the motion state and relative distance range of the excavator and truck, and combines the tracking trajectory and recognition result to construct a closed-loop analysis model, accurately calculating the single-machine efficiency and collaborative work efficiency. The system analyzes the video stream of the construction site in real time, outputs the equipment utilization rate and scheduling suggestions, optimizes vehicle resource allocation, reduces empty load and waiting time, and thus improves the overall construction efficiency of earthwork engineering. The specific process is shown in the accompanying Figure 2 The implementation is as follows: S101, Data Acquisition and Processing: This stage acquires raw video data through monitoring equipment deployed at the construction site, covering typical interaction scenarios between excavators and trucks (such as loading, unloading, and movement). The video data is processed frame by frame to generate single-frame images, and each frame is labeled with the equipment location, category (excavator / truck), and action tag (such as "loading in progress" or "unloading completed"). For complex working conditions such as occlusion and dynamic blur, data augmentation strategies such as random occlusion and brightness perturbation are adopted to improve the generalization ability of subsequent models.

[0079] S1011) Video capture. Install multi-angle high-definition cameras in the earthwork operation area to continuously record the interaction process between excavators and trucks, ensuring coverage under different lighting, weather, and obstruction conditions.

[0080] S1012) Frame Segmentation and Annotation. The video is segmented into single-frame images, and annotation tools are used to annotate the bounding box coordinates, category, and action state of the device in each frame, forming a structured dataset.

[0081] S1013) Dataset partitioning. Divide the dataset into training, validation, and test sets proportionally to ensure a balanced distribution of category and action labels.

[0082] S102, based on the improved DeepSORT algorithm, achieves continuous multi-truck tracking.

[0083] S1021) Tracking module integration. The DeepSORT tracking algorithm is introduced based on the improved YOLOv8. This algorithm adopts a detection-driven mode, and the target detection quality directly affects the overall tracking performance. First, new targets are identified through the detection algorithm, and the appearance features of the target are calculated based on the position and size of the detection box. At the same time, the possible position in the next frame is predicted. As shown in formula (1), the kinematic distance and appearance feature distance are weighted and calculated, and then matched with the newly detected target in the next frame. Finally, the same target in different frames is associated through the Hungarian algorithm and cascaded matching to form a trajectory. Unassociated targets are included in the unconfirmed set to ensure the accuracy of target trajectory generation.

[0084] (1) D is the weighted distance; D 1 represents the kinematic distance; D 2 represents the appearance feature distance; a is the weighting coefficient of the kinematic distance; b is the weighting coefficient of the appearance feature distance. t 1, t 2 represents the threshold values ​​for kinematic distance and appearance feature distance, respectively.

[0085] S1022) Detection model optimization. In the original YOLOv8 architecture, a LightConv module is introduced. By point-wise convolution and depth convolution operations, the local salient features of the truck are fully extracted, and the detection accuracy and recall rate are ensured, as shown in equations (2) and (3). The number of feature maps generated by depth convolution corresponds to the number of convolution kernels, which determines the dimension of feature capture. Subsequent point-wise convolution assigns different weights according to the importance of each feature map, ensuring that key local features of the truck are preserved, thereby improving the detection success rate. In addition, each channel in the calculation process of this module has independent convolution kernels, rather than a single convolution kernel shared by multiple channels, significantly reducing the computational complexity.

[0086] (2) X i,j,c denotes the element of the input feature map c channel at coordinate (i, j); W i,j,c denotes the element of the input feature map c channel at coordinate (i, j);

[0087] (3) X i,j,c denotes the element of the input feature map c channel at coordinate (i, j); W i,j,c denotes the element of the input feature map c channel at coordinate (i, j);

[0088] S103, earthmoving equipment action perception. Excavators typically perform several basic actions, including scooping material from the ground, rotating the boom, unloading the truck, idling, and working in the yard. Occasionally, the truck may not arrive on time, at which point the excavator will prepare for the next loading cycle, a state referred to as "soil loosening condition." In addition, the excavator may leave the current location to find the next digging point, indicating that the equipment is in a moving state. For the action recognition model, a multi-scale spatial attention module (MSSAM) is designed in the TimeSformer framework, using a 3x3 small convolution kernel to capture detailed features such as the bucket pose, a 7x7 large convolution kernel to model the global semantics such as the rotation range, and a temporal attention mechanism to associate consecutive action segments (such as "loading-rotation-unloading"), achieving high-precision spatiotemporal feature modeling.

[0089] S104, calculation and analysis of the working efficiency of the excavator and the truck. The working state of the excavator needs to be determined according to its action cycle: from the "loading" action, through "rotating the arm → unloading → rotating the arm" to the next "loading", it is a complete working cycle. If multiple rotations or unloading actions occur in a single cycle in the actual working condition (such as transporting soil in batches), they are still counted as effective working time. If the soil is hard and needs to be loosened in advance, the loosening action is also included in the working time. Non-cyclic actions (such as empty moving and equipment failure) are classified as non-working time. For the truck, the moving state (transporting soil or going to the material yard) is directly regarded as the working state; the static state needs to be determined in combination with the spatial position - taking the working arm coverage area of the excavator as the center, defining the effective working range (0.5-2 times the arm length), if the truck is stationary and located within the range, further judgment is made through the multi-truck competition logic: when multiple trucks are simultaneously within the effective range, only the closest vehicle is marked as "being loaded" (because the excavator usually loads only one truck at a time), and the rest are considered as queuing. If the truck is stationary but beyond the working range, it is due to mechanical failure or queue waiting, and is counted as non-working time. When calculating the cooperative working efficiency, the working time of the excavator is based on the action cycle and the loosening operation, and the working time of the truck is based on the moving state time and the loading waiting time within the effective range. Abnormal interruptions (such as failure, out-of-range queuing) are excluded from the statistics to ensure the accuracy of the utilization analysis. The specific calculation steps are as follows: S1041) Calculate the spatial distance between the same truck in different video frames by the following formula.

[0090]

[0091] S1042) According to the results in S1041), preliminarily screen the moving or stationary trucks. For the trucks determined to be in the moving state, accumulate their working time, and the rest are classified as trucks whose working state has not been determined. Figure 3 The truck and excavator working state determination diagram based on local occlusion consideration framework.

[0092] S1043) According to the tracking results of each truck and each excavator, calculate the distance between a truck and an excavator in different video frames.

[0093] S1044) According to the relationship between the results in S1043) and the effective working range, screen the moving or stationary trucks. For the trucks determined to be in the moving state, accumulate their working time, and the rest are classified as trucks whose working state has not been determined.

[0094] S1045) Calculate the working time of the excavator, which is derived from the time between two "loading the bucket from the truck" actions according to the action recognition results of each excavator.

[0095] S1046) Calculate the working time and non-working time of the excavator and truck, respectively.

[0096] S105 Calculate the work productivity based on the perception module.

[0097] According to the actual engineering monitoring video, before action recognition and multi-self-tag tracking, two algorithms need to be trained respectively to obtain the weight file required for predicting unknown actions or self-tag categories and positions. The multi-target tracking algorithm training can be divided into two steps. For the target detection module, a truck data set covering truck images under various complex construction conditions is prepared, with a training set and test set ratio of 4:1, a total of 7500 images. During the training process, the training rounds are set to 120 rounds, the batch size is 16, the image size is 640x640 pixels, and the pre-trained YOLOv8n weight file is directly used.

[0098] To train the action recognition algorithm, the training rounds are set to 15 times, the spatial attention mechanism is used, and the image block size is 16. The image resolution is set to 256x256 pixels, and the adjusted size is 224x224 pixels. After screening, the data set contains a training set, a validation set, and a test set in a ratio of 3:1:1, a total of 10000 images, and a pre-trained weight file is obtained from the official website.

[0099] After training, the weight files of the target detection module and the action recognition module are obtained. Running the above algorithm, according to the obtained accurate observation results, the working time and non-working time of the truck and excavator are calculated and analyzed respectively. Further calculate its work productivity, and make corresponding scheduling recommendations for construction machinery to improve production efficiency. Figure 4 The action state recognition diagram of the excavator in the work cycle.

[0100] After calculating the working time and non-working time of the excavator and truck, respectively, the independent work efficiency of each can be calculated first through formula (4). Then use formula (5) to calculate the overall work efficiency of all trucks or excavators, and determine the collaborative work efficiency of the two according to formula (6).

[0101] (4) (5) (6) t 1 represents the working time of the excavator or truck; t 2 represents the non-working time of the equipment; represents the total idle time of the equipment; is the total working time of the truck or excavator; represents the collaborative working time of the excavator and truck; t totalThe total time length of the monitored video is referred to.

[0102] The rockfill excavation efficiency monitoring method based on deep visual multi-mechanical interactive action recognition provided by the present application quantifies the cooperation efficiency of the equipment in the "loading-unloading" process by calculating the cooperative operation cycle between the excavator and the truck, realizes fine evaluation of the overall process of the rockfill excavation operation, and thus optimizes construction scheduling and improves the comprehensive operation efficiency of the earthwork transportation system.

[0103] Embodiment 3 To achieve the above-mentioned embodiments, as Figure 5 indicated, the embodiment also provides a rockfill excavation efficiency monitoring device 10 based on deep visual multi-mechanical interactive action recognition, which comprises: A target detection module is configured to perform target detection on the excavator and the truck in the construction site by using an improved YOLOv8 model integrated with a lightweight deep separable convolution module, so as to enhance local discriminative feature extraction and improve detection stability in a shielding scene. A trajectory tracking module is configured to realize continuous trajectory tracking of multiple trucks by combining a DeepSORT algorithm based on a weighted matching strategy that fuses appearance features and kinematic distances, so as to maintain target correlation in a shielding state. An action recognition module is configured to recognize the action of the excavator by using a TimeSformer model with a multi-scale spatial attention mechanism, extract local details and global context features by using differentiated convolution kernels, and distinguish different interactive actions of the excavator by combining action timing logic modeling. A state determination module is configured to construct a dynamic state determination rule base according to the tracking trajectories of the excavator and the truck and the action recognition results, and dynamically determine the working state and non-working state of the equipment based on the motion state, relative distance and action cycle logic of the equipment.

[0104] Further, the target detection module is further configured to: adopt a combination structure of pointwise convolution and depth convolution, wherein the depth convolution is used to generate feature maps corresponding to the number of convolution kernels to capture multi-scale local features, and the pointwise convolution is used to assign different weights according to the importance of each feature map to retain key local features of the truck; reduce the computational complexity by independently setting the convolution kernel of each channel, and dynamically adjust the convolution kernel weight during the detection process to adapt to the feature extraction requirements under different shielding degrees.

[0105] Further, the action recognition module is further configured to: use a 3x3 small convolution kernel to extract local detail features such as the posture of the bucket, so as to enhance the recognition ability of the fine action of the excavator; The global context features such as the rotation range are modeled by using a 7*7 large convolution kernel, and the continuous action segments are modeled by combining the time sequence attention mechanism to identify the composite action sequence.

[0106] Further comprising: The cooperative cycle calculation module is configured to calculate the cooperative operation cycle between the devices according to the tracking trajectories of the excavator and the truck and the action recognition result, wherein the cooperative operation cycle is defined as the total time for the excavator to complete one loading-unloading action and the truck to be in the effective loading range, and is used to evaluate the cooperative efficiency of the devices.

[0107] The rock pile excavation efficiency monitoring device based on deep visual multi-machine interactive action recognition of the embodiment of the application further quantifies the cooperation efficiency of the devices in the "loading-unloading" process by calculating the cooperative operation cycle between the excavator and the truck, and realizes fine evaluation of the overall process of the rock pile excavation operation, so as to optimize construction scheduling and improve the comprehensive operation efficiency of the earthwork transportation system.

[0108] Embodiment 4 The application further provides an electronic device comprising a processor and a memory, wherein the memory stores executable instructions, and the processor implements the rock pile excavation efficiency monitoring method based on deep visual multi-machine interactive action recognition when executing the instructions.

[0109] Embodiment 5 The application further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the rock pile excavation efficiency monitoring method based on deep visual multi-machine interactive action recognition.

[0110] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0111] Furthermore, the terms "first", "second", "third", "fourth", "fifth" and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily presented chronologically. It is to be understood that the terms so used are interchangeable under appropriate circumstances and embodiments. By definition, the term "a" or "an" means "one or more". It is to be understood that the terms "comprising", "including", "incorporating", "having", "containing" or "featuring", etc., are used in the description and in the claims as equivalents of the term "consisting of". Furthermore, the term "comprising" is used in the description and in the claims to mean "consisting of, but also including additional, non-specified elements or steps".

Claims

1. A method for monitoring the efficiency of rockfill excavation based on depth vision and multi-mechanical interaction action recognition, characterized in that, include: S1 uses an improved YOLOv8 model with an integrated lightweight depthwise separable convolution module to detect excavators and trucks at a construction site, thereby enhancing local discriminative feature extraction and improving detection stability in occluded scenarios. S2, based on the detection results of improved YOLOv8, adopts a weighted matching strategy that integrates appearance features and kinematic distance, combined with the DeepSORT algorithm to achieve continuous trajectory tracking of multiple trucks, so as to maintain the target correlation under occlusion. S3 uses the TimeSformer model with a multi-scale spatial attention mechanism to identify excavator actions. It extracts local details and global context features through differentiated convolutional kernels and combines action temporal logic modeling to distinguish different interactive actions of the excavator. S4, based on the tracking trajectory and action recognition results of the excavator and truck, constructs a dynamic state determination rule base, and dynamically determines the working state and non-working state of the equipment based on the equipment's motion state, relative distance and action loop logic.

2. The method as described in claim 1, characterized in that, S1 includes: S11 employs a combination of pointwise convolution and depthwise convolution. The depthwise convolution is used to generate feature maps corresponding to the number of convolution kernels to capture multi-scale local features, while the pointwise convolution is used to assign different weights according to the importance of each feature map to preserve the key local features of the truck. S12 reduces computational complexity by setting convolution kernels independently for each channel, and dynamically adjusts the weights of the convolution kernels during detection to adapt to feature extraction requirements under different degrees of occlusion.

3. The method as described in claim 1, characterized in that, The S3 includes: S31 uses 3×3 small convolution kernels to extract local detail features such as bucket posture to enhance the ability to recognize the fine movements of excavators; S32 uses a 7×7 large convolutional kernel to model global context features such as rotation range, and combines a temporal attention mechanism to model continuous action segments in order to identify compound action sequences.

4. The method as described in claim 1, characterized in that, Also includes: S5 calculates the collaborative operation cycle between the equipment based on the tracking trajectory and motion recognition results of the excavator and the truck. The collaborative operation cycle is defined as the total time for the excavator to complete one loading-unloading action and for the truck to be within the effective loading range, which is used to evaluate the collaborative efficiency of the equipment.

5. A monitoring device for rockfill excavation efficiency based on depth vision and multi-mechanical interaction action recognition, characterized in that, include: The target detection module is used to detect excavators and trucks in construction sites using an improved YOLOv8 model that integrates a lightweight depthwise separable convolution module, in order to enhance local discriminative feature extraction and improve detection stability in occluded scenes. The trajectory tracking module is used to track the continuous trajectories of multiple trucks based on the detection results of the improved YOLOv8, using a weighted matching strategy that integrates appearance features and kinematic distance, combined with the DeepSORT algorithm, in order to maintain the target correlation under occlusion conditions. The action recognition module is used to recognize the excavator's actions using the TimeSformer model which introduces a multi-scale spatial attention mechanism. It extracts local details and global context features through differentiated convolutional kernels and combines action temporal logic modeling to distinguish different interactive actions of the excavator. The status determination module is used to build a dynamic status determination rule base based on the tracking trajectory and action recognition results of the excavator and truck. Based on the equipment's motion state, relative distance, and action loop logic, it dynamically determines the equipment's working state and non-working state.

6. The apparatus as claimed in claim 5, characterized in that, The target detection module is also used for: A combination structure of pointwise convolution and depthwise convolution is adopted, in which depthwise convolution is used to generate feature maps corresponding to the number of convolution kernels to capture multi-scale local features, and pointwise convolution is used to assign different weights according to the importance of each feature map to preserve the key local features of the truck. The computational complexity is reduced by setting convolution kernels independently for each channel, while the weights of the convolution kernels are dynamically adjusted during the detection process to adapt to the feature extraction requirements under different degrees of occlusion.

7. The apparatus as claimed in claim 5, characterized in that, The action recognition module is also used for: 3×3 small convolution kernels are used to extract local detail features such as bucket posture to enhance the ability to recognize the fine movements of excavators; A 7×7 large convolutional kernel is used to model global context features such as rotation range, and a temporal attention mechanism is combined to model continuous action segments in order to identify compound action sequences.

8. The apparatus as claimed in claim 5, characterized in that, Also includes: The collaborative cycle calculation module is used to calculate the collaborative operation cycle between the equipment based on the tracking trajectory and action recognition results of the excavator and the truck. The collaborative operation cycle is defined as the total time for the excavator to complete one loading-unloading action and for the truck to be within the effective loading range, which is used to evaluate the collaborative efficiency of the equipment.

9. An electronic device, comprising: processor; The memory stores executable instructions; when the processor executes the instructions, it implements the method as described in any one of claims 1-4.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1-4.

Images