Excavator operation intelligent monitoring and analysis system based on vision and large model

Abstract

The invention discloses an excavator operation intelligent monitoring and analysis system based on vision and a large model, and relates to the technical field of building construction. According to the method, an accessed monitoring video is automatically processed through a computing device deployed at the edge, and the method comprises multiple links of data preprocessing, action recognition, job analysis and report generation and block chain evidence storage. Firstly, based on a SlowFast-BiLSTM model, in combination with a target detector and a target tracker integrated in a network, real-time detection, tracking and fine-grained action recognition of multiple excavators in a picture are achieved, and a structured action sequence is output. An analysis task is autonomously planned through a multi-agent workflow driven by a large language model, and an external tool is scheduled to complete multi-dimensional quantitative and qualitative analysis. According to the system, tampering-free evidence storage is carried out on key data of the whole process from an original video, an action sequence, an inference process to a final report, it is ensured that the analysis process can be audited, the result can be traced, and interpretable and traceable intelligent analysis support is provided for construction management.

Description

Technical Field

[0001] This invention relates to the field of building construction technology, specifically to an intelligent monitoring and analysis system for excavator operations based on vision and large models. Background Technology

[0002] Construction machinery plays a crucial role in modern engineering and construction projects, directly impacting various aspects of the overall project, including operational efficiency, safety, and carbon emissions. Therefore, accurately identifying and understanding the real-time operational status of this equipment is essential for precisely controlling on-site progress, optimizing resource allocation, and conducting effective project management. Traditional operation management methods rely on manual supervision, with construction workers or on-site supervisors monitoring machinery operations. They primarily depend on personal experience to judge compliance and potential safety hazards, managing through verbal communication, gestures, or written notices. While these methods are still widely used on construction sites, they require significant manpower and resources, over-reliance on the subjective experience of supervisors, and are susceptible to the influence of personal judgment on recorded information. Furthermore, the entire process suffers from poor traceability, paper documents are easily lost, and in the event of safety disputes or cost controversies, effective tracing and liability determination are difficult.

[0003] With the deepening of smart construction site construction, a large number of surveillance cameras have been deployed on construction sites, continuously generating massive amounts of video data on construction machinery operations. This provides a solid data foundation for intelligent analysis of construction machinery operations. For project management, this method, which can automatically identify and analyze construction machinery operations from raw video, is crucial. This automated and intelligent motion analysis method has significant practical implications for improving the digitalization and intelligence level of engineering construction.

[0004] Excavators, as a typical type of construction machinery, are widely used in various construction scenarios due to their high flexibility. Currently, there is considerable research on excavator motion recognition and operational activity analysis, which typically includes three main stages: operational activity information extraction, operational activity analysis, and the processing and presentation of analysis results. However, existing methods have the following limitations: First, some studies use sensor-based methods, which, while highly accurate, require additional hardware installation on the equipment, resulting in high deployment costs, maintenance difficulties, and hindering large-scale application. Second, vision-based excavator motion recognition methods are currently mostly designed for single targets and short videos, unable to adapt to long videos recorded by surveillance cameras. Finally, existing methods often focus on optimizing individual stages, lacking intelligent analysis and a complete framework for the entire excavator operation process.

[0005] Therefore, there is an urgent need in this field for a technical solution that can overcome the above-mentioned technological gaps, realize end-to-end intelligent analysis of excavator operation activities, and has high precision, high reliability, and strong traceability, so as to meet the actual needs of excavator operation management in modern engineering. Summary of the Invention

[0006] The purpose of this invention is to propose an intelligent monitoring and analysis method for excavator operations based on vision and large models. This method can automatically extract the excavator's operating actions from videos and perform multi-dimensional operation analysis to meet the operation analysis needs of excavators in actual engineering projects.

[0007] To address the aforementioned issues, this application adopts the following technical solution: an excavator operation analysis system integrating computer vision, large-scale modeling, and blockchain evidence storage technologies, comprising the following modules: Cameras are installed near the construction area to collect real-time video streams of excavator operations on site.

[0008] Edge computing devices, linked to cameras, are used to receive video stream data in real time and upload it to a server.

[0009] The data preprocessing module, serving as the system's fundamental data entry point, is responsible for standardizing the input raw construction site monitoring video. It receives continuous video streams captured by on-site cameras and sequentially performs preprocessing methods such as invalid segment cropping, image resolution standardization, and invalid action tag removal. The output is a standardized video stream containing only work segments.

[0010] The action recognition module is built upon an improved SlowFast-BiLSTM two-stage deep learning model. This module receives a preprocessed keyframe sequence and first uses the dual-path architecture of the SlowFast network to extract spatial semantic features and fine-grained temporal motion features from the video. Then, the built-in YOLOv10 object detector accurately locates the bounding box of the excavator in each frame. To associate targets across frames, the module integrates the DeepSORT multi-object tracking algorithm, assigning a unique and continuous trajectory ID to each excavator. Finally, the BiLSTM network models the spatiotemporal feature sequence extracted by SlowFast, learns the periodic patterns of excavator operations, and outputs continuous atomic action labels with temporal IDs, forming a structured sequence of operational actions.

[0011] The large-model-driven workflow operation analysis module handles user requests and performs multi-dimensional analysis of excavator operations. It constructs a multi-agent collaborative workflow through prompting engineering, encompassing problem understanding, task planning, information retrieval, report generation, and report review and optimization nodes. This module receives analysis requests submitted by users in natural language (e.g., "Analyze the excavator's operational efficiency and carbon emissions in video V001"). The workflow begins with the problem understanding node parsing the user's intent; subsequently, the task planning node decomposes the complex requirements into executable sub-tasks and generates a structured JSON execution plan, specifying the external tools to be invoked (such as efficiency calculation models and carbon emission accounting modules) and the data sources to be queried; the information retrieval node extracts action sequences from the database and retrieves domain specifications from the knowledge base based on the execution plan; the report generation node, acting as a multi-modal large model, integrates all information and invokes tools to generate a preliminary analysis report; and the report review and optimization node verifies and refines the report to ensure its professionalism, accuracy, and readability.

[0012] The blockchain-based evidence storage and traceability module is built on the Hyperledger Fabric consortium blockchain. This module implements a layered evidence storage strategy: First, hash values ​​are calculated and uploaded to the blockchain for the raw video output from the data preprocessing module and the structured action sequences generated by the action recognition module, ensuring the integrity of the original data. Second, hash values ​​are calculated and uploaded to the blockchain for key intermediate data generated during the large-scale model workflow analysis module's operation, including JSON execution plans from task planning nodes, query logs from information retrieval nodes, and thought chains from report generation nodes. After the construction party confirms the report, the hash value and generation timestamp of the final report are anchored to the blockchain; after the supervisor approves, their approval result and timestamp are also recorded, forming an immutable and complete audit chain, achieving trusted traceability throughout the entire process from raw data to final insights.

[0013] The interaction and report generation module serves as the system's user interface and output terminal, provided as a web application. Users can upload videos, submit analysis requests, and view analysis results. The module receives the final analysis report, which is stored on the blockchain, and presents it to users in structured text format. The report not only includes quantitative results but also incorporates the large-scale model's deep understanding of the operational context, providing attribution analysis and optimization suggestions. Users can perform interactive queries through this module, and the system can trace and display various data and reasoning steps referenced during report generation, greatly enhancing the transparency and decision support value of the analysis results.

[0014] The method includes: preprocessing and motion recognition of the input construction site video to obtain the excavator operation motion sequence; constructing a workflow based on a large language model to perform demand-driven multi-dimensional analysis and report generation on the motion sequence; and performing layered blockchain notarization on the motion sequence, workflow reasoning process, and generated report to achieve full-process traceability and trustworthiness.

[0015] Advantages of this invention: 1. Acquiring on-site excavator operation information using computer vision. Traditional methods rely on manual recording or require the installation of sensors on the equipment, resulting in low efficiency, high cost, and strong invasiveness. This invention applies an advanced computer vision model based on SlowFast-BiLSTM to excavator action recognition, directly and automatically extracting structured atomic-level operation action sequences from widely deployed surveillance videos in a non-contact manner. This overcomes the drawbacks of traditional methods, such as reliance on manual labor, low efficiency, and strong subjectivity, achieving automatic processing of massive video data. It provides an objective, continuous, and fine-grained data foundation for subsequent analysis, which is a prerequisite for the digitalization of engineering management.

[0016] 2. Large-scale model analysis workflows can adapt to various changing analytical needs. Existing analysis systems mostly have fixed analysis processes, which cannot flexibly respond to users' dynamic and complex analytical requirements. This invention constructs a multi-agent workflow based on a large language model, decomposing analytical tasks into standardized nodes such as understanding, planning, retrieval, generation, and optimization. Through prompt engineering, the system can be "programmed" to adapt to various new tasks, while modular tool call interfaces allow for the rapid integration of new algorithms (such as new security rule bases). This fundamentally changes the way system functions are upgraded, shifting from rewriting code to adjusting prompts and access tools, giving the system agility and scalability to cope with future unknown analytical needs.

[0017] 3. Deeply integrated, context-aware, and interpretable intelligent insights. Traditional data analysis only outputs calculation results, failing to explain the causes of those results or provide relevant suggestions. This invention utilizes a multimodal large language model, which, when generating reports, not only calls upon quantitative calculation tools but also deeply integrates the visual context of video keyframes, domain specifications in the knowledge base, and the temporal logic of action sequences. This approach elevates analysis from describing phenomena to analyzing causes, significantly enhancing the practical guiding value of the results.

[0018] 4. The entire analysis process is traceable. This invention innovatively introduces a layered evidence storage mechanism based on a consortium blockchain. It calculates hashes and uploads them to the blockchain for all key data throughout the entire chain, from the original video, recognition results, large-scale model thought process, to the final report, ensuring that data at any stage cannot be unilaterally tampered with. The introduction of blockchain makes the report generation process traceable. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and approaches in the embodiments of the present invention, the overall system architecture and specific technical implementation details of the solution will be given below with reference to the accompanying drawings. These drawings illustrate the embodiments of the present invention in an exemplary and not restrictive manner, and the specific meaning of each drawing is given below: Figure 1 This is a schematic diagram of the module composition of an intelligent monitoring and analysis system for excavator operations based on vision and large models, provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the step-by-step data processing flow in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram showing the motion cycle transition characteristics of an excavator in working and non-working states.

[0022] Figure 4 The diagram illustrates the internal node structure and task division of a large-scale workflow, and also provides a schematic diagram of the communication architecture with external tools and systems.

[0023] Figure 5 The document showcases the evidence storage content of the blockchain layered evidence storage mechanism, which is divided into basic data evidence storage and large model reasoning process evidence storage. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be fully and clearly described below with reference to the accompanying drawings and specific implementation examples, so as to enable those skilled in the art to understand the present invention.

[0025] See Figure 1 and Figure 2 Content, Figure 1 This diagram illustrates the modules involved in an excavator operation activity result analysis method based on computer vision and large-scale models. Cameras should be installed at the work site, ensuring their image capture range covers the excavator's working area. Edge computing devices are used to upload raw video to the analysis platform. The data preprocessing module performs preprocessing steps on the raw video, including cropping invalid segments, unifying resolution, and filtering invalid action tags, ensuring the video data meets model requirements. The action recognition module takes the preprocessed video as input, extracting the excavator's bounding box and atomic actions, and obtaining action sequences. The large-scale model-driven workflow operation analysis module combines keyframe images and action sequence files to respond to various user analysis needs. The blockchain notarization and traceability module ensures data integrity and the traceability of the reasoning process. Finally, all analysis results and uploaded videos are integrated into the interaction and report generation module, displayed as a web application with interactive pages.

[0026] Figure 2 This is an overview of the system architecture, which can be roughly divided into four steps: video evidence storage and preprocessing, excavator action recognition, intelligent analysis of excavator operations based on a large model workflow, and evidence storage during the inference process. The following specific examples illustrate the system's operation flow.

[0027] Example This embodiment focuses on calculating the operational efficiency and carbon emissions of excavators during operation. Four construction videos of different excavator models were selected as data sources. These videos were all taken at a frame rate of 30 frames per second and a resolution of 852x480, with each video exceeding 10 minutes in length. To derive the actual operational efficiency value, the testing process employed manual video observation, taking into account the excavator's operating time, model, and complete work cycle. Conversely, the actual carbon emission values ​​were derived from real fuel consumption data and mechanical parameters. Subsequently, the detected efficiency and carbon emissions were compared with their actual values ​​to evaluate the accuracy of the efficiency calculation module and the carbon emission analysis module. This excavator operation analysis method embodiment includes the following steps: Step 1: Video Data Storage and Preprocessing. For each segment of on-site excavator monitoring video data uploaded to the system, the system first calculates the encrypted hash value (e.g., SHA-256) of the video file and stores this hash value, upload timestamp, file metadata, and other information as evidence. This information is then written to the Hyperledger Fabric consortium blockchain via a smart contract call. This process is completed before the data enters the analysis process, ensuring the integrity and immutability of the original video data and providing a reliable data source for subsequent analysis. Afterward, the video data is fed into the preprocessing module. Preprocessing steps include invalid segment cropping, resolution unification, and invalid action tag removal. Finally, the video data and keyframes are stored in the MogoDB database as the data foundation for subsequent large-scale excavator operation analysis workflows.

[0028] Step 2: Excavator Action Recognition. To extract the excavator's operational action sequence from the video, the preprocessed video needs to be input into the action recognition model for inference, resulting in a processed video with action bounding boxes and an action sequence file. The SlowFast-BiLSTM two-stage model can learn from visual features and the excavator's operational action cycle pattern, respectively. First, the preprocessed video is input into the SlowFast backbone network fused with YOLOv10. The dual-path structure of the SlowFast network can extract spatial semantic features and high frame rate temporal motion features, respectively, to achieve inference of the excavator's operational actions. SlowFast performs second-level detection of the video frame, outputting a detection box and corresponding action label for each second. Then, the detection results of adjacent frames are associated using the DeepSORT target tracking algorithm to form a continuous excavator trajectory. Using the DeepSORT algorithm, based on appearance features and motion consistency constraints, the target is associated across frames to generate a trajectory with a temporal ID. Combined with the action labels output by SlowFast, a preliminary temporal action sequence for each excavator is generated. Finally, the initial action sequence is input into the temporal correction module. This module is built on a bidirectional long short-term memory network (BiLSTM), which learns the contextual dependencies of actions through its bidirectional gating mechanism, and can effectively correct erroneous labels caused by transient occlusion, target blurring, or instantaneous misjudgment by the model. Figure 3 The diagram illustrates the transitions in the excavator's action modes during operation. When in working condition, its actions typically cycle through digging, transporting, dumping, and slewing. When the excavator is moving or not in working condition, it exhibits moving and idling actions. Therefore, the excavator's operational behavior exhibits a significant sequential relationship and cyclical pattern. This sequential relationship and cyclical pattern can be used to further refine the action labels. After refinement, the final structured sequence file with timing IDs and action labels is output, along with a complete video annotated with bounding boxes and action labels.

[0029] Step 3: Intelligent analysis of excavator operations based on a large-scale model workflow. For example... Figure 4 As shown, the excavator operation analysis workflow includes multiple large-scale model analysis nodes with different roles: problem understanding node, task planning node, information retrieval node, report generation node, and report review and optimization node. These nodes are used to understand user needs, plan task processing flows, retrieve data from databases and knowledge bases, generate analysis reports, and optimize report content. User-inputted needs or problems are processed sequentially through these nodes. Simultaneously, instructions are transmitted to external interfaces via JSON messages for information retrieval and tool invocation. Through preprocessing of the operation activities using external tools, the large-scale model can obtain the final analysis results, such as operation efficiency and carbon emission calculations, which are ultimately presented in report form.

[0030] Step 4: Report Generation and Reasoning Process Evidence Storage. After the analysis of the large-scale model workflow described above, a satisfactory excavator operation analysis report will be obtained. In this embodiment, this includes an excavator operation efficiency calculation report and a carbon emission analysis report. To ensure the credibility of the analysis results and the traceability of the entire process, a blockchain-based evidence storage and traceability mechanism is also introduced into the large-scale model reasoning process. Specifically, the report generation node will simultaneously trigger the evidence storage process while outputting the final analysis report. This process includes two levels: First, a digital digest of the report itself is calculated and bound to information such as the report version, generation timestamp, and generation node identity to form a report integrity evidence stored and anchored to the blockchain. Second, process evidence storage is performed on the key reasoning links of the aforementioned large-scale model workflow. This involves calculating hash values ​​for the requirement analysis results at the problem understanding node, the execution plan generated at the task planning node, the query and return content at the information retrieval node, the key thought chain at the report generation node, and all inputs and outputs of external tools called, and then associating them on the blockchain according to the execution sequence.

[0031] Figure 5 The diagram illustrates the layered notarization mechanism of blockchain: First, there's the data layer notarization mechanism, primarily ensuring data integrity. This layer completes notarization before data is entered into the database, ensuring the data hasn't been tampered with or lost. On-site operation videos captured by surveillance cameras have their hash values ​​calculated and notarized in the blockchain network. These video streams are then uploaded to the database for storage. This uploaded video data is further input into the excavator's action recognition module to obtain action sequence files, which are also notarized and stored. Second, there's the notarization mechanism for the analysis process, used to trace the generation process of the large-scale model's analysis results. When a user submits an operation analysis request to the system, the large-scale model workflow automatically parses the user's request and initiates query requests to the database and knowledge base. The returned data first has its hash value calculated and matched against records in the blockchain network to ensure data integrity before operation analysis is performed. Data query logs, tool call logs, and the thought chains of each node during the large-scale model workflow analysis process are also notarized on the blockchain. This information comprehensively records the generation process of the analysis results, facilitating later process traceability. Similarly, the report content generated by the large model will also be stored on the blockchain, forming a complete and verifiable audit chain from raw data, intermediate processes to final conclusions.

[0032] The aforementioned evidence records form an immutable and verifiable complete audit trail on the blockchain through transaction IDs. Any authorized participant (such as the construction party or the supervision party) can independently verify whether the report has been tampered with through this on-chain record, and fully trace every layer of data, every reasoning step, and every tool call on which the report was based, thereby constructing a complete and traceable chain of analysis processes.

[0033] In this example, specifically: In step 1, the data preprocessing steps include three sub-steps: invalid segment cropping, image resolution unification, and invalid action label removal. First, the acquired raw surveillance video is sampled at a frequency of 1 frame per second to generate a time-series image sequence. Second, the pre-trained lightweight object detection model YOLOv10-N is used to perform excavator target detection and bounding box localization on each frame. Then, the validity of the segment is judged based on the target vanishing criterion and the target stationary criterion: if no excavator is detected within 30 consecutive seconds, or if the change in the intersection-over-union ratio (IoU) of the target bounding box is less than the threshold IoU=0.9 within 30 consecutive seconds, it is determined to be an invalid segment. The formula for calculating the intersection-over-union ratio (IoU) is:

[0034] Secondly, the acquired videos underwent resolution standardization. To unify the input size of videos from different sources, the resolution of all video frames was adjusted to 640×640 pixels. When the original image size was smaller than the target size, a bicubic interpolation algorithm was used for upscaling; when the original image size was larger than the target size, a bilinear interpolation algorithm was used for downscaling, ensuring consistent input data scale.

[0035] Finally, invalid action tags need to be cleaned. This mainly includes two types of cleaning rules: first, filtering out short-duration action segments by setting a minimum duration threshold. First, action tags with durations shorter than the threshold are removed to eliminate noise caused by momentary false detections and tracking drift; second, small target detection boxes are filtered out by setting a minimum relative area threshold. The bounding box area is set at 5% of the total image area. Detection results with a bounding box area ratio lower than this threshold are removed to avoid unreliable annotations that interfere with model training due to targets that are too small or have blurred features.

[0036] In this embodiment, specifically: In step 2, the dataset used for training the action recognition model is the AVA2.2 dataset. This dataset requires keyframe sampling at 1-second intervals and labeling of its atomic action categories. In this embodiment, six excavator action categories are set: idling, moving, turning, transporting, digging, and dumping. Three truck action categories are also set: standby, moving, and loading. The truck actions are used to correct the excavator's action sequence in the post-processing stage and are not the primary recognition target.

[0037] The dataset annotation process uses ffmpeg to capture keyframes, and then the labelImg annotation tool is used to annotate the bounding boxes of excavators and trucks in the keyframes. This process requires accurately selecting all relevant construction machinery in the image and assigning a target category label to each bounding box. The output of this step is to generate an XML file containing target location and category information for each keyframe, thus completing the spatial positioning information annotation. Finally, to assign specific action labels to the targets within each bounding box, we further use the VGG Image Annotator (VIA) tool for temporal action classification annotation. In VIA, a custom attribute is created with its input type set to checkboxes, and the options define all the action categories to be annotated for the excavator or truck, such as: digging, transporting, dumping, turning, moving, idling, etc. Converting the annotation results into a CSV file yields the AVA2.2 dataset required for training.

[0038] The core annotation files for the dataset include ava_train_v2.2.csv and ava_val_v2.2.csv. Each line of data has the following format: video_id, middle_frame_timestamp, x1, y1, x2, y2, action_id, person_id. Here, video_id is the video identifier, middle_frame_timestamp is the timestamp (in seconds) of the keyframe, x1, y1, x2, y2 are the bounding box coordinates, action_id corresponds to the action category ID, and person_id is used to distinguish different people in the video; in this example, it is the ID of the construction machinery.

[0039] In this embodiment, specifically: In step 3, the large model workflow integrates a data augmentation retrieval mechanism, as well as external interfaces such as tool calls and database queries. This enables the large model not only to possess understanding, reasoning, and generation capabilities, but also to access external tools and data as needed for more precise task analysis and content generation. Specifically, this workflow uses prompt templates to restrict the roles, task boundaries, and output formats of each node. For example, the prompt template for the task planning node might include: "You are a task flow planner. Your core responsibility is to determine whether a given subtask can be supported by the system. If the task is not feasible, you need to explain the specific reasons; if it is feasible, please generate a detailed execution plan containing ordered steps, clearly listing the analysis tools to be called, the data sources to be queried, and their specific parameters. Your output must be machine-readable instructions that strictly conform to the following JSON Schema, and must not contain any free text descriptions, in the format..." Through such constraints, the node will output a structured sequence of JSON instructions, which explicitly includes the step order, tool identifiers, parameter lists, and data dependencies.

[0040] The knowledge base primarily records engineering-related documents, such as excavator specifications, operating standards, carbon emission calculation standards, project schedules, and geological condition descriptions. The database, on the other hand, records excavation operation videos, action sequence information, and keyframe images. The interaction between the large model and the database is achieved by generating query statements based on requirements. When constructing the knowledge base, the input documents need to be extracted and segmented into appropriately sized fragments according to semantic logic. These fragments can be transformed from textual information into high-dimensional semantic vectors through an embedding model. To improve query efficiency, this embodiment uses an approximate nearest neighbor algorithm to construct an index of the semantic vectors of document fragments. First, K-Means is used to cluster the semantic vectors, obtaining k cluster centers. During the indexing phase, each semantic vector is matched with its nearest cluster center, constructing an inverted list for each cluster center. During querying, the query statement is also converted into a semantic vector through an embedding model. Find the n nearest clusters and merge the vectors from these clusters into candidate vectors. Then, find the semantic vector that is closest to the query vector among the candidate vectors. The following formula describes how to find the semantic vector that is closest to the query vector. The process of clustering by nearest neighbor:

[0041] There are k cluster centers in total. It is the j-th cluster center. This is the index of the cluster center most similar to the query vector. In the code implementation, an inverted index is used. The following formula describes the process of finding the nearest neighbor vector from the candidate vectors:

[0042] in, The collection of all inverted lists. These are all semantic vectors that are assigned to cluster centers c. In practice, this algorithm typically searches through n selected clusters, calculates distances, and selects the k closest segment vectors as output. The top-K most relevant text segments retrieved above will be used as enhanced context, integrated along with the user's original query into the suggestions submitted to the large language model.

[0043] In this embodiment, specifically: In step 4, when the construction party generates an analysis report through the system, the client first verifies whether the original video hash value and action sequence hash match the records in the blockchain. If the verification passes, the client automatically calculates the encrypted hash values ​​of key data, including the output hashes of each node in the large model workflow, the thought chain hash, and the hash of the final report content. The construction party, as the initiator of evidence storage, submits the above hash values ​​and related metadata (such as timestamps and report IDs) to the blockchain via a smart contract as the transaction payload, completing the initial evidence locking.

[0044] Upon receiving the report notification, the supervising node invokes the verification function in the smart contract to verify the consistency between the report content and the on-chain hash value. After reviewing the report content and confirming its accuracy, the supervising party initiates a second transaction, recording the approval result, approval timestamp, and its digital signature hash on the blockchain. Through this dual operation of the construction party's evidence storage and the supervising party's approval, an immutable multi-party collaborative evidence storage record is formed.

[0045] The blockchain-based evidence stored in this embodiment can be audited and traced through a web application integrated into the system. This application is based on the layered evidence storage mechanism: at the data layer, the system can display the evidence hashes of original surveillance videos and action sequence files, and automatically perform integrity verification before the large model workflow calls relevant data; at the process layer, the application provides a function to query by report ID. Users can input the report ID to retrieve and visualize the complete inference chain evidence storage information corresponding to the report's generation in the interactive interface, including the on-chain hash records and temporal relationships of key process data such as the outputs of each node in the large model workflow, thought chain records, and tool call logs, thereby achieving transparent auditing of the entire intelligent analysis process.

[0046] This invention provides a method for analyzing excavator operation activities based on computer vision and a large-scale model. Through the collaborative work of multiple modules, including data preprocessing, action recognition, operation analysis workflow, and blockchain-based evidence storage, the system can achieve intelligent analysis of input monitoring videos. Due to the introduction of the multi-agent large-scale model workflow, it can respond to user-driven needs and perform multi-dimensional, customizable, in-depth analysis of operation activities. In this embodiment, the system focuses on two core analyses: operation efficiency calculation and carbon emission accounting. However, its modular design allows for flexible integration with other professional computing tools (such as safety risk identification and equipment health assessment modules), enabling continuous expansion of analytical capabilities.

[0047] This method leverages a large language model to deeply understand keyframes and action sequences in videos, elevating traditional numerical calculations to intelligent analysis with semantic interpretation capabilities. The system not only outputs specific quantitative indicators but also automatically analyzes the underlying causes of results by incorporating contextual information during operations, such as weather conditions, equipment coordination, and operational pauses and waiting times. It generates improvement suggestion reports with practical operational guidance. This makes the analysis conclusions not only quantifiable and traceable but also interpretable and valuable for decision support. The technical solution provided by this invention enables full-process, intelligent, and reliable analysis of construction machinery operations such as excavators, effectively addressing pain points in traditional engineering management such as reliance on manual inspections, fragmented data recording, limited analytical dimensions, and difficulty in result traceability. Through deep perception, accurate identification, intelligent analysis, and reliable evidence storage of the operation process, this system significantly improves the refinement of construction management, the scientific nature of decision-making, and the transparency of operations. It provides key technical support for promoting machinery operation management in smart construction sites, demonstrating outstanding innovation, engineering practicality, and industry promotion value.

[0048] For other details regarding the methods and implementation techniques of each module in the above embodiments, please refer to the detailed descriptions of data preprocessing, action recognition, large model workflow construction and blockchain notarization in the foregoing embodiments, which will not be repeated here.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A vision-based and large-scale model-based intelligent monitoring and analysis system for excavator operations, characterized in that: include, Cameras, deployed near the work area, can capture the entire operation of the excavator and are used to collect video streams of on-site operations. Edge computing devices, connected to cameras, are used to receive video stream data in real time and upload it to a server; The data preprocessing module, as the basic data entry point of the system, performs standardized processing on the original construction site monitoring videos uploaded to the system; it sequentially performs preprocessing methods such as invalid segment cropping, image resolution unification, and invalid action tag removal. Output video stream data in a uniform format; The action recognition module, connected to the data preprocessing module, uses an improved SlowFast-BiLSTM model to process the preprocessed video keyframes. First, it extracts spatiotemporal features through the SlowFast network, then detects the excavator bounding box using YOLOv10, and uses the DeepSORT algorithm for cross-frame target tracking and ID association. Finally, the BiLSTM network performs temporal modeling on the feature sequence and outputs a structured atomic action sequence with temporal IDs. The large-model-driven operation activity analysis module enables multi-dimensional intelligent analysis of excavator operations by prompting engineering to construct a multi-agent collaborative process. The large-model-driven operation activity analysis module receives user natural language requests and processes them sequentially through problem understanding, task planning, information retrieval, report generation, and review and optimization nodes. The task planning node generates a structured JSON execution plan and schedules external tools and data sources; the information retrieval node extracts action sequences and domain knowledge. The report generates comprehensive node information and uses tools to create the report; the nodes are reviewed for final verification and the report content is optimized to ensure the results are professional and reliable. The blockchain evidence storage and traceability module is based on the Hyperledger Fabric consortium blockchain and adopts a layered evidence storage strategy; it calculates hashes for basic data such as original videos and action sequences and puts them on the blockchain to ensure the integrity of the data source. Hash storage is also performed on key process data such as task plans, query logs, and inference chains in the workflow to achieve reliable traceability throughout the entire process. The interaction and report generation module provides a user interface, receives user input, and outputs analysis results. The interaction and report generation module is configured to present evidence-based intelligent analysis reports in a structured format, which include quantitative indicators, attribution information, and optimization suggestions generated based on operational context understanding. Furthermore, the interaction and report generation module supports tracing and querying the data and reasoning chain upon which the report generation process is based.

2. The excavator operation activity analysis system based on computer vision and large model as described in claim 1, characterized in that, The action recognition module is a two-stage action recognition model; it performs action recognition from visual features and excavator operation cycle patterns respectively; the model integrates the YOLOv10 target detector and DeepSORT target tracker to assign and maintain a unique trajectory identifier for each excavator in a long video, achieving stable target association across frames; and finally obtains the action sequence of each excavator.

3. The excavator operation activity analysis system based on computer vision and large model as described in claim 1, characterized in that, The large model-driven operation activity analysis module addresses users' multi-agent collaborative workflow for excavator operation analysis by prompting the engineering construction to include problem understanding nodes, task planning nodes, information retrieval nodes, report generation nodes, and report review and optimization nodes.

4. The excavator operation activity analysis system based on computer vision and large model according to claim 3, characterized in that, The large-model-driven operation activity analysis module achieves reliable interaction with external system resources through retrieval enhancement generation technology and an external communication protocol based on JSON structured messages. Specifically, the information retrieval node utilizes the retrieval enhancement generation technology to semantically retrieve documents related to equipment specifications, operating procedures, and safety guidelines from a pre-built domain knowledge base based on vector similarity, and queries a structured database for specified excavator action sequences and keyframe information. The report generation node parses the JSON structured messages and dynamically calls externally registered professional analysis tools as needed, including but not limited to operation efficiency calculation models, carbon emission accounting modules, and safety compliance checkers. It then deeply integrates the quantitative calculation results returned by these tools with domain knowledge to generate an intelligent analysis report with data support and logical explanation.

5. The excavator operation activity analysis system based on computer vision and large model according to claim 1, characterized in that, The blockchain evidence storage and traceability module is built on the Hyperledger Fabric consortium blockchain and adopts a layered evidence storage mechanism: in the data integrity protection layer, the hash value of the original video file output by the data preprocessing module and the structured action sequence data generated by the action recognition module are calculated and stored on the blockchain. In the trusted traceability layer of the analysis process, the hash values ​​of the task planning JSON execution plan, information retrieval node query log, report generation node thought chain record, external tool call parameters and results, and the final analysis report generated during the operation of the large model workflow analysis module are calculated and linked to the chain, forming an immutable and complete audit chain from the original data, intermediate reasoning to the final conclusion.

Images