A method, system, and portable terminal for offline security inspection and work order closed-loop management.

Abstract

This invention discloses an offline safety inspection and work order closed-loop management method, system, and portable terminal, including: acquiring video images and simultaneously obtaining sensor data through the portable terminal; aligning the video images and multi-source sensor data in multiple dimensions using a unified timestamp and spatial pose parameters; sending multimodal inputs to a multimodal large model deployed locally on the terminal, and performing logical discrimination in conjunction with a safety procedure expert database; real-time monitoring of the terminal's current temperature T, temperature change rate dT / dt, remaining battery power SoC, and safety risk level R of the operating scenario, performing three-level predictive scheduling based on T, dT / dt, SoC, and R, and dynamically switching inference modes; when a safety hazard is identified, generating an offline work order package with tamper-proof encryption features, and triggering in-situ review and automatic cancellation based on geofencing. This invention solves the problem of thermal runaway and missed detection of the large model on the edge under extreme equipment environments through multi-dimensional parameter prediction and dynamic scheduling of risk levels.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing, and particularly relates to the fields of industrial safety inspection and artificial intelligence technology. Specifically, it relates to an offline safety inspection and work order closed-loop management method, system and portable terminal based on a multimodal large model on the edge side. Background Technology

[0002] In special industrial settings such as ocean-going vessels, dredging projects, and deep-buried tunnels, safety supervision faces the following reconstructive technological bottlenecks:

[0003] First, the communication environment is extreme. The aforementioned scenarios are often in a network vacuum or with weak network connectivity, making existing cloud-based AI security monitoring solutions highly dependent on stable network connections. In offshore or underground environments, real-time early warning functions are completely ineffective, and cloud computing power support is unavailable. Furthermore, special operational scenarios involve core process data, and transmitting raw images back to the cloud poses risks of leaking trade secrets and compliance issues related to cross-border data transmission.

[0004] Second, hardware deployment is limited. Existing edge computing devices (such as AI boxes) have high power consumption, requiring additional wiring and power supply, resulting in high deployment and maintenance costs. Because these devices are located in fixed positions, they cannot cover blind spots in the cabin, ballast tanks, or mobile construction sites, which are often unregulated areas. In a high-altitude environment, it is extremely difficult to obtain technical maintenance and replacement after hardware failure.

[0005] Third, there is a lack of regulatory logic. Traditional computer vision models are mostly based on static feature detection (such as only identifying whether a safety helmet is being worn), belonging to the "fragmented recognition" stage, and cannot understand the "behavioral semantic logic" that includes spatiotemporal causal relationships. For example, in dredging vessel lifting operations, existing models can only identify independent targets such as "crane boom" and "personnel," but cannot determine the high-risk behavior of "personnel entering the dynamic blind spot of the crane boom." However, edge-side multimodal large models have the ability to understand image semantics and contextual relationships, and can accurately identify such correlated safety hazards, achieving a fundamental breakthrough from "single-point feature detection" to "behavioral logic judgment."

[0006] Fourth, the conflict between edge thermal management and business continuity is prominent in extreme environments. Existing computing scheduling strategies for general-purpose mobile devices (such as frequency throttling in mobile games and power-saving modes for laptops) are all passive response-based. Their core logic is: once the temperature or power consumption exceeds a threshold, the computing load is immediately reduced to protect the hardware. However, in extreme industrial scenarios such as ocean-going vessels and deep-buried tunnels, such strategies have fundamental flaws:

[0007] (1) Lag in passive response: The general strategy only triggers frequency reduction after the temperature threshold is reached, at which point the equipment is already on the verge of thermal runaway. In high-temperature engine room or summer deck environment, the interval between triggering frequency reduction and thermal shutdown is often only a few minutes, which cannot support continuous industrial inspection operations for 4-8 hours. Frequent thermal shutdowns will lead to large-scale interruption of supervision.

[0008] (2) The blindness of degradation mode: General degradation strategies (such as frequency reduction and core locking) only focus on "reducing power consumption" without considering "the continuity of monitoring services". After degradation, severe lag, frame drops or even application crashes are often accompanied by severe lag, frame drops or even application crashes, resulting in the failure to detect critical safety hazards (such as not wearing safety ropes at height or personnel entering the blind spot of the crane boom). In the field of industrial safety, a single missed detection may cause serious casualties, and general degradation strategies cannot meet the needs of industrial safety scenarios.

[0009] (3) Lack of risk prioritization: The existing strategy treats all computing tasks the same. When equipment overheats, the system does not distinguish between "low-risk general inspection" and "high-risk high-altitude hoisting operation" that is currently being monitored. This "one-size-fits-all" degradation logic may lead to the loss of monitoring capabilities at the highest risk moment, which runs counter to the basic requirement of "high risk and high protection" in industrial safety supervision.

[0010] (4) Inefficient asynchronous auditing: Some existing solutions adopt the extreme degradation method of "recording first and then reviewing", but the review process relies on manual frame-by-frame screening or requires full AI re-analysis of all recordings. In long-term inspection scenarios, this will generate huge delay computing power burden and power consumption, and cannot achieve efficient and automated hazard tracing and work order closure.

[0011] Therefore, how to ensure the continuous and stable operation of multimodal large models in extreme environments, ensure all-time inference capability during high-risk operation periods, and maintain efficient automated hazard detection capability in degraded mode on portable terminals is a special technical challenge that distinguishes this field from the field of general computing scheduling. Summary of the Invention

[0012] Based on this, the present invention aims to provide an offline security inspection and work order closed-loop management method, system, and portable terminal based on a multimodal large model on the edge side, in order to solve the following technical problems existing in the prior art:

[0013] (1) The general end-side passive response thermal management strategy has lag and blindness. It cannot support continuous industrial inspection operations when the equipment is in a high-temperature environment, and the downgrade leads to the failure to detect key safety hazards.

[0014] (2) Existing scheduling strategies lack the ability to differentiate risk priorities for industrial safety and cannot prioritize ensuring the continuity of reasoning during high-risk operation periods;

[0015] (3) Asynchronous auditing in the downgrade mode relies on manual or full AI re-analysis, which is inefficient and cannot achieve automated hazard tracing;

[0016] (4) Real-time semantic-level security auditing cannot be achieved in a completely offline environment. Traditional solutions have drawbacks such as high deployment costs, many regulatory blind spots, and high data compliance risks.

[0017] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0018] The first objective of this invention is to provide an offline safety inspection and work order closed-loop management method, comprising the following steps:

[0019] Step S1: Acquire video images through a portable terminal and simultaneously acquire sensor data, including one or more of position coordinates, height, and attitude;

[0020] Step S2: Align the video images and sensor data using a unified timestamp to form a multi-modal input;

[0021] Step S3: The multimodal input is sent to the multimodal large model deployed locally on the portable terminal, and logical judgment is performed in combination with the preset security procedure expert database to output the security risk identification result;

[0022] Step S4: Monitor the current temperature T, temperature change rate dT / dt, and remaining battery power SoC of the portable terminal in real time, and simultaneously obtain the safety risk level R of the current working scenario; the safety risk level R is generated by the multimodal large model dynamically assessing the frequency and severity of potential hazards output in the previous stage; based on T, dT / dt, SoC, and R, perform three-level predictive scheduling and dynamically switch inference modes; the inference modes include real-time streaming inference mode, intermittent frame extraction mode, and asynchronous audit mode;

[0023] The three-level predictive scheduling includes:

[0024] (i) When condition A1 or condition A2 is met, switch to or maintain the real-time streaming inference mode:

[0025] Condition A1: T < T1 and the estimated time t_pred to reach T2 based on dT / dt prediction is greater than the safe operation duration threshold;

[0026] Condition A2: Safety risk level R is high risk;

[0027] In the real-time streaming inference mode, the entire video stream is sampled and inferred in real time, and millisecond-level alarms are output.

[0028] (ii) When condition B is met, switch to or maintain the intermittent frame skipping mode:

[0029] Condition B: T1 ≤ T < T2 or t_pred ≤ safe operation time threshold, and R is not high risk;

[0030] In the intermittent frame-skipping mode, the calculation frequency is reduced, and inference is performed by sampling images at fixed periods.

[0031] (iii) When condition C1 or condition C2 is met, switch to or maintain the asynchronous audit mode:

[0032] Condition C1: T ≥ T2;

[0033] Condition C2: SoC < preset battery threshold;

[0034] In the asynchronous audit mode, the process switches to recording the video as evidence first, and then performing a delayed audit in the background after the device has cooled down or been powered on.

[0035] Step S5: When a security risk is identified, an offline work order package with tamper-proof encryption features is generated, and on-site review and automatic cancellation are triggered based on geofencing technology.

[0036] Optionally, the portable terminal is a general-purpose mobile device equipped with a smart operating system, including a smartphone or tablet, and has the capabilities of image acquisition, data processing, local storage, and sensor access.

[0037] Optionally, in step S2, the multidimensional alignment includes: simultaneously recording the corresponding GPS / BeiDou coordinates, barometric altitude, and gyroscope attitude data when acquiring each frame or key frame of the video image, and assigning a unified timestamp.

[0038] Optionally, in step S3, the logical discrimination includes: jointly verifying the height change in sensor data with the lack of safety equipment in visual semantics to determine the behavior of potential safety hazards; the safety procedure expert database contains multi-constraint discrimination rules, and the multi-constraint conditions include one or more of spatial position constraints, height threshold constraints, and attitude constraints.

[0039] Optionally, the determination of the safety hazard behavior includes: when the barometric pressure sensor detects that the relative altitude rise exceeds a preset threshold, the high-altitude inspection mode is activated; if the multimodal large model identifies that the operator is not connected to safety equipment, it is automatically determined to be a safety hazard.

[0040] Optionally, in step S4, the safe operation time threshold refers to the time required to complete the current inspection task. The safe operation time threshold is determined based on the historical time of the inspection path and the inspection task, or judged based on historical experience, and is adaptively corrected as historical data of the inspection task is collected. T1 is 45℃, T2 is 55℃, and the preset power threshold is 15%.

[0041] Optionally, in step S4, the high-risk determination in condition A2 includes: when the multimodal large model identifies that the current work scenario contains one or more high-risk semantic tags such as high-altitude operation, hoisting operation or hot work, the safety risk level R is assessed as high risk.

[0042] Optionally, in step S4, the three-level predictive scheduling further includes a threshold exemption mechanism: when the security risk level R is high risk, the values ​​of T1 and / or T2 are dynamically increased to prioritize the continuous operation of the real-time streaming inference mode.

[0043] Optionally, in step S4, the intermittent frame-skipping mode adopts an event-driven adaptive sampling strategy: when the rate of change of the values ​​of one or more sensors, such as the gyroscope, barometric altimeter, or accelerometer, exceeds a preset threshold, or when the inference confidence of the multimodal large model for the previous frame is within a preset fuzzy range, the sampling frequency is temporarily increased.

[0044] Optionally, in step S4, in the asynchronous audit mode, a two-stage evidence storage with semantic pre-labeling is performed:

[0045] Phase 1: Run a lightweight anomaly detection network with power consumption not exceeding 10% of the power consumption of the multimodal large model. During the recording process, detect sensor anomaly events in real time and insert semantic timestamp markers.

[0046] Phase 2: During asynchronous auditing in the background, the multimodal large model prioritizes processing video clips with semantic timestamps, and generates work orders after confirming potential risks.

[0047] Optionally, in step S4, the switching between the three inference modes adopts a seamless switching with state inheritance: when switching from the first inference mode to the second inference mode, the tracking state of the multimodal large model is frozen and serialized to the local persistent storage medium; when switching back from the second inference mode to the first inference mode, the serialized tracking state recovery model is loaded to maintain the continuity of the target identification ID and the motion trajectory.

[0048] Optionally, in step S5, the offline work order package includes screenshots of violations, a short video evidence chain, tamper-proof encryption features, and standard operating procedure repair suggestions; the encryption features include at least a timestamp, location coordinates, and device identifier.

[0049] Optionally, in step S5, the in-situ review and automatic cancellation triggered based on geofencing technology includes:

[0050] Asynchronous resume upload: The system continuously sniffs the network environment, and once an available network signal is detected, it automatically uploads the encrypted offline work order package asynchronously;

[0051] In-situ re-inspection trigger: Based on geofencing technology, when the inspection equipment re-enters the preset geofence area, the re-inspection mode is automatically triggered;

[0052] Automatic verification: By using a multimodal large model to infer from the images collected during the review, the work order is automatically closed after determining that the hidden danger has been eliminated.

[0053] Optionally, it also includes step S6: all raw image data is inferred instantaneously only in the local memory of the portable terminal, and after the inference is completed, it is desensitized and encrypted, and only the desensitized structured audit results are retained, thus blocking the information leakage path from a physical level.

[0054] The second objective of this invention is to provide an offline safety inspection and work order closed-loop management system based on an edge-side multimodal large model, for executing the above-mentioned method, including:

[0055] Multi-source sensing module: configured to acquire video images and simultaneously obtain sensor data via a portable terminal;

[0056] Spatiotemporal alignment module: configured to perform multidimensional alignment of video images and sensor data using a unified timestamp;

[0057] Edge-side inference module: Configured to deploy a multimodal large model and perform logical discrimination in conjunction with a security procedure expert database;

[0058] Dynamic scheduling module: configured to monitor the current temperature T, temperature change rate dT / dt, remaining power SoC and safety risk level R of the current working scenario in real time, and perform three-level predictive scheduling, dynamically switching inference mode;

[0059] Offline closed-loop module: Configured to generate offline work order packages and trigger in-situ review and automatic cancellation based on geofence.

[0060] A third objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0061] A fourth objective of this invention is to provide a portable terminal, comprising:

[0062] The image acquisition unit is used to acquire video images;

[0063] The sensor unit is used to synchronously acquire position coordinates, height, and attitude data;

[0064] The memory stores a multimodal large model and a database of safety procedures experts.

[0065] The processor is configured to execute the methods described above.

[0066] Due to the adoption of the above technical solution, the beneficial effects obtained by the present invention include:

[0067] 1. Eliminate reliance on satellite traffic, additional hardware, and dedicated cabling; full-site monitoring can be achieved using portable work terminals such as smartphones or tablets already available to on-site personnel, significantly reducing deployment and maintenance costs.

[0068] 2. By combining visual semantics with physical sensors, we can achieve a leap from target recognition to complex behavioral semantic understanding, accurately identifying dynamic logical vulnerabilities that contain spatiotemporal causal relationships.

[0069] 3. The dynamic reasoning mode scheduling based on joint thermal-risk prediction proposed in this invention breaks through the limitations of conventional equipment state management strategies: by integrating temperature change rate prediction with operational safety risk levels, it achieves a leap from "passive response" to "predictive proactive scheduling" and "high-risk exemption"; through event-driven adaptive sampling, it can still capture critical hidden danger moments in intermittent frame-skipping mode; through two-stage semantic pre-labeling and evidence storage, it achieves automated hidden danger detection after degradation in asynchronous audit mode; and through seamless switching with state inheritance, it ensures the spatiotemporal continuity of target tracking before and after mode switching. The above mechanisms jointly solve the problem of thermal runaway and missed detection of large-scale end-side models in high-temperature and vibration environments, ensuring the monitoring stability of industrial-grade continuous operation.

[0070] 4. It solves the pain point of not being able to record, trace, and verify data in real time in special industrial scenarios such as ocean-going vessels, dredging projects, and deep-buried tunnels, and realizes a fully automated regulatory closed loop of "offline discovery, asynchronous transmission, in-situ review, and automatic verification".

[0071] 5. All original image data remains within the terminal, fundamentally resolving the compliance risks of trade secret leakage and cross-border data transmission. Attached Figure Description

[0072] Figure 1 This is an overall flowchart of the method of the present invention, which shows the timing relationship and dynamic scheduling branches of steps S1 to S5.

[0073] Figure 2 This is the state transition diagram of the dynamic scheduling module of the present invention, which shows the switching conditions between real-time streaming inference mode, intermittent frame extraction mode, and asynchronous audit mode.

[0074] Figure 3This is a sequence diagram of the offline work order closed-loop management of the present invention, which shows the complete process of hidden danger discovery, work order generation, asynchronous upload, on-site review, and automatic cancellation.

[0075] Figure 4 This is a schematic diagram of the module structure of the system of the present invention, which shows the data flow and control flow relationship between the multi-source sensing module, the spatiotemporal alignment module, the edge inference module, the dynamic scheduling module, and the offline closed-loop module. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0077] For example, an offline safety inspection and work order closed-loop management method based on an edge-side multimodal large model includes the following steps:

[0078] Step S1 (Multi-source sensing): Acquire video images through a portable terminal and simultaneously acquire sensor data, which includes one or more of position coordinates, height, and attitude.

[0079] Specifically, the portable terminal is a general-purpose mobile device equipped with a smart operating system, including a smartphone or tablet computer, which has the capabilities of image acquisition, data processing, local storage, and sensor access, and can be deployed and run without dedicated hardware. This invention extracts GPS / BeiDou coordinates, barometric altimeter data, and gyroscope attitude data in real time while acquiring video streams.

[0080] Step S2 (Spatiotemporal Alignment): Align the video images and sensor data in multiple dimensions using a unified timestamp to form a multimodal input.

[0081] Specifically, the system establishes a synchronous acquisition matrix. When acquiring each frame or keyframe of a video image, the corresponding sensor data is recorded synchronously and assigned a unified timestamp, achieving multi-dimensional alignment between visual frames and sensor features. This alignment ensures a strict correspondence between image semantics and spatiotemporal physical conditions during subsequent inference.

[0082] Step S3 (End-side reasoning): The multimodal input is sent to the multimodal large model deployed locally on the portable terminal, and logical judgment is performed in combination with the preset security procedure expert database to output the security risk identification result.

[0083] Specifically, this invention integrates a lightweight inference engine (preferably an open-source framework such as Mobile Neural Network or MNN) locally on a portable terminal, constructing an edge-side inference environment independent of the public network. Through preset model quantization and compression configurations, it ensures that the multimodal large model (Vision-Language Model, VLM) can run stably on the portable terminal's general-purpose processor (CPU) or hardware acceleration unit (GPU, NPU). The system injects unstructured image semantics and structured sensor values ​​into the local inference engine, and performs multi-constraint discrimination through a preset safety procedure expert library. The safety procedure expert library contains multi-constraint discrimination rules, including one or more of spatial location constraints, height threshold constraints, and attitude constraints.

[0084] For example, when the barometric pressure sensor detects a relative altitude increase Δh ≥ 2m, the high-altitude inspection mode is automatically activated; when the multimodal large model identifies that a worker's waist is not connected to the lifeline, it automatically determines it as a safety hazard by combining spatial height and behavioral semantics. This process achieves a fundamental leap from "target detection" to "behavioral logic judgment".

[0085] Step S4 (Scheduling of Dynamic Inference Mode Based on Joint Heat-Risk Prediction):

[0086] The system monitors the following multi-dimensional status indicators of the portable terminal in real time:

[0087] Thermal parameters: current temperature T (unit: °C) and temperature change rate dT / dt (unit: °C / min), obtained through the operating system's underlying interface or an external temperature sensor;

[0088] Energy efficiency indicator: Remaining power SoC (State of Charge, unit: %)

[0089] Business semantic indicator: The safety risk level R of the current work scenario. This level is generated by the multimodal large model through dynamic assessment of the frequency and severity of potential hazards output in the previous stage (e.g., the first 30 seconds), and is divided into three levels: high, medium, and low. For example, when the model continuously identifies high-risk semantic tags such as "high-altitude operation", "crane boom rotation", and "hot work", R is automatically determined to be high risk.

[0090] Based on the above metrics, the system performs a three-level dynamic scheduling system that combines predictive and task semantic awareness, specifically including:

[0091] 4.1 Real-time streaming inference mode (high-performance mode)

[0092] Triggering condition: Meets any of the following conditions—

[0093] Condition A1: T < T1 (preferably 45°C), and the estimated time t to reach T2 (preferably 55°C) is predicted based on dT / dt pred > the safety operation duration threshold (the safety operation duration threshold is determined according to the historical time of the inspection path and inspection tasks or judged by historical experience);

[0094] Condition A2: The current safety risk level R is high risk.

[0095] Behavior definition: Real-time sampling and inference are performed on the full video stream (such as 20fps), and voice or picture alarms at the millisecond level are output.

[0096] Innovation mechanism: When R is high risk, the system dynamically raises the temperature tolerance threshold (for example, temporarily raises T1 to 48°C) to prioritize ensuring the full-time real-time inference ability during high-risk periods, and then restores the original threshold after R drops to medium and low risks. This is a "risk-first" thermal management strategy, different from the passive degradation strategy of "protecting hardware" for general devices.

[0097] 4.2 Intermittent frame sampling mode (balanced mode)

[0098] Trigger condition: T1 ≤ T < T2 or t pred ≤ the safety operation duration threshold, and R is not high risk.

[0099] Behavior definition: Reduce the computing frequency, but instead of fixed-period sampling, an event-driven adaptive sampling strategy is adopted:

[0100] (a) Sensor event-driven: When the gyroscope detects severe shaking (angular velocity change rate > threshold), the barometric altimeter changes rapidly (Δh / Δt > 0.5m / s), or the accelerometer detects a falling posture, temporarily increase the sampling frequency (for example, temporarily restore from 5 seconds / frame to 1 second / frame), and re-evaluate after 3 seconds;

[0101] (b) Semantic uncertainty-driven: When the inference result of the previous frame by the multi-modal large model contains "suspicious target" (such as the confidence level of not wearing a safety rope is 0.45, lower than the confirmation threshold of 0.7 but higher than the ignoring threshold of 0.3), perform attention-guided differential sampling on several subsequent frames, increase the sampling frequency (such as 2 frames / second) to confirm potential hazards and avoid missing detections at critical moments;

[0102] (c) Steady-state annealing: Only when the device is continuously in a stable and low-risk scenario (no mutations in sensors, high confidence in model output without potential hazards), degrade to the lowest frequency sampling (such as 1 frame every 5 seconds).

[0103] 4.3 Asynchronous auditing mode (safety degradation mode)

[0104] Trigger condition: T ≥ T2 (preferably 55°C), or SoC < preset power threshold (preferably 15%).

[0105] Behavior definition: Switch to "record evidence first and then conduct asynchronous auditing", but it is not a simple recording. Instead, it performs two-stage evidence storage with semantic pre-marking:

[0106] The first stage (real-time lightweight detection): The system records videos in an ultra-low power consumption mode (such as turning off large model inference and only enabling video recording and sensor sampling), while running a lightweight anomaly detection network (such as a comparator based on a single sensor rule, or a binary classification CNN model with a parameter count < 1M and a power consumption of only 5% of the complete multimodal large model). When this network detects possible security events (such as a height mutation > 2m, an abnormal attitude angle > 30°, continuous gyroscope oscillation, etc.), semantic timestamp marks (such as "suspicious high-altitude movement" or "suspected fall") are inserted into the video file in real time, and the start and end times of the corresponding sensor data segments are recorded synchronously.

[0107] The second stage (background asynchronous auditing): After the device cools down (T < T1) or is powered on, the system automatically starts the background asynchronous auditing program. The multimodal large model preferentially processes the video segments with timestamp marks (such as only processing 10 seconds before and after the marks), quickly confirms potential hazards and generates offline work orders; the unmarked video segments can be processed with extremely low priority or automatically overwritten when the storage space is insufficient.

[0108] 4.4 Seamless state inheritance mechanism for mode switching

[0109] The switching between the three inference modes adopts seamless switching with state inheritance, specifically including:

[0110] State freezing and serialization: When switching from the real-time stream inference mode to the intermittent frame extraction mode, the memory state of the large model (including target tracking IDs, motion feature vectors, historical attention weights, etc.) is frozen and serialized to a local temporary file in a local persistent storage medium.

[0111] State restoration: When switching back from the intermittent frame extraction mode to the real-time stream inference mode, the system directly loads the serialized state file to restore the model state, without re-initializing the model or re-identifying existing targets, avoiding target re-identification errors (such as the same person being assigned a new ID) or broken motion trajectories caused by mode switching.

[0112] Context anchor transfer: When switching from the intermittent frame extraction mode to the asynchronous auditing mode, the system stores the last N frames (such as 30 frames) of key frames and their intermediate inference results (such as detection boxes, classification confidence levels) as context anchors in the video file header for quickly restoring scene understanding during asynchronous auditing and shortening the warm-up time when the large model re-analyzes.

[0113] 4.5 Example Description

[0114] Taking ocean-going vessel inspection as an example: After 30 minutes of continuous operation, the hull temperature rose to 46℃, exceeding T1=45℃, and dT / dt was +0.5℃ / min. It was predicted that the temperature would reach 55℃ in 10 minutes, and since R was not considered high-risk, the estimated time to complete the inspection (safe operation time threshold) was 40 minutes remaining. The system predictively triggered an intermittent frame-sampling mode, reducing the sampling frequency from 20fps to 5 seconds / frame. At this time, the gyroscope detected severe boom swaying, and the system temporarily restored the sampling frequency to 1 second / frame for 3 seconds, successfully capturing the critical moment when a worker was not wearing a safety rope. After another 20 minutes of operation, the temperature rose to 54℃, and the system prematurely triggered the asynchronous audit mode, starting video recording and activating the lightweight detector. After the inspection was completed and the system returned to port for power restoration, the system background automatically audited the marked segments and generated work orders. The entire process did not result in any thermal shutdowns, and no potential hazards were missed.

[0115] Comparative example (maintaining real-time streaming inference mode): After the device ran continuously for 30 minutes, the body temperature rose to 46℃, exceeding T1=45℃, and dT / dt was +0.5℃ / min. It was predicted that it would reach 55℃ in 10 minutes. The system continued to use real-time streaming inference mode. After running for another 10 minutes, the temperature rose to 56℃. Because the dynamic inference mode of this solution was not adopted, only the device's general low-power strategy was triggered, causing the inspection process to be interrupted and the risk of missed inspections. After running for another 8 minutes, a thermal shutdown occurred.

[0116] Step S5 (Offline Closed Loop): When a security risk is identified, an offline work order package with tamper-proof encryption features is generated, and on-site review and automatic cancellation are triggered based on geofencing technology.

[0117] Specifically, after identifying potential hazards, the system automatically captures a chain of evidence including screenshots and short videos, generating a structured offline work order package with tamper-proof encryption features. These encryption features include at least a timestamp, location coordinates, and device identification. The terminal displays real-time standard operating procedure (SOP) repair suggestions to assist operators in immediately correcting their actions.

[0118] The system continuously sniffs the network environment, and once a usable signal (satellite, 4G, or WiFi) is detected, it automatically transmits encrypted work orders asynchronously to the management platform. Based on geofencing technology, when the inspection equipment reaches the same coordinate point again, a re-inspection is automatically triggered. After AI determines that the hidden danger has been eliminated, the work order is automatically closed, realizing a fully automated closed-loop supervision process of "offline discovery, asynchronous transmission, on-site re-inspection, and automatic cancellation".

[0119] Step S6 (Privacy Isolation): All raw image data is inferred instantaneously only in the local memory of the portable terminal. After the inference is completed, the data is de-identified and encrypted, and only the de-identified structured audit results are retained, thus blocking the path of information leakage at the physical level.

[0120] Example 1 (High-altitude inspection of ocean-going vessels)

[0121] This embodiment uses deck hoisting operations on ocean-going vessels as an application scenario to describe in detail the specific implementation process of the present invention.

[0122] Scenario Description: A dredging vessel is conducting lifting operations in international waters, in an area completely without network coverage. A worker enters the hazardous area beneath the crane boom without being secured by a safety rope.

[0123] Implementation steps:

[0124] Step 101: Equipment Deployment and Initialization

[0125] The portable work terminal has a variety of built-in sensors, including but not limited to:

[0126] GPS / BeiDou positioning module is used to obtain geographic location coordinates (records the last known coordinates when offline);

[0127] A barometric altimeter is used to measure relative altitude and determine the status of high-altitude operations.

[0128] Gyroscopes are used to measure angular velocity and detect changes in equipment attitude and shaking.

[0129] Accelerometers are used to measure linear acceleration and detect abnormal movements such as falls and impacts.

[0130] A magnetometer (electronic compass) is used to obtain directional information and assist in determining the orientation of a task.

[0131] An ambient light sensor is used to assist in adjusting image acquisition parameters.

[0132] The inspector boards the ship carrying a portable work terminal with the application of this invention installed. After the application is launched, it automatically loads the quantified multimodal large model into the phone's memory and loads the preset expert database of ship operation safety procedures. The expert database contains the following rules: {"High-altitude operation": height > 2m, "Safety rope": must be identified to the waist connection, "Crane blind spot": distance between personnel and crane < 3m is high risk}.

[0133] Step 102: Multi-source sensing and multi-dimensional alignment

[0134] The inspector activates inspection mode. The portable work terminal's rear camera begins capturing a video stream (20fps). Simultaneously, the system activates GPS (even without signal, it can record the last known coordinates), barometric altimeter, and gyroscope. The system adds a unique timestamp to each frame and records the corresponding sensor values.

[0135] When the workers climbed onto the platform 2.5 meters above the deck, the barometric altimeter detected a relative altitude increase Δh = 2.5m, exceeding the preset threshold of 2m. The system automatically activated the "high-altitude inspection mode".

[0136] Step 103: End-side reasoning and hazard identification

[0137] The multimodal large model performs inference on video frames. The model identifies a "person" target in the image, but no "safety rope" connection is detected at the waist position. At the same time, the model combines spatial location semantics to determine that the person is located within the projection blind zone of the "crane boom".

[0138] The system jointly verifies visual semantics (not secured with a safety rope) with sensor data (height 2.5m), and matches the high-risk rules in the expert database. The system determines it to be a combined safety hazard of "not secured with a safety rope at height + entering the blind spot of the boom".

[0139] Step 104: Dynamic Scheduling

[0140] The system synchronously acquires the safety risk level R of the current work scenario (generated by a multimodal large model dynamically assessing the frequency and severity of potential hazards output in the previous stage). If the current R is determined to be high-risk (high-altitude, hoisting, hot work, and other high-risk operations), the system switches to or maintains the real-time streaming inference mode. If the current R is not high-risk and the equipment triggers the power consumption threshold, the system automatically switches to intermittent frame-skipping mode or asynchronous audit mode to ensure task continuity.

[0141] Step 105: Offline work order generation and closed loop

[0142] Upon detecting a potential hazard, the system automatically captures three screenshots of the violation and a short video clip of the preceding and following 10 seconds, generating an offline work order package with tamper-proof encryption. The encryption feature can be a watermark, which includes: current GPS coordinates (last known latitude and longitude), barometric altitude of 2.5m, timestamp (UTC), and the device's unique identifier.

[0143] A red warning box immediately popped up on the phone screen, accompanied by a voice announcement: "Working at height without a safety rope, please connect the lifeline immediately!" The SOP (Standard Operating Procedure) repair guide was also displayed with images and text.

[0144] Step 106: Asynchronous Upload and Review / Verification

[0145] Portable devices can directly notify the ship's safety supervisor via LAN, Bluetooth, short-range communication protocols, or near-field communication technology, providing real-time alarms. The safety supervisor can review the work order on the platform and immediately rectify the issue, or assign the same inspector to conduct a follow-up inspection. When the inspector returns to the original work location (within a 10-meter radius of the geofence) or through image feature recognition, the mobile phone automatically triggers "follow-up inspection mode." The system re-captures on-site images, and the multimodal large model determines that personnel are correctly wearing safety harnesses and have not entered the crane's blind spot, automatically canceling the work order and completing the closed loop.

[0146] Upon returning to port, only the complete work order needs to be uploaded for easy traceability and review. The system automatically and asynchronously resumes the encrypted offline work order package to the shore-based management platform.

[0147] Example 2 (Inspection of the mobile construction face of a deep-buried tunnel)

[0148] This embodiment uses a deep-buried tunnel excavation face as an application scenario to illustrate the mobility and degradation strategy of the present invention.

[0149] Scenario Description: There is no network signal inside a deeply buried tunnel, and the construction face continues to move forward as the excavation progresses. Traditional AI boxes cannot be moved, resulting in numerous blind spots for monitoring.

[0150] Implementation steps:

[0151] Step 201: Mobile Inspection

[0152] Safety officers, armed with portable work terminals, move alongside the construction team. The tablets are loaded with a database of tunnel safety regulations experts, including rules on "not wearing a safety helmet," "walking on tracks," and "blasting warning zones."

[0153] Step 202: Battery and Thermal Management Degradation

[0154] After the portable work terminal operated continuously for 30 minutes, the body temperature rose to 46℃, exceeding the first threshold of 45℃. The dynamic scheduling module detected the temperature exceeding the standard, and the temperature change rate dT / dt was +0.5℃ / min, predicting that it would reach 55℃ in 10 minutes. The system synchronously obtained the safety risk level R of the current work scenario (generated by the multimodal large model based on the frequency and severity of hidden dangers output in the previous stage). It was determined that the current R was of medium to low risk (no high-altitude, hoisting, hot work, or other high-risk operations), and the estimated overheating time t_pred = 10 minutes was less than the safe operation time threshold. The safe operation time threshold refers to the time required to complete the inspection task. The safe operation time threshold can be determined based on the inspection path and the historical time of the inspection task, or it can be judged based on historical experience, or it can be adaptively corrected based on the historical data of the inspection task. In this embodiment, the safe operation time threshold is set to 30 minutes.

[0155] Based on the above multidimensional state indicators (T=46℃, dT / dt=+0.5℃ / min, t_pred=10 minutes, R=low to medium risk), the system performs predictive feedforward scheduling—before the temperature reaches the second threshold, it actively switches from "real-time streaming inference mode" to "intermittent frame skipping mode", reducing the basic sampling frequency from 30fps to one frame every 5 seconds.

[0156] Unlike traditional feedback control (which only responds after the temperature reaches a threshold), this system predicts overheating trends in advance based on the rate of temperature change, at t _pred Degradation is proactively triggered when the duration of safe operation is ≤ a safety operation time threshold, avoiding the lag of passive response. The degradation decision incorporates the safety risk level R, and degradation is only executed when R is not high-risk. If the current operation is high-risk, the system will implement a threshold exemption mechanism (dynamically increasing T1) to prioritize real-time inference. After degradation, the system maintains a sensitive response to sensor events. When the gyroscope detects severe shaking or a rapid change in the barometric altimeter, the sampling frequency is temporarily increased (e.g., restored to 1 second / frame) to ensure that critical potential hazards are not missed in an instant.

[0157] After the switch, the temperature stabilized at around 48℃, without triggering the second-level degradation (asynchronous audit mode), and the inspection continued. The entire process achieved a leap from "passive response" to "feedforward proactive scheduling," ensuring both equipment thermal safety and the continuity of regulatory operations.

[0158] After one hour of continuous inspection, the portable terminal's battery level dropped to 12%, below the 15% battery threshold. The system automatically switched from "intermittent frame skipping mode" to "asynchronous auditing mode." The tablet stopped performing intermittent frame skipping inference and instead continuously recorded and stored video in a low-power mode.

[0159] Step 203: Asynchronous Auditing

[0160] After the safety officer finished the inspection, he plugged the portable work terminal into a power source. The system detected the power connection and the temperature drop, automatically initiating an asynchronous background audit program to perform time-delay inference on the recorded video. Three violations of "not wearing a safety helmet" were detected, and offline work orders were automatically generated.

[0161] Step 204: In-situ re-examination

[0162] When the safety officer revisits the vicinity of the violation site, the system automatically prompts a re-inspection based on geofencing (last known coordinates + inertial navigation calculations). After the photo is uploaded, the model confirms the hazard has been eliminated, and the work order is automatically cancelled.

[0163] The foregoing descriptions and embodiments are provided to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be easily made to these contents, and the general principles described herein can be applied to other embodiments without creative effort. Therefore, the present invention is not limited to the foregoing descriptions and embodiments. Improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from its scope should be within the protection scope of the present invention.

Claims

1. A method for offline safety inspection and work order closed-loop management, characterized in that, Includes the following steps: Step S1: Acquire video images through a portable terminal and simultaneously acquire sensor data, including one or more of position coordinates, height, and attitude; Step S2: Align the video images and sensor data using a unified timestamp to form a multi-modal input; Step S3: The multimodal input is sent to the multimodal large model deployed locally on the portable terminal, and logical judgment is performed in combination with the preset security procedure expert database to output the security risk identification result; Step S4: Monitor the current temperature T, temperature change rate dT / dt, and remaining battery power SoC of the portable terminal in real time, and simultaneously obtain the safety risk level R of the current working scenario; the safety risk level R is generated by the multimodal large model dynamically assessing the frequency and severity of potential hazards output in the previous stage; based on T, dT / dt, SoC, and R, perform three-level predictive scheduling and dynamically switch inference modes; the inference modes include real-time streaming inference mode, intermittent frame extraction mode, and asynchronous audit mode; The three-level predictive scheduling includes: (i) When condition A1 or condition A2 is met, switch to or maintain the real-time streaming inference mode: Condition A1: T < T1 and the estimated time t_pred to reach T2 based on dT / dt prediction is greater than the safe operation duration threshold; Condition A2: Safety risk level R is high risk; In the real-time streaming inference mode, the entire video stream is sampled and inferred in real time, and millisecond-level alarms are output. (ii) When condition B is met, switch to or maintain the intermittent frame skipping mode: Condition B: T1 ≤ T < T2 or t_pred ≤ safe operation time threshold, and R is not high risk; In the intermittent frame-skipping mode, the calculation frequency is reduced, and inference is performed by sampling images at fixed periods. (iii) When condition C1 or condition C2 is met, switch to or maintain the asynchronous audit mode: Condition C1: T ≥ T2; Condition C2: SoC < preset battery threshold; In the asynchronous audit mode, the process switches to recording the video as evidence first, and then performing a delayed audit in the background after the device has cooled down or been powered on. Step S5: When a security risk is identified, an offline work order package with tamper-proof encryption features is generated, and on-site review and automatic cancellation are triggered based on geofencing technology.

2. The method according to claim 1, characterized in that, In step S1, the portable terminal is a general-purpose mobile device equipped with a smart operating system, including a smartphone or tablet, which has the ability to acquire images, process data, store locally, and access sensors.

3. The method according to claim 1, characterized in that, In step S2, the multidimensional alignment includes: simultaneously recording the corresponding GPS / BeiDou coordinates, barometric altitude, and gyroscope attitude data when acquiring each frame or key frame of the video image, and assigning a unified timestamp.

4. The method according to claim 1, characterized in that, In step S3, the logical discrimination includes: jointly verifying the height change in sensor data with the lack of safety equipment in visual semantics to determine the behavior of potential safety hazards; the safety procedure expert database contains multi-constraint discrimination rules, and the multi-constraint conditions include one or more of spatial position constraints, height threshold constraints and attitude constraints.

5. The method according to claim 4, characterized in that, The determination of the safety hazard behavior includes: when the barometric pressure sensor detects that the relative altitude rise exceeds a preset threshold, the high-altitude inspection mode is activated; if the multimodal large model identifies that the operator is not connected to safety equipment, it is automatically determined as a safety hazard.

6. The method according to claim 1, characterized in that, In step S4, the safe operation time threshold refers to the time based on the current completion of the inspection task. The safe operation time threshold is determined based on the inspection path and the historical time of the inspection task or determined by historical experience. T1 is 45℃, T2 is 55℃, and the preset power threshold is 15%.

7. The method according to claim 1, characterized in that, In step S4, the high-risk determination in condition A2 includes: when the multimodal large model identifies that the current work scenario contains one or more high-risk semantic tags such as high-altitude operation, hoisting operation or hot work, the safety risk level R is assessed as high risk.

8. The method according to claim 1, characterized in that, In step S4, the three-level predictive scheduling further includes a threshold exemption mechanism: when the security risk level R is high risk, the values ​​of T1 and / or T2 are dynamically increased to prioritize the continuous operation of the real-time streaming inference mode.

9. The method according to claim 1, characterized in that, In step S4, the intermittent frame-skipping mode adopts an event-driven adaptive sampling strategy: when the rate of change of the values ​​of one or more sensors, such as the gyroscope, barometric altimeter, or accelerometer, exceeds a preset threshold, or when the inference confidence of the multimodal large model for the previous frame is within a preset fuzzy range, the sampling frequency is temporarily increased.

10. The method according to claim 1, characterized in that, In step S4, the asynchronous audit mode performs two-stage evidence storage with semantic pre-labeling: Phase 1: Run a lightweight anomaly detection network with power consumption not exceeding 10% of the power consumption of the multimodal large model. During the recording process, detect sensor anomaly events in real time and insert semantic timestamp markers. Phase 2: During asynchronous auditing in the background, the multimodal large model prioritizes processing video clips with semantic timestamps, and generates work orders after confirming potential risks.

11. The method according to claim 1, characterized in that, In step S4, the switching between the three inference modes adopts a seamless switching with state inheritance: when switching from the first inference mode to the second inference mode, the tracking state of the multimodal large model is frozen and serialized to the local persistent storage medium; when switching back from the second inference mode to the first inference mode, the serialized tracking state recovery model is loaded to maintain the continuity of the target identification ID and the motion trajectory.

12. The method according to claim 1, characterized in that, In step S5, the offline work order package includes screenshots of violations, short video evidence chains, tamper-proof encryption features, and standard operating procedure repair suggestions; the encryption features include at least timestamps, location coordinates, and device identifiers.

13. The method according to claim 1, characterized in that, In step S5, the in-situ review and automatic cancellation triggered by geofencing technology includes: Asynchronous resume upload: The system continuously sniffs the network environment, and once an available network signal is detected, it automatically uploads the encrypted offline work order package asynchronously; In-situ re-inspection trigger: Based on geofencing technology, when the inspection equipment re-enters the preset geofence area, the re-inspection mode is automatically triggered; Automatic verification: By using a multimodal large model to infer from the images collected during the review, the work order is automatically closed after determining that the hidden danger has been eliminated.

14. The method according to claim 1, characterized in that, It also includes step S6: all raw image data is inferred instantaneously only in the local memory of the portable terminal. After the inference is completed, the data is desensitized and encrypted, and only the desensitized structured audit results are retained, thus blocking the information leakage path from a physical level.

15. An offline safety inspection and work order closed-loop management system based on an edge-side multimodal large model, characterized in that, For performing the method according to any one of claims 1 to 14, comprising: Multi-source sensing module: configured to acquire video images and simultaneously obtain sensor data via a portable terminal; Spatiotemporal alignment module: configured to perform multidimensional alignment of video images and sensor data using a unified timestamp; Edge-side inference module: Configured to deploy a multimodal large model and perform logical discrimination in conjunction with a security procedure expert database; Dynamic scheduling module: configured to monitor the current temperature T, temperature change rate dT / dt, remaining power SoC and safety risk level R of the current working scenario in real time, and perform three-level predictive scheduling, dynamically switching inference mode; Offline closed-loop module: Configured to generate offline work order packages and trigger in-situ review and automatic cancellation based on geofence.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 14.

17. A portable terminal, characterized in that, include: The image acquisition unit is used to acquire video images; The sensor unit is used to synchronously acquire position coordinates, height, and attitude data; The memory stores a multimodal large model and a database of safety procedures experts. A processor configured to perform the method of any one of claims 1 to 14.

Images