A power plant high-altitude operation safety risk determination method based on multi-modal large model and small model cooperation

Abstract

The application provides a power plant high-altitude operation safety risk judgment method based on multi-modal large model and small model cooperation, belongs to the technical field of power plant high-altitude operation safety, and realizes adaptive two-stage frame extraction through dynamic adjustment of a trigger threshold network by reinforcement learning, traces a key frame sequence through a ring buffer supplement recording mechanism, extracts space-time features and carries out target detection and posture estimation through a multi-scale space-time fusion detection network, smoothes human body key point coordinates through a time sequence consistency Kalman filter, judges operation sequence compliance through a time sequence causal reasoning network, carries out compliance review through a multi-modal artificial intelligence large model and generates a violation text explanation, realizes automatic judgment and archiving of a violation behavior through comprehensive risk scoring and evidence bag data, and solves the technical problem that power plant high-altitude operation safety monitoring is difficult to realize accurate and real-time violation behavior detection under limited computing resource conditions.

Description

Technical Field

[0001] This invention belongs to the field of power plant high-altitude operation safety technology, specifically, it relates to a method for determining the safety risks of power plant high-altitude operations based on the collaboration of multimodal large and small models. Background Technology

[0002] Safety monitoring of high-altitude operations in power plants is a crucial aspect of ensuring power production safety. Traditional monitoring technologies primarily rely on manual inspections combined with fixed camera surveillance, using manual review of video recordings to identify violations. Existing automated monitoring methods employ deep learning object detection networks to perform full-frame analysis of monitoring videos, extracting image features through convolutional neural networks and determining whether safety belts are worn and whether work postures comply with regulations. However, existing technologies suffer from excessive computational resource consumption. High-frequency full-frame detection leads to exponentially increasing computing power requirements, making deployment on edge devices difficult. Simultaneously, fixed threshold triggering mechanisms cannot adapt to the dynamic changes in different work scenarios, resulting in persistently high false alarm and false negative rates. In current power plant high-altitude operation monitoring practices, due to the massive amount of monitoring video data and the complex and variable nature of work scenarios, existing fixed frame rate sampling and fixed threshold determination methods struggle to balance detection accuracy and computational efficiency. In other words, existing technologies present a technical challenge in achieving accurate and real-time violation detection in power plant high-altitude operation safety monitoring under conditions of limited computational resources. Summary of the Invention

[0003] In view of this, the present invention provides a method for determining the safety risks of high-altitude operations in power plants based on the collaboration of multimodal large and small models, which can solve the technical problem in the prior art that it is difficult to achieve accurate and real-time detection of violations in the safety monitoring of high-altitude operations in power plants under the condition of limited computing resources.

[0004] This invention is implemented as follows: This invention provides a method for determining the safety risks of high-altitude operations in power plants based on the collaboration of a multimodal large model and a small model, comprising the following steps: acquiring the monitoring video stream of the work area and performing image stabilization preprocessing and time synchronization preprocessing, while simultaneously acquiring the equipment operation logs with corresponding timestamps to form a time-aligned original monitoring dataset; performing adaptive two-stage frame extraction on the original monitoring dataset, and using a reinforcement learning-based dynamic trigger threshold network to calculate the sharpness score, motion score, and attention score in real time; when any of the sharpness score, motion score, and attention score exceeds the dynamic trigger threshold and remains stable, high frame rate sampling is initiated and a circular buffer is executed to obtain a high frame rate sampling frame sequence; the high frame rate sampling frame sequence is input into a multi-scale spatiotemporal fusion detection network for target detection and pose estimation, outputting the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area, and the coordinates of the human body key points, and smoothing the human body key point coordinates using a time-consistent Kalman filter. The smoothed human body key point coordinates are obtained through smoothing. Exponential smoothing and window statistical calculations are performed on the personnel bounding box coordinates, safety belt anchor point coordinates, ladder area coordinates, and smoothed human body key point coordinates to obtain escalator contact stability and tether connectivity stability. When either escalator contact stability or tether connectivity stability exceeds the entry threshold and the duration reaches the preset minimum trigger duration, key frame images and temporal summary data from the high frame rate sampled frame sequence are input into the temporal causal inference network for action sequence semantic segmentation and causal relationship determination to obtain state transition path encoding. Based on the state transition path encoding, it is determined whether the operation sequence conforms to the compliant operation sequence. If the operation sequence does not conform to the compliant operation sequence, the violation fragment images and evidence element data are submitted to the multimodal large model for compliance review and a violation text explanation is generated. When both escalator contact stability and tether connectivity stability are below the exit threshold and the fall duration exceeds the preset minimum fall duration, the system automatically closes the loop and falls back to the normal monitoring mode.

[0005] Among them, the reinforcement learning dynamic adjustment trigger threshold network adopts the Double DQN architecture, which constructs the reward function value with historical trigger accuracy and computing power consumption value. The input includes sliding window sequences of clarity score, motion score, attention score, job scenario type encoding and environmental parameter vector, and the output is the action value estimate of the dynamic trigger threshold.

[0006] Among them, the reinforcement learning dynamic adjustment trigger threshold network sets up an experience playback buffer to store state transition quadruples, adopts a dual network structure of target network and evaluation network to reduce overestimation bias, and introduces a confidence interval estimation module to dynamically adjust the trigger sensitivity coefficient according to the historical variance value.

[0007] The steps for establishing the training dataset for the reinforcement learning-based dynamic adjustment of the trigger threshold network include collecting historical monitoring video data under different work scenario categories, different time period categories, and different weather condition categories; labeling the start and end times of real violations and violation type labels; and recording the trigger time data, false trigger counts, and missed detection counts under different fixed threshold values.

[0008] The sharpness score is obtained by calculating the variance value after applying the Laplacian operator to the gray intensity matrix of the frame image. The Laplacian operator performs a second-order difference operation on the frame image to highlight the edge sharpness. The larger the variance value, the sharper the frame image.

[0009] The motion score is obtained by calculating the structural similarity value between the frame image and the previous frame image and taking its complement. A structural similarity value of 1 indicates that the two frames are completely identical, and the larger the complement value, the more obvious the changes between frames.

[0010] The attention score is calculated by using a fast target detection network to detect people, ladders, and safety belts in low-resolution images. The maximum value among the confidence values ​​of the person, ladder, and safety belt in the high-altitude work area is extracted as the attention score.

[0011] Among them, the circular buffer supplementation maintains the latest low frame rate sampled frames continuously in the circular buffer for a preset duration. When any of the sharpness score, motion score and attention score triggers high frame rate sampling, the contents of the circular buffer are first replayed to complete the frame sequence before the trigger.

[0012] Among them, the multi-scale spatiotemporal fusion detection network uses ResNet3D-50 as the backbone network to extract spatiotemporal features. It captures motion patterns by performing convolution operations in both time and space dimensions simultaneously through 3D convolution kernels, and internally combines a feature pyramid network to construct multi-scale feature maps.

[0013] Among them, the multi-scale spatiotemporal fusion detection network design uses skip connections to fuse shallow high-resolution features with deep semantic features, and inserts a temporal attention module in front of the detection head to dynamically weight feature maps at different times to achieve enhanced detection of moving targets.

[0014] This also includes: calculating a comprehensive risk score based on the compliance review results output by the multimodal large model, combined with escalator contact stability, tethering connection stability, duration and operation level parameters; automatically generating evidence package data containing keyframe images, explanations of violations, violation timestamps and equipment location information, and archiving it in the database.

[0015] Among them, the temporal consistency Kalman filter applies kinematic constraints to the coordinates of human key points, assuming that the motion of human key point coordinates between adjacent frames conforms to a uniform motion model or a uniformly accelerated motion model. It eliminates the coordinate jump caused by detection noise and occlusion through two recursive estimation steps: state prediction step and observation update step.

[0016] Among them, the escalator contact stability is obtained by performing exponential smoothing on the escalator contact frame-level score and calculating the proportion of frames exceeding the entry threshold within the sliding window. The escalator contact frame-level score is obtained by calculating the spatial overlap between the person and the escalator based on the coordinates of the person's bounding box and the escalator area.

[0017] Among them, the stability of the connection between the seat belt and the human body is obtained by performing exponential smoothing on the connection frame-level score and calculating the proportion of frames exceeding the entry threshold within the sliding window. The connection frame-level score is obtained by calculating the connectivity between the seat belt and the human body based on the anchor point coordinates of the seat belt and the smoothed key point coordinates of the human body.

[0018] Among them, the temporal causal reasoning network uses a temporal convolutional network to extract long-term dependencies. Through causal convolution operations, it ensures that the output at the current moment depends only on the input at the past moment and the current moment and does not use information from the future moment. The internal design of the state transition matrix learns the compliant operation sequence path from the seat belt fastening state to the ladder climbing state.

[0019] Among them, the temporal causal reasoning network uses a reinforcement learning reward mechanism to optimize the accuracy of causal chain recognition. The output is a semantic segmentation label and state transition path encoding of the action sequence, which is used to determine whether the operation order conforms to the compliant operation order.

[0020] The evidence package data includes five types of elements: keyframe images, explanations of violations, violation timestamps, segment indexes, and device location information. Keyframe images are screenshots of video frames taken at the moment the violation occurred, and explanations of violations are violation description text and suggestion text generated by the multimodal large model.

[0021] This invention employs an adaptive two-stage frame extraction combined with a reinforcement learning-based dynamic trigger threshold network. Based on real-time changes in clarity score, motion score, and attention score, it intelligently switches between low-frame-rate routine monitoring and high-frame-rate intensive monitoring modes, significantly reducing average computational power consumption while ensuring accurate capture of violations. By introducing a circular buffering mechanism and a multi-scale spatiotemporal fusion detection network, this invention automatically backtracks the keyframe sequence before triggering high-frame-rate sampling and utilizes 3D convolutional kernels to simultaneously extract spatiotemporal features, enhancing moving target detection capabilities and resolving the issues of keyframe omissions and wasted computational resources caused by fixed-frame-rate sampling. Through a temporal causal reasoning network learning the state transition paths in safety operation specifications, combined with a multimodal large model for compliance verification of violation segments, this invention achieves accurate judgment and interpretable output of violations. In summary, this invention solves the technical problem mentioned in the background art of difficulty in achieving accurate and real-time violation detection in power plant high-altitude operation safety monitoring under limited computational resources. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention.

[0023] Figure 2 To reinforce learning, the trigger threshold is dynamically adjusted, and the adaptive adjustment curves are shown in different task scenarios. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0025] like Figure 1 The diagram shown is a flowchart of a method for determining the safety risks of high-altitude operations in power plants based on the synergy of multimodal large and small models, provided by this invention. This method includes the following steps:

[0026] S1. Acquire the monitoring video stream of the work area and perform image stabilization preprocessing and time synchronization preprocessing. At the same time, acquire the device operation log with the corresponding timestamp to form a time-aligned original monitoring dataset.

[0027] S2. Adaptive two-stage frame extraction is performed on the original monitoring dataset. The sharpness score, motion score and attention score are calculated in real time through a dynamic adjustment network of reinforcement learning. When any of the sharpness score, motion score and attention score exceeds the dynamic trigger threshold and remains stable, high frame rate sampling is started and a circular buffer is executed to obtain a high frame rate sampling frame sequence.

[0028] S3. Input the high frame rate sampling frame sequence into a multi-scale spatiotemporal fusion detection network for target detection and attitude estimation, output the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area and the coordinates of the human body key points, and smooth the human body key point coordinates by using a temporal consistent Kalman filter to obtain the smoothed human body key point coordinates.

[0029] S4. Perform exponential smoothing and window statistical calculations on the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area, and the coordinates of the smoothed human body key points to obtain the escalator contact stability and the mooring connection stability. When either the escalator contact stability or the mooring connection stability exceeds the entry threshold and the duration reaches the preset minimum trigger duration, input the key frame images and temporal summary data in the high frame rate sampling frame sequence into the temporal causal reasoning network to perform action sequence semantic segmentation and causal relationship determination to obtain the state transition path encoding.

[0030] S5. Determine whether the operation order conforms to the compliant operation order based on the state transition path code. If the operation order does not conform to the compliant operation order, submit the violation fragment image and evidence element data to the multimodal big model for compliance review and generate a violation text explanation.

[0031] S6. Based on the compliance review results output by the multimodal large model, combined with the escalator contact stability, the tether connection stability, the duration and operation level parameters, calculate a comprehensive risk score, automatically generate evidence package data containing the key frame image, the explanation of the violation text, the violation timestamp and equipment location information and archive it in the database. When the escalator contact stability and the tether connection stability are both lower than the exit threshold and the fall-down duration exceeds the preset minimum fall-down duration, automatically close the loop and fall back to the normal monitoring mode.

[0032] The reinforcement learning-based dynamically adjusted trigger threshold network employs a Double DQN architecture, constructing a reward function value based on historical trigger accuracy and computational cost. The inputs to the network include sliding window sequences of the clarity score, motion score, and attention score, as well as the task scenario type encoding and environmental parameter vector. The output is an estimate of the action value for the dynamic trigger threshold. Internally, the network includes an experience replay buffer to store state transition quadruples, employs a dual-network structure of a target network and an evaluation network to reduce overestimation bias, and introduces a confidence interval estimation module to dynamically adjust the trigger sensitivity coefficient based on historical variance values.

[0033] The steps for establishing the training dataset for the reinforcement learning-based dynamically adjusted trigger threshold network specifically include: collecting historical monitoring video data with a total duration of 5000 hours under different job scenario categories, different time period categories, and different weather condition categories; labeling each video segment in the historical monitoring video data with the start and end times of real violations and violation type labels; recording the trigger time data, false trigger counts, and missed detection counts under different fixed threshold values; calculating the peak and average computing power consumption after each trigger; organizing the start and end time data, violation type labels, trigger time data, false trigger counts, missed detection counts, peak computing power consumption, and average computing power consumption into a state-action-reward-next state quadruple format and dividing the training set data and validation set data in an 8:2 ratio.

[0034] The specific steps of the reinforcement learning dynamic adjustment trigger threshold network training include: initializing the network parameters of the evaluation network and the target network and setting the learning rate parameter to 0.0001; randomly sampling training sample data with a batch size of 64 from the experience replay buffer; calculating the mean squared error loss value between the action value output by the evaluation network and the target value calculated by the target network; updating the network parameters of the evaluation network using the Adam optimizer; copying the network parameters of the evaluation network to the target network every 200 training steps; and saving the optimal model parameters after the average reward value on the validation set data converges or the maximum number of iterations of 10,000 is reached.

[0035] The sharpness score is obtained by calculating the variance of the grayscale intensity matrix of the t-th frame image after applying a Laplacian operator. The Laplacian operator performs a second-order difference operation on the t-th frame image to highlight edge sharpness. The larger the variance value, the sharper the t-th frame image. The formula for calculating the sharpness score is as follows: ,in The grayscale intensity matrix of the t-th frame image is in units of grayscale values. For the Laplace operator, This is the variance operator. For reference, the grayscale intensity matrix of a clear image is in grayscale value units. The sharpness score of the t-th frame image is a dimensionless value.

[0036] The motion score is obtained by calculating the structural similarity value between the t-th frame and the (t-1)-th frame and taking their complement. A structural similarity value of 1 indicates that the two frames are completely identical, and a larger complement value indicates a more significant difference between the frames. The formula for calculating the motion score is as follows: ,in The structural similarity function output ranges from 0 to 1. For the t-th frame image, For the (t-1)th frame image, The motion score of the t-th frame image is a dimensionless value.

[0037] The attention score is calculated by using a fast object detection network to detect personnel, ladders, and safety belts in low-resolution images. The maximum value among the following three values ​​is extracted: the confidence value of the personnel located in the working area, the confidence value of the ladder detection, and the confidence value of the safety belt detection. The formula for calculating the attention score is as follows: ,in This indicates that the confidence level of the person is in the range of 0 to 1. This indicates that the confidence value of the ladder ranges from 0 to 1. This indicates that the confidence value of the seat belt ranges from 0 to 1. This indicates pre-marked areas of interest related to the risks of climbing. The attention score of the t-th frame image is a dimensionless value.

[0038] The ring buffer recording duration is: A circular buffer continuously stores the latest low frame rate sampled frames. When any of the sharpness score, motion score, or attention score triggers the high frame rate sampling, the contents of the circular buffer are first replayed to fill in the content before the trigger. A sequence of frames lasting seconds, followed by continuous high frame rate sampling in subsequent videos until all scores fall below the dynamic trigger threshold, at which point the sampling rate drops back to the low frame rate. The value ranges from 3 seconds to 10 seconds, and is configured according to the operation type parameter and risk level parameter.

[0039] The multi-scale spatiotemporal fusion detection network uses ResNet3D-50 as the backbone to extract spatiotemporal features, and captures motion patterns by simultaneously performing convolution operations in the temporal and spatial dimensions using 3D convolutional kernels. Internally, the network incorporates a Feature Pyramid Network (FPN) to construct multi-scale feature maps, designs skip connections to fuse shallow high-resolution features with deep semantic features, and inserts a Temporal Attention Module (TAM) before the detection head to dynamically weight feature maps at different times, thereby enhancing the detection of moving targets. The detection output of the multi-scale spatiotemporal fusion detection network includes the coordinates of the person's bounding box, the coordinates of the seatbelt anchor point, the coordinates of the ladder region, and the coordinates of the human body's key points and their confidence values.

[0040] The specific structure of the multi-scale spatiotemporal fusion detection network is as follows: the input layer receives a sequence of 16 consecutive RGB images with a size of 224×224×3; the backbone network ResNet3D-50 contains 4 residual block groups, each group using 3×3×3 3D convolutional kernels; the feature pyramid network FPN extracts residual block features from the 2nd, 3rd, and 4th residual block groups of the backbone network ResNet3D-50 and fuses them through upsampling and lateral connection operations to form a 5-layer feature pyramid; the temporal attention module TAM applies softmax attention weights to each layer of the feature pyramid in the time dimension; the detection head is divided into a classification branch and a regression branch, which output the target category probability value and the bounding box coordinates of the person, respectively; the pose estimation branch adopts the HRFormer structure to output the heatmap response of the coordinates of 17 human key points; in the post-processing stage, non-maximum suppression (NMS) is used to remove duplicate detection boxes, and the intersection-union (IU) threshold is set to 0.5.

[0041] The steps for establishing the training dataset of the multi-scale spatiotemporal fusion detection network specifically include: collecting 2000 hours of monitoring video data of high-altitude operation scenarios in power plants, covering different lighting conditions and different work postures; extracting keyframes at a sampling rate of 2 frames per second and labeling the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area, and the coordinates of 17 key points of the human body; performing data augmentation on each labeled sample, including random cropping transformation, horizontal flip transformation, color dithering transformation, and motion blur transformation; constructing a negative sample set containing scene frames without personnel or high-altitude operation behavior; organizing the labeled data into COCO format and dividing it into training set, validation set, and test set in a ratio of 7:2:1.

[0042] The specific steps for training the multi-scale spatiotemporal fusion detection network include: initializing the backbone network ResNet3D-50 using ResNet3D-50 weights pre-trained on the Kinetics-400 action recognition dataset; setting the batch size to 8, the learning rate to 0.001, and employing a cosine annealing learning rate scheduling strategy; the loss function consists of three parts: classification loss, bounding box regression loss, and keypoint heatmap loss, with a weight ratio of 1:1:0.5; using a stochastic gradient descent (SGD) optimizer with a momentum parameter set to 0.9 and a weight decay parameter set to 0.0001; training for a total of 100 epochs, evaluating the mean accuracy (mAP) on the validation set every 10 epochs; and saving the model weights with the highest mean accuracy (mAP) on the validation set as the final detection network parameters.

[0043] The temporally consistent Kalman filter applies kinematic constraints to the coordinates of the human keypoints, assuming that the motion of the keypoints between adjacent frames conforms to a uniform motion model or a uniformly accelerated motion model. It eliminates coordinate jumps caused by detection noise and occlusion through a two-step recursive estimation process involving state prediction and observation update. The temporally consistent Kalman filter maintains the position and velocity state vectors of each human keypoint. The state prediction step extrapolates the predicted coordinates for the next frame based on the uniform motion model or the uniformly accelerated motion model. The observation update step fuses the human keypoint coordinates and the predicted coordinates for the next frame and adjusts the Kalman gain weights based on the keypoint confidence value.

[0044] The escalator contact stability is obtained by performing exponential smoothing on the escalator contact frame-level score and calculating the proportion of frames exceeding the entry threshold within the sliding window. The exponential smoothing formula for the escalator contact frame-level score is as follows: ,in The frame-level score for the escalator contact in the t-th frame image is a confidence level ranging from 0 to 1. The maximum confidence level is set to 1 for dimensionless transformation. The escalator contact smoothness fraction of the t-th frame image is a dimensionless value. The escalator contact smoothness score of the (t-1)th frame image is a dimensionless value. The smoothing coefficient ranges from 0 to 1. The window statistical formula for the escalator contact stability is expressed as follows: ,in Indicates the most recent frame window, This indicates the number of frames within the frame window. This indicates that the indicator function takes a value of 1 if the condition is true and 0 if it is false. The entry threshold value for escalator contact ranges from 0 to 1. The range of the escalator contact stability value is 0 to 1.

[0045] The system connectivity stability is obtained by performing exponential smoothing on the system connectivity frame-level score and calculating the proportion of frames exceeding the entry threshold within the sliding window. The exponential smoothing formula for the system connectivity frame-level score is as follows: ,in The frame-level score for the t-th frame image is defined as a confidence level ranging from 0 to 1. The maximum confidence level is set to 1 for dimensionless transformation. The connectivity smoothness fraction of the t-th frame image is a dimensionless value. The connectivity smoothness fraction of the (t-1)th frame image is a dimensionless value. The smoothing coefficient ranges from 0 to 1. The window statistical formula for the stability of the mooring connectivity is expressed as follows: ,in This refers to the most recent frame window. This indicates the number of frames within the frame window. This indicates that the condition of the indicator function is 1 if true and 0 if false. The threshold value for indicating a connection between the tether and the tether is between 0 and 1. The stability of the connection of the system is defined as ranging from 0 to 1.

[0046] The escalator contact frame score is obtained by calculating the spatial overlap between the person and the ladder based on the coordinates of the person's bounding box and the coordinates of the ladder area. The spatial overlap is obtained by calculating the intersection-union ratio (IUU) of the person's bounding box coordinates and the ladder area coordinates. The larger the IUU is, the higher the degree of contact between the person and the ladder.

[0047] The connection frame-level score is obtained by calculating the connectivity between the seat belt and the human body based on the coordinates of the seat belt anchor point and the smoothed human body key point coordinates. The connectivity is obtained by calculating the Euclidean distance between the coordinates of the seat belt anchor point and the coordinates of the waist key point in the smoothed human body key point coordinates. The smaller the Euclidean distance value, the higher the degree of connection between the seat belt and the human body.

[0048] The temporal causal reasoning network employs a temporal convolutional network (TCN) to extract long-term dependencies. Causal convolution operations ensure that the output at the current moment depends only on past moments and the current input, without using information from future moments. Internally, the network designs a state transition matrix to learn the compliant operation sequence path from the seatbelt fastening state to the ladder climbing state, and uses a reinforcement learning reward mechanism to optimize the accuracy of causal chain recognition. The output of the network consists of semantic segmentation labels for the action sequence and the state transition path encoding, used to determine whether the operation sequence conforms to the compliant operation sequence.

[0049] The specific structure of the temporal causal inference network is as follows: the input layer receives data with a length of... The frame contains the escalator contact stability sequence and the mooring connectivity stability sequence; the temporal convolutional network (TCN) contains 6 causal convolutional layers, each with a kernel size of 3 and dilation rates of 1, 2, 4, 8, 16, and 32 to achieve exponentially growing receptive fields; each causal convolutional layer is followed by weight normalization and ReLU activation functions, and residual connections are used; the fully connected layer maps the feature vector output by the TCN to the state transition matrix with a dimension of the number of states multiplied by the number of states; each element of the state transition matrix represents the transition probability value from state i to state j; the decoding layer uses the Viterbi algorithm to infer the most likely state path based on the state transition matrix and the current observation sequence; the output layer generates the semantic segmentation labels for the action sequence, including unmoored state, moored state, moored but not climbing state, and climbing state.

[0050] The steps for establishing the training dataset for the temporal causal reasoning network specifically include: extracting 3000 segments of data containing the complete high-altitude operation process from historical monitoring videos; labeling each segment with a state sequence according to the timeline, including the preparation stage, the safety harness fastening stage, the inspection stage, the climbing stage, and the operation stage; labeling each segment with a compliance tag to distinguish between compliant and non-compliant operations; recording key time points and transition type labels for state transitions; constructing negative sample data, including sequences of climbing before harnessing and sequences of climbing directly without harnessing; and organizing the labeled data into a sequence-to-sequence format and dividing it into training and validation sets in an 8:2 ratio.

[0051] The specific steps for training the temporal causal inference network include: initializing the network parameters of the temporal causal inference network using the Xavier initialization method; setting the batch size to 16 and the learning rate to 0.0005 using an exponential decay learning rate scheduling strategy; the loss function consists of two parts: a sequence classification loss value and a state transition loss value, with a weight ratio of 1:1; the sequence classification loss value uses the cross-entropy function to calculate the difference between the predicted state label and the true state label; the state transition loss value is calculated through the reinforcement learning reward mechanism, with positive rewards for compliant paths and negative rewards for non-compliant paths; training with the Adam optimizer for a total of 80 rounds, evaluating the sequence accuracy on the validation set every 5 rounds; and saving the weights of the model with the highest sequence accuracy on the validation set as the final inference network parameters.

[0052] The Temporal Convolutional Network (TCN) achieves a large receptive field by stacking multiple causal convolutional layers and introducing a dilation rate. The receptive field of the causal convolution operation with a dilation rate of 'd' in the i-th layer is the kernel size minus 1 multiplied by the dilation rate plus 1. After stacking multiple layers, the receptive field grows exponentially. The causal convolution operation ensures that the output at time t depends only on the input from time 1 to time t. This is achieved by padding the input sequence before the causal convolution operation, with the padding length being the kernel size minus 1 multiplied by the dilation rate.

[0053] The state transition matrix is ​​a square matrix with dimensions equal to the number of state types. The matrix elements of the state transition matrix... The state transition matrix represents the transition probability value from state i to state j. All matrix elements range from 0 to 1, and the sum of the elements in each row is 1. The initial value of the state transition matrix is ​​obtained through statistical analysis of the state sequences in the training dataset, and it is updated and optimized through backpropagation during training. The compliant operation sequence path is defined as a path whose state transition sequence conforms to the safe operation specifications, and the non-compliant path is defined as a path whose state transition sequence does not conform to the safe operation specifications.

[0054] The comprehensive risk score is calculated by weighting four factors: escalator contact stability, tethering connection stability, duration, and operational level parameters. The formula for calculating the comprehensive risk score is as follows: ,in The unit for the escalator contact stability is a proportional value ranging from 0 to 1. The maximum stability of the escalator contact is set to 1 for dimensionless calculation. The unit for the stability of the connection between the mooring and the suspension system is a proportional value ranging from 0 to 1. The maximum stability of the system is set to 1 for dimensionless transformation. The duration is in seconds. The maximum duration is measured in seconds, with a value of 300 for dimensionless conversion. The value range for the job level parameter is 1 to 5. The maximum job level is set to 5 for dimensionless processing. , , , For the weighting coefficients to satisfy , The comprehensive risk score is a dimensionless value ranging from 0 to 1.

[0055] The evidence package data includes five elements: the keyframe image, the violation text explanation, the violation timestamp, the segment index, and the device location information. The keyframe image is a screenshot of a video frame at the moment the violation occurred. The violation text explanation consists of the violation description text and suggested text generated by the multimodal large model. The violation timestamp records the start and end times of the violation to the second. The segment index identifies the location of the violation segment in the original monitoring dataset. The device location information includes the camera number and the name of the monitoring area.

[0056] The spatiotemporal feature adjustment function calculates the feature extraction depth adjustment value based on three data points: the motion intensity value, the illumination change rate value, and the target density value of the current frame. The calculation formula for the spatiotemporal feature adjustment function is as follows: ,in The motion intensity value of the current frame is expressed in pixel displacement. The reference motion intensity value is in pixels, and the displacement value is 50 for dimensionless conversion. The illumination change rate value for the current frame is expressed in units of brightness difference. The reference value for the rate of change of illumination is in units of brightness difference, with a value of 30 for dimensionless calculation. The target density value for the current frame is expressed in units of units per square meter. The target density value is set to 5 units per square meter for dimensionless calculation. , , For the weighting coefficients to satisfy , The feature extraction depth adjustment value is a dimensionless value.

[0057] When the feature extraction depth adjustment value When the data falls within different ranges, different residual block group depths are used to adjust the layer parameter for extracting the spatiotemporal features from the 3D convolutional kernel in the multi-scale spatiotemporal fusion detection network. When using the shallow feature extraction mode to activate the first and second residual block groups of the ResNet3D-50, the total number of layers is 20; When using the standard feature extraction mode to activate the first, second, and third residual block groups of the ResNet3D-50, the total number of layers is 35; When using the deep feature extraction mode, all four residual block groups of the ResNet3D-50 are activated, with a total of 50 layers.

[0058] The motion intensity value is obtained by calculating the average amplitude of the optical flow field between the current frame and the previous frame. The optical flow field describes the motion vector of each pixel in the image, and the amplitude of the optical flow field represents the magnitude of the motion speed. The larger the motion intensity value, the more intense the motion in the image, requiring a deeper multi-scale spatiotemporal fusion detection network to extract the spatiotemporal features to capture complex motion patterns.

[0059] The illumination change rate value is obtained by calculating the average brightness difference between the current frame and the previous frame, where the average brightness is the average grayscale value of all pixels in the image. A larger illumination change rate value indicates more drastic fluctuations in scene lighting conditions, requiring adjustment of the feature extraction depth adjustment value to adapt to target detection under different lighting conditions.

[0060] The target density value is obtained by summing the number of people, ladders, and seat belts detected in the current frame and dividing by the area of ​​the monitored region. A higher target density value indicates that there are more targets to be tracked in the image, requiring a deeper multi-scale spatiotemporal fusion detection network to extract rich semantic features to distinguish multiple targets.

[0061] The automatic closed-loop fallback mechanism continuously monitors the escalator contact stability and the mooring connection stability for violations. When both the escalator contact stability and the mooring connection stability are below the exit threshold and the fallback duration exceeds the preset minimum fallback duration, the system determines that the violation has been resolved. The system automatically disables the alarm and falls back from the high frame rate sampling mode to the normal monitoring mode. The exit threshold is set to 80% of the entry threshold to create a hysteresis characteristic and avoid frequent switching.

[0062] As a preferred embodiment, the present invention also provides a computer-based method for determining the safety risks of high-altitude operations in power plants, which is based on the collaboration of a multimodal large model and a small model. The computer is equipped with a readable storage medium, which stores program instructions. When the program instructions are run on the computer, they are used to execute the aforementioned method for determining the safety risks of high-altitude operations in power plants based on the collaboration of a multimodal large model and a small model.

[0063] The specific implementation methods of the above steps are described in detail below.

[0064] The specific implementation of step S1 is as follows: First, real-time video stream data of the work area is collected through a monitoring camera at a frame rate of 25 frames per second and a resolution of 1920×1080 pixels. Then, the collected video stream is subjected to image stabilization preprocessing. An image registration algorithm based on feature point matching is used to eliminate the image shift caused by camera shake. Frame alignment is achieved by extracting SIFT feature points between adjacent frames and calculating the homography matrix. Next, time synchronization preprocessing is performed to synchronize the timestamp of the video stream with the Network Time Protocol (NTP) server to ensure that the time accuracy is within 50 milliseconds. At the same time, log records with corresponding timestamps are extracted from the equipment operation log system, including equipment status identifiers, operator numbers, and job task numbers. The video data and log data are associated according to timestamps to form a time-aligned original monitoring dataset, providing stable and reliable data input for subsequent adaptive frame extraction and target detection.

[0065] The specific implementation of step S2 is as follows: First, the original monitoring dataset is sampled globally using a low frame rate sampling method, with the low frame rate set to 2 frames per second. For each frame, three lightweight indicators are calculated: sharpness score, motion score, and attention score. The sharpness score is obtained by performing a second-order differential operation on the grayscale image using the Laplacian operator to obtain the edge response intensity; the larger the variance value, the richer the image details. The motion score is obtained by comparing the similarity of the brightness, contrast, and structure components of two adjacent frames using a structural similarity algorithm; the lower the similarity, the more significant the inter-frame changes. The attention score is obtained by performing fast target detection on the low-resolution image using a lightweight YOLO-nano network, extracting the detection confidence of people, ladders, and safety belts, and taking the maximum value. Then, the three scores are input into a reinforcement learning network that dynamically adjusts the trigger threshold for real-time evaluation. This network is based on Double... The DQN architecture learns the optimal triggering strategy by maintaining an experience replay buffer and a dual-network structure. When any score exceeds a dynamically adjusted trigger threshold and remains stable for 5 consecutive frames, high frame rate sampling is triggered. Upon triggering, a circular buffer supplementation mechanism is immediately started, replaying the first 5 seconds of low frame rate sampled frames stored in the circular buffer, and switching to high frame rate sampling mode to continuously sample frames at a rate of 15 frames per second until all scores fall below the trigger threshold, at which point it falls back to low frame rate sampling. The reference value of the trigger threshold is between 0.65 and 0.75, which is adaptively adjusted according to the scene.

[0066] The specific implementation of step S3 involves inputting the high frame rate sampled frame sequence into a multi-scale spatiotemporal fusion detection network in groups of 16 consecutive frames. This network uses ResNet3D-50 as its backbone, extracting features simultaneously in both time and space through 3D convolution to capture the motion trajectory and pose changes of the person. Internally, the network incorporates a Feature Pyramid Network (FPN) to extract multi-scale features from residual block groups of different depths. Shallow features retain high-resolution spatial information for accurate localization, while deep features provide rich semantic information for target classification. A Temporal Attention Module (TAM) is inserted before the detection head to apply attention weights to the feature maps at different times, highlighting... The system identifies significant temporal features of motion; it detects the bounding box coordinates of the person, the anchor point coordinates of the seatbelt, the ladder area coordinates, and the coordinates of 17 key points on the human body, including the head, shoulders, elbows, wrists, hips, knees, and ankles; then, it applies a temporally consistent Kalman filter to the detected key point coordinates, with the filter maintaining the position and velocity state vectors of each key point. A prediction step extrapolates the coordinates of the next frame based on a uniform motion model, and an update step fuses the detected and predicted values, dynamically adjusting the Kalman gain based on the key point confidence level to eliminate coordinate jumps caused by limb occlusion, resulting in smooth and continuous key point coordinates.

[0067] The specific implementation of step S4 is as follows: First, the intersection-union ratio (IUU) is calculated based on the coordinates of the personnel bounding box and the ladder area to obtain the escalator contact frame-level score. The IUU is the area of ​​the intersection of the personnel frame and the ladder frame divided by the area of ​​the union, with a value ranging from 0 to 1. Then, the Euclidean distance is calculated based on the coordinates of the safety belt anchor point and the waist key point in the smoothed human body key point coordinates to obtain the harness connectivity frame-level score. The smaller the distance, the higher the harness connection. Next, exponential smoothing is performed on the escalator contact frame-level score and the harness connectivity frame-level score respectively, with the smoothing coefficient set to 0.3, so that the current frame score and the historical score are merged with a weight of 3:7 to suppress instantaneous fluctuations. Then, within a sliding window of 30 frames, the proportion of frames whose smoothed scores exceed the entry threshold is counted. The reference value for the entry threshold is 0.6, and the escalator contact stability and mooring connection stability are obtained. When any stability exceeds 0.7 and lasts for 2 seconds, the trigger condition is determined to be met. Keyframe images and temporal summary data from the high frame rate sampling frame sequence are extracted and input into the temporal causal inference network. The network adopts the temporal convolutional network (TCN) architecture, which extracts long-term dependencies through 6 layers of causal convolution. Causal convolution ensures that the output at the current moment only depends on historical information and does not use future information. The expansion rate increases exponentially so that the network receptive field covers the entire window. The network outputs a state transition path code to identify whether the action sequence conforms to the compliant transition order from the unmoored state to the moored state, then to the moored but not climbing state, and finally to the climbing state.

[0068] The specific implementation of step S5 involves parsing the state transition path encoding output by the temporal causal reasoning network, determining whether there are any illegal paths in the state transition sequence where the unsecured state directly transitions to the climbing state or the secured state transitions to the climbing state. If an illegal path is detected, video clips 3 seconds before and after the time of the violation are extracted from the high frame rate sampled frame sequence as illegal fragment images. At the same time, the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area, and the coordinates of the smoothed human key points are extracted as evidence element data. The illegal fragment images and evidence element data are encoded into a multimodal input format, where the image data is converted to base64 encoding and the coordinate data is organized in JSON format. The data is submitted to a multimodal large model for compliance review. The large model receives the image sequence and structured data and understands the temporal logic of the personnel's actions. It outputs a compliance review result indicating whether the behavior violates the safety regulations for high-altitude operations, and generates a textual explanation of the violation, describing the type of violation, the time of the violation, and the potential risks.

[0069] The specific implementation of step S6 involves using the compliance review results output by the multimodal large model. If a violation is determined, the escalator contact stability, tethering connectivity stability, duration, and operational level parameters are extracted and weighted to calculate a comprehensive risk score. The weighting coefficients are set to 0.3, 0.3, 0.25, and 0.15, respectively. The comprehensive risk score ranges from 0 to 1, with higher values ​​indicating more severe risks. Then, evidence package data is constructed, keyframe images are converted to JPEG format for compression and storage, violation text explanations are encoded into UTF-8 text, and violation timestamps are recorded to the second level, generating... The fragment index identifies the start and end frame numbers of the violation fragment in the original monitoring dataset, extracts equipment location information including camera number and monitoring area name, writes the evidence package data into the database for archiving and triggers an alarm notification, and simultaneously activates an automatic closed-loop fallback mechanism to continuously monitor escalator contact stability and tethering connection stability. When both are below the exit threshold of 0.48 and the fallback duration exceeds 5 seconds, it is determined that the violation status has been restored, the alarm is automatically turned off, and the monitoring mode is switched from high frame rate sampling to low frame rate normal monitoring mode. The exit threshold is set to 80% of the entry threshold to form a hysteresis characteristic to avoid frequent switching.

[0070] It should be noted that the key technical ideas of this invention include an adaptive two-stage frame extraction mechanism based on reinforcement learning, multi-scale spatiotemporal fusion detection and temporal consistency filtering collaboration, and action sequence semantic segmentation using a temporal causal reasoning network. The reinforcement learning-based adaptive two-stage frame extraction mechanism learns the optimal trigger threshold strategy under different work scenarios and environmental conditions through a Double DQN network, overcoming the limitation of traditional fixed threshold methods that cannot adapt to scene differences. It achieves a dynamic balance of trigger sensitivity by constructing a reward function based on historical trigger accuracy and computational cost, avoiding both the problem of missing critical moments due to low frame rates and the waste of computational power caused by excessively long high frame rate durations. The multi-scale spatiotemporal fusion detection and temporal consistency filtering collaboration mechanism combines 3D convolutional spatiotemporal feature extraction with Kalman filter kinematic constraints. 3D convolution captures the movement patterns of personnel simultaneously in both time and space dimensions. The feature pyramid network fuses multi-scale information to balance positioning accuracy and semantic richness. The temporal consistency Kalman filter applies smoothing constraints to the detection results using the motion continuity assumption, effectively solving the problem of key point coordinate jumps caused by limb self-occlusion when high-altitude workers are in a distorted posture. The temporal causal reasoning network extracts long-term dependencies through causal convolution and dilated convolution in a temporal convolutional network. The state transition matrix learns the temporal logic of compliant operation paths, overcoming the shortcomings of traditional sliding window statistical methods that only calculate frame-level score averages and cannot distinguish the order of actions. It accurately identifies the temporal causal differences between climbing before securing and securing before climbing. The synergistic effect of these three key technical approaches constructs a complete technical chain from data acquisition optimization to feature extraction enhancement and temporal logic reasoning. Adaptive frame extraction provides high-quality input data for subsequent detection, multi-scale fusion detection provides stable and reliable keypoint coordinates for temporal reasoning, and temporal causal reasoning transforms spatial detection results into temporal semantic understanding. The collaboration of these three elements enables accurate assessment of safety risks in high-altitude operations at power plants, significantly reducing false positive and false negative rates compared to traditional methods.

[0071] It should be noted that this invention also solves the following technical problem: Existing technologies suffer from a lack of interpretability in the detection results of violations during high-altitude operations, making it difficult for regulators to quickly pinpoint the causes of violations. Traditional deep learning detection methods only output a binary judgment of violation or compliance, failing to provide a specific description of the violation and a chain of evidence. Regulators need to manually review videos to confirm the details of the violation, reducing regulatory efficiency. This invention uses a temporal causal reasoning network to semantically segment the operation sequence and extract state transition path codes. It compares the detected violations with the compliant operation sequence in the safety operation specifications, clearly indicating which stage of the operation the violation occurred in. Furthermore, it submits the violation fragment images and evidence element data to a multimodal large model for compliance review. Utilizing the large model's language generation capabilities, it automatically generates a textual explanation of the violation, including a violation timestamp, a description of the violation action, and safety recommendations. The keyframe images, the textual explanation of the violation, the violation timestamp, the fragment index, and equipment location information are integrated into a structured evidence package and archived in a database. This provides regulators with a complete chain of evidence and traceable review basis, significantly improving the efficiency and accuracy of handling violations.

[0072] Specifically, the principle of this invention is as follows: The invention solves the aforementioned technical problems by employing a combination of adaptive sampling and hierarchical detection. The reinforcement learning-based dynamic trigger threshold adjustment network constructs a reward function based on historical trigger accuracy and computational cost. During training, it learns the optimal trigger threshold strategy for different operational scenarios, enabling the system to reduce sampling frequency during low-risk periods to save computational power, and immediately increase the sampling frequency when a suspected violation is detected to ensure complete capture of the violation process. The circular buffering mechanism automatically backtracks the frame sequence before triggering high frame rate sampling, compensating for the keyframe omission problem caused by trigger delay in traditional methods. The multi-scale spatiotemporal fusion detection network uses three-dimensional convolutional kernels to extract features simultaneously in the time and spatial dimensions. Through a temporal attention module, it dynamically weights feature maps at different times, enhancing the detection capability of moving targets and enabling the system to accurately identify seatbelt fastening status and climbing posture. The temporal causal reasoning network learns the state transition paths in safety operating procedures through causal convolution operations and uses the Viterbi algorithm to infer the most likely operation sequence. This achieves causal logic judgment of violations rather than simple appearance feature matching, improving judgment accuracy and reducing false alarm rate.

[0073] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0074] The specific implementation of step S2 involves adaptive two-stage frame extraction of the original monitoring dataset, followed by real-time calculation of the sharpness score, motion score, and attention score using a reinforcement learning-based dynamic threshold adjustment network. The formula for calculating the sharpness score is as follows:

[0075] ;

[0076] In the formula, Let be the grayscale intensity matrix of the t-th frame image, in units of grayscale values; The Laplace operator is implemented using a 3×3 convolution kernel; This is the variance operator; The grayscale intensity matrix of the reference clear image, in units of grayscale values, is obtained through offline calibration; Let be the sharpness score of the t-th frame, and be a dimensionless value. The formula for calculating the motion score is as follows:

[0077] ;

[0078] In the formula, This represents the structural similarity function, with an output range of 0 to 1. This is the image of the (t-1)th frame; Let be the motion score of the image in frame t, which is a dimensionless value. The formula for calculating the attention score is as follows:

[0079] ;

[0080] In the formula, This represents the confidence value of personnel located in the elevated work area, ranging from 0 to 1, and is output by the fast target detection network. This represents the ladder confidence value, ranging from 0 to 1, output by the fast object detection network; This represents the seatbelt confidence value, ranging from 0 to 1, and is output by the fast target detection network. This indicates pre-marked areas of interest related to the risks of climbing. Let be the attention score of the image in frame t, a dimensionless value. High frame rate sampling is initiated when any of the sharpness score, motion score, or attention score exceeds the dynamic trigger threshold and remains stable. The high frame rate sampling rate is 30 frames per second, and the low frame rate sampling rate is 2 frames per second. The circular buffer backup duration is [length missing]. A circular buffer continuously stores the latest low frame rate sampled frame. When a high frame rate sampling is triggered, the contents of the circular buffer are first replayed to fill in the frame before the trigger. A sequence of frames per second is used, followed by continuous high frame rate sampling in subsequent videos until all scores fall below the dynamic trigger threshold, at which point the sampling rate drops back to low frame rate. The empirical value is 3 to 10 seconds.

[0081] The specific implementation of step S3 involves inputting a high frame rate sampled frame sequence into a multi-scale spatiotemporal fusion detection network for target detection and pose estimation. This network uses ResNet3D-50 as its backbone to extract spatiotemporal features and captures motion patterns through simultaneous convolution operations in both the temporal and spatial dimensions using 3D convolutional kernels. The input layer receives a sequence of 16 consecutive RGB image frames, each with a size of 224×224×3. ResNet3D-50 contains four residual block groups, each using a 3×3×3 3D convolutional kernel. The feature pyramid network extracts residual block features from the second, third, and fourth residual block groups of ResNet3D-50 and fuses them through upsampling and lateral connection operations to form a five-layer feature pyramid. The temporal attention module applies softmax attention weights to each layer of the feature pyramid in the temporal dimension. The detection head is divided into classification and regression branches, which output target category probability values ​​and personnel bounding box coordinates, respectively. The pose estimation branch uses the HRFormer structure to output the heatmap response of 17 human keypoint coordinates. In the post-processing stage, non-maximum suppression is used to remove duplicate detection boxes, and the intersection-over-union (IoU) threshold is set to 0.5. The output includes personnel bounding box coordinates, seat belt anchor point coordinates, ladder area coordinates, and human keypoint coordinates. The human keypoint coordinates are smoothed by temporal consistent Kalman filtering. The temporal consistent Kalman filtering maintains the position and velocity state vectors of each human keypoint. The state prediction step extrapolates the predicted coordinates of the next frame based on a uniform motion model or a uniformly accelerated motion model. The observation update step fuses the human keypoint coordinates and the predicted coordinates of the next frame and adjusts the Kalman gain weights according to the keypoint confidence value to obtain the smoothed human keypoint coordinates.

[0082] The specific implementation of step S4 involves performing exponential smoothing and window statistical calculations on the coordinates of the personnel bounding box, the safety belt anchor point, the ladder area, and the smoothed human key point coordinates to obtain the escalator contact stability and the tethered connectivity stability. The escalator contact frame-level score is obtained by calculating the spatial overlap between the personnel and the ladder based on the personnel bounding box coordinates and the ladder area coordinates. The spatial overlap is obtained by calculating the intersection-union ratio (IUU) of the personnel bounding box coordinates and the ladder area coordinates; a larger IUU indicates a higher degree of contact between the personnel and the ladder. The exponential smoothing formula for the escalator contact frame-level score is as follows:

[0083] ;

[0084] In the formula, is the escalator contact frame-level score of the t-th frame image, in confidence level, with a value from 0 to 1; The maximum confidence level is set to 1, and it is used for dimensionless transformation. Let be the escalator contact smoothness fraction of the t-th frame image, which is a dimensionless value; Let be the escalator contact smoothness score of the (t-1)th frame image, which is a dimensionless value; The smoothness coefficient for escalator contact ranges from 0 to 1, with an empirical value of 0.7. The window statistical formula for escalator contact stability is expressed as follows:

[0085] ;

[0086] In the formula, Represents the most recent frame window for escalator contact statistics; This indicates the number of frames within the frame window used for escalator contact statistics; the empirical value is 30 frames. This indicates an indicator function; the condition is 1 for true and 0 for false. This represents the entry threshold for escalator contact, ranging from 0 to 1, with an empirical value of 0.6. The escalator contact stability is represented by a value ranging from 0 to 1. The harness connectivity frame-level score is calculated based on the anchor point coordinates of the safety belt and the smoothed key point coordinates of the human body. Connectivity is obtained by calculating the Euclidean distance between the anchor point coordinates of the safety belt and the smoothed key point coordinates of the human body, specifically the waist key point coordinates. A smaller Euclidean distance indicates a higher degree of harness connectivity between the safety belt and the human body. The exponential smoothing formula for the harness connectivity frame-level score is as follows:

[0087] ;

[0088] In the formula, is the hook connectivity frame-level score of the t-th frame image, in confidence level, with a value from 0 to 1; The maximum confidence level is set to 1, and it is used for dimensionless transformation. Let be the t-th frame image's connectivity smoothness fraction, which is a dimensionless value; Let be the connectivity smoothness fraction of the image in frame t-1, which is a dimensionless value; The smoothing coefficient for mooring connectivity ranges from 0 to 1, with an empirical value of 0.7. The window statistical formula for mooring connectivity stability is expressed as follows:

[0089] ;

[0090] In the formula, Represents the most recent frame window for connectivity statistics; This represents the number of frames within the frame window used for connectivity statistics; the empirical value is 30 frames. This represents the threshold for entering the system connection, with a value ranging from 0 to 1, and an empirical value of 0.65. The stability of the escalator's connection is defined as 0 to 1. When either the escalator's contact stability or the connection stability exceeds the threshold and the duration reaches the preset minimum trigger duration (empirically 2 seconds), keyframe images and temporal summary data from the high-frame-rate sampled frame sequence are input into the temporal causal inference network for action sequence semantic segmentation and causal relationship determination. The temporal causal inference network uses a temporal convolutional network to extract long-term dependencies, ensuring that the output at the current moment depends only on the inputs at the past and current moments through causal convolution operations. The temporal convolutional network contains 6 causal convolutional layers, each with a kernel size of 3, and dilation rates of 1, 2, 4, 8, 16, and 32 to achieve exponentially growing receptive fields. The fully connected layer maps the feature vector output by the temporal convolutional network to a state transition matrix, with a dimension equal to the number of states multiplied by the number of states (empirically 4). The decoding layer uses the Viterbi algorithm to infer the most likely state path based on the state transition matrix and the current observation sequence, and outputs semantic segmentation labels for the action sequence, including unattached state, attached state, attached but not climbing state, and climbing state, thus obtaining the state transition path encoding.

[0091] The specific implementation of step S6 involves calculating a comprehensive risk score based on the compliance review results output by the multimodal large model, combined with parameters such as escalator contact stability, tether connectivity stability, duration, and operational level. The formula for calculating the comprehensive risk score is as follows:

[0092] ;

[0093] In the formula, The maximum stability of the escalator contact is set to 1 for dimensionless calculation. The maximum stability of the connection between the suspension and the device is set to 1, which is used for dimensionless transformation. Duration, in seconds; The maximum duration, in seconds, is 300 and is used for dimensionless conversion. The job level parameter is coded, with a value range of 1 to 5; This is the highest job level, with a value of 5, used for dimensionless processing; , , , For the weighting coefficients, satisfying The empirical values ​​are 0.3, 0.3, 0.2, and 0.2 respectively. The comprehensive risk score is a dimensionless value ranging from 0 to 1. Evidence packages containing keyframe images, explanations of violations, violation timestamps, and equipment location information are automatically generated and archived. These evidence packages include five elements: keyframe images, explanations of violations, violation timestamps, segment indexes, and equipment location information. When both the escalator contact stability and the tethering connectivity stability are below the exit threshold and the duration of the fall exceeds the preset minimum fall time, the system automatically closes the loop and returns to normal monitoring mode. The exit threshold is set to 80% of the entry threshold to create a hysteresis characteristic and avoid frequent switching. The preset minimum fall time is empirically set to 3 seconds.

[0094] To better understand and implement this invention, the following is a specific application scenario of this invention, Example 2:

[0095] To verify the effectiveness of this invention, technicians set up a test environment and conducted practical application tests by deploying a monitoring system in the boiler maintenance work area of ​​a power plant. This work area included three maintenance platforms with heights of 12m, 18m, and 25m respectively, equipped with four network cameras with a resolution of 1920×1080 and a video stream frame rate of 25fps. The test period was selected during seven consecutive days of operation during the equipment overhaul, collecting a total of 168 hours of monitoring video data.

[0096] Technicians first performed image stabilization preprocessing on the acquired monitoring video stream, using a feature point matching-based image stabilization algorithm to eliminate image shifts caused by camera shake. Time synchronization preprocessing synchronized the video timestamps with the equipment operation log timestamps, controlling the time deviation within 100ms. Simultaneously, operation logs from key equipment such as boiler blowers, induced draft fans, and feedwater pumps were acquired to form a time-aligned raw monitoring dataset. In the adaptive two-stage frame extraction stage, a reinforcement learning network that dynamically adjusts the trigger threshold was pre-trained based on 5000 hours of historical training data. The network input included a 30-frame sliding window sequence of sharpness score, a 30-frame sliding window sequence of motion score, a 30-frame sliding window sequence of attention score, a work scene type encoding set to boiler maintenance scene code 2, and an environmental parameter vector containing light intensity of 450 lux, ambient temperature of 32℃, and humidity of 68%.

[0097] During a task at 10:15 AM on the third day, the system detected that the sharpness score reached 0.87, the motion score reached 0.64, and the attention score reached 0.92. All three scores exceeded the dynamic trigger threshold of 0.75 and remained stable for more than 2 seconds. The system immediately initiated high frame rate sampling, increasing the frame rate from the normal 2fps to 25fps. Simultaneously, the circular buffer recording mechanism played back the low frame rate sampled frames from the previous 5 seconds to complete the frame sequence before the trigger. Figure 2As shown, the adaptive adjustment process of the dynamic trigger threshold under different job scenarios demonstrates the reinforcement learning network's ability to balance computational consumption and detection accuracy. High frame rate sampling continued until the end of the job, resulting in a total of 3780 high frame rate sampled frame sequences.

[0098] The multi-scale spatiotemporal fusion detection network receives a sequence of 16 consecutive RGB image frames as input. After the ResNet3D-50 backbone network extracts spatiotemporal features, the feature pyramid network extracts residual block features from the 2nd, 3rd, and 4th residual block groups and fuses them to form a 5-layer feature pyramid. The temporal attention module applies attention weights to each layer of the feature pyramid in the time dimension, giving higher weight coefficients to frames with rapid motion. The detection head outputs the coordinates of the person bounding box. Pixel coordinates and seat belt anchor point coordinates are Pixel coordinates, ladder area coordinates are The pixel coordinates and coordinates of 17 human keypoints, including the top of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles, along with their keypoint confidence values, are obtained. Temporally consistent Kalman filtering applies kinematic constraints to the human keypoint coordinates. The state prediction step extrapolates the predicted coordinates for the next frame, and the observation update step fuses the detected coordinates and predicted values, adjusting the Kalman gain weights based on the keypoint confidence values. This successfully eliminates coordinate jumps caused by occlusion, resulting in smoothed human keypoint coordinates.

[0099] Based on the smoothed human body keypoint coordinates, the system calculates the escalator contact frame score and the harness connectivity frame score. The escalator contact frame score is calculated as 0.78 using the intersection-union ratio (OCR) of the human bounding box coordinates and the escalator area coordinates. The harness connectivity frame score is calculated using the coordinates of the safety belt anchor point and the waist keypoint coordinates within the smoothed human body keypoint coordinates. The Euclidean distance value of the pixel coordinates was calculated, with an Euclidean distance value of 18 pixels corresponding to a mooring connectivity frame-level score of 0.92. Exponential smoothing was applied using a smoothing coefficient of 0.6 to smooth the frame-level scores, resulting in an escalator contact smoothness score of 0.75 and a mooring connectivity smoothness score of 0.89. Window statistics were performed using the most recent 120 frames. The percentage of frames within the window whose escalator contact smoothness score exceeded the entry threshold of 0.65 yielded an escalator contact stability of 0.83, and the percentage of frames within the window whose mooring connectivity smoothness score exceeded the entry threshold of 0.70 yielded a mooring connectivity stability of 0.76.

[0100] Because both the escalator contact stability (0.83) and the mooring connectivity stability (0.76) exceeded the entry threshold and lasted for 8 seconds, exceeding the preset minimum trigger duration of 5 seconds, the system inputs keyframe images and temporal summary data from the high frame rate sampling frame sequence into the temporal causal inference network. The temporal causal inference network receives the escalator contact stability sequence and mooring connectivity stability sequence within a window of 150 frames. It extracts long-term dependencies through 6 causal convolutional layers with dilation rates of 1, 2, 4, 8, 16, and 32, respectively, ensuring the receptive field covers the entire 150-frame window. The fully connected layer maps the feature vector output by the temporal convolutional network into a 4×4 state transition matrix. The matrix elements represent the transition probability values ​​between the unmoored state, the moored state, the moored but not climbing state, and the climbing state. As shown in Table 1, the actual detected state transition sequence is an unmoored state lasting 12 seconds and a climbing state lasting 26 seconds, without any intermediate steps between the moored state and the moored but not climbing state.

[0101] Table 1. Detection Results of State Transition Sequences

[0102]

[0103] The Viterbi algorithm infers the most probable state path based on the state transition matrix and the current observation sequence, encoding the state transition path as a direct jump from the unmoored state to the climbing state. The system compares the compliant operation sequence path, which should be a complete flow from the unmoored state to the mooring state to the moored but not climbing state to the climbing state. If the system determines that the operation sequence does not conform to the compliant operation sequence, it will be deemed not to be compliant. The temporal causal reasoning network ensures, through causal convolution operations, that the output at the current moment depends only on the inputs at the past and current moments, effectively identifying the violation pattern of climbing before mooring.

[0104] The system submitted the violation footage, including three keyframe images (frames 752, 968, and 1245), and evidence data, including personnel bounding box coordinates, safety belt anchor point coordinates, ladder area coordinates, human body key point coordinates, escalator contact stability sequence, and harness connectivity stability sequence, to a multimodal large-scale model for compliance review. Based on the image content and evidence data analysis, the multimodal large-scale model confirmed the violation and generated a textual explanation: "The system detected that a worker climbed the ladder directly into the 18m maintenance platform without wearing a safety belt, violating Article 3 of the power plant's high-altitude work safety operating procedures, which mandates that safety belts be worn before climbing. The system recommended immediately stopping the work and providing safety training to the worker."

[0105] The system calculates a comprehensive risk score based on the compliance review results output by the multimodal large model, combined with the escalator contact stability (0.83), tethering connection stability (0.76), duration (26 seconds), and the boiler maintenance high-altitude operation with a work level parameter code of 4. The weighting coefficients are set to... , , , The calculated comprehensive risk score was 0.68. The system automatically generated evidence package data, which included 3 keyframe images, 1 paragraph of text explanation of the violation, a violation timestamp from 10:15:20 to 10:15:46, a segment index of the 10th hour, 15th minute, and 20th second segment of the 3rd day of the monitoring video, and equipment location information of the 18m platform area east of the boiler on camera No. 3. The evidence package data was stored and archived in the safety monitoring database.

[0106] After the worker successfully fastened their safety belt, the system continuously monitored changes in the escalator contact stability and safety belt connection stability. Starting at 10:16:05, the escalator contact stability gradually decreased to 0.42, while the safety belt connection stability remained at 0.81, indicating that the worker had boarded the platform and correctly fastened their safety belt. At 10:16:18, the escalator contact stability dropped to 0.35, below the exit threshold of 0.52, and the safety belt connection stability dropped to 0.48, below the exit threshold of 0.56. Both stability values ​​were below the exit threshold, and the decline lasted for 15 seconds, exceeding the preset minimum decline time of 10 seconds. The system determined that the violation had been resolved. The automatic closed-loop decline mechanism was activated, the system disabled alarms, and the monitoring mode was reduced from high frame rate sampling (25fps) to normal monitoring mode (2fps), completing a full violation detection and closed-loop processing cycle.

[0107] The advancement of this invention compared to traditional methods lies in its intelligent allocation of monitoring resources through a dynamic adjustment network of trigger thresholds using reinforcement learning. This avoids the high false alarm rate and high computational cost associated with fixed-threshold methods in complex operational scenarios. Traditional methods typically employ fixed-threshold triggering mechanisms, which cannot adapt to scene changes under varying lighting conditions, job types, and environmental parameters. This results in numerous invalid triggers consuming computational resources or insufficient trigger sensitivity leading to missed violations. This invention, through a Double DQN architecture, learns a balance strategy between historical trigger accuracy and computational cost, dynamically adjusting the trigger threshold based on the current scene. This significantly reduces computational cost while maintaining detection accuracy. The multi-scale spatiotemporal fusion detection network, combining a ResNet3D-50 backbone network and a feature pyramid network, captures motion patterns by simultaneously performing convolution operations in the temporal and spatial dimensions using 3D convolutional kernels. Compared to traditional 2D detection networks that only focus on single-frame image information, this network can better identify subtle pose changes in continuous action sequences. The temporal causal reasoning network ensures that the output depends only on the inputs of the past and present moments through causal convolution operations, effectively identifying the sequence of violations such as climbing before tying, overcoming the limitations of traditional rule-based matching methods in handling complex temporal dependencies. The automatic closed-loop fallback mechanism avoids frequent switching of the system near threshold boundaries through its hysteresis characteristic. Compared to traditional fixed monitoring modes, it can promptly release computing resources and return to normal monitoring after the violation state is restored, improving the overall system stability and resource utilization efficiency.

[0108] It should be noted that the variables involved in this invention are explained in detail in Table 2.

[0109] Table 2 Variable Explanation Table

[0110]

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining the safety risks of high-altitude operations in power plants based on the synergy of multimodal large and small models, characterized in that, Includes the following steps: Acquire the monitoring video stream of the work area and perform image stabilization preprocessing and time synchronization preprocessing. At the same time, acquire the device operation logs with corresponding timestamps to form a time-aligned raw monitoring dataset. An adaptive two-stage frame extraction process is performed on the original monitoring dataset. A reinforcement learning-based dynamic trigger threshold network calculates sharpness score, motion score, and attention score in real time. When any of these scores exceeds the dynamic trigger threshold and remains stable, high frame rate sampling is initiated, and a circular buffer is used for supplementary recording, resulting in a high frame rate sampled frame sequence. This high frame rate sampled frame sequence is then input into a multi-scale spatiotemporal fusion detection network for target detection and pose estimation. The network outputs coordinates of personnel bounding boxes, seatbelt anchor points, ladder areas, and human keypoints, which are then processed using a temporal consistency Kalman filter. The wave smooths the coordinates of key human body points to obtain smoothed human body key point coordinates; exponential smoothing and window statistical calculations are performed on the coordinates of the personnel bounding box, the coordinates of the safety belt anchor point, the coordinates of the ladder area, and the smoothed human body key point coordinates to obtain the escalator contact stability and the mooring connection stability. When either the escalator contact stability or the mooring connection stability exceeds the entry threshold and the duration reaches the preset minimum trigger duration, the key frame images and temporal summary data in the high frame rate sampled frame sequence are input into the temporal causal reasoning network for action sequence semantic segmentation and causal relationship determination to obtain the state transition path encoding. The operation sequence is determined based on the state transition path code to determine whether it conforms to the compliant operation sequence. If the operation sequence does not conform to the compliant operation sequence, the violation fragment image and evidence element data are submitted to the multimodal big model for compliance review and a violation text explanation is generated. When the escalator contact stability and the tethering connection stability are both lower than the exit threshold and the fall duration exceeds the preset minimum fall duration, the loop is automatically closed and the system falls back to the normal monitoring mode.

2. The method according to claim 1, characterized in that, The reinforcement learning-based dynamic trigger threshold adjustment network adopts a Double DQN architecture. It constructs a reward function value based on historical trigger accuracy and computational cost. The input includes a sliding window sequence of clarity score, a sliding window sequence of motion score, a sliding window sequence of attention score, a task scenario type encoding, and an environmental parameter vector. The output is an estimate of the action value for the dynamic trigger threshold.

3. The method according to claim 2, characterized in that, The reinforcement learning-based dynamic trigger threshold adjustment network has an internal experience playback buffer for storing state transition quadruples. It adopts a dual-network structure of target network and evaluation network to reduce overestimation bias, and introduces a confidence interval estimation module to dynamically adjust the trigger sensitivity coefficient based on historical variance values.

4. The method according to claim 3, characterized in that, The steps for establishing the training dataset for a reinforcement learning network that dynamically adjusts the trigger threshold are as follows: collect historical monitoring video data under different work scenario categories, different time period categories, and different weather condition categories; label the start and end times and violation type labels of real violation events; and record the trigger time data, false trigger counts, and missed detection counts under different fixed threshold values.

5. The method according to claim 1, characterized in that, The sharpness score is obtained by calculating the variance value after applying the Laplacian operator to the gray intensity matrix of the frame image. The Laplacian operator performs a second-order difference operation on the frame image to highlight the edge sharpness. The larger the variance value, the sharper the frame image.

6. The method according to claim 1, characterized in that, The motion score is obtained by calculating the structural similarity value between a frame image and the previous frame image and taking its complement. A structural similarity value of 1 indicates that the two frames are completely identical, and the larger the complement value, the more obvious the changes between frames.

7. The method according to claim 1, characterized in that, Attention score is calculated by using a fast target detection network to detect people, ladders, and safety belts in low-resolution images. The maximum value among the confidence values ​​of the person, ladder, and safety belt in the high-altitude work area is extracted as the attention score.

8. The method according to claim 1, characterized in that, The circular buffer is maintained for a preset duration and continuously stores the latest low frame rate sampled frames. When any of the sharpness score, motion score, or attention score triggers high frame rate sampling, the contents of the circular buffer are first replayed to complete the frame sequence before the trigger.

9. The method according to claim 1, characterized in that, The multi-scale spatiotemporal fusion detection network uses ResNet3D-50 as the backbone network to extract spatiotemporal features. It captures motion patterns by performing convolution operations in both the temporal and spatial dimensions simultaneously through 3D convolution kernels. Internally, it combines a feature pyramid network to construct multi-scale feature maps.

10. The method according to claim 9, characterized in that, The multi-scale spatiotemporal fusion detection network design uses skip connections to fuse shallow high-resolution features with deep semantic features, and inserts a temporal attention module before the detection head to dynamically weight feature maps at different times to achieve enhanced detection of moving targets.

11. The method according to claim 1, characterized in that, Also includes: Based on the compliance review results output by the multimodal large model, combined with escalator contact stability, tether connection stability, duration and operation level parameters, a comprehensive risk score is calculated. Evidence package data containing keyframe images, explanations of violations, violation timestamps and equipment location information is automatically generated and archived.

Images