A video segment extraction method based on operation and maintenance key points

Abstract

The application provides a video segment extraction method based on operation and maintenance key points, and belongs to the technical field of video segment extraction based on operation and maintenance key points. The application realizes video time sequence adaptive segmentation through recursive splitting based on a Mahalanobis distance minimum spanning tree, fuses frame-by-frame visual features and physical sensor signals to construct a multi-modal joint feature vector, adopts a Gaussian mixture model and a Bayesian deep learning to probabilistically model key point boundaries, uses an Actor-Critic reinforcement learning framework and an artificial intelligence strategy network to model key point search as a Markov decision process to realize adaptive frame skipping and close reading, determines video segment start and end frames in combination with boundary probability distribution based on the located key point moment, and then completes end-to-end inference output through a three-stage asynchronous pipeline, thereby solving the technical problem of insufficient operation and maintenance video key point moment positioning accuracy.

Description

Technical Field

[0001] This invention belongs to the field of video segment extraction technology based on key operation and maintenance points, and specifically relates to a method for extracting video segments based on key operation and maintenance points. Background Technology

[0002] In the field of industrial operations and maintenance (O&M), accurately locating the moment of critical operation or failure in equipment operation videos is a fundamental task for achieving automated O&M analysis. Current technologies typically employ frame differencing based on fixed sliding windows or image classification methods with manually set thresholds to segment O&M videos and extract keyframes. These methods are applicable to simple scenarios with fixed cameras and uniform equipment status, and have already been applied in some factory quality inspection and equipment patrol systems.

[0003] However, the aforementioned methods exhibit significant limitations when dealing with complex operation and maintenance scenarios involving multiple types of industrial equipment and coexisting multimodal signals. Fixed window segmentation cannot adaptively match the feature distribution changes caused by different fault types, and the setting of manual thresholds relies heavily on prior experience, leading to frequent missed detections or boundary shifts in areas of subtle feature changes in progressive faults. Furthermore, purely visual features ignore crucial information contained in physical sensor signals such as vibration, temperature, and current, resulting in a lack of multimodal evidence to support the determination of key point boundaries.

[0004] In existing technologies, video temporal segmentation relies on manually preset window numbers, and the key point search process lacks an adaptive frame skipping mechanism for non-critical areas. This leads to a significant decrease in the accuracy of key point location during operation and maintenance under conditions of multiple devices and multiple fault types operating concurrently. In other words, existing technologies suffer from insufficient accuracy in locating key points in maintenance videos. Summary of the Invention

[0005] In view of this, the present invention provides a video segment extraction method based on key points of operation and maintenance, which can solve the technical problem of insufficient positioning accuracy of key points in operation and maintenance videos in the prior art.

[0006] This invention is implemented as follows: This invention provides a video segment extraction method based on key operation and maintenance points, including the following steps:

[0007] The input operation and maintenance video stream is divided into windows of equal length. The high-dimensional visual feature vector of each window is extracted. The feature map is constructed with the Mahalanobis distance between windows as the edge weight. The Kruskal algorithm is used to generate the minimum spanning tree. The maximum weight edge is recursively deleted to complete the video temporal segmentation.

[0008] Multi-scale visual features are extracted frame by frame from the segmented video, and physical sensor signals are acquired simultaneously. The visual features and sensor signals are fused in a multi-modal manner to form a multi-modal joint feature vector for each frame.

[0009] The key point time boundary is modeled as a probability distribution, and a Gaussian mixture model is used to model the boundary uncertainty. Sensor signals are used as prior anchor boundaries, and Bayesian deep learning is combined to output confidence intervals to obtain the frame-level boundary probability distribution.

[0010] The key point search process of each video segment is input into the adaptive key frame active sampling model. The model outputs the sampling action and key point annotation results based on the multimodal joint feature vector of the current frame and historical action records, thus completing the key point moment localization.

[0011] Based on the key moments of the positioning, the start and end frames of the video segment are determined by combining the boundary probability distribution. Layer normalization is used instead of batch normalization. Dynamic filling and bucket batch processing are used to batch process the variable-length segments to complete the extraction of video segments.

[0012] The extracted video clips are fed into a three-level asynchronous pipeline inference architecture, which sequentially performs video decoding, feature extraction, and key point judgment. Operator-level fusion and mixed-precision inference are used to complete the end-to-end output, resulting in the final set of video clips for key maintenance points.

[0013] Specifically, the step of recursively deleting the edge with the largest weight to complete the video temporal segmentation involves deleting the edge with the largest weight in each step to split the tree into two subtrees, and recursively executing this step until the average Mahalanobis distance within a segment is lower than the threshold of the average Mahalanobis distance within a segment or the number of segments reaches the upper limit of the number of segments.

[0014] The formula for calculating the Mahalanobis distance is as follows: ,in and The first With the High-dimensional visual feature vectors of each window The characteristic covariance matrix is ​​estimated incrementally online using a sliding window.

[0015] The edge weight calculation of the feature map adopts a hierarchical navigable small world graph structure for approximate nearest neighbor indexing acceleration, which supports parallel segmentation processing of large-scale video data streams.

[0016] The method of modeling boundary uncertainty in the Gaussian mixture model refers to using multiple labeled moments of the same type of key points as samples, fitting the Gaussian mixture model with the expectation-maximization algorithm, and using soft labels and label smoothing techniques during training to replace single-moment labels with frame-level soft labels with the output probability distribution of the Gaussian mixture model as weights.

[0017] The smoothing formula for the label smoothing technique is as follows: ,in For smoothing coefficients, This represents the number of categories.

[0018] The method of outputting confidence intervals in Bayesian deep learning refers to using Monte Carlo random deactivation in the fully connected layer of the feature extraction network, performing multiple forward propagations during inference, using the mean of the output distribution as the prediction time, and using the standard deviation as the width of the uncertainty interval.

[0019] The adaptive keyframe active sampling model adopts the Actor-Critic reinforcement learning framework, which includes a state encoding module, an Actor policy network, and a Critic value network. The state encoding module concatenates the multi-scale visual feature vector of the current frame with the historical action record and then sends it to the induction control recurrent unit to output the state representation vector.

[0020] Among them, the Actor policy network takes the state representation vector as input and outputs the probability distribution of four discrete actions: reading the current frame carefully, jumping to the next N frames, marking the current frame as a key point, and terminating the current segment scan. The Critic value network takes the state representation vector and the device topology causal graph as dual inputs. The graph attention network encodes the device topology causal graph and concatenates it with the state representation vector to output a scalar value estimate.

[0021] The reward function of the adaptive keyframe active sampling model is a weighted joint reward of the keypoint localization time intersection-union ratio and inference latency, as shown in the formula: ,in For accuracy weighting coefficients, This is the efficiency penalty coefficient.

[0022] The adaptive keyframe active sampling model also includes a dynamic weight adjustment function. This function calculates a comprehensive quality and efficiency index based on the average time intersection-over-union ratio and average inference latency of the current batch, and dynamically adjusts the weights according to the comprehensive quality and efficiency index. and The value of .

[0023] The adaptive keyframe active sampling model adopts a model-independent meta-learning mechanism. When encountering a new device type, it only needs a small number of labeled video segments to complete the policy transfer through a two-step gradient inner loop update.

[0024] The three-level asynchronous pipelined inference architecture designs the three stages of video decoding, feature extraction and key point judgment as independent CUDA streams to be executed concurrently, and operator-level fusion merges adjacent operators such as convolution, batch normalization and activation into a single operator at the CUDA level.

[0025] The adaptive keyframe active sampling model training adopts a near-end strategy optimization algorithm, and the batch samples in the data loop adopt a priority experience replay mechanism, with priority weights dynamically allocated by the time difference error.

[0026] The threshold for the mean Mahalanobis distance within a segment is determined through statistical analysis of the time intersection-comparison ratio (TUC) of different thresholds on a labeled dataset, with at least five rounds of iterative experiments; the smoothing coefficient... The value is taken as 0.05 to 0.15; the precision weighting coefficient. The initial value range is 0.6 to 0.8, and the efficiency penalty coefficient is... The initial value range is 0.2 to 0.4; the pruning coefficient of the near-end policy optimization algorithm is 0.1 to 0.2; the frame skipping step size N is output by the Actor policy network, and the maximum number of frame skips ranges from 20 to 50.

[0027] This invention models video temporal segmentation as a minimum spanning tree recursive splitting problem, automatically determining segment boundaries using a Mahalanobis distance-driven feature map structure, thus eliminating boundary offset errors caused by manually preset window numbers. Based on segmentation, it fuses frame-by-frame visual features and physical sensor signals to construct a multimodal joint feature vector, ensuring that the estimation of boundary probability distributions is supported by both visual and sensor evidence. This allows for accurate capture of distribution jumps even in regions of subtle feature changes in progressive faults.

[0028] This invention further employs an adaptive keyframe active sampling model, modeling keypoint search as a Markov decision process. This allows the model to adaptively balance precise reading and frame skipping, avoiding redundant computation of non-critical areas and reducing inference latency while maintaining positioning accuracy. The graph attention network introduces device topology causality as a structural prior for value estimation, enabling credit allocation to accurately reflect the causal links of fault propagation in multi-device systems and improving the localization generalization capability across device scenarios.

[0029] In summary, this invention solves the technical problem of insufficient positioning accuracy of key points in operation and maintenance videos mentioned in the background art. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention. Figure 2 A schematic diagram of the time-series segmentation and minimum spanning tree recursive splitting process for compressor operation and maintenance video. Figure 3 A comparison of the action distribution of the adaptive keyframe active sampling model on different device types. Figure 4 This is a schematic diagram of the CUDA flow timing for each stage of a three-stage asynchronous pipeline. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0032] like Figure 1 The diagram shows a flowchart of a video segment extraction method based on key operation and maintenance points provided by this invention. This method includes the following steps:

[0033] S01. Divide the input operation and maintenance video stream into equal-length windows, extract the high-dimensional visual feature vector of each window, construct a feature map with the Mahalanobis distance between windows as the edge weight, generate the minimum spanning tree using the Kruskal algorithm, and recursively delete the maximum weight edge to complete the video temporal segmentation.

[0034] S02. For the segmented video, extract multi-scale visual features frame by frame, simultaneously acquire physical sensor signals, and fuse the visual features and sensor signals in a multi-modal manner to form a multi-modal joint feature vector for each frame.

[0035] S03. The key point time boundary is modeled as a probability distribution. A Gaussian mixture model is used to model the boundary uncertainty. The sensor signal is used as the prior anchor boundary. The confidence interval is output by combining Bayesian deep learning to obtain the frame-level boundary probability distribution.

[0036] S04. Input the key point search process of each video segment into the adaptive key frame active sampling model. The model outputs the sampling action and key point annotation results based on the multimodal joint feature vector of the current frame and the historical action record, and completes the key point moment localization.

[0037] S05. Based on the key points of the positioning, the start and end frames of the video segment are determined by combining the boundary probability distribution. Layer normalization is used to replace batch normalization to eliminate the need for cross-device synchronization. Dynamic filling and bucket batch processing are used to batch process the variable-length segments to complete the extraction of video segments.

[0038] S06. The extracted video segments are fed into a three-level asynchronous pipeline inference architecture to sequentially complete video decoding, feature extraction and key point judgment. Operator-level fusion and mixed-precision inference are used to complete end-to-end output and obtain the final set of video segments for key maintenance points.

[0039] The minimum spanning tree is the core data structure of the video temporal segmentation algorithm based on dynamic recursive splitting of the minimum spanning tree. After dividing the video sequence into several windows of equal length, the high-dimensional feature vector of each window is used as a graph node. The Mahalanobis distance between windows is used as the edge weight, and the minimum spanning tree is constructed on all nodes using the Kruskal algorithm. The Mahalanobis distance formula is expressed as follows: ;in and The first With the High-dimensional feature vectors of each window The feature covariance matrix is ​​estimated incrementally online using a sliding window. Keypoint detection is transformed into a recursive maximum weight edge deletion problem of minimum spanning tree. In each step, the edge with the largest weight is deleted, splitting the tree into two subtrees. The corresponding video is divided into two segments. This process is repeated recursively until the mean Mahalanobis distance within a segment is lower than a threshold or the number of segments reaches the upper limit. The threshold for the mean Mahalanobis distance within a segment is obtained by running the segmentation algorithm on a labeled dataset of various types of maintenance equipment for different thresholds, calculating the temporal intersection-union ratio (TIU) of the segmentation results and the manually labeled boundaries, and using the mean value corresponding to the maximum TPU as the final threshold. The iteration experiment is conducted for no less than 5 rounds. The approximate nearest neighbor graph index uses an HNSW structure to accelerate edge weight calculation and supports parallel segmentation processing of large-scale video data streams. The algorithm does not require a preset number of segments; the segmentation results are automatically determined by the inherent structure of the data, and it has strong adaptability to feature distribution changes caused by different fault types. The minimum spanning tree dynamic recursive splitting algorithm transforms the video temporal segmentation problem into a graph-structured recursive splitting problem, making the segment boundaries entirely driven by the statistical distance in the feature space, thus avoiding boundary offset caused by manually pre-setting the number of windows. The online incremental estimation of the covariance matrix enables the algorithm to adaptively match the statistical characteristics of signals from different devices, thereby accurately capturing distribution jumps even in areas of weak feature changes in gradual faults, improving the temporal positioning accuracy of key point boundaries.

[0040] The multimodal joint feature vector refers to the joint representation vector obtained by concatenating the visual feature vector extracted frame by frame with the synchronously acquired physical sensor signals (continuous quantities such as vibration, temperature, and current) in the feature dimension and then normalizing it. This vector is used for subsequent boundary modeling and key point localization.

[0041] The Gaussian mixture model models boundary uncertainty by using multiple annotation times of the same type of keypoints as samples, fitting the Gaussian mixture model with the expectation-maximization algorithm to obtain a multi-peak probability density distribution of the boundary. During training, soft labels and label smoothing techniques are used to replace single-time annotations with frame-level soft labels weighted by the probability distribution output by the Gaussian mixture model. The label smoothing technique refers to replacing hard labels (0 or 1) in the training target with... ,in Take a value of 0.05 to 0.15. The number of categories; The range of values ​​is determined by applying different values ​​to the labeled dataset. The statistical analysis of the lower boundary positioning error is determined, and the iterative experiment is conducted for no less than 3 rounds.

[0042] The Bayesian deep learning output confidence interval is achieved by using Monte Carlo random deactivation in the fully connected layer of the feature extraction network, performing multiple forward propagations during inference, using the mean of the output distribution as the prediction time, and using the standard deviation as the width of the uncertainty interval.

[0043] The layer normalization refers to normalizing the mean and variance of all feature dimensions of a single sample, which does not depend on other samples in the batch and therefore does not generate a collective communication requirement across devices; the bucket batch processing refers to assigning video segments with similar sequence lengths to the same batch, reducing the sequence length variance within the batch, reducing the padding ratio, and improving GPU utilization.

[0044] The three-level asynchronous pipelined inference architecture refers to designing video decoding, feature extraction, and keypoint judgment as independent CUDA streams, with the three stages executed concurrently. Feature extraction of the previous frame and video decoding of the next frame are performed simultaneously, eliminating serial waiting between stages. The operator-level fusion refers to using TensorRT to merge adjacent operators such as convolution, batch normalization, and activation into a single operator at the CUDA level, reducing the number of GPU memory read / write operations. The mixed-precision inference refers to using INT8 precision for computationally intensive operators and retaining FP16 precision for the precision-sensitive keypoint judgment layer. The end-to-end latency target is less than 500ms for online operation and maintenance response requirements. The specific latency threshold is determined through the analysis of measured data on different precision configurations on the target embedded GPU platform.

[0045] The specific structure of the adaptive keyframe active sampling model is as follows: The model adopts the Actor-Critic reinforcement learning framework, and the overall structure is divided into three parts: a state encoding module, an Actor policy network, and a Critic value network. The state encoding module takes as input the multi-scale visual feature vector of the current frame and the concatenated vector of historical action records. The visual features are extracted by the MobileNetV3 backbone network, and the output is a 512-dimensional feature vector. The historical action records are represented by 32-dimensional one-hot encoding of the most recent 8 actions. After concatenation, the vectors are fed into a single-layer gated recurrent unit for temporal modeling. The hidden layer dimension of the gated recurrent unit is 256, and the output is a 256-dimensional state representation vector. The Actor policy network takes the state representation vector as input and outputs the probability distribution of four types of discrete actions through two layers of fully connected networks (dimensions 128 and 64, respectively). The four types of actions are: carefully reading the current frame, jumping to the next N frames (N is output by a separate fully connected network with a dimension of 1, and the activation function is Sigmoid multiplied by the maximum number of jump frames, 20-50), marking the current frame as a key point, and terminating the current segment scan. The Critic value network takes the state representation vector and the device topology causal graph as dual inputs. The graph attention network encodes the device topology causal graph. The graph attention network has 2 to 3 layers, with 4 to 8 attention heads per layer. The output node embedding dimension is 128. After being concatenated with the state representation vector, it is output as a scalar value estimate through two fully connected networks (dimensions of 256 and 128, respectively). The connection weights between neurons employ a sparse initialization strategy with a sparsity of 0.3–0.5 to reduce redundant computation. Gradient propagation between layers uses gradient pruning (pruning threshold of 0.5–1.0) to prevent gradient explosion. In-batch data loops utilize a priority-based empirical replay mechanism, with priority weights dynamically allocated based on temporal difference errors. GPU memory allocation employs a mixed-precision strategy, using FP16 for Actor forward propagation and FP32 for Critic gradient backpropagation. Memory allocation uses a pre-allocated buffer pool to avoid latency jitter from dynamic memory allocation. In CUDA stream allocation, Actor inference and Critic updates are assigned to independent CUDA streams for parallel execution. In CUDA thread block allocation, convolution operators and fully connected operators use different thread block sizes (128 and 256 threads respectively) to match their respective computation-to-memory access ratios. In terms of layer allocation, the backbone network feature extraction layer is fixed on GPU 0, while the policy network and value network are distributed on GPU 1, achieving pipelined parallelism. The reward function is designed as a weighted joint reward of the keypoint localization time intersection-union ratio and inference latency, expressed as follows: ;in For the current step, locate the key points in time and compare the intersection and union of the time. As a reference time intersection and comparison benchmark value, For the time delay of this inference, As a reference for inference delay baseline value, Take a value of 0.6 to 0.8. The parameters are set to 0.2 to 0.4, and were determined through grid search experiments on labeled datasets with different weight combinations, with at least 5 iterations. A fast adaptation mechanism based on model-independent meta-learning is introduced in the outer layer. When encountering a new device type, only 5 to 10 labeled video segments are needed to complete the policy transfer through a two-step gradient inner loop update. The CUDA stream allocation has a direct functional relationship with the frame skipping step size parameter N of the Actor policy network: the N value output by the Actor network determines the number of frames skipped in the current step. The CUDA stream scheduler dynamically adjusts the prefetch window length of the data prefetch stream based on the N value. The prefetch window length is equal to N multiplied by the estimated single-frame feature extraction delay, thus keeping the prefetch depth of the data loop synchronized with the policy decision.

[0046] The steps for establishing the training dataset for the adaptive keyframe active sampling model specifically include: collecting original videos from various types of industrial operation and maintenance videos; having professional annotators annotate the key moments of each video frame by frame; having the annotation results cross-validated by two people; and arbitrating samples with boundary discrepancies exceeding 10 frames by a third party; dividing the annotated videos into layers according to equipment type, with 80% used for the meta-learning training task set and 20% used for the meta-learning testing task set; constructing an independent Markov decision process trajectory for each type of equipment, recording the state vector, action, reward, and next state at each step, and storing the trajectory data in a priority experience replay buffer pool; and using oversampling and cost-sensitive weight compensation for imbalanced data categories (the ratio of key frame to non-key frame is approximately 1:20 to 1:50).

[0047] The specific steps of training the adaptive keyframe active sampling model include: training the Actor-Critic network using a near-end policy optimization algorithm with a pruning coefficient of 0.1 to 0.2; in each meta-learning outer loop iteration, performing two inner loop gradient updates for each type of device task with a learning rate of 0.01 to 0.05 and an outer loop meta-learning rate of 0.001 to 0.005; the above learning rate and meta-learning rate are determined through joint analysis of convergence speed and generalization error on the validation set, with at least 10 iterations; the total number of training steps is 5 million to 10 million steps, and the generalization performance is evaluated on the test task set every 100,000 steps, using the joint index of temporal intersection-over-union ratio and inference latency as the early stopping criterion.

[0048] The technical principle behind the adaptive keyframe active sampling model is as follows: The model models the video keypoint search process as a Markov decision process, enabling the model to adaptively balance between close reading and frame skipping, avoiding redundant calculations for non-critical areas; the graph attention network introduces device topology causality as a structural prior for value estimation, making credit allocation more accurately reflect the causal links of fault propagation in multi-device systems, and improving the generalization ability of keypoint localization in cross-device scenarios; the model-independent meta-learning mechanism enables the policy network to quickly complete migration with a small number of labeled samples when facing new device types, reducing the deployment cost of new scenarios; the collaborative allocation strategy of CUDA streams and data loops deeply couples inference scheduling and policy decision-making, reducing pipeline bubbles and memory fragmentation, and improving overall inference throughput.

[0049] The dynamic weight adjustment function is used to adjust the accuracy and efficiency weight ratio of the reward function in the adaptive keyframe active sampling model. The function calculates the comprehensive quality and efficiency index based on the average time intersection-union ratio and average inference latency of the current batch, as expressed in the following formula: ;in The average time for the current batch to crossover and merge (CLOSE) is given. This represents the average inference latency for the current batch. The current efficiency penalty coefficient is set to 0.2–0.6. For comprehensive quality and efficiency index; when At that time, reduce To 0.15–0.25, increase Towards 0.75–0.85, guide the model towards higher accuracy; when At that time, keep and Within the initial settings range, maintain a balance between accuracy and efficiency; when At that time, improve To 0.45–0.55, decrease The threshold value is set to 0.45–0.55, guiding the model to prioritize reducing inference latency. The boundary values ​​for each interval are determined by statistical analysis of the model's convergence stability under different boundary values ​​on a multi-type device validation set, with at least 5 rounds of iterative experiments.

[0050] The Mahalanobis distance is a statistical distance that measures the correlation between two high-dimensional feature vectors, taking into account the correlation between feature dimensions. Compared with Euclidean distance, Mahalanobis distance normalizes the variance and covariance of each dimension by using the inverse matrix of the covariance matrix, thus eliminating the influence of the difference in the dimensions of different feature dimensions on the distance calculation.

[0051] HNSW is a hierarchical navigable small-world graph structure, which is an approximate nearest neighbor indexing method. It achieves logarithmic-level approximate nearest neighbor queries by constructing a multi-layer sparse graph, and is used to accelerate the edge weight calculation of large-scale feature graphs.

[0052] The gated recurrent unit is a recurrent neural network unit that controls the retention and forgetting of historical information through reset gates and update gates, and is used to model the temporal dependencies between frames.

[0053] The graph attention network is a graph neural network that uses an attention mechanism to assign different weights to the neighboring nodes of each node in the graph to aggregate information, and is used to model the node representation in the device topology causal graph.

[0054] The model-independent meta-learning is a meta-learning method that learns a good initialization parameter on multiple tasks, enabling the model to quickly adapt when encountering new tasks with only a small number of gradient updates.

[0055] The priority experience replay is a reinforcement learning training technique that assigns sampling priority to experience samples according to the size of the temporal difference error. Samples with larger errors are sampled more frequently, thereby accelerating the model's learning of difficult samples.

[0056] The proximal policy optimization algorithm is a reinforcement learning policy gradient method that improves training stability by pruning the policy update step size to limit the magnitude of each parameter update.

[0057] Optionally, the present invention also provides a method for implementing a video segment extraction system based on key operation and maintenance points using a computer. The computer is equipped with a readable storage medium, which stores program instructions. When the program instructions are run on the computer, they can execute the above-described method.

[0058] The specific implementation of step S01 is as follows: The input maintenance video stream is divided into several equal-length windows of fixed duration. High-dimensional visual feature vectors are extracted from the frame sequences within each window. Feature extraction is performed using a pre-trained convolutional neural network backbone, with an output dimension typically of 512 or 1024. Subsequently, a complete graph is constructed using the feature vectors of all windows as nodes and the Mahalanobis distance between any two window feature vectors as edge weights. The calculation of the Mahalanobis distance depends on the feature covariance matrix. The matrix employs online incremental estimation using a sliding window to adapt to variations in the statistical characteristics of signals from different devices. A minimum spanning tree is constructed on the complete graph using Kruskal's algorithm, preserving the connection structure with the minimum statistical distance between nodes. Keypoint segmentation is transformed into a recursive maximum-weighted edge deletion problem on the minimum spanning tree. Each time, the edge with the largest weight is deleted, splitting the tree into two subtrees, corresponding to two time segments in the video. This process is recursively executed until the mean Mahalanobis distance within a segment falls below a threshold (approximately 0.3 times the mean global Mahalanobis distance in the feature space) or the number of segments reaches its upper limit. Edge weight calculation is accelerated using a hierarchical, navigable small-world graph structure index, supporting parallel processing of large-scale video data streams.

[0059] The specific implementation of step S02 is as follows: Multi-scale visual features are extracted frame-by-frame for each segmented video. These multi-scale features are obtained by parallel extraction and concatenation of convolutional layers with different receptive fields, enabling simultaneous capture of local texture changes and global structural changes. Simultaneously, physical sensor signals from the device are acquired, including continuous quantities such as vibration, temperature, and current. After normalization, these sensor signals are directly concatenated with the visual feature vector along the feature dimension to form a multi-modal joint feature vector for each frame. After concatenation, the joint feature vector is processed... Normalization eliminates the influence of differences in the dimensions of different modalities. The joint feature vector of multimodal features serves as a unified input representation for subsequent boundary probability modeling and keypoint localization.

[0060] The specific implementation of step S03 is as follows: Collect multiple manually annotated timestamps of the same type of keypoints. Use these annotated timestamps as samples and fit a Gaussian mixture model using the expectation-maximization algorithm to obtain the multi-peak probability density distribution of the boundary timestamps. During training, a soft-label technique is introduced, replacing the hard labels at a single time point with frame-level soft labels weighted by the probability distribution output by the Gaussian mixture model. Simultaneously, a label smoothing technique is used to smooth the hard labels according to the formula... Smoothing process, smoothing coefficient The range is 0.05 to 0.15. Sensor signals are used as prior information for boundary anchoring, enhancing boundary probabilities near moments of abrupt changes in physical signals. The Bayesian deep learning part employs Monte Carlo random deactivation in the fully connected layers of the feature extraction network, performs multiple forward propagations during inference, uses the mean of the output distribution as the prediction time, and the standard deviation as the width of the uncertainty interval, ultimately outputting the boundary probability distribution and confidence interval for each frame.

[0061] The specific implementation of step S04 is as follows: The adaptive keyframe active sampling model adopts the Actor-Critic reinforcement learning framework. The state encoding module concatenates the multi-scale visual feature vector of the current frame (extracted by the MobileNetV3 backbone network, outputting 512 dimensions) with the most recent 8 steps of historical action records (32-dimensional one-hot encoding), and then sends it to the gated recurrent unit for temporal modeling. The hidden layer dimension of the gated recurrent unit is 256, and it outputs a 256-dimensional state representation vector. The Actor policy network takes the state representation vector as input and outputs the probability distribution of four types of discrete actions through two fully connected layers. The four types of actions are: reading the current frame carefully, jumping to the next N frames, marking the current frame as a key point, and terminating the current segment scan. N is output by a single fully connected layer, and the maximum number of skipped frames is 20-50. The Critic value network takes the state representation vector and the device topology causal graph as dual inputs. The graph attention network encodes the device topology causal graph and concatenates it with the state representation vector. After passing through two fully connected layers, it outputs a scalar value estimate. The reward function is... , Take a value of 0.6 to 0.8. The value is set to 0.2–0.4. Training employs a near-end policy optimization algorithm, and within-batch samples, a priority-based empirical replay mechanism is used, with priority weights dynamically allocated based on temporal difference error. An outer layer introduces a model-independent meta-learning mechanism, which allows policy transfer to be completed through two-step gradient inner loop updates when encountering new device types, requiring only 5–10 labeled video clips.

[0062] The specific implementation of step S05 is as follows: Based on the key point time located in step S04, and combined with the frame-level boundary probability distribution output in step S03, the frame with the highest probability within the confidence interval is searched as the start and end frames of the video segment, thereby determining the boundary of each video segment. To eliminate the synchronization requirements during cross-device batch processing, layer normalization is used instead of batch normalization. Layer normalization normalizes the mean and variance of all feature dimensions of a single sample, without depending on other samples within the batch. A dynamic padding strategy is adopted for variable-length video segments, allocating segments with similar sequence lengths to the same batch (i.e., bucket batch processing), reducing the variance of sequence lengths within the batch, reducing the padding ratio, improving GPU utilization, and completing efficient batch processing and video segment extraction of variable-length segments.

[0063] The specific implementation of step S06 is as follows: The extracted video segments are fed into a three-level asynchronous pipelined inference architecture. The three stages of video decoding, feature extraction, and key point judgment are designed as independent CUDA streams executed concurrently. Feature extraction of the previous frame and video decoding of the next frame are performed simultaneously, eliminating serial waiting between stages. Operator-level fusion utilizes TensorRT to merge adjacent operators such as convolution, batch normalization, and activation into a single operator at the CUDA level, reducing the number of GPU memory read / write operations. Hybrid precision inference uses INT8 precision for computationally intensive operators and retains FP16 precision for the precision-sensitive key point judgment layer, with an end-to-end latency target of less than 500ms. Actor inference and Critic update are allocated to independent CUDA streams for parallel execution. The backbone network feature extraction layer is fixed on GPU 0, while the policy network and value network are distributed on GPU 1, achieving pipeline parallelism and finally outputting a set of video segments for key points of operation and maintenance.

[0064] It should be noted that the key technologies of this invention include: an adaptive video temporal segmentation technology based on minimum spanning tree recursive splitting, which drives the segmentation boundary entirely by the statistical distance in the feature space; online incremental estimation of the covariance matrix enables the algorithm to adaptively match the statistical characteristics of signals from different devices, accurately capturing distribution jumps even in areas of weak feature changes in progressive faults, avoiding boundary offset caused by manually preset window numbers; a multimodal boundary probability modeling technology based on Gaussian mixture models and Bayesian deep learning, which uses sensor signals as prior anchoring boundaries; soft labels and label smoothing mechanisms make the model robust to boundary uncertainties; and Monte Carlo random deactivation provides confidence interval quantization; and an adaptive keyframe active sampling technology based on Actor-Critic reinforcement learning and graph attention networks, which models keypoint search as a Markov decision process and introduces device topological causality as a structural prior for value estimation. The three technologies work together to enhance each other’s segmentation accuracy, boundary probability modeling and key point search efficiency. The segmentation results directly improve the input quality of boundary probability modeling, while the boundary probability distribution provides a more accurate state representation for the active sampling model, thus significantly improving the positioning accuracy and inference throughput of key points in the operation and maintenance video as a whole.

[0065] It's important to note that in maintenance environments characterized by concurrent operation of multiple types of industrial equipment, diverse failure modes, and scarce labeled data, models often struggle to quickly adapt their strategies when faced with new equipment types, leading to a significant drop in keypoint localization accuracy in new scenarios. This technical problem arises because the parameter space of deep learning models is highly dependent on the distribution of training data. When the sensor signal distribution, visual appearance, or fault propagation patterns of new equipment differ significantly from the training set, the model's feature extraction layer and policy network cannot accurately adapt to the statistical characteristics of the new scenario. Furthermore, large-scale re-labeling and retraining are limited by the time and manpower costs at the maintenance site. A common solution to this problem is to collect a large amount of labeled data for the new equipment and then perform full fine-tuning or retraining of the model. However, the deployment cycle of new equipment at maintenance sites is short, and labeling resources are limited. Full fine-tuning often requires hundreds or even thousands of labeled samples, and there is a risk of catastrophic forgetting of knowledge about the old equipment during the fine-tuning process, causing a simultaneous decline in the model's localization performance on the old equipment. This invention effectively solves this technical problem by introducing a model-independent meta-learning mechanism into the outer layer of the adaptive keyframe active sampling model. This mechanism learns a good initialization parameter across multiple device tasks, enabling the policy network to complete parameter transfer with only 5-10 labeled video clips through a two-step gradient inner loop update when facing new device types. The core principle of model-independent meta-learning lies in the fact that the meta-training phase repeatedly simulates the few-shot adaptation process on a large number of heterogeneous device tasks, placing the initialization parameter in a parameter space that is sensitive to the gradient direction of various device tasks. This allows for adaptation to the statistical characteristics of new devices within a few gradient update steps without modifying the overall model structure. Simultaneously, the priority experience replay mechanism prioritizes learning difficult samples with large temporal difference errors during training, further accelerating policy convergence in few-shot scenarios.

[0066] Specifically, the principle of this invention is:

[0067] This invention can solve the technical problem of insufficient positioning accuracy at key moments in operation and maintenance videos. Its principle can be explained from the following aspects.

[0068] First, at the video temporal segmentation level, this invention transforms the segmentation problem into a recursive splitting problem with a graph structure. Using the high-dimensional feature vectors of each time window as nodes and the Mahalanobis distance between windows as edge weights, a minimum spanning tree is constructed. Mahalanobis distance, through the inverse of the covariance matrix, eliminates the influence of differences in the dimensionality of different features, and more accurately reflects statistical changes in the feature space than Euclidean distance. The recursive deletion of the largest weight edge essentially segments the video sequence where the feature distribution difference is greatest, ensuring that the segmentation boundaries are entirely driven by the inherent structure of the data, rather than relying on a manually preset number of windows. Online incremental estimation of the covariance matrix enables the algorithm to adaptively match the statistical characteristics of signals from different devices, accurately capturing distribution jumps even in regions of subtle feature changes in progressive faults.

[0069] Secondly, at the multimodal boundary modeling level, this invention concatenates visual features and physical sensor signals along the feature dimension and normalizes them to form a multimodal joint feature vector. A Gaussian mixture model is used to model boundary uncertainty, with sensor signals serving as prior anchor boundaries, combined with Bayesian deep learning to output confidence intervals. A Monte Carlo random deactivation mechanism performs multiple forward propagations during inference, using the mean of the output distribution as the prediction time and the standard deviation as the uncertainty interval width. This ensures that the estimation of the boundary probability distribution is supported by both visual and sensor evidence, significantly reducing positioning errors caused by the lack of single-modal information.

[0070] Furthermore, at the keypoint active search level, this invention employs the Actor-Critic reinforcement learning framework, modeling the keypoint search process as a Markov decision process. The Actor policy network outputs four discrete actions—precise reading, frame skipping, labeling, and termination—based on the current frame's multimodal joint feature vector and historical action records. This allows the model to perform large-step frame skipping in non-critical areas and switch to frame-by-frame precise reading in suspected critical areas, thus avoiding latency accumulation caused by redundant computation. The Critic value network introduces a graph attention network to encode the device topology causal graph, incorporating the causal links of fault propagation in multi-device systems into value estimation. This makes credit allocation more accurately reflect cross-device relationships, improving the model's localization generalization ability in multi-device concurrent fault scenarios. The model-independent meta-learning mechanism enables the policy network to complete policy transfer through two-step gradient inner loop updates with only a small number of labeled samples when facing new device types, lowering the deployment threshold for new scenarios.

[0071] Finally, at the inference acceleration level, the three-stage asynchronous pipeline architecture designs video decoding, feature extraction, and keypoint judgment as independent concurrent streams, eliminating serial waiting between stages; layer normalization replaces batch normalization, eliminating the need for cross-device synchronization; and bucket batch processing reduces the fill ratio and improves GPU utilization. These engineering optimizations collectively ensure that end-to-end inference latency meets the requirements of online operation and maintenance response, enabling the high-precision positioning algorithm to operate stably in actual industrial deployments.

[0072] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0073] The specific implementation of step S01 is as follows: After dividing the input operation and maintenance video stream into windows of equal length, extract the high-dimensional visual feature vector of each window. ,in The feature dimension is defined as follows. The statistical distance between windows is measured using Mahalanobis distance, and the formula is as follows:

[0074] ;

[0075] In the formula, and The first With the High-dimensional visual feature vectors of each window The characteristic covariance matrix is ​​estimated incrementally online using a sliding window. This represents the dimensionless statistical distance. The covariance matrix... The online incremental estimation formula is expressed as follows:

[0076] ;

[0077] In the formula, For the first The covariance matrix estimated step by step, For the first The covariance matrix estimated step by step, For the front The mean vector of the step feature vectors, For the first The newly arrived feature vector, The current total number of windows. Using the window nodes and Mahalanobis distance edge weights, a minimum spanning tree is constructed using Kruskal's algorithm. Then, the edge with the largest weight is recursively deleted, splitting the tree into two subtrees. The corresponding video is divided into two segments. This process is repeated recursively until the mean Mahalanobis distance within a segment falls below a threshold or the number of segments reaches its maximum. The mean Mahalanobis distance threshold within a segment is determined on a multi-type maintenance equipment labeled dataset. The segmentation algorithm is run for different thresholds, and the temporal intersection-union ratio (TIU) of the segmentation results and the manually labeled boundaries is calculated. The mean TU corresponding to the maximum TU is used as the final threshold. The iteration experiment is conducted for at least 5 rounds. Edge weight calculation is accelerated using a hierarchical, navigable small-world graph structure, supporting parallel segmentation processing of large-scale video data streams.

[0078] The specific implementation of step S02 is as follows: extract multi-scale visual features frame by frame from the segmented video, simultaneously collect physical sensor signals such as vibration, temperature, and current, and convert the visual feature vectors... With sensor signal vector After concatenating the feature dimensions and normalizing, the multimodal joint feature vector of each frame is obtained. The splicing and normalization formulas are expressed as follows:

[0079] ;

[0080] In the formula, For the first Multimodal joint feature vector of a frame. This involves concatenating the visual feature vector with the sensor signal vector. It is a norm 2, after normalization , Each component is a dimensionless value. For visual feature dimensions, the empirical value is 512. The dimension of the sensor signal is determined by the actual number of acquisition channels. For frame index.

[0081] The specific implementation of step S03 is as follows: the time boundary of key points is modeled as a probability distribution, a Gaussian mixture model is used to model the uncertainty of the boundary, multiple annotation times of the same type of key points are used as samples, and the Gaussian mixture model is fitted with the expectation-maximization algorithm to obtain the multi-peak probability density distribution of the boundary, as expressed in the following formula:

[0082] ;

[0083] In the formula, For the first The dimensionless boundary probability density of the frame satisfies , The Gaussian component number has an empirical value of 3 to 8 and is determined by fitting the target dataset using the expectation-maximization algorithm. For the first The mixed weights of the components satisfy the following conditions: , For the first The mean of the Gaussian components, For the first The variance of the Gaussian components, The mean ,variance The Gaussian probability density function in the first place The value at the frame. During training, soft labels and label smoothing techniques are used to replace single-timepoint annotations with frame-level soft labels weighted by the probability distribution output by the Gaussian mixture model. The label smoothing formula is expressed as follows:

[0084] ;

[0085] In the formula, For smoothed soft labels, This is the original hard tag, with a value of 0 or 1. This is a smoothing coefficient, with a value ranging from 0.05 to 0.15. The number of categories, in the Gaussian mixture model The meanings are different here. The total number of classification categories is represented. Sensor signals serve as prior anchoring boundaries. Combining the Monte Carlo random deactivation method in Bayesian deep learning, the signal is propagated multiple times in the fully connected layer of the feature extraction network. The mean of the output distribution is used as the prediction time, and the standard deviation is used as the width of the uncertainty interval to obtain the frame-level boundary probability distribution.

[0086] The specific implementation of step S04 is as follows: The adaptive keyframe active sampling model adopts an actor-critic reinforcement learning framework. The state encoding module concatenates the 512-dimensional visual feature vector of the current frame (extracted by a lightweight backbone network) with the 32-dimensional historical action one-hot encoding, and then feeds it into a gated recurrent unit with a hidden layer dimension of 256, outputting a 256-dimensional state representation vector. The formula for the update process of the gated loop unit is expressed as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] In the formula, For the first Step reset gate vector, For the first Update the gate vector step by step. For the first Step candidate hidden state vector For the first The hidden state vector output by the step is the state representation vector. For the first Hidden state vector, , , This is the corresponding input weight matrix. , , This is the corresponding hidden state weight matrix. , , For the corresponding bias vector, It is the sigmoid activation function. The hyperbolic tangent activation function is used. For element-wise multiplication, For the first The frame multimodal joint feature vector is consistent with the definition in step S02. The actor policy network uses... As input, the network outputs the probability distributions of four discrete actions through a two-layer fully connected network, with a frame skipping step size. The output is a single fully connected network with the activation function being... Then multiply by the maximum number of frame skips (an empirical value is 20-50). The critic value network uses... The device topology causal graph encoding vector has two inputs. The graph attention network performs node embedding encoding on the device topology causal graph. Layer nodes The aggregation formula is expressed as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] In the formula, For the first Layer nodes with neighboring nodes Attention score between (dimensionless scalar). For the first The layer can learn attention vectors. For the first The layer can learn linear transformation matrices. and These are the graph attention network's first and second generation numbers. Layer nodes With nodes The embedding vector, here With the output of the gated loop unit For different variables, This represents vector concatenation. For nodes The set of neighboring nodes, The normalized attention coefficient (dimensionless). For the first Layer nodes The output embedding vector, For a leaky linear rectified activation function, For nodes The traversal index of all neighboring nodes, and the Gaussian mixture model For different index variables. The graph attention network has 2-3 layers, with 4-8 attention heads per layer, and the output node embedding dimension is 128. After splicing, a scalar value estimate is output through a two-layer fully connected network. The reward function formula is expressed as follows:

[0096] ;

[0097] In the formula, For dimensionless joint rewards, For the current step, locate the key points in time and compare the intersection and union of the time. As a reference time intersection and comparison benchmark value, For the time delay of this inference, As a reference for inference delay baseline value, and These are dimensionless weighting coefficients. Take a value of 0.6 to 0.8. The parameters, ranging from 0.2 to 0.4, were determined through grid search experiments on the labeled dataset, with at least five iterations. The dynamic weight adjustment function is based on the comprehensive quality and efficiency index of the current batch. Adjustment and , The formula is expressed as follows:

[0098] ;

[0099] In the formula, It is a dimensionless comprehensive quality and efficiency index. The average time for the current batch to crossover and merge (CLOSE) is given. This represents the average inference latency for the current batch. This is a dimensionless efficiency penalty coefficient, ranging from 0.2 to 0.6. When... hour, It decreased to 0.15-0.25. Increase to 0.75–0.85; when hour, and Maintain the initial setting range; when hour, Increase to 0.45–0.55, The value was reduced to 0.45–0.55, and the boundary values ​​for each interval were determined through statistical analysis on a multi-type device validation set, with at least five rounds of iterative experiments. The pruning objective function formula for the near-end policy optimization algorithm is expressed as follows:

[0100] ;

[0101] In the formula, Let the objective function be the dimensionless pruning strategy. For the current parameters of the actor strategy network, These are the policy network parameters from the previous update. For the current policy in state Select action The probability, The probability corresponding to the old strategy. For the first The dimensionless dominance estimate of the step. This is the cutting factor, with a value ranging from 0.1 to 0.2. For the expectation of the time step, This means cropping the probability ratio to... Within the interval, For the first The model selects actions step by step. It is trained using a proximal policy optimization algorithm and introduces a fast adaptation mechanism based on model-independent meta-learning. When encountering a new device type, only 5 to 10 labeled video clips are needed to complete the policy transfer through a two-step gradient inner loop update.

[0102] The specific implementation of step S05 is as follows: Based on the key point time of positioning, the start and end frames of the video segment are determined by combining the boundary probability distribution. Layer normalization is used instead of batch normalization to eliminate the need for cross-device synchronization. Dynamic filling and bucket batch processing are used to batch process variable-length segments. Video segments with similar sequence lengths are assigned to the same batch to reduce the variance of sequence length within the batch, reduce the filling ratio, improve the utilization of the graphics processor, and complete the extraction of video segments.

[0103] The specific implementation of step S06 is as follows: The extracted video segments are sent into a three-level asynchronous pipeline inference architecture. The three stages of video decoding, feature extraction and key point judgment are designed as independent computing unified device architecture streams to be executed concurrently. Operator-level fusion is used to merge adjacent operators into a single operator at the bottom layer. Hybrid precision inference is adopted (INT8 precision is used for computationally intensive operators, and FP16 precision is retained for the key point judgment layer). The end-to-end latency target is less than 500ms, and the final set of operation and maintenance key point video segments is output.

[0104] To better understand and implement this invention, the following is a specific application scenario of the invention, Example 2: To verify the effectiveness of the invention, technicians set up a test environment and deployed the method of the invention on a multi-type equipment operation and maintenance video dataset in an industrial park. Key point video segments were extracted from the operation and maintenance videos of three types of equipment: compressors, motors, and pump sets, to verify the functional performance of each module.

[0105] The test environment collected 180 maintenance videos, each 3-8 minutes long, with a frame rate of 25fps and a resolution of 1920×1080. Each device simultaneously collected signals from three physical sensors: vibration (accelerometer, sampling frequency 500Hz), temperature (thermocouple, sampling frequency 1Hz), and current (current transformer, sampling frequency 200Hz). The timestamp error between the video and sensor signals was controlled within 10ms. Annotators labeled key moments in each video frame-by-frame. The annotation results underwent cross-validation by two researchers; samples with a boundary discrepancy exceeding 10 frames were arbitrated by a third party, ultimately forming a standard dataset containing 432 key point annotations. The dataset was stratified by device type, with 80% used for training and 20% for testing.

[0106] In step S01, the time window length is set to 2 seconds (i.e., 50 frames). Feature extraction uses the ResNet-50 backbone network, and the output 2048-dimensional feature vector is reduced to 256 dimensions by principal component analysis before being used in Mahalanobis distance calculation. The covariance matrix is ​​estimated online incrementally using a sliding window with a window length of 20 time windows. The Kruskal algorithm constructs a minimum spanning tree on all window nodes, recursively deleting the largest weight edge. The threshold for the mean Mahalanobis distance within a segment is 0.3 times the mean global Mahalanobis distance, determined through 5 rounds of iterative experiments. The time intersection-union ratio (OCU) of the segmentation results for the three types of devices with the manually labeled boundaries is shown in Table 1.

[0107] Table 1. Results of Time Intersection and Union (TIU) of Video Time Sequence Segments for Each Device Type

[0108]

[0109] As shown in Table 1, the average time intersection-over-union ratio (TUIR) of the three types of devices is above 0.84, and the average boundary offset is less than 5 frames. This indicates that the adaptive segmentation method based on minimum spanning tree recursive splitting can accurately capture the characteristic distribution jumps of different devices, and the segmentation result is automatically determined by the inherent structure of the data. Figure 2 The diagram shows the time-series segmentation and recursive splitting process of the compressor operation and maintenance video. It can be seen intuitively that the video is divided into several independent time periods after the maximum weight edge is gradually deleted.

[0110] In step S02, multi-scale visual features are extracted in parallel by three convolutional layers with different receptive fields, with output dimensions of 128, 256, and 128 respectively. These are concatenated to obtain a 512-dimensional visual feature vector, which is then concatenated with the normalized features from the three sensor signals (16 dimensions per signal, totaling 48 dimensions) along the feature dimension to form a 560-dimensional multimodal joint feature vector. This vector is then processed... Normalization process.

[0111] In step S03, a Gaussian mixture model is fitted at multiple annotation times for keypoints of the same type, with the number of Gaussian components set to 3, aiming to maximize the iterative convergence of the algorithm. Label smoothing coefficient. Set to 0.1, number of categories The number of frames is 2 (keyframes and non-keyframes). Monte Carlo random inactivation performs 30 forward propagations during inference, outputting the mean and standard deviation of the boundary probabilities for each frame, forming a frame-level boundary probability distribution.

[0112] In step S04, in the adaptive keyframe active sampling model, the MobileNetV3 backbone network extracts 512-dimensional visual features, the gated recurrent unit hidden layer dimension is 256, the Actor policy network contains two fully connected layers (dimensions 128 and 64), and the maximum number of frame skips is 30. The graph attention network has 2 layers, each with 4 attention heads, and the node embedding dimension is 128. The total number of training steps is 8 million steps, with 2 steps of meta-learning inner loop update, an inner loop learning rate of 0.02, an outer loop meta-learning rate of 0.002, and a proximal policy optimization algorithm pruning coefficient of 0.15. In the reward function... Initially set to 0.7, The initial value was set to 0.3. In the rapid adaptation experiment on the new equipment type (pump set), the model completed the policy transfer using only 8 labeled video clips, and the cross-union ratio of key point localization time on the test set reached 0.82. The rapid adaptation performance under different numbers of labeled samples is shown in Table 2.

[0113] Table 2. Rapid Adaptability of New Equipment Types (Pump Sets)

[0114]

[0115] As shown in Table 2, the temporal intersection-over-union (TUI) of the test set steadily increased as the number of labeled video segments increased from 3 to 10, indicating that the model-independent meta-learning mechanism can effectively support policy transfer in scenarios with few samples. Figure 3 As shown, this is a comparison of the action distribution of the adaptive keyframe active sampling model on different device types. It can be observed that the model tends to output frame skipping actions in non-critical areas and switches to detailed reading actions in suspected critical areas, demonstrating the effectiveness of the adaptive trade-off mechanism.

[0116] In steps S05 and S06, after layer normalization replaces batch normalization, cross-device batch processing no longer requires collective communication synchronization. Bucket-based batch processing controls the sequence length difference to within 10 frames, significantly improving GPU utilization compared to the unbucketed scheme. The end-to-end latency test results of the three-level asynchronous pipeline inference architecture on the test device (embedded GPU platform, 8GB of video memory) are shown in Table 3.

[0117] Table 3. Test Results of Inference Latency in a Three-Level Asynchronous Pipeline

[0118]

[0119] As shown in Table 3, the three-level asynchronous pipelined inference architecture reduces the average end-to-end latency to below 500ms. The latency is further reduced after adding mixed-precision inference, meeting the requirements for online operation and maintenance response. Figure 4 As shown, this is a schematic diagram of the CUDA flow timing of each stage of the three-stage asynchronous pipeline. It can be intuitively observed that the concurrent execution relationship of the three stages of video decoding, feature extraction and key point judgment, as well as the pipeline overlap between adjacent frames.

[0120] Compared to traditional video analysis methods based on fixed sliding windows and manual thresholds, the advancements of this invention are reflected in the following aspects: Traditional methods rely on manually preset parameters for segmentation boundaries, failing to adaptively match feature distribution changes caused by different fault types. This invention, however, uses recursive splitting of minimum spanning trees to drive segmentation boundaries entirely by statistical distance in the feature space. Online estimation of the covariance matrix enables the algorithm to accurately capture distribution jumps even in regions with subtle feature changes in progressive faults. Traditional methods rely solely on single-modal visual information, easily missing detections in the early stages of faults where visual features are insignificant. This invention, through multimodal joint feature vectors and Gaussian mixture model boundary probability modeling, introduces the physical prior of sensor signals into boundary estimation, significantly enhancing the robustness of boundary localization. Traditional methods involve frame-by-frame traversal of videos, resulting in significant redundant computation. This invention, through a reinforcement learning-driven adaptive frame-skipping mechanism, allows the model to skip frames rapidly in non-critical areas, switching to detailed reading only in suspected critical areas. This fundamentally reduces computational overhead through inference scheduling, rather than relying on simple hardware acceleration.

[0121] It should be noted that the variables involved in this invention are explained in detail in Tables 4 and 5.

[0122] Table 4. Variable Explanation Table (Part 1)

[0123]

[0124] Table 5. Variable Explanation Table (Part Two)

[0125]

[0126] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting video segments based on key operation and maintenance points, characterized in that, Includes the following steps: The input operation and maintenance video stream is divided into windows of equal length. The high-dimensional visual feature vector of each window is extracted. The feature map is constructed with the Mahalanobis distance between windows as the edge weight. The Kruskal algorithm is used to generate the minimum spanning tree. The maximum weight edge is recursively deleted to complete the video temporal segmentation. Multi-scale visual features are extracted frame by frame from the segmented video, and physical sensor signals are acquired simultaneously. The visual features and sensor signals are fused in a multi-modal manner to form a multi-modal joint feature vector for each frame. The key point time boundary is modeled as a probability distribution, and a Gaussian mixture model is used to model the boundary uncertainty. Sensor signals are used as prior anchor boundaries, and Bayesian deep learning is combined to output confidence intervals to obtain the frame-level boundary probability distribution. The key point search process of each video segment is input into the adaptive key frame active sampling model. The model outputs the sampling action and key point annotation results based on the multimodal joint feature vector of the current frame and historical action records, thus completing the key point moment localization. Based on the key moments of the positioning, the start and end frames of the video segment are determined by combining the boundary probability distribution. Layer normalization is used instead of batch normalization. Dynamic filling and bucket batch processing are used to batch process the variable-length segments to complete the extraction of video segments. The extracted video clips are fed into a three-level asynchronous pipeline inference architecture, which sequentially performs video decoding, feature extraction, and key point judgment. Operator-level fusion and mixed-precision inference are used to complete the end-to-end output, resulting in the final set of video clips for key maintenance points.

2. The video segment extraction method based on key operation and maintenance points according to claim 1, characterized in that, The step of recursively deleting the edge with the largest weight to complete the video temporal segmentation is as follows: in each step, the edge with the largest weight is deleted and the tree is split into two subtrees. This process is repeated recursively until the average Mahalanobis distance within a segment is lower than the threshold of the average Mahalanobis distance within a segment or the number of segments reaches the upper limit of the number of segments.

3. The video segment extraction method based on key operation and maintenance points according to claim 2, characterized in that, The adaptive keyframe active sampling model also includes a dynamic weight adjustment function, which calculates the comprehensive quality and efficiency index based on the average time intersection-union ratio and average inference latency of the current batch.

4. The video segment extraction method based on key operation and maintenance points according to claim 3, characterized in that, The edge weight calculation of the feature map adopts a hierarchical navigable small world graph structure for approximate nearest neighbor indexing acceleration, which supports parallel segmentation processing of large-scale video data streams.

5. The video segment extraction method based on key operation and maintenance points according to claim 4, characterized in that, The method of modeling boundary uncertainty in the Gaussian mixture model refers to using multiple labeled moments of the same type of key points as samples, fitting the Gaussian mixture model with the expectation-maximization algorithm, and using soft labels and label smoothing techniques during training to replace single-moment labels with frame-level soft labels with the output probability distribution of the Gaussian mixture model as weights.

6. The video segment extraction method based on key operation and maintenance points according to claim 5, characterized in that, The adaptive keyframe active sampling model adopts a model-independent meta-learning mechanism. When encountering a new device type, only a small number of labeled video segments are needed to complete the policy transfer through a two-step gradient inner loop update.

7. The video segment extraction method based on key operation and maintenance points according to claim 6, characterized in that, The method of outputting confidence intervals in Bayesian deep learning refers to using Monte Carlo random deactivation in the fully connected layer of the feature extraction network, performing multiple forward propagations during inference, using the mean of the output distribution as the prediction time, and using the standard deviation as the width of the uncertainty interval.

8. The video segment extraction method based on key operation and maintenance points according to claim 7, characterized in that, The adaptive keyframe active sampling model adopts the Actor-Critic reinforcement learning framework, which includes a state encoding module, an Actor policy network and a Critic value network. The state encoding module concatenates the multi-scale visual feature vector of the current frame with the historical action record and then sends it to the induction control recurrent unit to output the state representation vector.

9. The video segment extraction method based on key operation and maintenance points according to claim 8, characterized in that, The Actor policy network takes the state representation vector as input and outputs the probability distribution of four discrete actions: reading the current frame carefully, jumping to the next N frames, marking the current frame as a key point, and terminating the current segment scan. The Critic value network takes the state representation vector and the device topology causal graph as dual inputs. The graph attention network encodes the device topology causal graph and concatenates it with the state representation vector to output a scalar value estimate.

10. The video segment extraction method based on key operation and maintenance points according to claim 9, characterized in that, The three-level asynchronous pipelined inference architecture designs the three stages of video decoding, feature extraction and key point judgment as independent CUDA streams to be executed concurrently. Operator-level fusion merges adjacent operators such as convolution, batch normalization and activation into a single operator at the CUDA level.

Images