A power maintenance action prediction method based on target consistency screening and bidirectional state space

Abstract

The application discloses a power maintenance action prediction method based on target consistency screening and a bidirectional state space, first constructs an observation side video feature sequence, then carries out action semantic token coding, probability weighted mapping and time sequence modeling, obtains action semantic context representation, constructs a joint input vector based on the action semantic context representation, carries out time sequence modeling and distribution construction through a gate recurrent unit, obtains a feature side target distribution, and extracts an observation action representation; subsequently, a future action prediction model based on a bidirectional selective state space is used for prediction, and multiple candidate future action sequences are output; finally, an action side posterior distribution is constructed, a target consistency score is calculated, and the candidate future action sequence with the minimum score is selected as an optimal action prediction result. The application explicitly models statistical correlation and potential target distribution of maintenance actions, introduces a bidirectional selective state space diffusion generation architecture, and realizes accurate and logical prediction of power maintenance actions.

Description

Technical Field

[0001] This invention relates to the field of power maintenance video data processing technology, specifically a power maintenance action prediction method based on target consistency screening and bidirectional state space. Background Technology

[0002] With the rapid development of the Global Energy Internet and the continuous improvement of the intelligence and automation level of power systems, power equipment maintenance has become a crucial link in ensuring the safe and stable operation of the power grid. In recent years, video surveillance technology has been widely used in power maintenance. By collecting on-site video data in real time through fixed or wearable camera equipment, the maintenance process can be recorded and analyzed, achieving comprehensive perception of work behavior, equipment status, and on-site environment, providing a data foundation for intelligent analysis and auxiliary decision-making in the maintenance process.

[0003] However, existing power maintenance video applications mainly rely on post-event review, manual spot checks, and simple event recording, lacking sufficient ability to understand the timing of maintenance processes and predict future states. In complex power maintenance scenarios, there are strong logical connections and highly standardized processes among work actions. At the same time, the same current operation may correspond to different potential maintenance objectives, leading to drastically different subsequent processes. This diversity of future outcomes poses a significant uncertainty challenge to the prediction of long-duration, intensive actions.

[0004] In recent years, video understanding research has gradually evolved from action recognition to future action prediction, focusing on inferring the categories and execution order of subsequent actions based on observed video content. Existing methods mainly include temporal modeling methods based on recurrent networks such as LSTM and GRU, long sequence modeling methods based on TCN, and long-range dependency modeling methods based on Transformer. In recent years, probabilistic modeling methods for generating future action sequences have also emerged, aiming to improve the ability to represent the diversity and uncertainty of future evolution. While these methods have achieved some success in everyday behavioral scenarios, they still have shortcomings when applied to power maintenance videos: First, maintenance processes have strong process constraints and goal orientation, and existing methods do not adequately model the action transfer rules and task objectives; second, power maintenance videos generally suffer from long durations, sparse effective actions, numerous occlusions, and complex perspectives, easily leading to blurred boundaries of future actions and long-term accumulation of prediction errors; third, most existing methods focus on general scenarios and lack targeted modeling of professional semantics such as power equipment, tools, and maintenance specifications, thus failing to meet the actual needs of future action prediction and auxiliary analysis of work processes in power maintenance scenarios. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a power maintenance action prediction method based on target consistency screening and bidirectional state space. By decoupling action identification and sequence reasoning, the statistical correlation and potential target distribution of maintenance actions are explicitly modeled, and a bidirectional selective state space diffusion generation architecture with linear complexity is introduced to achieve accurate and logical prediction of power maintenance actions.

[0006] The technical solution of this invention is as follows: A power maintenance action prediction method based on target consistency screening and bidirectional state space specifically includes the following steps: (1) Divide the power maintenance video into segments and label the maintenance action categories, and divide and combine all video segments into observation side video segment set and prediction side video segment set. Then encode the observation side video segment set to obtain the observation side video feature sequence. (2) Encode each observation video feature vector in the observation side video feature sequence with action semantic tokens, perform probability weighting mapping on the probability distribution of action categories of all video segments, obtain the observation side action semantic token sequence, perform temporal modeling on the observation side action semantic token sequence, and obtain the action semantic context representation corresponding to all observation video feature vectors. (3) The feature vector of each observed video is concatenated with the corresponding action semantic context representation to obtain a joint input vector. The joint input vector is then subjected to temporal modeling and distribution construction by a gated loop unit to obtain the feature-side target distribution. The observed action representation is then extracted based on the feature-side target vector corresponding to the last video segment on the observation side. (4) The video feature sequence, action semantic token sequence and potential maintenance target vector of the last video segment on the observation side are fused to obtain a joint conditional sequence. The action tags of all video segments are encoded and then noise is added forward to obtain a noisy action sequence. Based on the joint conditional sequence and the noisy action sequence, a future action prediction model based on bidirectional selective state space is used to predict and output multiple candidate future action sequences. (5) Extract the action semantic summary of each candidate future action sequence, and then construct the action-side posterior distribution based on the observed action representation and action semantic summary. Finally, calculate the target consistency score through bidirectional KL divergence, and select the candidate future action sequence with the smallest target consistency score as the optimal action prediction result output.

[0007] The process of dividing power maintenance videos into segments yields... A sequence of non-overlapping video clips. Each video clip All contain continuous Frame images; then for each video segment Label each video segment with tags indicating the type of maintenance action. Each action has a corresponding action category label, which includes known action category labels. and unknown action category tags The action category labels of all video clips constitute the action annotation sequence. ; In all video clips, ordered chronologically, the first... The observation-side video clip set is composed of several video clips, and then... The prediction side video clip set consists of several video clips. ; The collection of video clips from the observation side Each video segment uses a pre-trained visual encoder. Extracting spatiotemporal features, see the following formula (1): (1); In equation (1), Represents each video segment in the set of video segments observed. The encoded observation-side video features, and the observation-side video feature sequence. .

[0008] Each observation video feature vector in the observation-side video feature sequence is encoded with an action semantic token. The probability distributions of action categories in all video segments are then subjected to probability weighting mapping to obtain an observation-side action semantic token sequence. Temporal modeling is then performed on the observation-side action semantic token sequence to obtain the action semantic context representation corresponding to all observation video feature vectors. Specifically: S21. Perform action semantic token encoding on each observation video feature vector in the observation-side video feature sequence to obtain the action category probability distribution of all video segments. Action semantic token encoding is essentially the process of encoding each observation-side video feature vector... Action Classification Head Classify and process the data, and classify the actions. It consists of two fully connected layers and a Softmax activation function, as shown in equation (2) below: (2); In equation (2), This represents the total number of maintenance action categories. This represents the probability distribution of the action category corresponding to the t-th video segment in the set of video segments observed on the observation side. S22. Perform probability weighting mapping on the action category probability distribution of each video segment to obtain the action semantic token, as shown in the following formula (3): (3); In equation (3), The maintenance action in the t-th video segment of the observation side video segment set belongs to the t-th video segment. Predicted probability of action class Indicates the first Learnable action embedding vectors corresponding to class actions; The action semantic token represents the t-th video segment in the observation-side video segment set; the action semantic tokens of all video segment actions in the observation-side video segment set constitute the observation-side action semantic token sequence. ; S23, Action semantic token sequence on the observation side Temporal modeling is performed to obtain the semantic context representation of the action, as shown in equation (4) below: (4); In equation (4), The representative action semantic sequence encoder uses a causal mask self-attention mechanism to perform temporal modeling of the action semantic token sequence on the observation side. The action semantic context representation sequence is composed of the action semantic context representations of all video segments in the observation-side video segment set. composition.

[0009] The process involves concatenating the feature vector of each observed video with its corresponding action semantic context representation to obtain a joint input vector. This joint input vector is then subjected to temporal modeling and distribution construction via a gated recurrent unit to obtain the feature-side target distribution. Finally, the observed action representation is extracted based on the feature-side target vector corresponding to the last video segment observed. Specifically: S31. Concatenate the feature vector of each observed video with the corresponding action semantic context representation to obtain the joint input vector. , It represents splicing. and These are the observed video feature vectors. and action semantic context representation The dimension; S32. The joint input vector is used for time-series modeling by a gated recurrent unit (GRU) to obtain the recurrent hidden state at the current time step. ,based on Potential maintenance target variables The distribution is estimated, that is, prior distributions based on historical information are constructed respectively. and combined with the posterior distribution of current observations See equations (5) and (6) below for details: (5); (6); In equations (5) and (6), , These represent the mean and standard deviation of the prior distribution, respectively. , These represent the mean and standard deviation of the posterior distribution, respectively, derived from the recurrent hidden state of the previous time step. and the joint input vector at the current time step Joint estimation; Represents a Gaussian distribution; S33, at the last time step of the observation sequence, i.e., the first time step in the set of video clips on the observation side. Based on all observation information, the feature-side target distribution is obtained for each time step corresponding to a video segment. ,Right now , The first video clip in the representative observation side set The posterior distribution of each video segment at each time step; S34, For the first video clip in the observation side video clip set The potential maintenance target variable at each time step corresponding to a video segment With cyclic hidden state Construct observation action representation vector This is used to encode the action execution state and historical semantic context up to the current moment, as shown in the following formula (7): (7); In equation (7), , and All are linear mapping layers.

[0010] During the training phase, the gated recurrent unit obtains the joint input vector at the current time step. Potential maintenance target variables are sampled from the posterior distribution using reparameterization techniques, specifically as shown in equation (8): (8); In equation (8), Represents element-wise multiplication. Represents a standard Gaussian noise vector. The unit covariance matrix is ​​then combined with the joint input vector at the current time step. Together, update the cyclic hidden state.

[0011] The process involves fusing the observation-side video feature sequence, action semantic token sequence, and the potential maintenance target vector of the last video segment on the observation side to obtain a joint conditional sequence. Then, after encoding the action tags labeled in all video segments, noise is added forward to obtain a noisy action sequence. Based on the joint conditional sequence and the noisy action sequence, a future action prediction model based on bidirectional selective state space is used for prediction, outputting multiple candidate future action sequences. Specifically: S41. The video feature sequence, action semantic token sequence and potential maintenance target vector of the last video segment on the observation side are fused to obtain the joint condition sequence, as shown in the following formula (9): (9); In equation (9), the joint conditional sequence The t-th fusion feature , and All are learnable parameters; due to future... Since the joint conditional sequence of the video segments is unavailable, zeros are used for padding. S42. Encode the action tags of all video segments and then perform forward diffusion noise addition. This represents the set of one-hot encoded vectors for actions, i.e., the sequence of actual action labels. From action annotation sequence The action category tags in the code are obtained by one-hot encoding, and a length of is defined. Forward diffusion Markov chains, for diffusion steps Gradually towards Injecting Gaussian noise into the middle, the specific process is shown in the following equation (10): (10); In equation (10), During the forward diffusion noise addition process, the first... Noisy action label sequence Transfer to the Noisy action label sequence The conditional probability distribution, The representative mean is The covariance matrix is Gaussian distribution, , For noise scheduling parameters; Represents the cumulative signal retention rate. Represents the single-step signal retention rate. , Represents the sequence of retained real action labels at the j-th diffusion step. The proportion; Arbitrary diffusion step Noisy action tag sequences Depend on The sampled data is shown in the following formula (11): (11); In equation (11), Represents a given sequence of real action labels Under the conditions, the first Noisy action label sequence The conditional probability distribution; The representative mean is The covariance matrix is Gaussian distribution; Represents a sequence of real action labels After the first The noisy action tag sequence obtained by forward diffusion and noise addition. The construction process is shown in the following formula (12): (12); In equation (12), Represents Gaussian noise; when the diffusion step hour, It approximately follows a standard Gaussian distribution; S43, in any diffusion step Noisy action label sequence With joint conditional sequence The features are concatenated and mapped to the hidden layer dimension via a linear projection layer. The initial input feature sequence of the bidirectional selective state space layer is obtained. Initial input feature sequence Initial input features ; S44. The future action prediction model uses a bidirectional selective state space layer, namely the BSSL layer, for bidirectional temporal modeling. The processing of the forward branch of the BSSL layer is shown in the following equation (13), and the processing of the backward branch of the BSSL layer is shown in the following equation (14): (13); (14); In equations (13) and (14), and These represent two independent unidirectional state-space mapping processes. This represents reversing the sequence along the time dimension. These are the output features of the forward and backward branches at the current time step, respectively. After summing the output features of the forward and backward branches element by element, and mapping them back to the initial feature space, see the following equation (15): (15); In equation (15), and These are the learnable weight matrix and bias terms of the output layer of the BSSL layer. The output features represent the mapped features, and the output features corresponding to all video segments form the BSSL layer output sequence. ; Construct a B-layer BSSL layer, and add a feedforward network to each BSSL layer to form a B-layer BSSL-FFN layer. The processing procedure of the BSSL-FFN layer is shown in the following formula (16): (16); In equation (16), This is the input for the first BSSL layer; Representative layer normalization; This represents a feedforward network, consisting of two linear layers and an activation function. This represents an addition operation, i.e., residual connection; After processing by the BSSL-FFN layer (B layer), the final feature representation is obtained. ; S45. The future motion prediction model uses a prediction head to predict real motion label sequences. The reconstruction estimate is shown in the following formula (17): (17); In equation (17), and For the linear layer weights and biases of the prediction head, For future action prediction models in diffusion steps right The reconstruction results at each time step represent the probability distribution of each action category. S46. Independent sampling from a standard Gaussian distribution An initial Gaussian noise Subsequently, using the same joint condition sequence To guide, Each initial Gaussian noise is input into the trained future action prediction model for progressive inverse denoising. After one iteration, output Parallel candidate future action sequences .

[0012] The unidirectional state space mapping process specifically involves: defining the hidden state vector. , Given the state space dimension, the process of updating the unidirectional state space mapping is shown in equation (18): (18); In equation (18), The forget gate represents the control over the retention of historical information. This represents the input gate that controls the current input. The output gate that controls the output selection. , and Each represents a learnable linear projection weight matrix, used to weight the initial input features. Mapped to a gating signal; Represents the Sigmoid function; This represents the hidden state vector at the previous time step. This represents the hidden state vector at the current time step. The output features represent the unidirectional state-space mapping; The and These represent two independent unidirectional state-space mapping processes, namely When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix. , When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix. .

[0013] The process involves extracting an action semantic summary for each candidate future action sequence, constructing an action-side posterior distribution based on the observed action representation and the action semantic summary, and finally calculating the target consistency score using bidirectional KL divergence. The candidate future action sequence with the smallest target consistency score is then selected as the optimal action prediction result. Specifically: S51, the kth candidate future action sequence , Each candidate future action sequence represents the next... The action representation at each time step, and then The action representation at each time step is achieved through the action embedding matrix. The mapping is to a series of action semantic tokens, as shown in equation (19) below: (19); In equation (19), represents the future The action semantic tokens corresponding to each time step are combined to obtain the action semantic token sequence for the future segment. ; Represents transposition; S52, Observer-side action semantic token sequence Sequence of semantic tokens for future actions The action semantic sequence encoder of input formula (4) after time-sequential concatenation Temporal modeling is performed, and the semantic context representation of the action in the future segment is averaged to obtain the future action summary vector, as shown in the following formula (20): (20); In equation (20), represents the future The semantic context representation of the action at each time step. Representative future action sequence Future action summary vector; S53, The observation action representation vector calculated based on equation (7) and future action summary vector , build The corresponding action-side target posterior distribution is shown in the following equation (21): (twenty one); In equation (21), represent The corresponding potential maintenance target variables, and Represent The mean and standard deviation of the posterior distribution of the target on the action side; S54, Calculate the first Candidate future action sequences Target consistency score , see the following formula (22) for details: (twenty two); In equation (22), KL divergence is used to measure the difference between two distributions. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of feature-side targets To approximate the action-side target posterior distribution Information loss; The candidate future action sequence with the lowest target consistency score is selected as the optimal candidate future action sequence, and this optimal candidate future action sequence is taken as the optimal action prediction result. , see the following formula (23) for details: (twenty three); In equation (23), Indicates the index of the optimal candidate future action sequence. Representing the A sequence of candidate future actions.

[0014] The power maintenance action prediction method is trained using a constructed joint optimization objective function. See the following formula (24) for details: (twenty four); In equation (24), The diffusion reconstruction loss function is calculated by the following equation (25); The cross-entropy loss function is calculated by the following equation (26); The target consistency loss function is calculated by the following equation (27); The time series KL divergence regularization loss is calculated by the following equation (28); , and The balance weight coefficients representing each loss term; (25); In equation (25), Represents a sequence of actual action labels. The model represents the future action prediction model in the diffusion step. right The reconstruction results Represents the square of the Euclidean distance. Represents the diffusion step. Represents Gaussian noise. Representative to The distribution of the three factors takes the expected value; by minimizing... This optimizes the parameters in the future action prediction model. (26); In equation (26), Representative action category probability distribution In real action category labels The component value at the location; This represents the total number of video clips in the observed video clip set; (27); In equation (27), Represents a sequence of actions in the real future The corresponding action-side target posterior distribution, the true future action sequence It is a sequence of real action tags The middle and rear F real action tags are composed of and Represent The mean and standard deviation of the posterior distribution of the target on the action side; Represents the target distribution on the characteristic side; Represents the KL divergence. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of feature-side targets To approximate the action-side target posterior distribution Information loss; To make the posterior distribution Approximate prior distribution The time-series Kullback-Leibler divergence is introduced as a regularization term, as shown in equation (28): (28); In equation (28), This represents the Kullback-Leibler divergence, or KL divergence, which measures the difference between the posterior and prior distributions. The smaller the KL divergence, the higher the consistency between the two distributions.

[0015] Advantages of this invention: (1) This invention constructs an action semantic token encoding and causal mask self-attention mechanism. In view of the strong process standardization of power maintenance operations, an action classification head is set on the video features of the observation side. Combined with the learnable action embedding matrix, discrete action categories are mapped to continuous action semantic token sequences. Furthermore, an action semantic sequence encoder based on the causal mask self-attention mechanism is introduced to perform temporal modeling of historical action semantics, thereby forming an action semantic representation that has both category discrimination information and context dependency information, providing a unified semantic foundation for the construction of future action summaries and target consistency evaluation.

[0016] (2) The present invention constructs a potential maintenance target distribution modeling and target consistency screening mechanism. By using a gated loop unit to estimate the feature-side target distribution based on historical observation information and the action-side target posterior distribution based on candidate future action sequences, the target consistency score is calculated by using bidirectional KL divergence. From the multiple candidate future action sequences generated by the diffusion model, the optimal prediction result that best matches the current maintenance target is accurately selected, effectively avoiding the problem of logical separation in the prediction sequence.

[0017] (3) The present invention constructs a future action prediction model based on bidirectional selective state space. It replaces the dilated convolution and Transformer structure in the traditional diffusion model with a bidirectional selective state space layer, namely the BSSL layer. This not only maintains linear computational complexity for long-term video sequences and significantly reduces memory consumption, but also enables the generation of high-quality future action sequences through an input-dependent dynamic gating mechanism and a bidirectional global receptive field.

[0018] (4) This invention establishes a multi-source spatiotemporal information fusion and unified joint optimization framework, which integrates the video feature sequence, action semantic token sequence and the potential maintenance target vector of the last video segment on the observation side into a unified feature space for time-by-time splicing and linear projection, and serves as the joint guiding condition for the diffusion network. At the same time, during the training phase, a joint optimization objective function is constructed, which includes the diffusion reconstruction loss function, the cross-entropy loss function, the target consistency loss function and the time series KL divergence regularization loss, so as to comprehensively improve the prediction accuracy and robustness when dealing with multi-class and unbalanced power maintenance data. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] A power maintenance action prediction method based on target consistency screening and bidirectional state space includes the following steps: (1) The power maintenance video is segmented and labeled with maintenance action categories. All video segments are then divided and combined into an observation-side video segment set and a prediction-side video segment set. The observation-side video segment set is then encoded to obtain the observation-side video feature sequence, specifically: S11. Divide the power maintenance video into segments to obtain... A sequence of non-overlapping video clips. Each video clip All contain continuous Frame images; then for each video segment Label each video segment with tags indicating the type of maintenance action. Each action has a corresponding action category label, which includes known action category labels. and unknown action category tags The action category labels of all video clips constitute the action annotation sequence. ; S12. In all video clips, ordered chronologically, the first... The observation-side video clip set is composed of several video clips, and then... The prediction side video clip set consists of several video clips. ; S13, The video clips from the observation side set Each video segment uses a pre-trained visual encoder. Extracting spatiotemporal features, see the following formula (1): (1); In equation (1), Represents each video segment in the set of video segments observed. The encoded observation-side video features, and the observation-side video feature sequence. ; (2) Encode each observation video feature vector in the observation-side video feature sequence with action semantic tokens, perform probability weighting mapping on the probability distribution of action categories of all video segments to obtain the observation-side action semantic token sequence, perform temporal modeling on the observation-side action semantic token sequence to obtain the action semantic context representation corresponding to all observation video feature vectors, specifically: S21. Perform action semantic token encoding on each observation video feature vector in the observation-side video feature sequence to obtain the action category probability distribution of all video segments. Action semantic token encoding is essentially the process of encoding each observation-side video feature vector... Action Classification Head Classify and process the data, and classify the actions. It consists of two fully connected layers and a Softmax activation function, as shown in equation (2) below: (2); In equation (2), This represents the total number of maintenance action categories. This represents the probability distribution of the action category corresponding to the t-th video segment in the set of video segments observed on the observation side. S22. Perform probability weighting mapping on the action category probability distribution of each video segment to obtain the action semantic token, as shown in the following formula (3): (3); In equation (3), The maintenance action in the t-th video segment of the observation side video segment set belongs to the t-th video segment. Predicted probability of action class Indicates the first Learnable action embedding vectors corresponding to class actions; The action semantic token represents the t-th video segment in the observation-side video segment set; the action semantic tokens of all video segment actions in the observation-side video segment set constitute the observation-side action semantic token sequence. ; S23, Action semantic token sequence on the observation side Temporal modeling is performed to obtain the semantic context representation of the action, as shown in equation (4) below: (4); In equation (4), The representative action semantic sequence encoder uses a causal mask self-attention mechanism to perform temporal modeling of the action semantic token sequence on the observation side. The action semantic context representation sequence is composed of the action semantic context representations of all video segments in the observation-side video segment set. composition; (3) The feature vector of each observed video is concatenated with the corresponding action semantic context representation to obtain a joint input vector. The joint input vector is then subjected to temporal modeling and distribution construction by a gated recurrent unit to obtain the feature-side target distribution. The observed action representation is then extracted based on the feature-side target vector corresponding to the last video segment on the observation side, specifically: S31. Concatenate the feature vector of each observed video with the corresponding action semantic context representation to obtain the joint input vector. , It represents splicing. and These are the observed video feature vectors. and action semantic context representation The dimension; S32. The joint input vector is used for time-series modeling by a gated recurrent unit (GRU) to obtain the recurrent hidden state at the current time step. ,based on Potential maintenance target variables The distribution is estimated, that is, prior distributions based on historical information are constructed respectively. and combined with the posterior distribution of current observations See equations (5) and (6) below for details: (5); (6); In equations (5) and (6), , These represent the mean and standard deviation of the prior distribution, respectively. , These represent the mean and standard deviation of the posterior distribution, respectively, derived from the recurrent hidden state of the previous time step. and the joint input vector at the current time step Joint estimation; Represents a Gaussian distribution; During the training phase, the gated recurrent unit obtains the joint input vector at the current time step. Potential maintenance target variables are sampled from the posterior distribution using reparameterization techniques, specifically as shown in equation (8): (8); In equation (8), Represents element-wise multiplication. Represents a standard Gaussian noise vector. The unit covariance matrix is ​​then combined with the joint input vector at the current time step. Together, update the hidden state of the cycle; S33, at the last time step of the observation sequence, i.e., the first time step in the set of video clips on the observation side. Based on all observation information, the feature-side target distribution is obtained for each time step corresponding to a video segment. ,Right now , The first video clip in the representative observation side set The posterior distribution of each video segment at each time step; S34, For the first video clip in the observation side video clip set The potential maintenance target variable at each time step corresponding to a video segment With cyclic hidden state Construct observation action representation vector This is used to encode the action execution state and historical semantic context up to the current moment, as shown in the following formula (7): (7); In equation (7), , and All are linear mapping layers; (4) The video feature sequence, action semantic token sequence, and potential maintenance target vector of the last video segment on the observation side are fused to obtain a joint conditional sequence. After encoding the action tags of all video segments, noise is added forward to obtain a noisy action sequence. Based on the joint conditional sequence and the noisy action sequence, a future action prediction model based on bidirectional selective state space is used to predict and output multiple candidate future action sequences, specifically: S41. The video feature sequence, action semantic token sequence and potential maintenance target vector of the last video segment on the observation side are fused to obtain the joint condition sequence, as shown in the following formula (9): (9); In equation (9), the joint conditional sequence The t-th fusion feature , and All are learnable parameters; due to future... Since the joint conditional sequence of the video segments is unavailable, zeros are used for padding. S42. Encode the action tags of all video segments and then perform forward diffusion noise addition. This represents the set of one-hot encoded vectors for actions, i.e., the sequence of actual action labels. From action annotation sequence The action category tags in the code are obtained by one-hot encoding, and a length of is defined. Forward diffusion Markov chains, for diffusion steps Gradually towards Injecting Gaussian noise into the middle, the specific process is shown in the following equation (10): (10); In equation (10), During the forward diffusion noise addition process, the first... Noisy action label sequence Transfer to the Noisy action label sequence The conditional probability distribution, The representative mean is The covariance matrix is Gaussian distribution, , For noise scheduling parameters; Represents the cumulative signal retention rate. Represents the single-step signal retention rate. , Represents the sequence of retained real action labels at the j-th diffusion step. The proportion; Arbitrary diffusion step Noisy action tag sequences Depend on The sampled data is shown in the following formula (11): (11); In equation (11), Represents a given sequence of real action labels Under the conditions, the first Noisy action label sequence The conditional probability distribution; The representative mean is The covariance matrix is Gaussian distribution; Represents a sequence of real action labels After the first The noisy action tag sequence obtained by forward diffusion and noise addition. The construction process is shown in the following formula (12): (12); In equation (12), Represents Gaussian noise; when the diffusion step hour, It approximately follows a standard Gaussian distribution; S43, in any diffusion step Noisy action label sequence With joint conditional sequence The features are concatenated and mapped to the hidden layer dimension via a linear projection layer. The initial input feature sequence of the bidirectional selective state space layer is obtained. Initial input feature sequence Initial input features ; S44. The future action prediction model uses a bidirectional selective state space layer, namely the BSSL layer, for bidirectional temporal modeling. The processing of the forward branch of the BSSL layer is shown in the following equation (13), and the processing of the backward branch of the BSSL layer is shown in the following equation (14): (13); (14); In equations (13) and (14), and These represent two independent unidirectional state-space mapping processes. This represents reversing the sequence along the time dimension. These are the output features of the forward and backward branches at the current time step, respectively. The one-way state space mapping process is as follows: Define the hidden state vector. , Given the state space dimension, the process of updating the unidirectional state space mapping is shown in equation (18): (18); In equation (18), The forget gate represents the control over the retention of historical information. This represents the input gate that controls the current input. The output gate that controls the output selection. , and Each represents a learnable linear projection weight matrix, used to weight the initial input features. Mapped to a gating signal; Represents the Sigmoid function; This represents the hidden state vector at the previous time step. This represents the hidden state vector at the current time step. The output features represent the unidirectional state-space mapping; and These represent two independent unidirectional state-space mapping processes, namely When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix. , When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix.

[0022] After summing the output features of the forward and backward branches element by element, and mapping them back to the initial feature space, see the following equation (15): (15); In equation (15), and These are the learnable weight matrix and bias terms of the output layer of the BSSL layer. The output features represent the mapped features, and the output features corresponding to all video segments form the BSSL layer output sequence. ; Construct a B-layer BSSL layer, and add a feedforward network to each BSSL layer to form a B-layer BSSL-FFN layer. The processing procedure of the BSSL-FFN layer is shown in the following formula (16): (16); In equation (16), This is the input for the first BSSL layer; Representative layer normalization; This represents a feedforward network, consisting of two linear layers and an activation function. This represents an addition operation, i.e., residual connection; After processing by the BSSL-FFN layer (B layer), the final feature representation is obtained. ; S45. The future motion prediction model uses a prediction head to predict real motion label sequences. The reconstruction estimate is shown in the following formula (17): (17); In equation (17), and For the linear layer weights and biases of the prediction head, For future action prediction models in diffusion steps right The reconstruction results at each time step represent the probability distribution of each action category. S46. Independent sampling from a standard Gaussian distribution An initial Gaussian noise Subsequently, using the same joint condition sequence To guide, Each initial Gaussian noise is input into the trained future action prediction model for progressive inverse denoising. After one iteration, output Parallel candidate future action sequences ; (5) Extract the action semantic summary of each candidate future action sequence, and then construct the action-side posterior distribution based on the observed action representation and the action semantic summary. Finally, calculate the target consistency score through bidirectional KL divergence, and select the candidate future action sequence with the smallest target consistency score as the optimal action prediction result output, specifically: S51, the kth candidate future action sequence , Each candidate future action sequence represents the next... The action representation at each time step, and then The action representation at each time step is achieved through the action embedding matrix. The mapping is to a series of action semantic tokens, as shown in equation (19) below: (19); In equation (19), represents the future The action semantic tokens corresponding to each time step are combined to obtain the action semantic token sequence for the future segment. ; Represents transposition; S52, Observer-side action semantic token sequence Sequence of semantic tokens for future actions The action semantic sequence encoder of input formula (4) after time-sequential concatenation Temporal modeling is performed, and the semantic context representation of the action in the future segment is averaged to obtain the future action summary vector, as shown in the following formula (20): (20); In equation (20), represents the future The semantic context representation of the action at each time step. Representative future action sequence Future action summary vector; S53, The observation action representation vector calculated based on equation (7) and future action summary vector , build The corresponding action-side target posterior distribution is shown in the following equation (21): (twenty one); In equation (21), represent The corresponding potential maintenance target variables, and Represent The mean and standard deviation of the posterior distribution of the target on the action side; S54, Calculate the first Candidate future action sequences Target consistency score , see the following formula (22) for details: (twenty two); In equation (22), KL divergence is used to measure the difference between two distributions. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of feature-side targets To approximate the action-side target posterior distribution Information loss; The candidate future action sequence with the lowest target consistency score is selected as the optimal candidate future action sequence, and this optimal candidate future action sequence is taken as the optimal action prediction result. , see the following formula (23) for details: (twenty three); In equation (23), Indicates the index of the optimal candidate future action sequence. Representing the A sequence of candidate future actions.

[0023] The model corresponding to the power maintenance action prediction method is trained through a constructed joint optimization objective function. See the following formula (24) for details: (twenty four); In equation (24), The diffusion reconstruction loss function is calculated by the following equation (25); The cross-entropy loss function is calculated by the following equation (26); The target consistency loss function is calculated by the following equation (27); The time series KL divergence regularization loss is calculated by the following equation (28); , and The balance weight coefficients representing each loss term; (25); In equation (25), Represents a sequence of actual action labels. The model represents the future action prediction model in the diffusion step. right The reconstruction results Represents the square of the Euclidean distance. Represents the diffusion step. Represents Gaussian noise. Representative to The distribution of the three factors takes the expected value; by minimizing... This optimizes the parameters in the future action prediction model. (26); In equation (26), Representative action category probability distribution In real action category labels The component value at the location; This represents the total number of video clips in the observed video clip set; (27); In equation (27), Represents a sequence of actions in the real future The corresponding action-side target posterior distribution, the true future action sequence It is a sequence of real action tags The middle and rear F real action tags are composed of and Represent The mean and standard deviation of the posterior distribution of the target on the action side; Represents the target distribution on the characteristic side; Represents the KL divergence. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of feature-side targets To approximate the action-side target posterior distribution Information loss; To make the posterior distribution Approximate prior distribution The time-series Kullback-Leibler divergence is introduced as a regularization term, as shown in equation (28): (28); In equation (28), This represents the Kullback-Leibler divergence, or KL divergence, which measures the difference between the posterior and prior distributions. The smaller the KL divergence, the higher the consistency between the two distributions.

[0024] The overall method of this invention is implemented using the PyTorch deep learning framework and deployed on a server environment equipped with high-performance GPUs. The pre-trained visual encoder uses a video Transformer model pre-trained on a large-scale video dataset. During training, the backbone parameters are frozen, and only the adapter layer is fine-tuned to balance computational efficiency and feature representation capability.

[0025] During the training phase, parameters are updated according to the method steps of this invention, based on the video dataset collected and labeled by on-site power maintenance cameras; the initial learning rate is set to... The learning rate was decayed by combining a cosine annealing strategy, the batch size was set to 16, the total number of training rounds was set to 90, and the forward steps of the diffusion process were set to 300 to ensure that the model fully learns the future action distribution.

[0026] During the inference phase, for the same set of observed video clips, a joint conditional sequence is first obtained, followed by inverse diffusion denoising, with the diffusion step count set to 15. For each set of observation inputs, independent generation is performed. The candidate future action sequences are calculated, and the target consistency score is then used to select the candidate future action sequence that best matches the current potential maintenance target as the optimal action prediction result and output it.

[0027] The model's evaluation metrics include Mean over Classes (MoC) accuracy and Top-1 MoC accuracy. For the test set... There are n samples to be evaluated, and the model generates a total of n samples for each sample. The candidate future action sequence, number The future number corresponding to the candidate future action sequence The prediction results at each time step are denoted as The real label is recorded as , No. The candidate future action sequence in the 1st Class-by-class accuracy of maintenance actions See the following formula (29): (29); In equation (29), This represents the total number of samples to be evaluated in the test set; This represents the total number of future time steps that need to be predicted. This represents the analogy index corresponding to the maximum value in the vector; Representing the Maintenance-like actions; No. MoC accuracy of candidate future action sequences See the following formula (30): (30); In equation (30), The total number of maintenance action categories; Mean MoC accuracy The average class-by-class prediction performance of all candidate future action sequences is used to measure the following equation (31): (31); Top-1 MoC accuracy The class-wise prediction performance of the candidate sequence that best matches the ground truth label is shown in equation (32): (32); in, The higher the value, the stronger the model's overall ability to model the uncertainty of future actions; The higher the value, the higher the quality of the candidate sequence generated by the model, indicating that there are high-quality prediction results that are closest to the actual maintenance process. Both of the above indicators average the prediction results from the category dimension, which can weaken the dominant role of high-frequency action categories on the evaluation results and is more suitable for multi-category and unbalanced action prediction tasks in power maintenance scenarios.

[0028] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for predicting power maintenance actions based on target consistency screening and bidirectional state space, characterized in that: Specifically, it includes the following steps: (1) Divide the power maintenance video into segments and label the maintenance action categories, and divide and combine all video segments into observation side video segment set and prediction side video segment set. Then encode the observation side video segment set to obtain the observation side video feature sequence. (2) Encode each observation video feature vector in the observation side video feature sequence with action semantic tokens, perform probability weighting mapping on the probability distribution of action categories of all video segments, obtain the observation side action semantic token sequence, perform temporal modeling on the observation side action semantic token sequence, and obtain the action semantic context representation corresponding to all observation video feature vectors. (3) The feature vector of each observed video is concatenated with the corresponding action semantic context representation to obtain a joint input vector. The joint input vector is then subjected to temporal modeling and distribution construction by a gated loop unit to obtain the feature-side target distribution. The observed action representation is then extracted based on the feature-side target vector corresponding to the last video segment on the observation side. (4) The video feature sequence, action semantic token sequence and potential maintenance target vector of the last video segment on the observation side are fused to obtain a joint conditional sequence. The action tags of all video segments are encoded and then noise is added forward to obtain a noisy action sequence. Based on the joint conditional sequence and the noisy action sequence, a future action prediction model based on bidirectional selective state space is used to predict and output multiple candidate future action sequences. (5) Extract the action semantic summary of each candidate future action sequence, and then construct the action-side posterior distribution based on the observed action representation and action semantic summary. Finally, calculate the target consistency score through bidirectional KL divergence, and select the candidate future action sequence with the smallest target consistency score as the optimal action prediction result output.

2. The power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 1, characterized in that: The process of dividing power maintenance videos into segments yields... A sequence of non-overlapping video clips. Each video clip All contain continuous Frame images; then for each video segment Label each video segment with tags indicating the type of maintenance action. Each action has a corresponding action category label, which includes known action category labels. and unknown action category tags The action category labels of all video clips constitute the action annotation sequence. ; In all video clips, ordered chronologically, the first... The observation-side video clip set is composed of several video clips, and then... The prediction side video clip set consists of several video clips. ; The collection of video clips from the observation side Each video segment uses a pre-trained visual encoder. Extracting spatiotemporal features, see the following formula (1): （1）； In equation (1), Represents each video segment in the set of video segments observed. The encoded observation-side video features, and the observation-side video feature sequence. .

3. The power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 2, characterized in that: Each observation video feature vector in the observation-side video feature sequence is encoded with an action semantic token. The probability distributions of action categories in all video segments are then subjected to probability weighting mapping to obtain an observation-side action semantic token sequence. Temporal modeling is then performed on the observation-side action semantic token sequence to obtain the action semantic context representation corresponding to all observation video feature vectors. Specifically: S21. Perform action semantic token encoding on each observation video feature vector in the observation-side video feature sequence to obtain the action category probability distribution of all video segments. Action semantic token encoding is essentially the process of encoding each observation-side video feature vector... Action Classification Head Classify and process the data, including the action classification head. It consists of two fully connected layers and a Softmax activation function, as shown in equation (2) below: （2）； In equation (2), This represents the total number of maintenance action categories. This represents the probability distribution of the action category corresponding to the t-th video segment in the set of video segments observed on the observation side. S22. Perform probability weighting mapping on the action category probability distribution of each video segment to obtain the action semantic token, as shown in the following formula (3): （3）； In equation (3), The maintenance action in the t-th video segment of the observation side video segment set belongs to the t-th video segment. Predicted probability of action class Indicates the first Learnable action embedding vectors corresponding to class actions; The action semantic token represents the t-th video segment in the observation-side video segment set; the action semantic tokens of all video segment actions in the observation-side video segment set constitute the observation-side action semantic token sequence. ; S23, Action semantic token sequence on the observation side Temporal modeling is performed to obtain the semantic context representation of the action, as shown in equation (4) below: （4）； In equation (4), The representative action semantic sequence encoder uses a causal mask self-attention mechanism to perform temporal modeling of the action semantic token sequence on the observation side. The action semantic context representation sequence is composed of the action semantic context representations of all video segments in the observation-side video segment set. composition.

4. The power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 3, characterized in that: The process involves concatenating the feature vector of each observed video with its corresponding action semantic context representation to obtain a joint input vector. This joint input vector is then subjected to temporal modeling and distribution construction via a gated recurrent unit to obtain the feature-side target distribution. Finally, the observed action representation is extracted based on the feature-side target vector corresponding to the last video segment observed. Specifically: S31. Concatenate the feature vector of each observed video with the corresponding action semantic context representation to obtain the joint input vector. , Represents splicing, and These are the observed video feature vectors. and action semantic context representation The dimension; S32. The joint input vector is used for time-series modeling by a gated recurrent unit (GRU) to obtain the recurrent hidden state at the current time step. ,based on Potential maintenance target variables The distribution is estimated, that is, prior distributions based on historical information are constructed respectively. and combined with the posterior distribution of current observations See equations (5) and (6) below for details: （5）； （6）； In equations (5) and (6), , These represent the mean and standard deviation of the prior distribution, respectively. , These represent the mean and standard deviation of the posterior distribution, respectively, derived from the recurrent hidden state of the previous time step. and the joint input vector at the current time step Joint estimation; Represents a Gaussian distribution; S33, at the last time step of the observation sequence, i.e., the first time step in the set of video clips on the observation side. Based on all observation information, the feature-side target distribution is obtained for each time step corresponding to a video segment. ,Right now , The first video clip in the representative observation side set The posterior distribution of each video segment at each time step; S34, For the first video clip in the observation side video clip set The potential maintenance target variable at each time step corresponding to a video segment With cyclic hidden state Construct observation action representation vector This is used to encode the action execution state and historical semantic context up to the current moment, as shown in the following formula (7): （7）； In equation (7), , and All are linear mapping layers.

5. A power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 4, characterized in that: During the training phase, the gated recurrent unit obtains the joint input vector at the current time step. Potential maintenance target variables are sampled from the posterior distribution using reparameterization techniques, specifically as shown in equation (8): （8）； In equation (8), Represents element-wise multiplication. Represents a standard Gaussian noise vector. The unit covariance matrix is ​​then combined with the joint input vector at the current time step. Together, update the cyclic hidden state.

6. The power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 4, characterized in that: The process involves fusing the observation-side video feature sequence, action semantic token sequence, and the potential maintenance target vector of the last video segment on the observation side to obtain a joint conditional sequence. Then, after encoding the action tags labeled in all video segments, noise is added forward to obtain a noisy action sequence. Based on the joint conditional sequence and the noisy action sequence, a future action prediction model based on bidirectional selective state space is used for prediction, outputting multiple candidate future action sequences. Specifically: S41. The video feature sequence, action semantic token sequence and potential maintenance target vector of the last video segment on the observation side are fused to obtain the joint condition sequence, as shown in the following formula (9): （9）； In equation (9), the joint conditional sequence The t-th fusion feature , and All are learnable parameters; due to future... Since the joint conditional sequence of the video segments is unavailable, zeros are used for padding. S42. Encode the action tags of all video segments and then perform forward diffusion noise addition. This represents the set of one-hot encoded vectors for actions, i.e., the sequence of actual action labels. From action annotation sequence The action category tags in the code are obtained by one-hot encoding, and a length of is defined. Forward diffusion Markov chains, for diffusion steps Gradually towards Injecting Gaussian noise into the middle, the specific process is shown in the following equation (10): （10）； In equation (10), During the forward diffusion noise addition process, the first... Noisy action label sequence Transfer to the Noisy action label sequence The conditional probability distribution, The representative mean is The covariance matrix is Gaussian distribution, , For noise scheduling parameters; Represents the cumulative signal retention rate. Represents the single-step signal retention rate. , Represents the sequence of retained real action labels at the j-th diffusion step. The proportion; Arbitrary diffusion step Noisy action tag sequences Depend on The sampled data is shown in the following formula (11): （11）； In equation (11), Represents a given sequence of real action labels Under the conditions, the first Noisy action label sequence The conditional probability distribution; The representative mean is The covariance matrix is Gaussian distribution; Represents a sequence of real action labels After the first The noisy action tag sequence obtained by forward diffusion and noise addition. The construction process is shown in the following formula (12): （12）； In equation (12), Represents Gaussian noise; when the diffusion step hour, It approximately follows a standard Gaussian distribution; S43, in any diffusion step Noisy action label sequence With joint conditional sequence The features are concatenated and mapped to the hidden layer dimension via a linear projection layer. The initial input feature sequence of the bidirectional selective state space layer is obtained. Initial input feature sequence Initial input features ; S44. The future action prediction model uses a bidirectional selective state space layer, namely the BSSL layer, for bidirectional temporal modeling. The processing of the forward branch of the BSSL layer is shown in the following equation (13), and the processing of the backward branch of the BSSL layer is shown in the following equation (14): （13）； （14）； In equations (13) and (14), and These represent two independent unidirectional state-space mapping processes. This represents reversing the sequence along the time dimension. These are the output features of the forward and backward branches at the current time step, respectively. After summing the output features of the forward and backward branches element by element, and mapping them back to the initial feature space, see the following equation (15): （15）； In equation (15), and These are the learnable weight matrix and bias terms of the output layer of the BSSL layer. The output features represent the mapped features, and the output features corresponding to all video segments form the BSSL layer output sequence. ; Construct a B-layer BSSL layer, and add a feedforward network to each BSSL layer to form a B-layer BSSL-FFN layer. The processing procedure of the BSSL-FFN layer is shown in the following formula (16): （16）； In equation (16), This is the input for the first BSSL layer; Representative layer normalization; This represents a feedforward network, consisting of two linear layers and an activation function. This represents an addition operation, i.e., residual connection; After processing by the BSSL-FFN layer (B layer), the final feature representation is obtained. ; S45. The future motion prediction model uses a prediction head to predict real motion label sequences. The reconstruction estimate is shown in the following formula (17): （17）； In equation (17), and For the linear layer weights and biases of the prediction head, For future action prediction models in diffusion steps right The reconstruction results at each time step represent the probability distribution of each action category. S46. Independent sampling from a standard Gaussian distribution An initial Gaussian noise Subsequently, using the same joint condition sequence To guide, Each initial Gaussian noise is input into the trained future action prediction model for progressive inverse denoising. After one iteration, output Parallel candidate future action sequences .

7. A power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 6, characterized in that: The unidirectional state space mapping process specifically involves: defining the hidden state vector. , Given the state space dimension, the process of updating the unidirectional state space mapping is shown in equation (18): （18）； In equation (18), The forget gate represents the control over the retention of historical information. This represents the input gate that controls the current input. The output gate that controls the output selection. , and Each represents a learnable linear projection weight matrix, used to weight the initial input features. Mapped to a gating signal; Represents the Sigmoid function; This represents the hidden state vector at the previous time step. This represents the hidden state vector at the current time step. The output features represent the unidirectional state-space mapping; The and These represent two independent unidirectional state-space mapping processes, namely When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix. , When performing unidirectional state-space mapping, there is a corresponding learnable linear projection weight matrix. .

8. A power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 6, characterized in that: The process involves extracting an action semantic summary for each candidate future action sequence, constructing an action-side posterior distribution based on the observed action representation and the action semantic summary, and finally calculating the target consistency score using bidirectional KL divergence. The candidate future action sequence with the smallest target consistency score is then selected as the optimal action prediction result. Specifically: S51, the kth candidate future action sequence , Each candidate future action sequence represents the next... The action representation at each time step, and then The action representation at each time step is achieved through the action embedding matrix. The mapping is to a series of action semantic tokens, as shown in equation (19) below: （19）； In equation (19), represents the future The action semantic tokens corresponding to each time step are combined to obtain the action semantic token sequence for the future segment. ; Represents transposition; S52, Observer-side action semantic token sequence Sequence of semantic tokens for future actions The action semantic sequence encoder of input formula (4) after time-sequential concatenation Temporal modeling is performed, and the semantic context representation of the action in the future segment is averaged to obtain the future action summary vector, as shown in the following formula (20): （20）； In equation (20), represents the future The semantic context representation of the action at each time step. Representative future action sequence Future action summary vector; S53, The observation action representation vector calculated based on equation (7) and future action summary vector , build The corresponding action-side target posterior distribution is shown in the following equation (21): （21）； In equation (21), represent The corresponding potential maintenance target variables, and Represent The mean and standard deviation of the posterior distribution of the target on the action side; S54, Calculate the first Candidate future action sequences Target consistency score , see the following formula (22) for details: （22）； In equation (22), KL divergence is used to measure the difference between two distributions. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of characteristic-side targets To approximate the action-side target posterior distribution Information loss; The candidate future action sequence with the lowest target consistency score is selected as the optimal candidate future action sequence, and this optimal candidate future action sequence is taken as the optimal action prediction result. , see the following formula (23) for details: （23）； In equation (23), Indicates the index of the optimal candidate future action sequence. Representing the A sequence of candidate future actions.

9. A power maintenance action prediction method based on target consistency screening and bidirectional state space as described in claim 8, characterized in that: The power maintenance action prediction method is trained using a constructed joint optimization objective function. See the following formula (24) for details: （24）； In equation (24), The diffusion reconstruction loss function is calculated by the following equation (25); The cross-entropy loss function is calculated by the following equation (26); The target consistency loss function is calculated by the following equation (27); The time series KL divergence regularization loss is calculated by the following equation (28); , and The balancing weight coefficients representing each loss term; （25）； In equation (25), Represents a sequence of actual action labels. The model represents the future action prediction model in the diffusion step. right The reconstruction results Represents the square of the Euclidean distance. Represents the diffusion step. Represents Gaussian noise. Representative to The distribution of the three factors takes the expected value; by minimizing... This optimizes the parameters in the future action prediction model. （26）； In equation (26), Representative action category probability distribution In real action category labels The component value at the location; This represents the total number of video clips in the observed video clip set; （27）； In equation (27), Represents a sequence of actions in the real future The corresponding action-side target posterior distribution, the true future action sequence It is a sequence of real action tags It consists of F real action tags in the middle and later stages. and Represent The mean and standard deviation of the posterior distribution of the target on the action side; Represents the target distribution on the characteristic side; Represents the KL divergence. Used to measure the posterior distribution of action-side targets. To approximate the target distribution of the feature side Information loss Used to measure the distribution of characteristic-side targets To approximate the action-side target posterior distribution Information loss; To make the posterior distribution Approximate prior distribution The time-series Kullback-Leibler divergence is introduced as a regularization term, as shown in equation (28): （28）； In equation (28), This represents the Kullback-Leibler divergence, or KL divergence, which measures the difference between the posterior and prior distributions. The smaller the KL divergence, the higher the consistency between the two distributions.

Images