Cross-view-angle online action detection method based on probability time sequence mask attention

Abstract

The invention relates to a cross-view online motion detection method based on probability time sequence mask attention, and the method specifically comprises the steps: S1, obtaining training data, and obtaining video streams of a plurality of views in a target scene; s2, data preprocessing: extracting RGB and optical flow features from a video stream based on a pre-training model; s3, a double-branch network structure is constructed, probability branches are used for compressing the view-angle-level features into potential view-angle specific codes, and classification branches are used for conducting autoregressive classification through the codes; s4, a GRU-TMA module is constructed, and interference of remote historical information on current action detection is reduced; s5, performing model training and optimization, optimizing potential representation of probability branches by minimizing reconstruction loss and KL divergence loss, and optimizing action classification capability of classification branches through cross entropy loss; and S6, performing online action detection, performing action classification of sequence frames based on output of the double-branch network, and performing performance evaluation according to actual action labels. According to the invention, the accuracy and robustness of action detection are improved.

Description

Technical Field

[0001] The present invention relates to a cross-view online action detection technology based on probabilistic temporal masked attention, belonging to the technical field of action video understanding. Background Art

[0002] With the development of computer vision and deep learning technologies, the understanding and analysis of video content have become increasingly important. Especially in fields such as surveillance, human-computer interaction, and autonomous driving, real-time detection and recognition of actions in videos have become a key technical requirement.

[0003] In these application scenarios, the acquisition of multi-view video data has become increasingly common, because multi-view information can provide a more comprehensive understanding of the context, helping to improve the accuracy and robustness of action detection. For example, in shopping malls or public transportation hubs, multiple cameras capture scenes from different angles, enabling a more comprehensive monitoring of the flow of people; in sports events, multi-view cameras can capture every detail of the athletes' movements, providing rich data for sports analysis.

[0004] Traditional action detection methods usually train with a set of mixed camera views, with limited consideration of view-level cues and no attention to the performance of the model on unknown views. For example, as Figure 1 shown, when deployed in unseen views v 2 and v 3 , a model trained only on video data from a single viewpoint v 1 may have difficulty accurately detecting actions. Due to the inherent view diversity in the training dataset, this oversight is common, which masks the true complexity of adapting to new viewpoints.

[0005] In practical applications such as bank security surveillance, monitoring is usually performed by fixed cameras. The ability to generalize across different views is not only an academic issue but also a crucial operational necessity. This highlights the important research value in advancing online action detection (OAD) models to stand out under cross-view conditions. Summary of the Invention

[0006] Technical Problems:

[0007] Viewpoint differences: The same action may exhibit different characteristics under different viewpoints, which requires the model to be able to understand and fuse information from different viewpoints.

[0008] Time synchronization problem: In the training of actual models, the video streams of different viewpoints may not be completely synchronized due to reasons such as hardware or transmission, which brings additional complexity to the training of action detection.

[0009] Computing resource limitation: Analyzing multi-view video data consumes more computing resources. How to achieve efficient and accurate action detection with limited computing resources is a technical challenge.

[0010] To solve the above technical problems, the present invention proposes a cross-view online action detection method based on probabilistic temporal masked attention. By introducing a special network structure and training strategy, this method can ensure both the real-time performance and accuracy of the detection process.

[0011] Technical solution:

[0012] The present invention relates to a cross-view online action detection method based on probabilistic temporal masked attention, belonging to the field of action video understanding. This method includes the following steps:

[0013] S1, Training data acquisition: Obtain video streams from multiple perspectives in the target scene;

[0014] S2, Data preprocessing: Based on a pre-trained two-dimensional convolutional neural network (2D CNN), extract video features (RGB and optical flow features) from the video stream;

[0015] S3, Construct a two-branch network structure. The probability branch uses a variational autoencoder (VAE) to take the preprocessed video feature X -K+1～0 as input, compress the view-level video features into latent view-specific encodings. The classification branch then performs temporal masked attention interaction using these encodings, and finally mixes the outputs of each branch and calculates the cross-entropy loss according to the actual action labels;

[0016] S4, Construct a GRU-TMA module. By introducing a temporal masked attention mechanism in the classification branch, reduce the interference of distant historical information on the current action detection, and improve the real-time performance and accuracy of the model;

[0017] S5, Model training and optimization: Optimize the latent representation of the probability branch by minimizing the reconstruction loss and KL divergence loss, and at the same time optimize the action classification ability of the classification branch through the cross-entropy loss;

[0018] S6, Online action detection: Perform action classification on sequential frames based on the output of the two-branch network, use the output of the classification branch for action prediction, and evaluate the performance according to the actual action labels.

[0019] Further, the step S1 includes the following content:

[0020] (1) Check the view situation of the video dataset, which needs to satisfy that there are at least 3 different views monitoring or shooting the same action of the same sample;

[0021] (2) Check the length of the video dataset, ensuring that the video is in an unclipped state, with a background and several actions lasting for a certain period of time in the same video.

[0022] (3) Check the label situation of the video dataset, ensuring that the video has fine-grained action labels, and that there are identical labels in different perspectives that can form a mutual contrast for the training of the depth model.

[0023] Further, the step S2 includes the following content:

[0024] Using the real-time video stream as input data, every N frames of the video are grouped for processing, and the RGB and optical flow information of the frames within this group are extracted respectively. And a pre-trained convolutional neural network is used to extract the RGB and optical flow features in this part, where the convolutional neural network is a two-dimensional convolutional neural network (2D CNN), which is used to simultaneously capture the spatial information and temporal information of the video frames to achieve efficient recognition of the dynamic changes in the video content;

[0025] Further, the step S3 includes the following content:

[0026] (1) Obtain the feature subsequence using a sliding window. Sample the input using a window of length T to simulate the situation of long sequence inference during the training phase of the model; specifically: use a window of length T to sample the original feature input into several temporal windows At the same time, during model inference, it is assumed to infer the complete video sequence by default to simulate the real situation, and the original feature sequence is input where D is the dimension of the video feature, v i and v j represent different perspectives, x -K+1 is the Kth video frame in the past, and x 0 is the current frame.

[0027] (2) Construct an autoregressive model for the classification branch. Denote the online action detection model as Φ d , then the encoding process of the classification branch can be expressed as:

[0028]

[0029] where The autoregressive model can be constructed using typical recurrent neural network models, such as GRU, LSTM, etc.

[0030] GRU module construction:

[0031]

[0032] Among them, tanh represents the hyperbolic tangent activation function, W z , W r , W and U z , U r , U respectively represent the weight matrices of the update gate, the reset gate, and the candidate hidden state, b z , b r , b represents the corresponding bias vector, x t represents the input at time t, h t-1 represents the hidden state at time t-1, z t , r t respectively represent the outputs of the update gate and the reset gate at time t, represents the candidate hidden state at time t, h t represents the final hidden state at time t.

[0033] (3) Construct a probability model for the probability branch based on probability hypothesis, probability inference, and probability generation.

[0034] Probability hypothesis:

[0035] The present invention assumes that the features from different perspectives conform to a consistent distribution, that is, where p(z) is the probability distribution of the latent encoding. If the latent encoding z can effectively reconstruct the perspective v i , then this latent encoding can also represent some features of the viewpoint v j .

[0036] Probability inference:

[0037] The present invention approximates its posterior probability by maximizing the variational lower bound of the corresponding sequential features, and takes maximizing the likelihood probability as the optimization goal:

[0038]

[0039] where q φ is the variational distribution. To accelerate the approximation speed of the model, an MLP is used here.

[0040] Probability generation:

[0041] To promote a stable and differentiable sampling process and effectively reconstruct the sequential features after probability modeling, the present invention applies the reparameterization trick. Through a vector of random variables ε from , perform element-wise scaling and offset on it to align with the required Gaussian distribution parameters (mean μ and standard deviation σ), as follows:

[0042] z = μ + σ ⊙ ε,

[0043]

[0044] Among them, The probability generation process is composed of multiple linear cascades, and its output is used to calculate the reconstruction loss of the input video features by the probability model.

[0045] Furthermore, the step S4 includes the following contents:

[0046] Construct a GRU-TMA unit based on GRU and temporal masked attention mechanism - a neural network structure that combines the gated recurrent unit (GRU) and the temporal masked attention mechanism (TMA). The GRU part controls the flow of information by introducing an update gate and a reset gate, thus solving the long-term dependence problem of traditional RNNs while taking into account the inference speed of the model. The TMA module is responsible for applying the attention mechanism in the temporal dimension to highlight the features most relevant to the current action detection task and suppress irrelevant information. This attention mechanism is achieved by generating a mask matrix, which is formed by fusing distant memories, short-term memories, and future information, and performs targeted masking on different video frames in the long video sequence.

[0047] The output of TMA can be expressed by the following formula:

[0048]

[0049] Among them, F z = f q (z) represents the expansion of the latent code z, mapping it into the temporal coding space of the GRU output. At the same time, α is used to adjust the density of attention, is the temporal coding matrix, and H is the video feature coding sequence output by the GRU.

[0050] Furthermore, the step S5 includes the following contents:

[0051] The probability prediction part of the model is trained by minimizing the reconstruction error and the KL divergence. At the same time, the cross-entropy loss function is used to improve the action recognition accuracy of the classification branch. Finally, the present invention combines these three different loss functions to perform overall training on the model, enabling it to have the representation ability and action detection ability for video features from unseen perspectives at the same time.

[0052] Furthermore, the step S6 includes the following contents:

[0053] (1) Construct a classification MLP layer Φ for online action detection c .

[0054] (2) Combine the temporal codings of different branches and perform online action detection

[0055] (3) The mAP and mcAP metrics are used to measure the accuracy of online action detection, and the FPS and GFLOPs metrics are used to measure the complexity of model inference.

[0056] Beneficial effects:

[0057] The cross-view online action detection method of the present invention effectively improves the accuracy and robustness of action detection by introducing a probabilistic temporal mask attention mechanism. This method can integrate video information from different perspectives, enhance the model's adaptability to perspective differences, and at the same time, the optimized network design and training strategy ensure the real-time performance and computational efficiency of the detection process. In addition, the test results of the present invention on multiple datasets verify its good generalization ability and application potential in various monitoring and analysis tasks. It is easy to implement and integrate, providing an efficient and reliable technical solution for real-time action detection. Description of the drawings

[0058] Figure 1 Schematic diagram of cross-view online action detection in the present invention;

[0059] Figure 2 Framework diagram of the cross-view online action detection model based on probabilistic temporal mask attention of the present invention;

[0060] Figure 3 Framework diagram of the GRU-TMA unit structure of the present invention;

[0061] Figure 4 Framework diagram of the GRU module of the present invention;

[0062] Figure 5 Framework diagram of the TMA module of the present invention. Detailed implementation manners

[0063] In order to more clearly illustrate the technical solutions in the present invention, the present invention is described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0064] Refer to Figure 2 , a cross-view online action detection method based on probabilistic temporal mask attention, includes the following steps:

[0065] S1, Training data acquisition, obtaining videos that meet the conditions and making a supervised dataset to support cross-view online action detection experiments, as Figure 1 shown. Further, S1 includes the following contents:

[0066] (1) Check the perspective situation of the video dataset, which should satisfy that there are at least 3 different perspectives for monitoring or shooting the same action of the same sample, so as to ensure that the model can learn the spatial modeling ability for different perspectives from the training data;

[0067] (2) Check the length situation of the video dataset. In the actual application of online action detection, real-time action classification needs to be performed on the online video sequence. Therefore, the videos in the dataset should be in the unclipped state, and there are backgrounds and several actions lasting for a certain period of time in the same video to simulate the real situation;

[0068] (3) Check the label situation of the video dataset, which should satisfy that the videos have fine-grained action labels, and there are identical labels in different perspectives that are sufficient to form mutual comparison for the training of the deep model.

[0069] S2. Data preprocessing. In the multi-view online action detection system of the present invention, the real-time video stream is processed as input data. To effectively capture the action features in the video, the system divides the continuous N-frame video into a group, and marks the currently processed group as T. In addition, the system introduces the concepts of short-term memory and long-term memory to enhance the model's understanding and prediction ability of the dynamic changes in the video sequence.

[0070] Specifically, the first m s groups of video frames including the current group T are defined as the short-term memory fm s , and the first m l groups of the short-term memory are defined as the long-term memory fm l . The system extracts RGB and optical flow information from each memory group to obtain rich spatio-temporal features. For the short-term memory fm s and the long-term memory fm l , the system extracts RGB and optical flow features from each group respectively. The RGB features are extracted by selecting the RGB image of the [N / 2]-th frame in the group, while the optical flow features are extracted by calculating the cumulative optical flow field of the N - 1 optical flow images generated from the N-frame RGB images.

[0071] Further, the system uses a pre-trained two-dimensional convolutional neural network (2D CNN) to deeply extract the RGB and optical flow features of each group. This 2D CNN is specially designed to simultaneously capture the spatial information and temporal information of the video frames, so as to achieve efficient recognition of the dynamic changes in the video content. In each group, the RGB information is extracted by selecting the RGB image of the median frame in the group to ensure the representativeness of the features; while the optical flow information is extracted in the form of the cumulative optical flow field by superimposing the N - 1 optical flow images calculated from the N-frame RGB images, which helps to capture the motion information in the video.

[0072] The extracted RGB and optical flow features are concatenated along the channel dimension to form a set of comprehensive features. These features not only contain rich color and motion information, but also enhance the model's utilization of historical information in video sequences through the introduction of short-term and long-term memory, thereby improving the accuracy and robustness of action detection.

[0073] With this design, the system of the present invention can more comprehensively understand and analyze video content, especially in action detection tasks under multi-view and complex scenarios, showing excellent performance. In addition, the system can also be applied to various fields such as video surveillance, human-computer interaction, and autonomous driving, providing an efficient and reliable solution for real-time video analysis.

[0074] S3. Build a two-branch cross-view online action detection network based on the classification branch and the probability branch, specifically as follows:

[0075] (1) Obtain feature subsequences using a sliding window. Sample the input with a window of length T to enable the model to simulate the situation of long-sequence inference during the training phase.

[0076] (2) Construct an autoregressive model for the classification branch. Denote the online action detection model as Φ d , then the encoding process in the classification branch can be expressed as:

[0077]

[0078] where the autoregressive model can be constructed using typical recurrent neural network models such as GRU, LSTM, etc.

[0079] GRU module construction:

[0080]

[0081] where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, W z , W r , W and U z , U r , U represent the weight matrices of the update gate, reset gate, and candidate hidden state respectively, b z , b r , b represent the corresponding bias vectors, x t represents the input at time t, h t-1 represents the hidden state at time t - 1, z t , r t represent the outputs of the update gate and reset gate at time t respectively, represents the candidate hidden state at time t, h tRepresents the final hidden state at time t.

[0082] (3) Construct a probability model for the probability branches. The present invention models the temporal characteristics of the input video features respectively according to probability hypothesis, probability inference and probability generation.

[0083] Probability hypothesis:

[0084] The present invention assumes that the features from different perspectives conform to a consistent distribution, that is If the latent encoding z can effectively reconstruct the perspective v i , then this latent encoding can also represent part of the features of the viewpoint v j . In the present invention, z is initialized as a multi-dimensional Gaussian distribution.

[0085] Probability inference:

[0086] Due to the temporal motion relationship between adjacent video frames, their generation processes are not independent. Therefore, the present invention uses the average distribution of video frames within a time window to replace the original probability distribution of video frames. By maximizing the variational lower bound of the corresponding temporal features to approximate its posterior probability, and taking the maximization of the likelihood probability as the optimization objective:

[0087]

[0088] where q φ is the variational distribution. Simplifying the above formula, the following inequality relationship can be obtained:

[0089]

[0090] In the formula, D KL represents the KL divergence. The two terms on the right side are the KLD loss and the reconstruction loss To accelerate the approximation speed of the model, here an MLP is used to implement the mapping from the original feature distribution p(x) to the probability distribution p(z), that is, here the approximation of the variational function q φ is carried out.

[0091] Probability generation:

[0092] The probability generation part aims to sample and reconstruct the temporal features input to the probability model according to the latent variable z output in the probability inference stage. The present invention is completed using a decoder composed of multiple linear cascades, and its output is used to calculate the reconstruction loss of the probability model for the input video features.

[0093] In the present invention, the model enhances the temporal feature reconstruction ability of the model by introducing the reparameterization technique, and ensures the stability and differentiability of the sampling process. Specifically, using one from The random variable vector ε is scaled and translated for each of its tensor elements to match the parameters (mean μ and standard deviation σ) of the target Gaussian distribution. The mathematical expression for this process is as follows:

[0094] z = μ + σ ⊙ ε,

[0095]

[0096] where, In this way, the output X of the decoder (r) matches the dimension of the input, and can effectively reconstruct the temporal features after probability modeling, thus achieving better performance in the reconstruction loss calculation of video features.

[0097] S4. Construct the Figure 3 GRU-TMA module shown as follows. The specific steps are as follows:

[0098] When dealing with independent classification tasks, training the model from only one fixed perspective may cause the model to overly rely on the features of that perspective, resulting in a decline in generalization ability when facing other perspectives. To improve the model's performance on unseen perspectives, that is, its generalization ability on other perspectives, the present invention proposes an innovative method. By combining the gated recurrent unit (GRU) with temporal masking attention (TMA), a GRU-GMA unit is formed for the classification branch.

[0099] The GRU part of the GRU-GMA unit (see Figure 4 ) finely controls the flow of information by introducing an update gate and a reset gate, which not only solves the deficiencies of traditional recurrent neural networks (RNNs) in dealing with long-term dependence problems but also maintains the efficiency of model inference. At the same time, the temporal masking attention module (TMA) applies the attention mechanism in the temporal dimension. It generates a mask matrix to highlight the features most relevant to the current action detection task and suppress those irrelevant information. This mask matrix combines information from distant memories, short-term memories, and future information, and can specifically mask different video frames in a long video sequence, thereby improving the model's ability to detect actions.

[0100] Through this combination, the model can not only learn effective latent representations from visible perspectives but also use these representations to query and understand information from invisible perspectives, thus achieving better performance in the multi-view OAD task. This design enables the model to be more flexible and accurate when dealing with unseen perspectives, improving the applicability and robustness of the model in practical applications. The output of TMA can be expressed by the following formula:

[0101]

[0102] Among them, F z = f q (z) represents the expansion of the latent code z, mapping it into the temporal encoding space of the GRU output. At the same time, α is used to adjust the density of attention, is the temporal encoding matrix, and H is the video feature encoding sequence output by the GRU.

[0103] During the model training stage, a time window with a fixed length T is usually used for training. During inference, the model runs in an autoregressive manner, processing the entire video sequence from beginning to end. However, the interaction between model branches sometimes leads to unsatisfactory results. In particular, attention from the distant past may have a negative impact on the current OAD task. To avoid future information leakage, an attention masking matrix is usually used, setting the upper triangular region to -∞ to mask future time steps. However, when the video length K exceeds the window length T, it becomes impractical to consider frames beyond T time steps. Therefore, an attention mask needs to be formulated to reflect this limitation, ensuring that the model can effectively utilize relevant perspective information while suppressing the interference of irrelevant and future information when processing long video sequences. The present invention defines the following mask matrix:

[0104]

[0105] Among them, is the mask corresponding to the attention q i query temporal encoding h j , specifically as Figure 5 shown. The mask matrix designed by the present invention integrates the mask in an effective manner, allowing parallel computation and simplifying the entire OAD process.

[0106] S5, Model training and optimization, specifically the following steps:

[0107] The present invention defines the following losses according to the different stages of the probability branch and the output results of the classification branch:

[0108] (1) Temporal feature reconstruction loss:

[0109]

[0110] (2) KL divergence loss:

[0111]

[0112] (3) Online action detection cross-entropy loss:

[0113]

[0114] Among them, the reconstruction loss in (1) is calculated using MSE. During the training process of the model, different weights are assigned to the above losses for optimization. By minimizing the reconstruction loss and the KL divergence loss, the latent representation of the probability branch is optimized, and at the same time, the action classification ability of the classification branch is optimized through the cross-entropy loss. Specifically defined as:

[0115]

[0116] Among them, λ 1 , λ 2 and λ 3 are weight parameters.

[0117] S6, online action detection, specifically the following steps:

[0118] (1) Construct a classification MLP layer Φ c for cross-view online action detection. The input of this MLP is the temporal encoding dimension of the RNN, and the output dimension is the number of actions (including the background), aiming to use the MLP to compress the features in the complex temporal space into a low-dimensional action space.

[0119] (2) The classification branch effectively fuses the output of the probability branch using the temporal masking feature of TMA and performs online action detection The complete temporal encoding output by the GRU-TMA unit simultaneously has RGB information and optical flow information of both seen and unseen views. Using the classification layer MLP can cross-combine different views to make a more accurate judgment on the action category compared to single-view online action detection.

[0120] (3) Use the mean average precision (mAP) and the calibrated mean average precision (mcAP) as performance evaluation indicators to calculate the detection performance of the model. At the same time, use the number of parameters, GFLOPs (giga floating-point operations per second), and the inference speed (Frame Per Second, FPS) as efficiency evaluation indicators to calculate the efficiency performance of the model;

[0121] Through experimental verification on public datasets, it is proved the performance of the proposed method under different datasets and different modalities. In this embodiment, the DAHLIA and IKEAASM datasets are selected for experiments. The experimental results show that the method of the present invention has high effectiveness and generalization ability in dealing with long-time sequence action detection tasks. To verify the detection performance and running efficiency of the present invention, it is compared with traditional online action detection methods. The specific performance indicators are shown in Table 1, and the efficiency indicators are shown in Table 2.

[0122] Table 1 Comparison results of the online action detection performance between traditional online action detection methods and the present method in the cross-view scenario

[0123]

[0124] As can be seen from Table 1, the method of the present invention is superior to the traditional online action detection method in terms of both mAP and mcAP evaluation indicators. Therefore, this method has higher detection accuracy. The present invention not only solves the problems of insufficient background suppression and insufficient capture of temporal relationships, but also enables the system to have good generalization ability. At the same time, as can be seen from Table 2, the present invention effectively solves the problem of limited computing resources. Not only is the model scale greatly reduced, but the running speed is sufficient to meet the actual application requirements, and the effect on complex data sets exceeds that of traditional online action detection models, proving its effectiveness.

[0125] Table 2 Comparison results of online action detection efficiency indicators of this method in cross-view scenarios

[0126]

[0127] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any other changes or substitutions without adding substantial innovations should be covered by the protection scope of the present invention. Therefore, the specific protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A cross-view online action detection method based on probabilistic temporal mask attention, characterized in that: The steps include: S1, training data acquisition, obtains video streams from multiple perspectives of the target scene; S2, data preprocessing, extracts video features from the video stream based on a pre-trained two-dimensional convolutional neural network, including RGB and optical flow features; S3, build a dual-branch network structure, the probability branch uses the variational autoencoder to pre-process the video feature X -K+1～0 As input, the view-level video features are compressed into latent view-specific codes, which are then used by the classification branch to perform temporal masked attention interactions. Finally, the outputs of each branch are mixed and the cross-entropy loss is calculated based on the actual action labels. S4, builds the GRU-TMA module, and introduces the temporal mask attention mechanism in the classification branch to reduce the interference of distant historical information on current action detection and improve the real-time performance and accuracy of the model; S5, model training and optimization, optimizes the potential representation of the probability branch by minimizing the reconstruction loss and KL divergence loss, and optimizes the action classification ability of the classification branch by cross entropy loss; S6, online action detection, performs action classification of sequential frames based on the output of the dual-branch network, uses the output of the classification branch to perform action prediction, and performs performance evaluation based on the actual action labels.

2. The cross-view online action detection method based on probabilistic temporal mask attention according to claim 1 is characterized in that: The step S2 includes the following contents: Real-time video stream is used as input data, and every N frames of video constitute a group for processing. The RGB and optical flow information of the frames in this group are extracted using a pre-trained two-dimensional convolutional neural network. The two-dimensional convolutional neural network is used to simultaneously capture the spatial information and temporal information of the video frame, thereby achieving efficient recognition of dynamic changes in video content.

3. The cross-view online action detection method based on probabilistic temporal mask attention according to claim 1 is characterized in that: The step S3 specifically includes the following contents: (1) Sliding window to obtain feature subsequence; using a window of length T to input the original feature Sampling into several time windows At the same time, when the model is inferred, the full video sequence is inferred by default to simulate the real situation, and the original feature sequence is input Among them, D is the dimension of video features, v i With v j Indicates different viewing angles, x -K+1 is the Kth video frame in the past, and x0 is the current frame; (2) Construct an autoregressive model for the classification branch; the online action detection model is denoted as Φ d , the encoding process of the classification branch is expressed as: in, Autoregressive model - GRU module: Update gate z t : z t =Sigmoid(W z x t +U z h t-1 +b z ), Reset Gate t : r t =Sigmoid(W r x t +U r h t-1 +b r ), Candidate hidden states The final hidden state h t : Among them, tanh represents the hyperbolic tangent activation function, W z , W r , W and U z ,U r , U represent the weight matrices of the update gate, reset gate and candidate hidden state, respectively, b z ,b r ,b represents the corresponding bias vector, x t represents the input at time t, h t-1 represents the hidden state at time t-1, z t ,r t Respectively represent the outputs of the update gate and the reset gate at time t, represents the candidate hidden state at time t, h t represents the final hidden state at time t; (3) construct a probability model for the probability branch; Probability assumptions: Assume that the features of different perspectives conform to the consistent distribution, that is, Where p(z) is the probability distribution of the latent code. If the latent code z can effectively reconstruct the view v i , then it can also represent the viewpoint v j Some features of Probabilistic Inference: Use variational inference to approximate the posterior probability of the time series features, and maximize the likelihood probability as the optimization goal: Among them, q φ is the variational distribution, p(x) is the original feature distribution, and in order to speed up the approximation of the model, MLP is used here; probability generation: To facilitate a stable and differentiable sampling process, a reparameterization technique is applied; involving a The random variable vector ε of , rescaled and shifted to align with the desired Gaussian distribution parameters, is as follows: z=μ+σ⊙ε, Among them, mean μ and standard deviation σ are Gaussian distribution parameters; The probability generation process is completed by multiple linear layers, and its output is used to calculate the reconstruction loss of the probability model for the input video features.

4. The cross-view online action detection method based on probabilistic temporal mask attention according to claim 1 is characterized in that: The step S4 specifically includes the following contents: A GRU-TMA unit is constructed based on the gated recurrent unit GRU and the temporal mask attention mechanism TMA. In the GRU-TMA unit, the GRU part controls the flow of information by introducing update gates and reset gates, thereby solving the long-term dependency problem of traditional RNN and capturing the temporal dependency at the frame level; the TMA module is responsible for applying the attention mechanism in the temporal dimension to highlight the features most relevant to the current action detection task and suppress irrelevant information; This attention mechanism is implemented by generating a mask matrix that determines which historical information is important and which can be ignored at the current time step; The output of TMA is expressed as follows: Among them, F z =f q (z) represents the expansion of the latent code z, mapping it to the temporal coding space of the GRU output; at the same time, α is used to adjust the density of attention. is the temporal coding matrix, and H is the video feature coding sequence output by GRU.

5. The cross-view online action detection method based on probabilistic temporal mask attention according to claim 1, characterized in that: The step S5 specifically includes the following contents: The probability branch of the model is trained by minimizing the reconstruction loss and the KL divergence loss. At the same time, the action classification ability of the classification branch is optimized by the cross entropy loss. Finally, the model is globally trained by combining the three different losses, so that it has both the ability to represent the features of unseen perspective videos and the ability to detect actions.

6. The cross-view online action detection method based on probabilistic temporal mask attention according to claim 1, characterized in that: The step S6 specifically includes the following contents: (1) Construct the classification MLP layer Φ for online action detection c ; (2) Merge the temporal encoding of different branches and perform online motion detection (3) The mAP and mcAP indicators are used to measure the accuracy of online action detection, and the FPS and GFLOPs indicators are used to measure the complexity of model inference.

Images