A cross-view online action detection method based on probabilistic temporal mask attention

Abstract

This invention relates to a cross-view online action detection method based on probabilistic temporal mask attention. The specific steps include: S1, acquiring training data by obtaining video streams from multiple viewpoints in a target scene; S2, data preprocessing by extracting RGB and optical flow features from the video streams based on a pre-trained model; S3, constructing a dual-branch network structure, where the probabilistic branch compresses viewpoint-level features into latent viewpoint-specific codes, and the classification branch uses these codes for autoregressive classification; S4, constructing a GRU-TMA module to reduce interference from distant historical information on current action detection; S5, model training and optimization by minimizing reconstruction loss and KL divergence loss to optimize the latent representation of the probabilistic branch, while simultaneously optimizing the action classification capability of the classification branch through cross-entropy loss; S6, online action detection by classifying sequential frames based on the output of the dual-branch network and evaluating performance according to actual action labels. This invention improves the accuracy and robustness of action detection.

Description

Technical Field

[0001] This invention relates to a cross-view online action detection technology based on probabilistic temporal mask attention, belonging to the field of action video understanding technology. Background Technology

[0002] With the development of computer vision and deep learning technologies, the understanding and analysis of video content has become increasingly important. Especially in fields such as surveillance, human-computer interaction, and autonomous driving, real-time detection and recognition of actions in videos has become a critical technological requirement.

[0003] In these application scenarios, the acquisition of multi-view video data is becoming increasingly common because multi-view information can provide a more comprehensive understanding of the context, helping to improve the accuracy and robustness of motion detection. For example, in shopping malls or public transportation hubs, multiple cameras capture scenes from different angles, enabling more comprehensive monitoring of crowd dynamics; in sporting events, multi-view cameras can capture every detail of an athlete's movements, providing rich data for motion analysis.

[0004] Traditional action detection methods typically use a mixed set of camera views for training, offering limited consideration for viewpoint-level cues and neglecting model performance on unknown viewpoints. For example, ... Figure 1 As shown, when deployed in unseen viewpoints v2 and v3, models trained solely on video data from a single viewpoint v1 may struggle to accurately detect motion. This oversight is prevalent due to the inherent viewpoint diversity in the training dataset, masking the true complexity of adapting to new viewpoints.

[0005] In practical applications such as bank security monitoring, surveillance is typically performed by fixed cameras. The ability to generalize across different views is not only an academic issue but also a crucial operational necessity. This highlights the significant research value in advancing online action detection (OAD) models to excel under cross-view conditions. Summary of the Invention

[0006] Technical issues:

[0007] Perspective differences: The same action may exhibit different characteristics from different perspectives, which requires the model to understand and integrate information from different perspectives.

[0008] Time synchronization problem: In the training of actual models, video streams from different perspectives may not be fully synchronized due to hardware or transmission reasons, which brings additional complexity to the training of action detection.

[0009] Computational resource constraints: The analysis of multi-view video data requires more computing resources, and how to achieve efficient and accurate motion detection with limited computing resources is a technical challenge.

[0010] To address the aforementioned technical issues, this invention proposes a cross-view online action detection method based on probabilistic temporal mask attention. This method, by introducing a special network structure and training strategy, can simultaneously ensure the real-time performance and accuracy of the detection process.

[0011] Technical solution:

[0012] This invention relates to a cross-view online action detection method based on probabilistic temporal mask attention, belonging to the field of action video understanding. The method includes the following steps:

[0013] S1, Training data acquisition: Acquire video streams from multiple perspectives in the target scene;

[0014] S2, data preprocessing, based on a pre-trained two-dimensional convolutional neural network (2D CNN), extracts video features (RGB and optical flow features) from the video stream;

[0015] S3, construct a dual-branch network structure. The probabilistic branch utilizes a variational autoencoder (VAE) with preprocessed video features X. -K+1～0 As input, viewpoint-level video features are compressed into potential viewpoint-specific codes. The classification branch uses these codes to perform temporal mask attention interactions. Finally, the outputs of each branch are mixed and cross-entropy loss is calculated based on the actual action labels.

[0016] S4. Construct the GRU-TMA module, and reduce the interference of distant historical information on the detection of current actions by introducing a temporal mask attention mechanism in the classification branch, thereby improving the real-time performance and accuracy of the model.

[0017] S5, Model Training and Optimization: By minimizing reconstruction loss and KL divergence loss, the latent representation of the probability branch is optimized, while the action classification ability of the classification branch is optimized by cross-entropy loss.

[0018] S6, an online action detection system, classifies sequential frames based on the output of a dual-branch network, predicts actions using the output of the classification branches, and evaluates performance based on actual action labels.

[0019] Further, step S1 includes the following:

[0020] (1) Check the perspective of the video dataset. It is necessary to monitor or film the same action of the same sample from at least 3 different perspectives.

[0021] (2) Check the length of the video dataset. The video must be in an unedited state and contain background and several actions that last for a certain period of time.

[0022] (3) Check the labeling of the video dataset. The videos must have fine-grained action labels and have the same label that can be used to train the deep model in different perspectives.

[0023] Further, step S2 includes the following:

[0024] Real-time video streams are used as input data, and each group of N video frames is processed to extract RGB and optical flow information from each frame in the group. A pre-trained convolutional neural network (CNN) is then used to extract the RGB and optical flow features. This CNN is a two-dimensional CNN used to simultaneously capture the spatial and temporal information of the video frames, enabling efficient recognition of dynamic changes in the video content.

[0025] Further, step S3 includes the following:

[0026] (1) Obtain feature subsequences using a sliding window. Sample the input using a window of length T. This allows the model to simulate long sequence inference during the training phase; specifically, it uses a window of length T to input the original features. Sampling is performed in several time windows. Meanwhile, during model inference, the complete video sequence is inferred by default to simulate real-world situations, with the original feature sequence as input. Where D is the dimension of the video features, v i With v j To represent different perspectives, x -K+1 x is the Kth video frame from the past, and x0 is the current frame.

[0027] (2) Construct an autoregressive model for the classification branch. Let the online action detection model be Φ. d The coding process for the classification branch can then be represented as:

[0028]

[0029] in, The autoregressive model can be constructed using typical recurrent neural network models, such as GRU and LSTM.

[0030] GRU module construction:

[0031]

[0032] Where tanh represents the hyperbolic tangent activation function, W z W r ,W and U z U rU represents the weight matrix for the update gate, reset gate, and candidate hidden state, respectively. z ,b r ,b represents the corresponding bias vector, x t h represents the input at time t. t-1 Let z represent the hidden state at time t-1. t ,r t These represent the outputs of the update gate and the reset gate at time t, respectively. h represents the candidate hidden state at time t. t This represents the final hidden state at time t.

[0033] (3) Construct a probability model based on probability assumptions, probability inferences and probability generation as probability branches.

[0034] Probability assumptions:

[0035] This invention assumes that features from different perspectives conform to a consistent distribution, that is... Where p(z) is the probability distribution of the latent code. If the latent code z can effectively reconstruct the viewpoint v i Therefore, this latent encoding can also represent the viewpoint v. j Some of its features.

[0036] Probability inference:

[0037] This invention approximates the posterior probability by maximizing the variational lower bound of the corresponding temporal feature, and uses maximizing the likelihood probability as the optimization objective:

[0038]

[0039] Where, q φ Since it is a variational distribution, MLP is used here to speed up the model approximation.

[0040] Probability generation:

[0041] To facilitate a stable and differentiable sampling process and effectively reconstruct the temporal features after probabilistic modeling, this invention employs a reparameterization technique. This is achieved through a method derived from... The random variable vector ε is scaled and shifted at the tensor element level to align with the desired Gaussian distribution parameters (mean μ and standard deviation σ), as shown below:

[0042] z=μ+σ⊙ε,

[0043]

[0044] in, The probability generation process consists of multiple cascaded linear layers, and its output is used to calculate the reconstruction loss of the probabilistic model based on the input video features.

[0045] Further, step S4 includes the following:

[0046] A GRU-TMA unit is constructed based on GRU and temporal masked attention mechanism—a neural network structure combining GRU and Temporal Masked Attention (TMA). The GRU part controls the flow of information by introducing update and reset gates, thereby solving the long-term dependency problem of traditional RNNs while taking into account the inference speed of the model. The TMA module is responsible for applying the attention mechanism in the temporal dimension to highlight the features most relevant to the current action detection task and suppress irrelevant information. This attention mechanism is implemented by generating a mask matrix, which is a fusion of distant memory, short-term memory, and future information, to selectively mask different video frames in a long video sequence.

[0047] The output of TMA can be expressed by the following formula:

[0048]

[0049] Among them, F z =f q (z) represents the expansion of the latent code z, mapping it to the temporal coding space of the GRU output. Meanwhile, α is used to adjust the density of the attention. H is the temporal coding matrix, and H is the video feature coding sequence output by the GRU.

[0050] Further, step S5 includes the following:

[0051] The probabilistic prediction part of the model was trained by minimizing reconstruction error and KL divergence. Simultaneously, the cross-entropy loss function was used to improve the action recognition accuracy of the classification branch. Finally, this invention combines these three different loss functions to train the model as a whole, enabling it to simultaneously represent features from unseen video perspectives and detect actions.

[0052] Further, step S6 includes the following:

[0053] (1) Construct a classification MLP layer Φ for online action detection c .

[0054] (2) Merge the timing codes of different branches and perform online motion detection.

[0055] (3) Use mAP and mcAP metrics to measure the accuracy of online action detection, and use FPS and GFLOPs metrics to measure the complexity of model inference.

[0056] Beneficial effects:

[0057] The cross-view online action detection method of this invention effectively improves the accuracy and robustness of action detection by introducing a probabilistic temporal mask attention mechanism. This method integrates video information from different viewpoints, enhancing the model's adaptability to viewpoint differences. Meanwhile, the optimized network design and training strategy ensure the real-time performance and computational efficiency of the detection process. Furthermore, test results on multiple datasets verify its good generalization ability and application potential in various monitoring and analysis tasks. It is easy to implement and integrate, providing an efficient and reliable technical solution for real-time action detection. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of cross-view online motion detection in this invention;

[0059] Figure 2 This is a framework diagram of the cross-view online action detection model based on probabilistic temporal mask attention of the present invention;

[0060] Figure 3 This is a structural framework diagram of the GRU-TMA unit of the present invention;

[0061] Figure 4 This is a schematic diagram of the GRU module framework of the present invention;

[0062] Figure 5 This is a schematic diagram of the TMA module of the present invention. Detailed Implementation

[0063] To more clearly illustrate the technical solutions of this invention, the invention is described below in conjunction with the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention.

[0064] refer to Figure 2 A cross-view online action detection method based on probabilistic temporal mask attention includes the following steps:

[0065] S1, Training Data Acquisition: Acquire qualified videos and create a supervised dataset to support cross-view online action detection experiments, such as... Figure 1 As shown. Further, S1 includes the following:

[0066] (1) Check the perspective of the video dataset. It is necessary to monitor or film the same action of the same sample from at least 3 different perspectives to ensure that the model can learn the ability to model space from different perspectives from the training data.

[0067] (2) Check the length of the video dataset. In the actual application of online action detection, it is necessary to perform real-time action classification on the online video sequence. Therefore, the videos in the dataset must be in an unedited state, with background and several actions lasting a certain period of time in the same video to simulate the real situation.

[0068] (3) Check the labeling of the video dataset. The videos must have fine-grained action labels and have the same label that can be used to train the deep model in different perspectives.

[0069] S2, Data Preprocessing. In the multi-view online motion detection system of this invention, real-time video streams are processed as input data. To effectively capture motion features in the video, the system divides N consecutive video frames into a group, labeling the currently processed group as T. Furthermore, the system introduces the concepts of short-term memory and long-term memory to enhance the model's understanding and predictive ability regarding dynamic changes in video sequences.

[0070] Specifically, the top m, including the current group T s A group of video frames is defined as short-term memory (FM). s The first m of short-term memory l The group is defined as long-term memory fm l The system extracts RGB and optical flow information from each memory group to obtain rich spatiotemporal features. For short-term memory (fm)... s and long-term memory FM l The system extracts RGB and optical flow features from each group. The RGB features are extracted by selecting the RGB image of the [N / 2]th frame in the group, while the optical flow features are extracted by calculating the cumulative optical flow field of the N-1 frames of optical flow images generated from the N frames of RGB images.

[0071] Furthermore, the system utilizes a pre-trained two-dimensional convolutional neural network (2D CNN) to deeply extract RGB and optical flow features from each group. This 2D CNN is specifically designed to simultaneously capture spatial and temporal information from video frames, thereby achieving efficient recognition of dynamic changes in video content. In each group, RGB information is extracted by selecting the RGB image of the median frame within the group, ensuring the representativeness of the features; while optical flow information is extracted by superimposing N frames of RGB images to obtain N-1 frames of optical flow images, in the form of an accumulated optical flow field, which helps to capture motion information in the video.

[0072] The extracted RGB and optical flow features are concatenated along the channel dimension to form a set of comprehensive features. These features not only contain rich color and motion information, but also enhance the model's utilization of historical information in video sequences by introducing short-term and long-term memory, thereby improving the accuracy and robustness of action detection.

[0073] Through this design, the system of this invention can more comprehensively understand and analyze video content, especially demonstrating superior performance in motion detection tasks under multi-view and complex scenes. Furthermore, the system can be applied to various fields such as video surveillance, human-computer interaction, and autonomous driving, providing an efficient and reliable solution for real-time video analysis.

[0074] S3, a two-branch cross-view online action detection network is built based on classification and probability branches, specifically through the following steps:

[0075] (1) Obtain feature subsequences using a sliding window. Sample the input using a window of length T. This allows the model to simulate long sequence reasoning during the training phase.

[0076] (2) Construct an autoregressive model for the classification branch. Let the online action detection model be Φ. d Then the encoding process in the classification branch can be represented as:

[0077]

[0078] in, The autoregressive model can be constructed using typical recurrent neural network models, such as GRU and LSTM.

[0079] GRU module construction:

[0080]

[0081] Where σ represents the sigmoid activation function, tanh represents the hyperbolic tangent activation function, and W z W r ,W and U z U r U represents the weight matrix for the update gate, reset gate, and candidate hidden state, respectively. z ,b r ,b represents the corresponding bias vector, x t h represents the input at time t. t-1 Let z represent the hidden state at time t-1. t ,r t These represent the outputs of the update gate and the reset gate at time t, respectively. h represents the candidate hidden state at time t. t This represents the final hidden state at time t.

[0082] (3) Constructing a probabilistic model for the probabilistic branch. This invention models the temporal characteristics of the input video features based on probabilistic assumptions, probabilistic inferences, and probabilistic generation, respectively.

[0083] Probability assumptions:

[0084] This invention assumes that features from different perspectives conform to a consistent distribution, that is... If the latent encoding z can effectively reconstruct the viewpoint v i Therefore, this latent encoding can also represent the viewpoint v. j Some characteristics of z. In this invention, z is initialized as a multidimensional Gaussian distribution.

[0085] Probability inference:

[0086] Due to the temporal motion relationship between adjacent video frames, their generation processes are not independent. Therefore, this invention uses the average distribution of video frames within a time window to replace the original probability distribution of the video frames. It approximates the posterior probability by maximizing the variational lower bound of the corresponding temporal features, and uses maximizing the likelihood probability as the optimization objective.

[0087]

[0088] Where, q φ Let the distribution be a variational distribution. Simplifying the above equation, we obtain the following inequality:

[0089]

[0090] In the formula, D KL This represents the KL divergence. The two terms on the right are the KLD losses. and reconstruction losses To accelerate the model approximation, an MLP is used here to map the original feature distribution p(x) to the probability distribution p(z), i.e., the variational function q is mapped here. φ The approach.

[0091] Probability generation:

[0092] The probability generation part aims to sample and reconstruct the temporal features input to the probability model based on the latent variable z output from the probability inference stage. This invention uses a decoder composed of multiple cascaded linear layers to complete this, and its output is used to calculate the reconstruction loss of the probability model for the input video features.

[0093] In this invention, the model enhances its temporal feature reconstruction capability by introducing reparameterization techniques, while ensuring the stability and differentiability of the sampling process. Specifically, it utilizes a parameterization method derived from... The random variable vector ε is scaled and translated to match the parameters (mean μ and standard deviation σ) of the target Gaussian distribution. The mathematical expression for this process is as follows:

[0094] z=μ+σ⊙ε,

[0095]

[0096] in, In this way, the decoder's output X was modified. (r) Matching the dimension of the input allows for effective reconstruction of temporal features after probabilistic modeling, resulting in superior performance in video feature reconstruction loss calculation.

[0097] S4, build Figure 3 The GRU-TMA module shown below involves the following steps:

[0098] When dealing with independent classification tasks, training a model from only a fixed perspective may lead to over-reliance on features from that perspective, resulting in decreased generalization ability when facing other perspectives. To improve the model's performance in unseen perspectives, i.e., its generalization ability in other perspectives, this invention proposes an innovative method that combines a gated recurrent unit (GRU) with temporal masked attention (TMA) to form a GRU-GMA unit for classification branching.

[0099] The GRU portion of the GRU-GMA unit (see...) Figure 4 By introducing update and reset gates to finely control the flow of information, this not only addresses the shortcomings of traditional recurrent neural networks (RNNs) in handling long-term dependencies but also maintains the efficiency of model inference. Simultaneously, the Temporal Masking Attention (TMA) module applies an attention mechanism along the temporal dimension by generating a mask matrix. This method highlights the features most relevant to the current action detection task and suppresses irrelevant information. This mask matrix integrates information from distant memory, short-term memory, and future information, enabling targeted masking of different video frames in long video sequences, thereby improving the model's action detection capabilities.

[0100] Through this combination, the model can not only learn effective latent representations from the visible perspective, but also use these representations to query and understand information from the unseen perspective, thereby achieving better performance in multi-view OAD tasks. This design makes the model more flexible and accurate in handling unseen perspectives, improving its applicability and robustness in practical applications. The output of TMA can be represented by the following formula:

[0101]

[0102] Among them, F z =f q (z) represents the expansion of the latent code z, mapping it to the temporal coding space of the GRU output. Meanwhile, α is used to adjust the density of the attention. H is the temporal coding matrix, and H is the video feature coding sequence output by the GRU.

[0103] During model training, a fixed-length time window of T is typically used. During inference, the model operates in an autoregressive manner, processing the entire video sequence from beginning to end. However, interactions between model branches can sometimes lead to undesirable results, especially since attention from the distant past can negatively impact the current OAD task. To avoid leaking future information, an attention masking matrix is ​​usually used, setting the upper triangular region to -∞ to mask future time steps. However, when the video length K exceeds the window length T, considering frames beyond T time steps becomes impractical. Therefore, an attention mask needs to be designed to reflect this limitation, ensuring that the model can effectively utilize relevant viewpoint information while suppressing interference from irrelevant and future information when processing long video sequences. This invention defines the following mask matrix:

[0104]

[0105] in, This corresponds to attention q i Query timing code h j The mask, specifically as follows Figure 5 As shown. The mask matrix designed in this invention integrates masks in an efficient manner, allowing for parallel computation and simplifying the entire OAD process.

[0106] S5, Model Training and Optimization, specifically involves the following steps:

[0107] This invention defines the following loss based on the different stages of the probability branch and the output results of the classification branch:

[0108] (1) Temporal feature reconstruction loss:

[0109]

[0110] (2) KL divergence loss:

[0111]

[0112] (3) Cross-entropy loss for online action detection:

[0113]

[0114] In (1), the reconstruction loss is calculated using MSE. During training, the model optimizes the above losses by assigning different weights. By minimizing the reconstruction loss and KL divergence loss, the latent representation of the probability branch is optimized, while the action classification ability of the classification branch is optimized by using cross-entropy loss. Specifically, it is defined as:

[0115]

[0116] Where λ1, λ2 and λ3 are weight parameters.

[0117] S6, online motion detection, specifically involves the following steps:

[0118] (1) Construct a classification MLP layer Φ for cross-view online action detection c The input of this MLP is the temporal encoding dimension of the RNN, and the output dimension is the number of actions (including the background). The aim is to use MLP to compress the feature encoding in the complex temporal space into a low-dimensional action space.

[0119] (2) The classification branch utilizes the temporal masking characteristics of TMA to effectively fuse the outputs of the probability branch and perform online action detection. The complete temporal code output by the GRU-TMA unit contains RGB information and optical flow information from both seen and unseen viewpoints. Using the classification layer MLP, it can cross-combine different viewpoints to make a more accurate judgment on the category of action compared to single-view online action detection.

[0120] (3) The average accuracy (mAP) and calibration average accuracy (mcAP) are used as performance evaluation indicators to calculate the detection performance of the model. At the same time, the number of parameters, GFLOPs (1 billion floating-point operations per second), and inference speed (Frames Per Second, FPS) are used as efficiency evaluation indicators to calculate the efficiency performance of the model;

[0121] Experiments on publicly available datasets demonstrate the performance of the proposed method across different datasets and modalities. In this embodiment, the DAHLIA and IKEAASM datasets were selected for experiments. Experimental results show that the method of this invention has high effectiveness and generalization ability in handling long-term sequence action detection tasks. To verify the detection performance and operational efficiency of this invention, a comparison with traditional online action detection methods is provided. Specific performance indicators are shown in Table 1, and efficiency indicators are shown in Table 2.

[0122] Table 1. Performance comparison of traditional online motion detection methods and the proposed method in cross-view scenarios.

[0123]

[0124] As shown in Table 1, the method of this invention outperforms traditional online action detection methods in both mAP and mcAP evaluation metrics, thus demonstrating higher detection accuracy. This invention not only solves the problems of insufficient background suppression and inadequate capture of temporal relationships but also endows the system with good generalization ability. Furthermore, as shown in Table 2, this invention effectively addresses the problem of limited computational resources, significantly reducing the model size and ensuring sufficient running speed for practical applications. Its performance on complex datasets surpasses that of traditional online action detection models, proving its effectiveness.

[0125] Table 2 Comparison of online motion detection efficiency metrics of this method in cross-view scenarios.

[0126]

[0127] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any other changes or substitutions that do not involve substantial innovation should be covered within the scope of protection of the present invention. Therefore, the specific scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. A cross-view online action detection method based on probabilistic temporal mask attention, characterized in that, Includes the following steps: S1, Training data acquisition: Acquire video streams from multiple perspectives in the target scene; S2, data preprocessing, based on a pre-trained two-dimensional convolutional neural network, extracts video features from the video stream, including RGB and optical flow features; S3, construct a dual-branch network structure, the probabilistic branch utilizes a variational autoencoder to preprocess the video features X. -K+1～0 As input, viewpoint-level video features are compressed into potential viewpoint-specific codes. The classification branch uses these codes to perform temporal mask attention interactions. Finally, the outputs of each branch are mixed and cross-entropy loss is calculated based on the actual action labels. S4. Construct the GRU-TMA module, and reduce the interference of distant historical information on the detection of current actions by introducing a temporal mask attention mechanism in the classification branch, thereby improving the real-time performance and accuracy of the model. S5, Model Training and Optimization: By minimizing reconstruction loss and KL divergence loss, the latent representation of the probability branch is optimized, while the action classification ability of the classification branch is optimized by cross-entropy loss. S6, an online action detection system, classifies sequential frames based on the output of a dual-branch network, predicts actions using the output of the classification branches, and evaluates performance based on actual action labels.

2. The cross-view online action detection method based on probabilistic temporal mask attention as described in claim 1, characterized in that, Step S2 includes the following: Real-time video streams are used as input data, and each N-frame video is processed as a group. The RGB and optical flow information of the frames in this group are extracted using a pre-trained two-dimensional convolutional neural network. The two-dimensional convolutional neural network is used to simultaneously capture the spatial and temporal information of the video frames, so as to achieve efficient recognition of dynamic changes in video content.

3. The cross-view online action detection method based on probabilistic temporal mask attention as described in claim 1, characterized in that, Step S3 specifically includes the following: (1) Obtain feature subsequences using a sliding window; input the original features into a window of length T. Sampling is performed in several time windows. Meanwhile, during model inference, the complete video sequence is inferred by default to simulate real-world situations, with the original feature sequence as input. Where D is the dimension of the video features, v i With v j To represent different perspectives, x -K+1 x is the Kth video frame from the past, and x0 is the current frame; (2) Construct an autoregressive model for the classification branch; denote the online action detection model as Φ. d The coding process for the classification branches is represented as follows: in, Autoregressive model – GRU module: Update Gate Z t : z t =Sigmoid(W z x t +U z h t-1 +b z ), Reset door r t : r t =Sigmoid(W r x t +U r h t-1 +b r ), Candidate hidden state Final hidden state h t : Where tanh represents the hyperbolic tangent activation function, W z W r , W and U z U r U represents the weight matrix for the update gate, reset gate, and candidate hidden state, respectively. z ,b r ,b represents the corresponding bias vector, x t h represents the input at time t. t-1 Let z represent the hidden state at time t-1. t ,r t These represent the outputs of the update gate and the reset gate at time t, respectively. h represents the candidate hidden state at time t. t This represents the final hidden state at time t; (3) Construct a probability model for the probability branch; Probability assumptions: Assuming that the features from different perspectives conform to a consistent distribution, i.e. Where p(z) is the probability distribution of the latent code, if the latent code z can effectively reconstruct the viewpoint v i Therefore, it can also represent the viewpoint v. j Some features; Probability inference: Variational inference is used to approximate the posterior probability of time-series features, with maximizing the likelihood probability as the optimization objective: Where, q φ The distribution is a variational distribution, and p(x) is the original feature distribution. To speed up the model's approximation, an MLP is used here; probability generation: To facilitate a stable and differentiable sampling process, a reparameterization technique is applied; this involves a parameter derived from... The random variable vector ε is rescaled and shifted to align with the desired Gaussian distribution parameters, as shown below: z=μ+σ⊙ε, Wherein, the mean μ and the standard deviation σ are Gaussian distribution parameters; The probability generation process is completed by multiple linear layers, and their output is used to calculate the reconstruction loss of the probabilistic model based on the input video features.

4. The cross-view online action detection method based on probabilistic temporal mask attention as described in claim 1, characterized in that, Step S4 specifically includes the following: A GRU-TMA unit is constructed based on the Gated Recurrent Unit (GRU) and the Temporal Mask Attention Mechanism (TMA). In the GRU-TMA unit, the GRU part controls the flow of information by introducing update gates and reset gates, thereby solving the long-term dependency problem of traditional RNNs and capturing frame-level temporal dependencies. The TMA module is responsible for applying the attention mechanism in the temporal dimension to highlight the features most relevant to the current action detection task and suppress irrelevant information. This attention mechanism is achieved by generating a mask matrix that determines which historical information is important and which can be ignored at the current time step. The output of TMA is expressed by the following formula: Among them, F z =f q (z) represents the expansion of the latent code z, which is mapped to the temporal coding space of the GRU output; meanwhile, α is used to adjust the density of attention. H is the temporal coding matrix, and H is the video feature coding sequence output by the GRU.

5. The cross-view online action detection method based on probabilistic temporal mask attention as described in claim 1, characterized in that, Step S5 specifically includes the following: The probabilistic branch of the model is trained by minimizing the reconstruction loss and KL divergence loss; at the same time, the action classification ability of the classification branch is optimized by cross-entropy loss. Finally, the model is trained globally by combining the three different losses, so that it has the ability to represent video features from unseen perspectives and the ability to detect actions.

6. The cross-view online action detection method based on probabilistic temporal mask attention as described in claim 1, characterized in that, Step S6 specifically includes the following: (1) Construct a classification MLP layer Φ for online action detection c ; (2) Merge the timing codes of different branches and perform online motion detection. (3) Use mAP and mcAP metrics to measure the accuracy of online action detection, and use FPS and GFLOPs metrics to measure the complexity of model inference.

Images