Video monitoring personnel risk behavior identification method and system based on deep learning

Abstract

The invention discloses a video monitoring personnel risk behavior identification method and system based on deep learning, and belongs to the technical field of risk behavior identification, and the method comprises the steps: obtaining a plurality of monitoring videos, and carrying out the preprocessing of the monitoring videos, and obtaining a target data set; constructing a risk behavior recognition network, and training and testing the risk behavior recognition network according to the target data set; obtaining a target video frame set, and processing the target video frame set according to the risk behavior recognition network to obtain a risk behavior recognition result corresponding to the target video frame set; the multi-modal features and the spatial-temporal features are integrated through the fusion of the enhanced fusion module and the spatial-temporal feature extraction module, and the recognition capability of subtle actions and complex scenes can be improved, so that the recognition precision of the risk behavior recognition network is enhanced.

Description

Technical Field

[0001] This application belongs to the field of risk behavior recognition technology, specifically relating to a method and system for identifying risky behaviors of video surveillance personnel based on deep learning. Background Technology With the rapid development of artificial intelligence technology, the risk behavior of video surveillance personnel has gradually evolved from manual monitoring to intelligent analysis; and the methods for identifying the risk behavior of video surveillance personnel based on deep learning networks are also constantly being updated and iterated.

[0002] Currently, existing video surveillance personnel risk identification based on deep learning networks is generally implemented using simple convolutional neural networks. However, due to the problem of low recognition accuracy, the results in practical applications are not ideal. Summary of the Invention

[0003] The purpose of this application is to provide a deep learning-based method and system for identifying risky behaviors of video surveillance personnel, which can solve the technical problem of low identification accuracy in the prior art.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a method for identifying risky behaviors of video surveillance personnel based on deep learning, the method comprising: Multiple surveillance videos are acquired, and the multiple surveillance videos are preprocessed to obtain a set of video frames corresponding to each surveillance video. A target dataset is constructed based on each set of video frames. A risk behavior recognition network is constructed, which includes an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a spatiotemporal feature extraction module, an attention fusion module, and a classifier; The risk behavior recognition network is trained using a portion of the target dataset, and then tested using another portion of the dataset. Obtain the target video frame set, and process the target video frame set according to the risk behavior recognition network after it has passed the test to obtain the risk behavior recognition result corresponding to the target video frame set.

[0005] As an optional implementation of the first aspect of this application, the step of processing the target video frame set according to the risk behavior recognition network after passing the test to obtain the risk behavior recognition result of the monitoring personnel specifically includes: The target video frame set is processed by the RGB feature extraction module to obtain initial RGB features; The optical flow feature extraction module extracts optical flow features from the target video frame set to obtain initial optical flow features. The initial RGB features and the initial optical flow features are enhanced and fused according to the enhancement and fusion module to obtain the target enhanced features; The spatiotemporal feature extraction module extracts spatiotemporal features from the target video frame set to obtain target spatiotemporal features. The target enhancement features and the target spatiotemporal features are fused using the attention fusion module to obtain the target fusion features. The target fusion features are classified according to the classifier to obtain the risk behavior identification result corresponding to the target video frame set. As an optional implementation of the first aspect of this application, the RGB extraction module includes multiple multi-scale convolutional layers, an adaptive fusion layer, a local attention mechanism, and a first fully connected layer; The RGB feature extraction module performs feature extraction processing on the target video frame set to obtain initial RGB features; specifically: Each video frame in the target video frame set is downsampled multiple times by the multiple multi-scale convolutional layers to obtain multiple RGB scale features of different scales corresponding to each video frame; The adaptive fusion layer adaptively fuses the multiple RGB scale features corresponding to each video frame to obtain each first RGB fusion feature corresponding to each video frame. According to the local attention mechanism, each of the first RGB fusion features is subjected to local attention processing to obtain each RGB local feature corresponding to each of the first RGB fusion features. The channel dimension of each RGB local feature is processed according to the fully connected layer, and the element-wise summation of each RGB local feature after channel dimension processing is performed to obtain the second RGB fusion feature. The second RGB fusion feature and multiple first RGB fusion features are sequentially subjected to batch normalization, activation, and max pooling to obtain the second RGB pooling feature and multiple first RGB pooling features; The initial RGB feature is obtained by simultaneously performing element-wise multiplication on the second RGB pooling feature and all the first RGB pooling features. As an optional implementation of the first aspect of this application, the enhanced fusion module includes: a segmentation layer, a 3×3 convolutional layer, a splicing layer, an enhanced fusion layer, and a second fully connected layer; The initial RGB features and the initial optical flow features are fused and enhanced according to the fusion enhancement module to obtain the target enhanced features; specifically: The initial RGB features and the initial optical flow features are segmented according to the segmentation layer to obtain multiple segmented RGB features and multiple segmented optical flow features; The 3×3 convolutional layers are used to perform convolution processing on the multiple segmented RGB features and the multiple segmented optical flow features respectively, to obtain each convolutional RGB feature corresponding to each segmented RGB feature and each convolutional optical flow feature corresponding to each segmented optical flow feature. Based on the stitching layer, each of the convolutional RGB features and each of the convolutional optical flow features are stitched together to obtain stitched optical flow features and stitched RGB features; The spliced ​​optical flow features and the spliced ​​RGB features are enhanced and fused according to the enhancement and fusion layer to obtain the fused enhanced features; The fused enhancement features are sequentially batch normalized, activated, and pooled according to the second fully connected layer to obtain the target enhancement features. As an optional implementation of the first aspect of this application, the step of performing enhancement and fusion processing on the spliced ​​optical flow feature and the spliced ​​RGB feature according to the enhancement and fusion layer to obtain a fused enhancement feature is specifically as follows: The spliced ​​optical flow features are processed by feature slicing to obtain the corresponding horizontal optical flow features, vertical optical flow features, and optical strain features; The optical flow horizontal features, optical flow vertical features, and optical strain features are all subjected to 3×3 convolution, batch normalization, activation, and max pooling to obtain optical flow horizontal pooling features, optical flow vertical pooling features, and optical strain pooling features, respectively. The optical flow horizontal pooling feature, optical flow vertical pooling feature, and optical strain pooling feature are weighted and fused to obtain the enhanced optical flow feature. The spliced ​​RGB features are cut to obtain multiple spliced ​​RGB sub-features of the same size, and the spliced ​​RGB sub-features are mapped to obtain each spliced ​​RGB sub-feature corresponding to each mapped RGB feature; Clustering is performed on each of the mapped RGB features to obtain multiple different types of mapped RGB feature sets, and each similarity matrix corresponding to each mapped RGB feature set is calculated; Local enhancement processing is performed on the spliced ​​RGB features according to each of the aforementioned similarity matrices to obtain enhanced RGB features. These enhanced RGB features are then fused with the enhanced optical flow features to obtain the fused enhanced features. As an optional implementation of the first aspect of this application, the spatiotemporal feature extraction module includes: a 5×5 convolutional layer, a 1×1 convolutional layer, a pooling fusion layer, a multi-scale attention layer, and a third fully connected layer. The step involves extracting spatiotemporal features from the target video frame set using the spatiotemporal feature extraction module to obtain the target spatiotemporal features; specifically: Based on the 5×5 convolutional layer and the 1×1 convolutional layer, spatial and temporal features are extracted from the target video frame set to obtain the initial spatial and initial temporal features corresponding to the target video frame set. The initial spatial features and initial temporal features are subjected to global max pooling processing according to the pooling fusion layer, and the initial spatial features and initial temporal features after global max pooling are fused to obtain the initial spatiotemporal features. The initial spatiotemporal features are processed by the multi-scale attention layer to obtain an enhanced spatiotemporal feature map; The target spatiotemporal features are obtained by performing residual connections on the enhanced spatiotemporal feature map and the initial spatiotemporal feature map using the third fully connected layer. As an optional implementation of the first aspect of this application, the step of processing the initial spatiotemporal features according to the multi-scale attention layer to obtain enhanced spatiotemporal features specifically includes: The initial spatiotemporal features are processed by 7×1×1 three-dimensional convolution, 3×3×3 three-dimensional convolution and 7×7×1 three-dimensional convolution respectively to obtain long temporal features, medium spatial features and short temporal features respectively; The initial spatiotemporal features, long temporal features, medium spatial features, and short temporal features are processed sequentially with channel attention, spatial attention, and temporal attention to obtain weighted spatiotemporal features, weighted long temporal features, weighted medium spatial features, and weighted short temporal features, respectively. The weighted spatiotemporal features, weighted long temporal features, weighted spatial features, and weighted short temporal features are concatenated along the channel dimension to obtain the concatenated spatiotemporal features. The spliced ​​spatiotemporal features are sequentially processed by 1×1×1 three-dimensional convolution and global average pooling to obtain the enhanced spatiotemporal features. Secondly, embodiments of this application provide a video surveillance personnel risk behavior recognition system based on deep learning, the system comprising: Acquisition module: Acquires multiple surveillance video segments, preprocesses the multiple surveillance video segments to obtain a set of video frames corresponding to each surveillance video, and constructs a target dataset based on each video frame; Construction Module: Constructs a risk behavior recognition network, which includes an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a spatiotemporal feature extraction module, an attention fusion module, and a classifier; Testing module: Trains the risk behavior recognition network based on a portion of the target dataset, and tests the trained risk behavior recognition network based on another portion of the data; Prediction module: acquires the target video frame set, processes the target video frame set according to the risk behavior recognition network after it has passed the test, and obtains the risk behavior recognition result corresponding to the target video frame set.

[0006] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] In the embodiments of this application, compared with the prior art, the following technical effects are achieved: (1) By strengthening and fusing the initial RGB features and initial optical flow features of different modalities through the strengthening fusion module, key features can be highlighted and background noise can be suppressed, thereby improving the robustness of the risk behavior recognition network; (2) By strengthening the fusion of the fusion module and the spatiotemporal feature extraction module, multimodal features and spatiotemporal features are incorporated, which can improve the ability to recognize subtle movements and complex scenes, thereby enhancing the recognition accuracy of the risk behavior recognition network; (3) The initial spatiotemporal features are processed by a multi-scale attention layer to generate enhanced spatiotemporal features with multi-scale and attention weights, which improves the ability to identify features of different scales, complex behaviors and complex scenes, thereby enhancing the recognition efficiency of the risk behavior recognition network. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a deep learning-based method for identifying risky behaviors of video surveillance personnel, provided in some embodiments of this application. Figure 2 This is a network structure diagram of risk behavior recognition in a video surveillance personnel risk behavior recognition method based on deep learning, provided by some embodiments of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0012] The following description, in conjunction with the accompanying drawings, details a method and system for identifying risky behaviors of video surveillance personnel based on deep learning, through specific embodiments and application scenarios.

[0013] Example A deep learning-based method for identifying risky behaviors of video surveillance personnel includes the following steps: S100: Acquire multiple surveillance videos, preprocess the multiple surveillance videos to obtain each video frame set corresponding to each surveillance video, and construct the target dataset based on each video frame set; It is important to understand that by collecting multiple surveillance videos, extracting frames from the videos, and performing preprocessing such as noise reduction and size normalization on the extracted video frames, a corresponding set of video frames is obtained for each surveillance video. Finally, a target dataset is constructed based on each set of video frames corresponding to each surveillance video. Through noise reduction, size normalization, and other processing, the data format is unified and noise interference is eliminated, providing high-quality samples for the subsequent training of the risk behavior recognition network.

[0014] S200: Construct a risk behavior recognition network, which includes an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a spatiotemporal feature extraction module, an attention fusion module, and a classifier; S300: Train the risk behavior recognition network based on a portion of the target dataset, and test the trained risk behavior recognition network based on another portion of the dataset; It is important to understand that, firstly, the target dataset is divided according to a preset ratio; in this embodiment, the target dataset is divided into a training set and a test set in a 7:3 ratio. Then, a loss function is constructed, and the risk behavior recognition network is trained based on the loss function and the training set. The loss function optimizes the risk behavior network by optimizing the parameters through backpropagation. After training, the risk behavior recognition network is tested using the test set, thus completing the training and testing of the risk behavior recognition network. Through the training process, the risk behavior recognition network can learn the feature representation of risky behaviors, while the testing process is to select the optimal network.

[0015] S400: Obtain the target video frame set, process the target video frame set according to the risk behavior recognition network after passing the test, and obtain the risk behavior recognition result corresponding to the target video frame set.

[0016] The system acquires target video frame sets from real-time monitoring or offline video. These frames are then processed by a risk behavior recognition network that has passed testing. This process enables the identification of dangerous behaviors within the target video frame sets. When the target video frame sets are acquired from real-time monitoring, the system can be combined with an early warning system and a visualization system. When the risk behavior recognition network detects a dangerous behavior in the real-time monitoring, it immediately sends a signal to the early warning system. Upon receiving the signal, the early warning system activates an emergency warning and visualizes and tracks the real-time footage of the dangerous behavior on the background monitoring system. This achieves automated identification and early warning of risk behaviors.

[0017] Furthermore, in S400, the target video frame set is processed by the risk behavior recognition network after passing the test to obtain the risk behavior recognition result corresponding to the target video frame set, specifically as follows: S410: Perform feature extraction processing on the target video frame set according to the RGB feature extraction module to obtain the initial RGB features; S420: Optical flow features are extracted from the target video frame set by the optical flow feature extraction module to obtain the initial optical flow features; S430: The initial RGB features and initial optical flow features are enhanced and fused according to the enhancement fusion module to obtain the target enhanced features; S440: The spatiotemporal feature extraction module extracts spatiotemporal features from the target video frame set to obtain the target spatiotemporal features; S450: The target enhancement features and target spatiotemporal features are fused using the attention fusion module to obtain the target fusion features; S460: Classify the target fusion features according to the classifier to obtain the risk behavior identification result corresponding to the target video frame set.

[0018] It's important to understand that, firstly, the RGB feature extraction module extracts and processes the target video frame set frame by frame to obtain initial RGB features. These initial RGB features capture static visual information from the target video frame set, such as character outlines and scene layout. They focus on the spatial dimension, providing appearance context information for subsequent behavior recognition. By acquiring these initial RGB features, higher-level semantic features are extracted, thereby improving robustness to lighting changes and background complexity. Simultaneously, the optical flow feature extraction module calculates the pixel displacement field between consecutive frames in the target video frame set using an optical flow algorithm, generating initial optical flow features based on this field. These initial optical flow features focus more on the temporal dimension, characterizing dynamic behavior patterns. By acquiring these initial optical flow features, subtle movements are captured, compensating for the lack of dynamic information in the initial RGB features. Then, the enhancement fusion module performs enhancement fusion processing on the initial RGB features and initial optical flow features to obtain enhanced target features. Through feature enhancement fusion, redundant information between modalities is eliminated, focusing on key areas, thereby improving the recognition of risky behaviors. First, the network's ability to discriminate complex behaviors is enhanced. Second, the spatiotemporal feature extraction module extracts spatiotemporal features from the target video frame set to obtain target spatiotemporal features. These features capture temporal logic, thereby enhancing the network's ability to recognize and judge temporal logic. Third, the attention fusion module performs attention fusion processing on the target enhancement features and target spatiotemporal features to obtain target fusion features. By using the attention module to fuse the target enhancement features and target spatiotemporal features, the contribution of the two features is dynamically adjusted, achieving cross-modal feature complementarity and improving the network's adaptability to different scenarios. Simultaneously, the combination of target enhancement features and target spatiotemporal features covers the static, dynamic, and temporal dimensions of behavior recognition, thereby reducing the probability of missed detections of risky behaviors. Subsequently, the network uses a fully connected layer combined with a Softmax function to classify the target fusion features, mapping them to a risk behavior category probability distribution to obtain the risk behavior recognition results corresponding to the target video frame set.

[0019] Furthermore, the RGB extraction module in S410 includes multiple multi-scale convolutional layers, adaptive fusion layers, local attention mechanisms, and a first fully connected layer; the RGB feature extraction module performs feature extraction processing on the target video frame set to obtain initial RGB features; specifically: S411: Perform multiple downsampling processes on each video frame in the target video frame set using multiple multi-scale convolutional layers to obtain multiple RGB scale features of different scales corresponding to each video frame. S412: Adaptively fuse multiple RGB scale features corresponding to each video frame according to the adaptive fusion layer to obtain each first RGB fusion feature corresponding to each video frame. S413: Perform local attention processing on each first RGB fusion feature according to the local attention mechanism to obtain each RGB local feature corresponding to each first RGB fusion feature; S414: Process each RGB local feature according to the channel dimension based on the fully connected layer, and add each RGB local feature after channel dimension processing element-wise to obtain the second RGB fusion feature; S415: Perform batch normalization, activation and max pooling on the second RGB fusion feature and multiple first RGB fusion features in sequence to obtain the second RGB pooling feature and multiple first RGB pooling features; S416: Perform element-wise multiplication on the second RGB pooling feature and all first RGB pooling features simultaneously to obtain the initial RGB feature.

[0020] Specifically, the initial RGB features in S416 are represented by the following formula: , in, Indicates the initial RGB characteristics. This represents the second RGB pooling feature. This indicates element-wise multiplication. This indicates max pooling. Indicates activation process. This indicates batch normalization processing. Indicates the second RGB fusion feature. This represents element-wise addition. This indicates the total number of video frames. Indicates the first channel dimension after processing. RGB local features, This indicates channel-level processing. Indicates the first RGB local features, Indicates the first The first RGB pooling feature Indicates the first The first RGB fusion feature, Indicates the first The corresponding video frame Learnable weights for each RGB scale feature Indicates the first The corresponding video frame Each RGB scale feature This represents the total number of RGB scale features corresponding to each video frame.

[0021] It's important to understand that, firstly, multiple downsampling processes are performed on each video frame in the target video frame set using multiple multi-scale convolutional layers, resulting in multiple RGB scale features of different scales for each video frame. This yields RGB scale features at different levels for each video frame. Multi-scale processing enhances the adaptability of the risk behavior recognition network to complex scenes, while downsampling reduces the computational load of the risk behavior network and expands the receptive field to capture global contextual information. Subsequently, an adaptive fusion layer adaptively fuses the multiple RGB scale features corresponding to each video frame, resulting in each first RGB fusion feature for each video frame. For each video frame, learnable weights are used to dynamically weight and fuse the multiple RGB scale features of different scales, resulting in the first RGB fusion feature. Adaptive fusion eliminates modal redundancy, focusing on key features for identifying risk behaviors and improving the detection capability for small or occluded targets. Secondly, a local attention mechanism is used to process each first RGB fusion feature, resulting in each corresponding RGB local feature. Local attention suppresses background interference and improves the sensitivity of the risk behavior recognition network to subtle movements. Then, the root... The fully connected layer processes each RGB local feature by channel dimension, and then element-wise sums the channel-dimension-processed RGB local features to obtain the second RGB fusion feature. The fully connected layer adjusts the number of channels in the first RGB local features, and then element-wise sums the channel-adjusted first RGB local features to achieve cross-channel information interaction. Channel adjustment alleviates gradient vanishing and accelerates the convergence of the risk behavior recognition network, while element-wise summing of the first RGB local features enhances the robustness of the risk behavior recognition network. Finally, the second RGB fusion feature and multiple first RGB fusion features are compared. The features are sequentially processed by batch normalization, activation, and max pooling to obtain second RGB pooled features and multiple first RGB pooled features. Through batch normalization, activation, and max pooling, the training efficiency and generalization ability of the risk behavior recognition network are improved, as well as its robustness to translation and rotation are enhanced. Finally, the second RGB pooled features and each first RGB pooled feature are multiplied element-wise to obtain the initial RGB features. By multiplying all first RGB pooled features with the second RGB pooled features element-wise, common features can be strengthened and noise can be suppressed, thereby achieving global-local special augmentation synergy and improving the discriminative ability of the risk behavior recognition network.

[0022] Furthermore, the enhancement and fusion module in S430 includes: a segmentation layer, a 3×3 convolutional layer, a stitching layer, an enhancement and fusion layer, and a second fully connected layer; the initial RGB features and initial optical flow features are fused and enhanced according to the enhancement and fusion module to obtain the target enhanced features; specifically: S431: The initial RGB features and initial optical flow features are segmented according to the segmentation layer to obtain multiple segmented RGB features and multiple segmented optical flow features; S432: Perform convolution processing on multiple segmented RGB features and multiple segmented optical flow features using multiple 3×3 convolutional layers respectively, to obtain each convolutional RGB feature corresponding to each segmented RGB feature, and each convolutional optical flow feature corresponding to each segmented optical flow feature. S433: Based on the splicing layer, each convolutional RGB feature and each convolutional optical flow feature are spliced ​​together to obtain spliced ​​optical flow features and spliced ​​RGB features; S434: The spliced ​​optical flow features and spliced ​​RGB features are enhanced and fused according to the enhancement fusion layer to obtain the fused enhanced features; S435: The fused enhancement features are batch normalized, activated, and pooled sequentially according to the second fully connected layer to obtain the target enhancement features.

[0023] It should be noted that, firstly, the initial RGB features and initial optical flow features are segmented along the spatial dimension by the segmentation layer, resulting in multiple segmented RGB features and multiple segmented optical flow features. This segmentation process reduces computational complexity while focusing on key local regions and suppressing background interference. Then, multiple 3×3 convolutional layers are used to convolve the multiple segmented RGB features and multiple segmented optical flow features, resulting in each convolutional RGB feature corresponding to each segmented RGB feature, and each convolutional optical flow feature corresponding to each segmented optical flow feature. Performing 3×3 convolutions on the segmented RGB features enhances the texture details of the RGB segmentation features, while performing 3×3 convolutions on the segmented optical flow features captures local motion patterns within the segmented optical flow features. The kernel size of 3 is chosen because smaller kernels preserve spatial structure, making them more suitable for extracting detailed features, and compared to larger kernels, they reduce the number of parameters and improve computational efficiency. Secondly, based on the stitching... Each convolutional RGB feature and each convolutional optical flow feature are concatenated to obtain concatenated optical flow features and concatenated RGB features. By concatenating all convolutional RGB features and all convolutional optical flow features, corresponding concatenated RGB features and concatenated optical flow features are obtained, preserving the original information of the two modalities and providing a complete context for subsequent fusion. At the same time, feature concatenation expands the channel dimension, thereby enhancing the expressive power of the features. Then, the concatenated optical flow features and concatenated RGB features are enhanced and fused according to the enhancement fusion layer to obtain fused enhanced features. Through enhancement fusion processing, the adaptability of the risk behavior recognition network to different scenarios is improved. Finally, the fused enhanced features are batch normalized, activated, and pooled sequentially according to the second fully connected layer to obtain target optical flow features. Batch normalization, activation, and pooling processing can improve the generalization of the risk behavior recognition network, reduce the risk of overfitting, and generate compact target enhanced features for efficient processing by subsequent modules.

[0024] Furthermore, in S434, the spliced ​​optical flow features and spliced ​​RGB features are enhanced and fused according to the enhancement and fusion layer to obtain the fused enhancement features; specifically: S4341: The spliced ​​optical flow features are processed by feature slicing to obtain the corresponding horizontal optical flow features, vertical optical flow features and optical strain features; S4342: The optical flow horizontal features, optical flow vertical features, and optical strain features are all processed by 3×3 convolution, batch normalization, activation, and max pooling to obtain the optical flow horizontal pooling features, optical flow vertical pooling features, and optical strain pooling features, respectively. S4343: Weighted fusion of optical flow horizontal pooling features, optical flow vertical pooling features, and optical strain pooling features is performed to obtain enhanced optical flow features; S4344: The spliced ​​RGB features are cut to obtain multiple spliced ​​RGB sub-features of the same size, and the spliced ​​RGB sub-features are mapped to obtain each spliced ​​RGB sub-feature corresponding to each mapped RGB feature; S4345: Perform clustering processing on each mapped RGB feature to obtain multiple different types of mapped RGB feature sets, and calculate each similarity matrix corresponding to each mapped RGB feature set; S4346: Perform local enhancement processing on the spliced ​​RGB features according to each similarity matrix to obtain enhanced RGB features, and fuse the enhanced RGB features with the enhanced optical flow features to obtain fused enhanced features.

[0025] Specifically, the fusion enhancement feature in S4346 is represented by the following formula: , in, Indicates fusion enhancement features, This indicates an enhanced optical flow characteristic. Indicates the splicing of RGB features. Indicates the first A set of mapped RGB features, This represents the total number of mapped RGB feature sets. Indicates the first The first RGB feature set mapped The weights of the concatenated RGB sub-features corresponding to the mapped RGB features. Indicates the first The first RGB feature set mapped Mapping RGB features and concatenating RGB sub-features. This indicates element-wise multiplication. Represents the normalization function. Indicates the first The first RGB feature set mapped The sum of similarities between each mapped RGB feature and other mapped RGB features. In Indicates the first In the RGB feature set of the nth mapping, except for the first The first RGB mapping feature, in addition to the third Map RGB features.

[0026] It is important to understand that by stitching together optical flow features and performing feature slicing, corresponding horizontal, vertical, and optical strain features are obtained. Decoupling the motion direction and deformation information of the stitched optical flow features provides a foundation for refined analysis and reduces coupling interference between modes. Then, 3×3 convolution, batch normalization, activation, and max pooling are applied to the horizontal, vertical, and optical strain features to obtain horizontal, vertical, and optical strain pooled features, respectively. Finally, 3×3 convolution, batch normalization, activation, and max pooling are applied to these features. Pooling enhances the ability to represent direction-sensitive motion and accelerates the training convergence of the risk behavior recognition network, improving its stability. Subsequently, weighted fusion of optical flow horizontal pooling features, optical flow vertical pooling features, and optical strain pooling features yields enhanced optical flow features. Learnable weights are used to dynamically weight these features, allowing for adaptive adjustment to scene requirements and highlighting key motion components. Finally, the spliced ​​RGB features are cropped to obtain multiple splices of the same size. RGB sub-features are mapped using a spatial transformation network to obtain each mapped RGB feature corresponding to each concatenated RGB sub-feature. By cropping and mapping the concatenated RGB features, key local regions can be focused, background interference can be suppressed, and feature space invariance can be enhanced. Then, each mapped RGB feature is clustered to obtain multiple different types of mapped RGB feature sets, and each similarity matrix corresponding to each mapped RGB feature set is calculated. Through unsupervised clustering and intra-group similarity calculation, the correlation of local features can be uncovered, providing a structured prior for subsequent local enhancement. Finally, the concatenated RGB features are locally enhanced based on each similarity matrix to obtain enhanced RGB features, and these enhanced RGB features are fused with enhanced optical flow features to obtain fused enhanced features. By locally enhancing RGB features through the similarity matrix and fusing the enhanced RGB features with enhanced optical flow features, key regions in the RGB features can be highlighted while retaining optical flow motion information, achieving deep synergy between static appearance and dynamic motion, and improving the discriminative power of the risk behavior recognition network for risky behaviors.

[0027] Furthermore, the spatiotemporal feature extraction module in S440 includes: a 5×5 convolutional layer, a 1×1 convolutional layer, a pooling fusion layer, a multi-scale attention layer, and a third fully connected layer; the spatiotemporal feature extraction module extracts spatiotemporal features from the target video frame set to obtain the target spatiotemporal features; specifically: S441: Spatial and temporal features of the target video frame set are extracted by 5×5 convolutional layer and 1×1 convolutional layer respectively, to obtain the initial spatial and temporal features of the target video frame set; S442: Perform global max pooling on the initial spatial features and initial temporal features respectively according to the pooling fusion layer, and fuse the initial spatial features and initial temporal features after global max pooling to obtain the initial spatiotemporal features; S443: The initial spatiotemporal features are processed by the multi-scale attention layer to obtain the enhanced spatiotemporal feature map; S444: The target spatiotemporal features are obtained by performing residual connections on the enhanced spatiotemporal feature map and the initial spatiotemporal feature map based on the third fully connected layer.

[0028] It's important to understand that, firstly, spatial and temporal features are extracted from the target video frame set using 5×5 and 1×1 convolutional layers respectively, yielding the initial spatial and temporal features corresponding to the target video frame set. Then, spatial and temporal features are extracted from the target video frame set using convolutional layers with different kernels, decoupling spatial and temporal features and improving the module's flexibility. Next, global max pooling is applied to the initial spatial and temporal features using a pooling fusion layer, and the initial spatial and temporal features after global max pooling are fused to obtain the initial spatiotemporal features. Global max pooling retains the most significant global information in the initial spatial and temporal features while reducing computation and accelerating subsequent processing. The processed initial spatial and temporal features are fused to integrate global information, providing a basic context for the multi-scale attention layer. Secondly, the initial spatiotemporal features are processed by the multi-scale attention layer to obtain an enhanced spatiotemporal feature map. Through multi-scale attention processing, the spatiotemporal focus can be dynamically adjusted to adapt to different behavioral patterns, while suppressing background noise and redundant temporal segments. Finally, the enhanced spatiotemporal feature map and the initial spatiotemporal feature map are residually connected by the third fully connected layer to obtain the target spatiotemporal features. By residually connecting the enhanced spatiotemporal feature map and the initial spatiotemporal feature map, the original global features are combined with enhanced details, thereby improving feature robustness. Simultaneously, the residual connection can alleviate the gradient vanishing problem and accelerate the convergence of the risk behavior recognition network.

[0029] Furthermore, in S443, the initial spatiotemporal features are processed using a multi-scale attention layer to obtain enhanced spatiotemporal features; specifically: S4431: Perform 7×1×1 three-dimensional convolution, 3×3×3 three-dimensional convolution and 7×7×1 three-dimensional convolution on the initial spatiotemporal features respectively to obtain long temporal features, medium spatial features and short temporal features; S4432: Channel attention, spatial attention, and temporal attention are applied sequentially to the initial spatiotemporal features, long temporal features, medium spatial features, and short temporal features to obtain weighted spatiotemporal features, weighted long temporal features, weighted medium spatial features, and weighted short temporal features, respectively. S4433: The weighted spatiotemporal features, weighted long-time features, weighted spatial features, and weighted short-time features are concatenated along the channel dimension to obtain the concatenated spatiotemporal features; S4434: Perform 1×1×1 three-dimensional convolution and global average pooling on the spliced ​​spatiotemporal features in sequence to obtain enhanced spatiotemporal features.

[0030] It's important to understand that, firstly, the initial spatiotemporal features are processed using 7×1×1, 3×3×3, and 7×7×1 3D convolutions respectively to obtain long-term temporal features, mid-term spatial features, and short-term temporal features. By performing convolution processing on the initial spatiotemporal features using three 3D convolutions of different scales, coverage from local detailed features to global temporal features can be achieved, adapting to the analysis needs of different granularities. Simultaneously, decomposing the spatiotemporal convolution into three independent convolutions reduces the number of parameters, thereby improving computational efficiency. Secondly, the initial spatiotemporal features and temporal features... Long features, spatial features, and short temporal features are processed sequentially using channel attention, spatial attention, and temporal attention, respectively, to obtain weighted spatiotemporal features, weighted long temporal features, weighted spatial features, and weighted short temporal features. By applying these three attention mechanisms to both the initial spatiotemporal features and the features from the three convolutional branches at different scales—channel attention adjusting channel weights to enhance task-relevant features, spatial attention focusing on key spatial regions, and temporal attention highlighting key time points—the initial spatiotemporal features can be adaptively adjusted. This study focuses on the spatiotemporal attention of empty features, long temporal features, mid-space features, and short temporal features, thereby enhancing the ability of the risk behavior recognition network model to analyze complex behaviors. Subsequently, weighted spatiotemporal features, weighted long temporal features, weighted mid-space features, and weighted short temporal features are concatenated along the channel dimension to obtain multi-scale, multi-attention-enhanced spatiotemporal features. By concatenating these features along the channel dimension, information from different scales and attention enhancements can be integrated; simultaneously, redundant feature branches can be eliminated. To enhance the robustness of the high-risk behavior recognition network against interference from occlusion and lighting changes, the spliced ​​spatiotemporal features are subjected to 1×1×1 three-dimensional convolution and global average pooling to obtain enhanced spatiotemporal features. The 1×1×1 convolution process allows adjustment of the number of channels in the spliced ​​spatiotemporal features, enabling cross-channel information interaction. Global averaging aggregates the global spatiotemporal information of the spliced ​​spatiotemporal features to generate compact spatiotemporally enhanced features. The lightweight design of the 1×1×1 convolution reduces the number of parameters and accelerates the computation process.

[0031] According to a deep learning-based video surveillance personnel risk behavior recognition method of this embodiment, firstly, by strengthening and fusing the initial RGB features and initial optical flow features of different modalities through an enhanced fusion module, key features can be highlighted and background noise suppressed, thereby improving the robustness of the risk behavior recognition network; secondly, by incorporating a spatiotemporal feature extraction module to extract target spatiotemporal features, the risk behavior recognition network's ability to recognize short-term and long-term behaviors can be improved, thereby enhancing the recognition accuracy of the risk behavior recognition network; finally, by processing the initial spatiotemporal features through a multi-scale attention layer, enhanced spatiotemporal features with multi-scale and attention weights are generated, thereby enhancing the risk behavior recognition network's ability to recognize complex behaviors and complex scenes.

[0032] It should be noted that the video surveillance personnel risk behavior identification method based on deep learning provided in this application embodiment can be executed by a video surveillance personnel risk behavior identification system based on deep learning, or a control module within that system for executing and loading the deep learning-based video surveillance personnel risk behavior identification method. This application embodiment uses the execution of a deep learning-based video surveillance personnel risk behavior identification system loading the deep learning-based video surveillance personnel risk behavior identification method as an example to illustrate the deep learning-based video surveillance personnel risk behavior identification method provided in this application embodiment.

[0033] A deep learning-based video surveillance personnel risk behavior recognition system includes: Acquisition module: Acquires multiple surveillance video segments, preprocesses the multiple surveillance video segments to obtain a set of video frames corresponding to each surveillance video, and constructs a target dataset based on each video frame; Construction Modules: Construct a risk behavior recognition network, which includes an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a spatiotemporal feature extraction module, an attention fusion module, and a classifier; Test module: Train the risk behavior recognition network based on a portion of the target dataset, and test the trained risk behavior recognition network based on another portion of the dataset; Prediction module: Acquires the target video frame set, processes the target video frame set according to the risk behavior recognition network after passing the test, and obtains the risk behavior recognition result corresponding to the target video frame set.

[0034] The video surveillance personnel risk behavior recognition system based on deep learning in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0035] This application provides a deep learning-based video surveillance personnel risk behavior recognition system that can achieve... Figures 1 to 2 The various processes and technical effects of the deep learning-based video surveillance personnel risk behavior identification method implemented in the method embodiment are not described in detail here to avoid repetition.

[0036] Optionally, this application embodiment also provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a video surveillance personnel risk behavior recognition method based on deep learning, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0037] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a deep learning-based video surveillance personnel risk behavior identification method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0038] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0039] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0040] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0041] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for identifying risk behaviors of personnel in video surveillance based on deep learning, characterized in that, The method comprises: acquiring multiple monitoring videos, preprocessing the multiple monitoring videos to obtain each video frame set corresponding to each monitoring video, and constructing a target data set according to each video frame set; constructing a risk behavior recognition network, wherein the risk behavior recognition network comprises an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a spatiotemporal feature extraction module, an attention fusion module, and a classifier; training the risk behavior recognition network according to part of the data in the target data set, and testing the trained risk behavior recognition network according to another part of the data; acquiring a target video frame set, processing the target video frame set according to the risk behavior recognition network that has passed the test, and obtaining a risk behavior recognition result corresponding to the target video frame set.

2. The method according to claim 1, wherein, The processing of the target video frame set according to the risk behavior recognition network that has passed the test to obtain a monitoring personnel risk behavior recognition result comprises: performing feature extraction processing on the target video frame set according to the RGB feature extraction module to obtain initial RGB features; performing optical flow feature extraction on the target video frame set according to the optical flow feature extraction module to obtain initial optical flow features; performing reinforcement fusion processing on the initial RGB features and the initial optical flow features according to the reinforcement fusion module to obtain target reinforcement features; performing spatiotemporal feature extraction on the target video frame set according to the spatiotemporal feature extraction module to obtain target spatiotemporal features; performing attention fusion processing on the target reinforcement features and the target spatiotemporal features according to the attention fusion module to obtain target fusion features; performing classification processing on the target fusion features according to the classifier to obtain a risk behavior recognition result corresponding to the target video frame set.

3. The method according to claim 2, wherein, The RGB extraction module comprises multiple multi-scale convolution layers, an adaptive fusion layer, a local attention mechanism, and a first full connection layer; The RGB feature extraction module performs feature extraction processing on the target video frame set to obtain initial RGB features. Specifically, performing multiple times of down-sampling processing on each video frame in the target video frame set according to multiple multi-scale convolution layers to obtain multiple RGB scale features of different scales corresponding to each video frame; performing adaptive fusion on the multiple RGB scale features corresponding to each video frame according to the adaptive fusion layer to obtain each first RGB fusion feature corresponding to each video frame; performing local attention processing on each first RGB fusion feature according to the local attention mechanism to obtain each RGB local feature corresponding to each first RGB fusion feature; performing channel dimension processing on each RGB local feature according to a full connection layer, and performing element-level addition on each RGB local feature after the channel dimension processing to obtain a second RGB fusion feature; performing batch normalization, activation, and maximum pooling processing on the second RGB fusion feature and multiple first RGB fusion features in sequence to obtain a second RGB pooling feature and multiple first RGB pooling features; Element-level multiplication is simultaneously performed on the second RGB pooling feature and all the first RGB pooling features to obtain the initial RGB feature.

4. The method according to claim 2, wherein, The reinforcement fusion module comprises a segmentation layer, a 3*3 convolution layer, a splicing layer, a reinforcement fusion layer and a second full connection layer. The initial RGB feature and the initial optical flow feature are subjected to reinforcement fusion processing according to the reinforcement fusion module to obtain a target reinforcement feature. The initial RGB feature and the initial optical flow feature are subjected to segmentation processing according to the segmentation layer to obtain a plurality of segmented RGB features and a plurality of segmented optical flow features. The plurality of segmented RGB features and the plurality of segmented optical flow features are subjected to convolution processing according to a plurality of 3*3 convolution layers respectively to obtain each convolution RGB feature corresponding to each segmented RGB feature and each convolution optical flow feature corresponding to each segmented optical flow feature. The feature splicing is performed on each convolution RGB feature and each convolution optical flow feature according to the splicing layer to obtain spliced optical flow features and spliced RGB features. The reinforcement fusion processing is performed on the spliced optical flow features and the spliced RGB features according to the reinforcement fusion layer to obtain a fusion reinforcement feature. The batch normalization, activation and pooling processing are sequentially performed on the fusion reinforcement feature according to the second full connection layer to obtain the target reinforcement feature.

5. The method according to claim 4, wherein, The reinforcement fusion processing is performed on the spliced optical flow features and the spliced RGB features according to the reinforcement fusion layer to obtain a fusion reinforcement feature. Specifically, The feature slicing processing is performed on the spliced optical flow features to obtain corresponding optical flow horizontal features, optical flow vertical features and optical strain features. The 3*3 convolution, batch normalization, activation and maximum pooling processing are performed on the optical flow horizontal features, the optical flow vertical features and the optical strain features to obtain optical flow horizontal pooling features, optical flow vertical pooling features and optical strain pooling features respectively. The weighted fusion processing is performed on the optical flow horizontal pooling features, the optical flow vertical pooling features and the optical strain pooling features to obtain a reinforcement optical flow feature. The spliced RGB features are subjected to clipping processing to obtain a plurality of spliced RGB sub-features with the same size, and the spliced RGB sub-features are subjected to mapping to obtain each mapping RGB feature corresponding to each spliced RGB sub-feature. Each mapping RGB feature is subjected to clustering processing to obtain a plurality of mapping RGB feature sets of different types, and each similarity matrix corresponding to each mapping RGB feature set is calculated. The local reinforcement processing is performed on the spliced RGB features according to each similarity matrix to obtain a reinforcement RGB feature, and the reinforcement RGB feature and the reinforcement optical flow feature are subjected to feature fusion to obtain the fusion reinforcement feature.

6. The method according to claim 2, wherein, The spatio-temporal feature extraction module comprises a 5*5 convolution layer, a 1*1 convolution layer, a pooling fusion layer, a multi-scale attention layer and a third full connection layer. The spatio-temporal feature extraction is performed on the target video frame set according to the spatio-temporal feature extraction module to obtain a target spatio-temporal feature. According to the 5*5 convolution layer and the 1*1 convolution layer, spatial features and time sequence features of the target video frame set are extracted respectively, to obtain initial spatial features and initial time sequence features corresponding to the target video frame set; According to the pooling fusion layer, global maximum pooling processing is performed on the initial spatial features and the initial time sequence features respectively, and feature fusion is performed on the initial spatial features and the initial time sequence features after global maximum processing, to obtain initial space-time features; According to the multi-scale attention layer, the initial space-time features are processed to obtain reinforced space-time feature maps; According to the third full connection layer, residual connection is performed on the reinforced space-time feature maps and the initial space-time feature maps, to obtain target space-time features.

7. The method according to claim 6, wherein, According to the multi-scale attention layer, the initial space-time features are processed to obtain reinforced space-time features; Specifically: The initial space-time features are processed by 7*1*1 three-dimensional convolution, 3*3*3 three-dimensional convolution and 7*7*1 three-dimensional convolution respectively, to obtain time sequence long features, spatial medium features and time sequence short features respectively; The initial space-time features, the time sequence long features, the spatial medium features and the time sequence short features are sequentially processed by channel attention, spatial attention and time sequence attention respectively, to obtain weighted space-time features, weighted time sequence long features, weighted spatial medium features and weighted time sequence short features respectively; The weighted space-time features, the weighted time sequence long features, the weighted spatial medium features and the weighted time sequence short features are processed by channel dimension splicing, to obtain spliced space-time features; The spliced space-time features are sequentially processed by 1*1*1 three-dimensional convolution and global average pooling, to obtain the reinforced space-time features.

8. A deep learning-based video monitoring personnel risk behavior identification system capable of implementing the deep learning-based video monitoring personnel risk behavior identification method of any one of claims 1-7, characterized in that, The system comprises: An acquisition module: acquiring multiple monitoring videos, preprocessing the multiple monitoring videos, obtaining each video frame set corresponding to each monitoring video, and constructing a target data set according to each video frame; A construction module: constructing a risk behavior recognition network, the risk behavior recognition network comprising an RGB feature extraction module, an optical flow feature extraction module, a reinforcement fusion module, a space-time feature extraction module, an attention fusion module and a classifier; A test module: training the risk behavior recognition network according to part of the data in the target data set, and testing the trained risk behavior recognition network according to another part of the data; A prediction module: acquiring a target video frame set, processing the target video frame set according to the risk behavior recognition network that has passed the test, and obtaining a risk behavior recognition result corresponding to the target video frame set.

9. An electronic device, comprising: The program or instructions stored on the readable storage medium are executed by the processor to implement the steps of the deep learning-based video monitoring personnel risk behavior recognition method according to any one of claims 1-7.

10. A readable storage medium, characterized by, The program or instructions stored on the readable storage medium are executed by the processor to implement the steps of the deep learning-based video monitoring personnel risk behavior recognition method according to any one of claims 1-7.

Images