Space-time asynchronous dual-channel human action recognition method and device for monitoring video

Abstract

The application relates to a space-time asynchronous dual-channel human action recognition method and device for monitoring video, and belongs to the technical field of intelligent monitoring. The method comprises the following steps: acquiring a monitoring video stream, and performing time sequence downsampling on the monitoring video stream to obtain a sampling sequence. Video frames in the sampling sequence are pretreated through a time sequence-space enhancement strategy to extract space-time features. A cross-channel feature alignment mechanism is used to generate dynamic weights based on the space-time features, and the space-time features are subjected to multi-scale fusion according to the dynamic weights. Key points of a target object in the video frames are identified through OpenPose, and a spatial weight is generated based on a key point heat map in the video frames. Based on the multi-scale fused space-time features and the spatial weight, a convolutional neural network is trained to call the trained convolutional neural network to output an action category probability distribution according to the motion features of the target object, so that the action recognition accuracy in a complex scene is improved, and the real-time performance is good.

Description

Technical Field

[0001] This invention relates to the field of intelligent surveillance technology, and in particular to a spatiotemporal asynchronous dual-path human motion recognition method and device for surveillance video. Background Technology

[0002] With the widespread application of intelligent surveillance systems, video-based human motion recognition technology has become an important research direction in the field of computer vision. This technology has broad application prospects in public safety, smart homes, and human-computer interaction. However, existing surveillance video motion recognition methods still face many challenges, especially in complex scenarios where there is a contradiction between the spatiotemporal feature representation of motion and computational efficiency.

[0003] Traditional action recognition methods primarily employ single-modal feature extraction strategies, such as temporal feature modeling based on optical flow or spatial feature analysis based on static frames. However, these methods have significant limitations in complex surveillance scenarios: on the one hand, individual temporal or spatial features are insufficient to comprehensively represent the complete semantics of an action; on the other hand, the robustness of a single feature modality is inadequate when facing practical challenges such as changes in viewpoint, target occlusion, and illumination variations. Although existing dual-path models attempt to extract spatiotemporal features simultaneously through parallel architectures, their feature fusion mechanisms typically employ simple synchronization strategies, failing to fully consider the dynamic relationships between spatiotemporal features, thus limiting the model's ability to represent complex actions.

[0004] In recent years, deep learning-based methods (such as 3D convolutional networks and spatiotemporal graph convolutional networks) have made some progress in action recognition tasks, but problems such as high computational complexity and insufficient modeling of long-term action sequences still exist. Especially in surveillance scenarios, where real-time requirements are high, existing methods struggle to achieve a balance between accuracy and efficiency. Therefore, there is an urgent need for an action recognition method that can efficiently model the spatiotemporal features of surveillance videos to improve the accuracy of action recognition in complex scenarios while meeting the demands of real-time processing. Summary of the Invention

[0005] The present invention aims to provide a spatiotemporal asynchronous dual-path human motion recognition method and device for surveillance video, so as to overcome the shortcomings of the existing technology. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0006] This invention provides a spatiotemporal asynchronous dual-path human motion recognition method for surveillance video, the method comprising: Acquire the surveillance video stream and perform time-series downsampling on the surveillance video stream to obtain a sampling sequence; The video frames in the sampling sequence are preprocessed using a temporal-spatial enhancement strategy to extract spatiotemporal features; A cross-path feature alignment mechanism is used to generate dynamic weights based on the spatiotemporal features, and the spatiotemporal features are fused at multiple scales according to the dynamic weights. Key points of target objects in the video frame are identified using OpenPose, and spatial weights are generated based on the heatmap of key points in the video frame. Based on the spatiotemporal features and spatial weights after multi-scale fusion, a convolutional neural network is trained, and the trained convolutional neural network is invoked to output the probability distribution of action categories according to the motion features of the target object.

[0007] In one embodiment, the step of acquiring the surveillance video stream and performing temporal downsampling on the surveillance video stream to obtain a sampling sequence includes: The total number of original video frames is determined based on the monitoring video stream, and then time-series downsampling is performed by downsampling operation based on a preset downsampling factor and the total number of original video frames to obtain the downsampled video frame count. The subsampling sequence is determined based on the number of downsampled video frames. The subsampling sequence is used to characterize the downsampled video tensor and is composed of the height, spatial dimension, and number of RGB channels of the video frame.

[0008] In one embodiment, the preprocessing of video frames in the sampling sequence using a temporal-spatial enhancement strategy to extract spatiotemporal features includes: The spatial domain in the sampling sequence is randomly horizontally flipped, and the spatial domain is color-dithered according to the random perturbation value of the RGB channels; Random temporal segments are sampled from the time domain of the sampling sequence according to the number of randomly sampled video segments.

[0009] In one embodiment, the spatiotemporal features are divided into spatial path features and temporal path features; the step of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, includes the following prior steps: The spatial pathway features are obtained by extracting static spatial features from each video frame using a lightweight EfficientNet-B3. A dilated temporal convolution stacked structure combined with an activation function is used to extract temporal features from the downsampled video tensor to obtain the temporal path features; Based on the spatial and temporal path features, the convolutional neural network is pre-trained using the same weight matrix and joint backpropagation to update the underlying convolutional parameters.

[0010] In one embodiment, the step of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, includes: The spatial path features are extended to the time step by 3D interpolation to align the spatial path features and obtain aligned spatial path features. The temporal path features and the aligned spatial path features are concatenated, and the spatiotemporal coupling coefficient is calculated by combining a gated attention mechanism with a sigmoid function to generate the dynamic weights.

[0011] In one embodiment, the step of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, further includes: Based on the dynamic weights, the temporal path features and the aligned spatial path features are multiplied and summed element-wise to obtain the dynamic fusion features; Based on the dynamic fusion features, a nonlinear mapping relationship of spatiotemporal path features is established through a pyramid feature aggregation algorithm to adaptively adjust the contribution of different spatiotemporal regions and obtain spatiotemporal features after multi-scale fusion.

[0012] In one embodiment, training a convolutional neural network based on multi-scale fused spatiotemporal features and spatial weights, and then calling the trained convolutional neural network to output an action category probability distribution based on the motion features of the target object, includes: Temporal max pooling and temporal average pooling are set in the classification layer of the convolutional neural network, and the cross-entropy loss is improved by using a label smoothing strategy as the loss function. The temporal and spatial path features in the spatiotemporal features are concatenated, and the concatenated dual-path features are input into the fully connected layer of the convolutional neural network.

[0013] The present invention also provides a spatiotemporal asynchronous dual-path human motion recognition device for surveillance videos, used to implement the spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos described in any of the above claims, the device comprising: The video sampling module is used to acquire the monitoring video stream and perform time-series downsampling on the monitoring video stream to obtain a sampling sequence; The feature extraction module is used to preprocess the video frames in the sampling sequence using a temporal-spatial enhancement strategy to extract spatiotemporal features; The feature fusion module is used to generate dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and to perform multi-scale fusion of the spatiotemporal features according to the dynamic weights. The action recognition module is used to identify key points of target objects in the video frame using OpenPose, and generate spatial weights based on the heatmap of key points in the video frame. The action classification module is used to train a convolutional neural network based on spatiotemporal features and spatial weights after multi-scale fusion, so that the trained convolutional neural network can output the action category probability distribution according to the motion features of the target object.

[0014] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video as described above.

[0015] The present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video as described above.

[0016] The aforementioned spatiotemporal asynchronous dual-path human motion recognition method and device for surveillance videos have the following advantages compared to existing technologies: (1) This invention extracts spatiotemporal features from video frames in the sampling sequence through a temporal-spatial enhancement strategy, and generates dynamic weights based on the extracted spatiotemporal features using a cross-path feature alignment mechanism. It performs multi-scale fusion of spatiotemporal features and designs a dynamic spatiotemporal feature interaction process to enhance the global dependency modeling of long-term actions and the local correlation capture of short-term actions, thereby improving the recognition accuracy of complex actions.

[0017] (2) This invention identifies key points of target objects in video frames using OpenPose and generates spatial weights based on the heatmap of key points in video frames. It combines multi-scale fusion of dual-path features and dynamic weights with lightweight path design and dynamic calculation allocation, reducing model complexity while maintaining accuracy, further adapting to the deployment requirements of edge devices, and can also dynamically suppress noise features in the background area, focusing on key areas of human body movements. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 2 This is a schematic diagram of the overall motion recognition process of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video in a specific embodiment provided by the present invention; Figure 3 This is the second flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 4 This is the third flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 5 This is the fourth flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 6 This is the fifth flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 7 This is the sixth flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 8 This is the seventh flowchart of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention; Figure 9 This is a schematic diagram of the spatiotemporal asynchronous dual-path human motion recognition device for surveillance videos provided by the present invention; Figure 10 This is an internal structural diagram of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The following is combined with Figures 1 to 10 This invention describes a spatiotemporal asynchronous dual-path human motion recognition method and apparatus for surveillance video.

[0022] like Figure 1 As shown in one embodiment, a spatiotemporal asynchronous dual-path human motion recognition method for surveillance video includes the following steps: Step S110: Acquire the monitoring video stream and perform time-series downsampling on the monitoring video stream to obtain the sampling sequence.

[0023] Specifically, in combination Figure 2 As shown, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos provided by this invention focuses on recognizing more than ten common actions in surveillance scenarios, such as walking, running, falling, waving, carrying, fighting, etc. The overall architecture of this method adopts a three-level optimization strategy of "feature decoupling - interaction enhancement - noise suppression".

[0024] First, enter the video. and input video Temporal downsampling is represented as: In the formula, This indicates the total number of frames in the original video. Indicates the downsampling factor (values ​​are 2, 3, or 4); This indicates the number of video frames after downsampling; This indicates a downward fetching operation.

[0025] After the above temporal downsampling, the subsampled sequence is obtained: in, This represents the downsampled video tensor; This indicates the height and width of the video frame; the number 3 represents the number of RGB channels.

[0026] Step S120: The video frames in the sampled sequence are preprocessed using a temporal-spatial enhancement strategy to extract spatiotemporal features.

[0027] Specifically, in combination Figure 2 As shown, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos provided by this invention specifically employs a temporal-spatial joint enhancement strategy as follows: Spatial domain: random horizontal flipping, color jitter ( ), Random perturbation value of RGB channels (range ±15); Time domain: Random time series segment sampling (length) frame), : Length (number of frames) of the randomly sampled video segment.

[0028] During the spatiotemporal feature extraction process, a pre-trained ImageNet model needs to be used as the spatial path for initialization, while the temporal path needs to be configured with a dilated convolution mechanism.

[0029] First, spatial pathway features are used as a branch of the CNN, and the lightweight EfficientNet-B3 is used to extract static features for each frame: In the formula, Represents the static spatial characteristics of the t-th frame; This represents the image data of frame t; 384 represents the spatial dimension of the feature map; 384 represents the number of feature channels.

[0030] In this embodiment, the temporal path features are used as TCN branches, employing a dilated temporal convolution stacked structure: In the formula, Indicates the first l The temporal characteristics of the layer; This represents the input video sequence; This indicates the kernel size (3 in this case); Indicates the expansion rate (the first) l Layer is ); This is the activation function.

[0031] Finally, the temporal pathway characteristics are represented as follows: The temporal feature has 512 layers. The spatial path is downsampled at 32, while the temporal path maintains the original temporal resolution.

[0032] In this embodiment, the two paths share the underlying convolution parameters to reduce computational redundancy, specifically including: (1) Sharing mechanism: The two paths use the same weight matrix in shallow convolution (the first 3 layers of Conv1D); (2) Training strategy: During joint backpropagation, the gradient will update the underlying parameters of both pathways simultaneously.

[0033] Step S130: A cross-path feature alignment mechanism is used to generate dynamic weights based on spatiotemporal features, and the spatiotemporal features are fused at multiple scales according to the dynamic weights.

[0034] Specifically, in combination Figure 2 As shown, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos provided by this invention achieves dynamic spatiotemporal feature interaction by designing a cross-path feature alignment mechanism and a dynamic weight generator containing a gating unit.

[0035] In the feature alignment process, spatial features are first aligned using 3D interpolation. Expand to Time step: In the formula, Indicates the spatial characteristics after alignment; This indicates a trilinear interpolation operation.

[0036] Among them, spatial features are obtained through 3D interpolation. Expand to The time step specifically includes the following steps: (1) Spatial Dimension Expansion: Expanding spatial features Interpolation along the time dimension frame; (2) Trilinear interpolation: interpolation is performed on a 7×7 spatial grid; (3) Channel maintenance: Maintain the number of channels at 384.

[0037] Then, the spatiotemporal coupling coefficient is calculated using a gated attention mechanism: In the formula, Indicates the gating attention weight; It is the sigmoid function; A combination representing spatial and temporal characteristics; Indicates element-wise multiplication; This represents the features after dynamic fusion.

[0038] Finally, a nonlinear mapping relationship of spatiotemporal features is established through pyramid feature aggregation to adaptively adjust the contribution of different spatiotemporal regions, thereby achieving multi-scale feature fusion. In the formula, Corresponding pooling window sizes of 1 / 2 / 4 frames; This represents the spatiotemporal joint features after aggregation. This indicates average pooling.

[0039] Step S140: Key points of the target object in the video frame are identified using OpenPose, and spatial weights are generated based on the heatmap of key points in the video frame.

[0040] Specifically, in combination Figure 2 As shown, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos provided by this invention first uses OpenPose to detect 17 key points of the human body. For a Gaussian heatmap (7×7 resolution), call... The function generates spatial weights based on the key point heatmap. : After that, through Functions for motion saliency detection: In the formula, Indicates time weighting, Represents the feature difference between adjacent frames. This indicates the time interval between adjacent frames.

[0041] Based on the calculation and Perform joint masking: In the formula, This represents the outer product operation (spatial weight × time weight). This is an enhanced feature after background suppression.

[0042] This process can effectively suppress non-motion-related feature responses in the background region and enhance the representation focus of the human motion region.

[0043] Step S150: Based on the spatiotemporal features and spatial weights after multi-scale fusion, the convolutional neural network is trained so that the trained convolutional neural network can output the probability distribution of action categories according to the motion features of the target object.

[0044] Specifically, in combination Figure 2 As shown, the spatiotemporal asynchronous dual-path human action recognition method for surveillance videos provided by this invention includes multi-timescale pooling in the classification layer during the action classification process, and the loss function adopts a label smoothing strategy.

[0045] In this embodiment, the temporal max pooling is Output dimension 512; Temporal average pooling is The output dimension is 512.

[0046] During the classification decision-making process, spatial path features and temporal path features are concatenated and then input into the fully connected layer: In the formula, For the weights of the fully connected layer (C is the number of action categories); Action probability distribution , It is a constant.

[0047] The loss function is an improved cross-entropy loss function. : In the formula, This represents the total number of action categories; it is a positive integer, and in this example, it is 10+ categories. For the category of real action tags, ; For indicator functions, when The value is 1 if it is true, and 0 otherwise. This represents the probability that the model predicts the class as c. ; 0.9 is the confidence coefficient of the true label; 0.1 is the smooth distribution coefficient.

[0048] Finally, by combining global and local motion features, the probability distribution of action categories is output.

[0049] The aforementioned spatiotemporal asynchronous dual-path human motion recognition method for surveillance videos extracts spatiotemporal features from video frames in the sampling sequence through a temporal-spatial enhancement strategy. It then employs a cross-path feature alignment mechanism to generate dynamic weights based on the extracted spatiotemporal features, performs multi-scale fusion of these features, and designs a dynamic spatiotemporal feature interaction process. This enhances the global dependency modeling of long-term actions and the local correlation capture of short-term actions, improving the recognition accuracy of complex actions. Furthermore, the method uses OpenPose to identify key points of target objects in video frames and generates spatial weights based on the key point heatmap. Combining multi-scale fusion of dual-path features, dynamic weights, lightweight path design, and dynamic computation allocation, it reduces model complexity while maintaining accuracy, further adapting to the deployment requirements of edge devices. It can also dynamically suppress background noise features, focusing on key areas of human motion.

[0050] like Figure 3 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention includes the following steps in step S110: Step S111: Determine the total number of original video frames based on the monitoring video stream, and perform temporal downsampling based on the preset downsampling factor and the total number of original video frames through downsampling operation to obtain the downsampled video frame count.

[0051] Step S112: Determine the subsampling sequence based on the number of downsampled video frames. The subsampling sequence is used to characterize the downsampled video tensor and is composed of the height, spatial dimension, and number of RGB channels of the video frame.

[0052] like Figure 4 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention includes the following steps in step S120: Step S121: Randomly flip the spatial domain in the sampling sequence horizontally, and perform color jitter on the spatial domain according to the random perturbation value of the RGB channels.

[0053] Step S122: Random time-series segment sampling is performed on the time domain of the sampling sequence according to the number of randomly sampled video segment frames.

[0054] like Figure 5 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention further includes the following steps before step S130: Step S510: Extract static spatial features from each video frame using a lightweight EfficientNet-B3 to obtain spatial path features.

[0055] Step S520: Using a dilated temporal convolution stacked structure combined with an activation function, temporal features are extracted from the downsampled video tensor to obtain temporal path features.

[0056] Step S530: Based on spatial path features and temporal path features, the convolutional neural network is pre-trained by using the same weight matrix to jointly backpropagate and update the underlying convolutional parameters.

[0057] like Figure 6 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention includes the following steps in step S130: Step S131: Extend the spatial path features to the time step through 3D interpolation to align the spatial path features and obtain aligned spatial path features.

[0058] Step S132: The temporal path features and the aligned spatial path features are concatenated, and the spatiotemporal coupling coefficient is calculated by combining the gating attention mechanism with the sigmoid function to generate dynamic weights.

[0059] like Figure 7 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention further includes the following steps in step S130: Step S133: Based on the dynamic weights, the temporal path features and the aligned spatial path features are multiplied and summed element by element to obtain the dynamic fusion features.

[0060] Step S134: Based on dynamic fusion features, a nonlinear mapping relationship of spatiotemporal pathway features is established through a pyramid feature aggregation algorithm to adaptively adjust the contribution of different spatiotemporal regions and obtain spatiotemporal features after multi-scale fusion.

[0061] like Figure 8 As shown, in some embodiments, the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video provided by the present invention specifically includes the following steps in step S150: Step S151: Set temporal max pooling and temporal average pooling in the classification layer of the convolutional neural network, and improve the cross-entropy loss by using a label smoothing strategy as the loss function.

[0062] Step S152: The temporal path features and spatial path features in the spatiotemporal features are concatenated, and the concatenated dual-path features are input into the fully connected layer of the convolutional neural network.

[0063] The spatiotemporal asynchronous dual-path human motion recognition device for surveillance video provided by the present invention will be described below. The spatiotemporal asynchronous dual-path human motion recognition device for surveillance video described below can be referred to in correspondence with the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video described above.

[0064] like Figure 9 As shown, in one embodiment, a spatiotemporal asynchronous dual-path human motion recognition device for surveillance video includes a video sampling module 910, a feature extraction module 920, a feature fusion module 930, a motion recognition module 940, and a motion classification module 950.

[0065] The video sampling module 910 is used to acquire the monitoring video stream and perform time-series downsampling on the monitoring video stream to obtain a sampling sequence.

[0066] The feature extraction module 920 is used to preprocess video frames in the sampled sequence using a temporal-spatial enhancement strategy to extract spatiotemporal features.

[0067] The feature fusion module 930 is used to generate dynamic weights based on spatiotemporal features using a cross-path feature alignment mechanism, and to perform multi-scale fusion of spatiotemporal features according to the dynamic weights.

[0068] The motion recognition module 940 is used to identify key points of target objects in video frames using OpenPose, and generate spatial weights based on the key point heatmap in the video frames.

[0069] The action classification module 950 is used to train a convolutional neural network based on the spatiotemporal features and spatial weights after multi-scale fusion, so as to call the trained convolutional neural network to output the action category probability distribution according to the motion features of the target object.

[0070] In this embodiment, the spatiotemporal asynchronous dual-path human motion recognition device for surveillance videos provided by the present invention, the video sampling module 910 is specifically used for: The total number of original video frames is determined based on the monitored video stream, and then time-series downsampling is performed using a downsampling operation based on a preset downsampling factor and the total number of original video frames to obtain the downsampled video frame count.

[0071] The subsampling sequence is determined based on the number of downsampled video frames. The subsampling sequence is used to characterize the downsampled video tensor, which is composed of the height, spatial dimension, and number of RGB channels of the video frame.

[0072] In this embodiment, the feature extraction module 920 of the spatiotemporal asynchronous dual-path human motion recognition device for surveillance video provided by the present invention is specifically used for: The spatial domain in the sampling sequence is randomly horizontally flipped, and the spatial domain is color-dithered according to the random perturbation values ​​of the RGB channels.

[0073] Random temporal segments are sampled from the time domain of the sampling sequence based on the number of frames of the randomly sampled video segments.

[0074] In this embodiment, the spatiotemporal asynchronous dual-path human motion recognition device for surveillance videos provided by the present invention further includes a model pre-training module, used for: Spatial path features are obtained by extracting static spatial features from each video frame using a lightweight EfficientNet-B3.

[0075] A dilated temporal convolution stacked structure combined with an activation function is used to extract temporal features from the downsampled video tensor to obtain temporal path features.

[0076] Based on spatial and temporal path features, the convolutional neural network is pre-trained using the same weight matrix and joint backpropagation to update the underlying convolutional parameters.

[0077] In this embodiment, the feature fusion module 930 of the spatiotemporal asynchronous dual-path human motion recognition device for surveillance video provided by the present invention is specifically used for: Spatial pathway features are extended to the time step through 3D interpolation to align them, resulting in aligned spatial pathway features.

[0078] The temporal path features and aligned spatial path features are concatenated, and the spatiotemporal coupling coefficient is calculated by combining a gated attention mechanism with the sigmoid function to generate dynamic weights.

[0079] In this embodiment, the feature fusion module 930 of the spatiotemporal asynchronous dual-path human motion recognition device for surveillance video provided by the present invention is further used for: The dynamic fusion features are obtained by multiplying and summing the temporal path features and the aligned spatial path features element-wise based on dynamic weights.

[0080] Based on dynamic fusion features, a nonlinear mapping relationship of spatiotemporal pathway features is established through a pyramid feature aggregation algorithm to adaptively adjust the contribution of different spatiotemporal regions and obtain spatiotemporal features after multi-scale fusion.

[0081] In this embodiment, the spatiotemporal asynchronous dual-path human motion recognition device for surveillance video provided by the present invention, the motion classification module 950 is specifically used for: Temporal max pooling and temporal average pooling are set in the classification layer of the convolutional neural network, and the cross-entropy loss is improved by using a label smoothing strategy as the loss function.

[0082] The temporal and spatial pathway features in the spatiotemporal features are concatenated, and the concatenated dual-path features are input into the fully connected layer of the convolutional neural network.

[0083] Figure 10 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 10 As shown. The electronic device includes a processor, internal memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a spatiotemporal asynchronous dual-path human motion recognition method for surveillance video, which includes:

[0084] Acquire the surveillance video stream and perform temporal downsampling on the surveillance video stream to obtain the sampling sequence; A temporal-spatial enhancement strategy is used to preprocess video frames in the sampled sequence to extract spatiotemporal features; A cross-path feature alignment mechanism is adopted to generate dynamic weights based on spatiotemporal features, and multi-scale fusion of spatiotemporal features is performed according to the dynamic weights; OpenPose is used to identify key points of target objects in video frames, and spatial weights are generated based on the heatmap of key points in the video frames. Based on the spatiotemporal features and spatial weights after multi-scale fusion, a convolutional neural network is trained, and the trained convolutional neural network is invoked to output the probability distribution of action categories according to the motion features of the target object.

[0085] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0086] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a spatiotemporal asynchronous dual-path human motion recognition method for surveillance video, the method comprising: Acquire the surveillance video stream and perform temporal downsampling on the surveillance video stream to obtain the sampling sequence; A temporal-spatial enhancement strategy is used to preprocess video frames in the sampled sequence to extract spatiotemporal features; A cross-path feature alignment mechanism is adopted to generate dynamic weights based on spatiotemporal features, and multi-scale fusion of spatiotemporal features is performed according to the dynamic weights; OpenPose is used to identify key points of target objects in video frames, and spatial weights are generated based on the heatmap of key points in the video frames. Based on the spatiotemporal features and spatial weights after multi-scale fusion, a convolutional neural network is trained, and the trained convolutional neural network is invoked to output the probability distribution of action categories according to the motion features of the target object.

[0087] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements a spatiotemporal asynchronous dual-path human motion recognition method for surveillance video, the method comprising:

[0088] Acquire the surveillance video stream and perform temporal downsampling on the surveillance video stream to obtain the sampling sequence; A temporal-spatial enhancement strategy is used to preprocess video frames in the sampled sequence to extract spatiotemporal features; A cross-path feature alignment mechanism is adopted to generate dynamic weights based on spatiotemporal features, and multi-scale fusion of spatiotemporal features is performed according to the dynamic weights; OpenPose is used to identify key points of target objects in video frames, and spatial weights are generated based on the heatmap of key points in the video frames. Based on the spatiotemporal features and spatial weights after multi-scale fusion, a convolutional neural network is trained, and the trained convolutional neural network is invoked to output the probability distribution of action categories according to the motion features of the target object.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0090] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A spatiotemporal asynchronous dual-path human motion recognition method for surveillance video, characterized in that, The method includes: Acquire the surveillance video stream and perform time-series downsampling on the surveillance video stream to obtain a sampling sequence; The video frames in the sampling sequence are preprocessed using a temporal-spatial enhancement strategy to extract spatiotemporal features; The spatiotemporal characteristics are divided into spatial path characteristics and temporal path characteristics; The spatial pathway features are obtained by extracting static spatial features from each video frame using a lightweight EfficientNet-B3. A dilated temporal convolution stacked structure combined with an activation function is used to extract temporal features from the downsampled video tensor to obtain the temporal path features; A cross-path feature alignment mechanism is used to generate dynamic weights based on the spatiotemporal features, and the spatiotemporal features are fused at multiple scales according to the dynamic weights. Key points of target objects in the video frame are identified using OpenPose, and spatial weights are generated based on the heatmap of key points in the video frame. First, OpenPose was used to detect 17 key points on the human body. For Gaussian heatmaps, 7x7 resolution, call... The function generates spatial weights based on the key point heatmap. : After that, through Functions for motion saliency detection: In the formula, Indicates time weight, Represents the feature difference between adjacent frames. Indicates the time interval between adjacent frames; Based on the calculation and Perform joint masking: In the formula, This represents the outer product operation. Enhanced features after background suppression Spatial features; The enhanced features, derived from background suppression, are input into the classification layer to output the probability distribution of action categories.

2. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 1, characterized in that, The process of acquiring the surveillance video stream and performing temporal downsampling on the surveillance video stream to obtain a sampling sequence includes: The total number of original video frames is determined based on the monitored video stream, and then time-series downsampling is performed by rounding down according to the preset downsampling factor and the total number of original video frames to obtain the downsampled video frame count. The subsampling sequence is determined based on the number of downsampled video frames. The subsampling sequence is used to characterize the downsampled video tensor and is composed of the height, spatial dimension, and number of RGB channels of the video frame.

3. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 2, characterized in that, The step of preprocessing video frames in the sampled sequence using a temporal-spatial enhancement strategy to extract spatiotemporal features includes: The spatial domain in the sampling sequence is randomly horizontally flipped, and the spatial domain is color-dithered according to the random perturbation value of the RGB channels; Random temporal segments are sampled from the time domain of the sampling sequence according to the number of randomly sampled video segments.

4. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 2, characterized in that, The step of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, includes: Based on the spatial and temporal path features, the convolutional neural network is pre-trained using the same weight matrix and joint backpropagation to update the underlying convolutional parameters.

5. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 4, characterized in that, The method of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, includes: The spatial path features are extended to the time step by 3D interpolation to align the spatial path features and obtain aligned spatial path features. The temporal path features and the aligned spatial path features are concatenated, and the spatiotemporal coupling coefficient is calculated by combining a gated attention mechanism with a sigmoid function to generate the dynamic weights.

6. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 5, characterized in that, The method of generating dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and performing multi-scale fusion of the spatiotemporal features according to the dynamic weights, further includes: Based on the dynamic weights, the temporal path features and the aligned spatial path features are multiplied and summed element-wise to obtain the dynamic fusion features; Based on the dynamic fusion features, a nonlinear mapping relationship of spatiotemporal path features is established through a pyramid feature aggregation algorithm to adaptively adjust the contribution of different spatiotemporal regions and obtain spatiotemporal features after multi-scale fusion.

7. The spatiotemporal asynchronous dual-path human motion recognition method for surveillance video according to claim 1, characterized in that, The enhanced features based on background suppression are input into the classification layer to output an action category probability distribution, including: Temporal max pooling and temporal average pooling are set in the classification layer of the convolutional neural network, and the cross-entropy loss is improved by using a label smoothing strategy as the loss function. The output features of temporal max pooling and temporal average pooling are concatenated, and the concatenated dual-path features are then input into the fully connected layer of the convolutional neural network.

8. A spatiotemporal asynchronous dual-path human motion recognition device for surveillance video, characterized in that, The apparatus for implementing the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video as described in any one of claims 1 to 7, the apparatus comprising: The video sampling module is used to acquire the monitoring video stream and perform time-series downsampling on the monitoring video stream to obtain a sampling sequence; The feature extraction module is used to preprocess the video frames in the sampling sequence using a temporal-spatial enhancement strategy to extract spatiotemporal features; The feature fusion module is used to generate dynamic weights based on the spatiotemporal features using a cross-path feature alignment mechanism, and to perform multi-scale fusion of the spatiotemporal features according to the dynamic weights. The action recognition module is used to identify key points of target objects in the video frame using OpenPose, and generate spatial weights based on the heatmap of key points in the video frame. The action classification module is used to input enhanced features based on background suppression into the classification layer to output the action category probability distribution.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video as described in any one of claims 1 to 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the spatiotemporal asynchronous dual-path human motion recognition method for surveillance video as described in any one of claims 1 to 7.

Images