Behavior detection method and system based on skeleton thermodynamic diagram and frequency domain convolution

Abstract

The invention provides a behavior detection method and system based on a skeleton thermodynamic diagram and frequency domain convolution. The method comprises the following steps: acquiring a video sequence to be detected; extracting human skeleton key points from the to-be-detected video sequence to obtain a human skeleton key point sequence, and converting the human skeleton key point sequence to obtain a corresponding skeleton thermodynamic diagram; the skeleton thermodynamic diagram is input into a pre-trained behavior recognition model for feature extraction, behavior recognition is carried out according to the extracted features to obtain a recognition result corresponding to the to-be-detected video sequence, the behavior recognition model comprises an input layer, a hierarchical residual module and a classifier, the hierarchical residual module is embedded with a frequency domain convolution module, and the classifier is embedded with a frequency domain convolution module; the global features of the time-space domain and the local detail features of the frequency domain are fused; therefore, by extracting the global features and the local features of the skeleton thermodynamic diagram and introducing frequency domain analysis, the behavior detection effect is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a behavior detection method based on skeletal heatmap and frequency domain convolution, a behavior detection system based on skeletal heatmap and frequency domain convolution, a computer-readable storage medium, and a computer device. Background Technology

[0002] With the increasing demand for social security, automated violent behavior detection technology is becoming increasingly important in fields such as video surveillance and public safety. Traditional behavior detection methods mainly rely on two types of technologies: First, methods based on manually designed features, such as calculating low-level visual features like optical flow and gradient histograms of video sequences, combined with classifiers like support vector machines for judgment. These methods have limited feature representation capabilities, are highly sensitive to environmental background, lighting changes, and viewpoint shifts, and have poor robustness. Second, end-to-end methods based on deep learning, such as using 3D convolutional neural networks to directly learn the spatiotemporal features of original video frames. Although these methods have stronger feature extraction capabilities, their models often need to process massive amounts of pixel information, resulting in high computational costs and heavy reliance on large-scale, precisely labeled datasets; at the same time, the models are prone to overfitting to scene background information unrelated to the behavior itself, leading to poor generalization performance.

[0003] Meanwhile, with the maturation of human pose estimation technology, behavior recognition based on skeletal keypoints has received widespread attention as a new technical approach. This method abstracts the human body into a spatiotemporal sequence of keypoints, constructing a spatiotemporal graph model, thereby transforming the behavior recognition problem from pixel space to a pure human body space with more concise information and a more accurate structure. However, existing skeletal model-based methods still face significant challenges in the specific task of violent behavior detection: First, violent behavior is usually characterized by intense, rapid, and unstructured limb movements, and effectively capturing its unique spatiotemporal dynamic patterns from the skeletal sequence is a key difficulty. Second, existing models such as graph convolutional networks still lack the ability to model the interpersonal keypoint interactions when dealing with interactive violent behavior involving two or more people. Furthermore, in traditional graph neural network models, the relationships between skeletal keypoints are severely forgotten, which significantly limits the depth of behavior recognition networks based on graph neural networks. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, one objective of this invention is to propose a behavior detection method based on skeletal heatmaps and frequency domain convolution. By extracting global and local features from the skeletal heatmap and introducing frequency domain analysis, the accuracy, robustness, and generalization ability of behavior feature extraction are effectively improved, thereby enhancing the detection performance.

[0005] A second objective of this invention is to provide a computer-readable storage medium.

[0006] The third objective of this invention is to provide a computer device.

[0007] The fourth objective of this invention is to propose a behavior detection system based on skeletal heatmaps and frequency domain convolution.

[0008] To achieve the above objectives, a first aspect of this invention proposes a behavior detection method based on skeletal heatmaps and frequency domain convolution. The method includes the following steps: acquiring a video sequence to be detected; extracting human skeletal key points from the video sequence to obtain a human skeletal key point sequence, and transforming the human skeletal key point sequence to obtain a corresponding skeletal heatmap; inputting the skeletal heatmap into a pre-trained behavior recognition model for feature extraction, and performing behavior recognition based on the extracted features to obtain the recognition result corresponding to the video sequence to be recognized. The behavior recognition model includes an input layer, a hierarchical residual module, and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain. Therefore, by extracting global and local features from the skeletal heatmap and introducing frequency domain analysis, the accuracy, robustness, and generalization ability of behavior feature extraction are effectively improved, thereby enhancing the detection effect.

[0009] In addition, the behavior detection method based on bone heatmap and frequency domain convolution proposed in the above embodiments of the present invention may also have the following additional technical features:

[0010] Optionally, extracting human skeletal key points from the video sequence to be detected to obtain a human skeletal key point sequence, and transforming the human skeletal key point sequence to obtain a corresponding skeletal heatmap, includes: uniformly selecting a fixed number of frames in the video sequence to be detected along the time dimension to obtain a sampled video frame sequence; extracting human skeletal key points from the sampled video frame sequence to obtain a human skeletal key point sequence, and spatially normalizing the human skeletal key points to obtain a spatially normalized skeletal key point sequence; and transforming the spatially normalized skeletal key point sequence to obtain a corresponding skeletal heatmap.

[0011] Optionally, the skeletal heatmap is input into a pre-trained behavior recognition model for feature extraction, including: the input layer performing three-dimensional convolution, normalization, activation, and max pooling on the skeletal heatmap to obtain an initial feature map; and the hierarchical residual module performing channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain an enhanced feature tensor.

[0012] Optionally, the hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain an enhanced feature tensor, including: channel compression of the initial feature map to obtain compressed features; inputting the compressed features into a frequency domain convolution module to enhance the globally important parts of the compressed features using an attention modulation mechanism, followed by three-dimensional convolution to obtain global spatiotemporal response features; performing a fast Fourier transform on the compressed features to transform them to the frequency domain, taking the real part and averaging it in the time dimension to extract the dominant frequency response, and then projecting it through the channels. The original features are modulated to obtain frequency domain modulation features; the global spatiotemporal response features and the frequency domain modulation features are added to output fused features; channel recovery is performed on the fused features, and it is determined whether the number of channels / resolution of the channel-recovered features is the same as the number of channels / resolution of the initial feature map; if they are different, the initial feature map is adjusted by bypass convolution and then fused with the channel-recovered features to obtain an enhanced feature tensor; if they are the same, the initial feature map is directly fused with the channel-recovered features to obtain an enhanced feature tensor.

[0013] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a behavior recognition program based on skeletal heatmaps and frequency domain convolution, which, when executed by a processor, implements the behavior recognition method based on skeletal heatmaps and frequency domain convolution as described above.

[0014] To achieve the above objectives, a third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the behavior recognition method based on skeletal heatmap and frequency domain convolution as described above.

[0015] To achieve the above objectives, a fourth aspect of the present invention proposes a behavior recognition system based on skeletal heatmaps and frequency domain convolution, comprising: an acquisition module for acquiring a video sequence to be detected; a preprocessing module for extracting human skeletal key points from the video sequence to be detected to obtain a human skeletal key point sequence, and converting the human skeletal key point sequence to obtain a corresponding skeletal heatmap; and a behavior recognition module for inputting the skeletal heatmap into a pre-trained behavior recognition model for feature extraction, and performing behavior recognition based on the extracted features to obtain a recognition result corresponding to the video sequence to be detected. The behavior recognition model includes an input layer, a hierarchical residual module, and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain.

[0016] In addition, the behavior recognition system based on skeletal heatmap and frequency domain convolution proposed in the above embodiments of the present invention may also have the following additional technical features:

[0017] Optionally, the preprocessing module is further configured to: uniformly select a fixed number of frames in the video sequence to be detected along the time dimension to obtain a sampled video frame sequence; extract human skeletal key points from the sampled video frame sequence to obtain a human skeletal key point sequence, and perform spatial normalization on the human skeletal key points to obtain a spatially normalized skeletal key point sequence; and transform the spatially normalized skeletal key point sequence to obtain a corresponding skeletal heatmap.

[0018] Optionally, the behavior recognition module is further configured to: the input layer performs three-dimensional convolution, normalization, activation, and max pooling on the skeletal heatmap to obtain an initial feature map; and the hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain an enhanced feature tensor.

[0019] Optionally, the behavior recognition module is further configured to: compress the initial feature map into channels to obtain compressed features; input the compressed features into a frequency domain convolution module to enhance the globally important parts of the compressed features using an attention modulation mechanism, and then perform three-dimensional convolution to obtain global spatiotemporal response features; perform a fast Fourier transform on the compressed features to convert them to the frequency domain, take the real part and average it in the time dimension to extract the main frequency response, and then modulate it with the original features through channel projection to obtain frequency domain modulated features; add the global spatiotemporal response features and the frequency domain modulated features to output fused features; perform channel restoration on the fused features and determine whether the number of channels / resolution of the channel-restored features is the same as the number of channels / resolution of the initial feature map; if they are different, use bypass convolution to adjust the initial feature map and then fuse it with the channel-restored features to obtain an enhanced feature tensor; if they are the same, directly use the initial feature map and the channel-restored features to fuse them to obtain an enhanced feature tensor. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the behavior recognition method based on skeletal heatmap and frequency domain convolution according to an embodiment of the present invention.

[0021] Figure 2 This is an overall flowchart of the behavior recognition method based on skeletal heatmap and frequency domain convolution according to an embodiment of the present invention;

[0022] Figure 3 This is a network structure diagram of a behavior recognition model according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of the hierarchical residual module according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the frequency domain convolution module according to an embodiment of the present invention;

[0025] Figure 6 This is a block diagram of a behavior detection system based on skeleton heatmap and frequency domain convolution according to an embodiment of the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0028] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0029] refer to Figure 1 As shown, the behavior detection method based on skeleton heatmap and frequency domain convolution in this embodiment of the invention includes the following steps:

[0030] S101, Obtain the video sequence to be detected.

[0031] It should be noted that the video sequence to be detected is video from a real-world application scenario (such as real-time / recorded video captured by a surveillance camera).

[0032] S102, extract human skeletal key points from the video sequence to be detected to obtain a human skeletal key point sequence, and transform the human skeletal key point sequence to obtain the corresponding skeletal heat map.

[0033] As one embodiment, human skeletal key points are extracted from the video sequence to be detected to obtain a human skeletal key point sequence, and the human skeletal key point sequence is transformed to obtain a corresponding skeletal heatmap. This includes: uniformly selecting a fixed number of frames from the video sequence to be detected along the time dimension to obtain a sampled video frame sequence; extracting human skeletal key points from the sampled video frame sequence to obtain a human skeletal key point sequence, and spatially normalizing the human skeletal key points to obtain a spatially normalized skeletal key point sequence; and transforming the spatially normalized skeletal key point sequence to obtain a corresponding skeletal heatmap.

[0034] Specifically, firstly, temporal processing is performed by uniformly sampling the input video using UniformSampleFrames: A fixed sampling of 48 frames is applied, assuming the original video sequence contains N frames. The sampling interval is calculated (e.g., interval = total number of original frames / / 48). Then, starting from the first frame of the video, one frame is selected at each interval until 48 frames are collected. For example, if the original video has 96 frames and the interval is 2, frames 0, 2, 4...94 are selected, totaling 48 frames, ensuring a uniform distribution of frames along the timeline. Next, spatial processing is performed by spatially normalizing skeletal keypoints using PoseCompact and Resize: First, PoseCompact unifies the spatial reference of skeletal keypoints, then Resize fixes the spatial dimensions, achieving spatial normalization of the skeletal keypoints. Then, data augmentation is performed using RandomResizedCrop and Flip to enhance the model's robustness: the augmented skeletal keypoint spatial data, after random cropping, resizing, and flipping, preserves the structural relationships between joints while increasing data diversity. Finally, feature representation is performed. Based on GeneratePoseTarget, the enhanced skeletal keypoint sequence is converted into a skeletal heatmap, which serves as the input tensor of the network. For each skeletal keypoint, a response (usually in Gaussian distribution) is generated at the corresponding spatial location in the skeletal heatmap, so that the spatial location of the skeletal keypoint and the relative relationship between joints are presented in the grayscale distribution of the heatmap. In this way, discrete keypoint coordinates are converted into continuous heatmap tensors, which not only preserves the structural information of the skeleton, but also adapts to the processing requirements of convolutional networks for dense data.

[0035] S103, the skeleton heatmap is input into the pre-trained behavior detection model for feature extraction, and behavior recognition is performed based on the extracted features to obtain the recognition result corresponding to the video sequence to be detected. The behavior recognition model includes an input layer, a hierarchical residual module and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain.

[0036] As an example, the skeletal heatmap is input into a pre-trained behavior recognition model for feature extraction, including: the input layer performs 3D convolution, normalization, activation and max pooling on the skeletal heatmap to obtain an initial feature map; the hierarchical residual module performs channel compression, frequency domain transformation, channel restoration and residual fusion on the initial feature map to obtain an enhanced feature tensor.

[0037] Specifically, such as Figure 2-3 As shown, the behavior recognition model is a three-dimensional frequency domain residual network (SFNet). Based on the traditional three-dimensional residual network (ResNet3D), it introduces a frequency domain convolutional module (FDConv3D), thereby integrating a frequency domain feature enhancement mechanism into the spatiotemporal feature modeling. The network consists of an input layer, four residual stages composed of hierarchical residual modules (SFBlock), and a classification head.

[0038] The input layer consists of a 3D convolutional layer, a normalization layer, and an activation layer. Its computation process is as follows:

[0039] X1 = σ(BN(Conv3D(X0;W0)))

[0040] Where X0 represents the heatmap feature tensor, W0 is the convolution kernel weight, and σ() is the ReLU activation function. Then, max pooling is performed.

[0041] X2 = MaxPool3D(X1)

[0042] To reduce spatial resolution and enhance the receptive field.

[0043] The network consists of four stages connected in series, with the number of channels in each stage being 64, 128, 256, and 512, respectively, and the corresponding spatiotemporal step sizes being (1,1,1), (1,2,2), (1,2,2), and (1,2,2). The final output feature tensor is denoted as:

[0044] F out =SFNet(X0)=f stage4 (f stage3 (f stage2 (f stage1 (X2))))

[0045] This structure enhances the network's ability to express spatiotemporal frequency information in videos by introducing frequency domain modulation and channel adaptive mechanisms, making it particularly suitable for high-dynamic scenarios such as violent behavior recognition.

[0046] As one embodiment, the hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain an enhanced feature tensor. This includes: channel compression of the initial feature map to obtain compressed features; inputting the compressed features into a frequency domain convolution module to enhance the globally important parts of the compressed features using an attention modulation mechanism, followed by 3D convolution to obtain global spatiotemporal response features; performing a fast Fourier transform on the compressed features to convert them to the frequency domain, taking the real part and averaging it in the time dimension to extract the dominant frequency response, then modulating it with the original features through channel projection to obtain frequency domain modulated features; adding the global spatiotemporal response features and the frequency domain modulated features to output the fused features; performing channel restoration on the fused features and determining whether the number of channels / resolution of the restored features is the same as the number of channels / resolution of the initial feature map; if they are different, using bypass convolution to adjust the initial feature map and fusing it with the channel-restored features to obtain an enhanced feature tensor; if they are the same, directly fusing the initial feature map with the channel-restored features to obtain the enhanced feature tensor.

[0047] Specifically, such as Figure 4 As shown, the hierarchical residual module (SFBlock) is the basic residual unit that constitutes the three-dimensional frequency domain residual network (SFNet). This module consists of three convolutional layers and introduces a frequency domain modulation intermediate layer. Let the input be XR. Cin×T×H×W Its forward process is as follows:

[0048] The first step is to perform channel compression:

[0049] X1=δ(BN1(Conv 1×1×1 (X)))

[0050] The second step involves convolutional processing of the intermediate features in the frequency domain:

[0051] X2=δ(BN2(FDConv3D(X1)))

[0052] Among them, FDConv3D() is the core module of this application.

[0053] The third step is to restore the channel:

[0054] X3=BN3(Conv 1×1×1 (X2))

[0055] If there is a downsampling or channel number change, adjust it by bypass convolution I = BNd(Convd(X)); otherwise, I = X.

[0056] Finally, output the residual fusion result:

[0057] Y = δ(X³ + I)

[0058] Where δ() is the ReLU activation function.

[0059] In addition, such as Figure 5 As shown, the input of the FDConv3D module is the features of the SFBlock intermediate layer, and the output is formed by the sum and fusion of the features of the two parallel paths.

[0060] First Pathway (Spatiotemporal Attention Pathway):

[0061] Let the input feature map be XR C×T×H×W Global scale factors are extracted using the Key Scale Modulation (KSM) mechanism:

[0062] S = KSM(X)

[0063] Element-wise modulation is performed to obtain scale-enhanced features:

[0064] X s =XS

[0065] Here, represents element-wise multiplication.

[0066] For X s Perform spatiotemporal convolution operations:

[0067] Yconv = Conv(X s )

[0068] Obtain the global spatiotemporal response.

[0069] Second path (frequency domain analysis path):

[0070] Map the input features to the frequency domain. First, perform a Fast Fourier Transform (FFT) along the time dimension:

[0071]

[0072] Take the real part and average it over the time dimension to extract the dominant frequency response:

[0073]

[0074] The input features are mapped through a learnable channel projection layer Proj() and modulated with the frequency domain response:

[0075] Yfft = Proj(X)X fft

[0076] Feature fusion: To ensure spatiotemporal consistency, if the spatial dimensions of Yfft and Yconv do not match, trilinear interpolation is used for alignment.

[0077] Y fft ←Interp(Y fftsize(Y) conv ))

[0078] The final output feature is obtained by fusing the responses of the two pathways:

[0079] Y = Y conv +Y fft

[0080] It should be noted that when training the behavior recognition model, the loss function used is labeled smooth cross-entropy loss.

[0081] Let the output of the classification head be S = s i,c R B×C The true label is Y = y i .

[0082] After label smoothing:

[0083]

[0084] The loss function is:

[0085]

[0086] The evaluation metric is Top-k accuracy:

[0087]

[0088] Furthermore, practicality verification and inference optimization: To balance recognition accuracy and inference speed to adapt to real-world application scenarios, this application compared the performance of various input sequence lengths, including 1 frame, 3 frames, and 5 frames. Experimental results show that using 3-frame input can maintain high recognition accuracy while controlling the single inference time to approximately 1.34 seconds, achieving the best trade-off between accuracy and efficiency. This is superior to the 1-frame (approximately 0.71 seconds, but with low accuracy) and 5-frame (approximately 2.3 seconds, slow with no significant improvement in accuracy) solutions.

[0089] In summary, the behavior recognition method based on skeletal heatmaps and frequency domain convolution proposed in this application converts skeletal key points into skeletal heatmaps, transforming structured skeletal sequence data into a dense tensor form more suitable for convolutional network processing. This preserves the spatial structural relationships between joints while avoiding the problem of relationship forgetting in graph neural networks. The designed frequency domain convolution module (FDConv3D) combines spatiotemporal convolution with frequency domain analysis, enabling it to simultaneously capture macroscopic patterns of behavior in the spatiotemporal domain and subtle dynamic changes in the frequency domain, enhancing the model's ability to represent rapid and violent movements. The deep residual network using hierarchical residual modules (SFBlock) as basic units effectively increases network depth while ensuring training stability, allowing the model to learn more complex levels of violent behavior features. Through practical verification, three frames of input were determined as the optimal balance between accuracy and inference speed, making the proposed solution more suitable for real-time or near-real-time monitoring scenarios.

[0090] In addition, the present invention also proposes a computer-readable storage medium storing a behavior recognition program based on skeletal heatmap and frequency domain convolution, which, when executed by a processor, implements the behavior detection method based on skeletal heatmap and frequency domain convolution as described above.

[0091] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the behavior detection method based on skeletal heatmap and frequency domain convolution as described above.

[0092] Figure 6 This is a block diagram of a behavior detection system based on skeletal heatmaps and frequency domain convolution according to an embodiment of the present invention. Figure 6 As shown, the behavior detection system includes: an acquisition module 10, a preprocessing module 20, and a behavior recognition module 60;

[0093] The acquisition module 10 is used to acquire the video sequence to be detected; the preprocessing module 20 is used to extract human skeletal key points from the video sequence to be detected to obtain a human skeletal key point sequence, and to transform the human skeletal key point sequence to obtain the corresponding skeletal heatmap; the behavior recognition module 30 is used to input the skeletal heatmap into a pre-trained behavior recognition model for feature extraction, and to perform behavior recognition based on the extracted features to obtain the recognition result corresponding to the video sequence to be detected. The behavior recognition model includes an input layer, a hierarchical residual module and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain.

[0094] It should be noted that the foregoing explanation of the embodiment of the behavior detection method based on skeletal heatmap and frequency domain convolution also applies to the behavior detection system based on skeletal heatmap and frequency domain convolution in this embodiment, and will not be repeated here.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0100] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0102] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0103] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0104] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0105] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0106] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A behavior detection method based on skeletal heatmap and frequency domain convolution, characterized in that, Includes the following steps: Obtain the video sequence to be detected; Human skeletal key points are extracted from the video sequence to be detected to obtain a human skeletal key point sequence, and the human skeletal key point sequence is transformed to obtain a corresponding skeletal heat map. The skeletal heatmap is input into a pre-trained behavior recognition model for feature extraction, and behavior recognition is performed based on the extracted features to obtain the recognition result corresponding to the video sequence to be detected. The behavior recognition model includes an input layer, a hierarchical residual module and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain.

2. The behavior detection method based on skeleton heatmap and frequency domain convolution as described in claim 1, characterized in that, Human skeletal key points are extracted from the video sequence to be detected to obtain a human skeletal key point sequence, and the human skeletal key point sequence is transformed to obtain a corresponding skeletal heatmap, including: A fixed number of frames are uniformly selected from the video sequence to be detected along the time dimension to obtain a sampled video frame sequence; Human skeleton key points are extracted from the sampled video frame sequence to obtain a human skeleton key point sequence, and the human skeleton key points are spatially normalized to obtain a spatially normalized skeleton key point sequence. The spatially normalized sequence of skeletal key points is transformed to obtain the corresponding skeletal heatmap.

3. The behavior detection method based on skeleton heatmap and frequency domain convolution as described in claim 1, characterized in that, The skeletal heatmap is input into a pre-trained behavior recognition model for feature extraction, including: The input layer performs 3D convolution, normalization, activation, and max pooling on the bone heatmap to obtain an initial feature map; The hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain the enhanced feature tensor.

4. The behavior detection method based on skeleton heatmap and frequency domain convolution as described in claim 3, characterized in that, The hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain an enhanced feature tensor, including: The initial feature map is subjected to channel compression to obtain compressed features; The compressed features are input into the frequency domain convolution module to enhance the globally important parts of the compressed features using an attention modulation mechanism, and then three-dimensional convolution is performed to obtain the global spatiotemporal response features. The compressed features are subjected to a fast Fourier transform to convert them to the frequency domain. The real part is taken and averaged in the time dimension to extract the main frequency response. Then, the original features are modulated by channel projection to obtain the frequency domain modulation features. The global spatiotemporal response features and the frequency domain modulation features are added together to output the fused features; Channel restoration is performed on the fused features, and it is determined whether the number of channels / resolution of the restored features is the same as the number of channels / resolution of the initial feature map; If they are different, the initial feature map is adjusted by bypass convolution and then fused with the channel-restored features to obtain the enhanced feature tensor. If they are the same, the initial feature map is directly fused with the channel-recovered features to obtain the enhanced feature tensor.

5. A computer-readable storage medium, characterized in that, It stores a behavior detection program based on bone heatmap and frequency domain convolution, which, when executed by the processor, implements the behavior detection method based on bone heatmap and frequency domain convolution as described in any one of claims 1-4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the behavior detection method based on skeletal heatmap and frequency domain convolution as described in any one of claims 1-4.

7. A behavior detection system based on skeletal heatmaps and frequency domain convolution, characterized in that, include: The acquisition module is used to acquire the video sequence to be detected; The preprocessing module is used to extract human skeleton key points from the video sequence to be detected to obtain a human skeleton key point sequence, and to convert the human skeleton key point sequence to obtain the corresponding skeleton heatmap. The behavior recognition module is used to input the skeleton heatmap into a pre-trained behavior recognition model for feature extraction, and to perform behavior recognition based on the extracted features to obtain the recognition result corresponding to the video sequence to be detected. The behavior recognition model includes an input layer, a hierarchical residual module and a classifier. The hierarchical residual module embeds a frequency domain convolution module to fuse global features in the spatiotemporal domain and local detail features in the frequency domain.

8. The behavior detection system based on skeletal heatmap and frequency domain convolution as described in claim 7, characterized in that, The preprocessing module is also used for, A fixed number of frames are uniformly selected from the video sequence to be detected along the time dimension to obtain a sampled video frame sequence; Human skeleton key points are extracted from the sampled video frame sequence to obtain a human skeleton key point sequence, and the human skeleton key points are spatially normalized to obtain a spatially normalized skeleton key point sequence. The spatially normalized sequence of skeletal key points is transformed to obtain the corresponding skeletal heatmap.

9. The behavior detection system based on skeletal heatmap and frequency domain convolution as described in claim 7, characterized in that, The behavior recognition module is also used for, The input layer performs 3D convolution, normalization, activation, and max pooling on the bone heatmap to obtain an initial feature map; The hierarchical residual module performs channel compression, frequency domain transformation, channel restoration, and residual fusion on the initial feature map to obtain the enhanced feature tensor.

10. The behavior detection system based on skeletal heatmap and frequency domain convolution as described in claim 9, characterized in that, The behavior detection module is also used for, The initial feature map is subjected to channel compression to obtain compressed features; The compressed features are input into the frequency domain convolution module to enhance the globally important parts of the compressed features using an attention modulation mechanism, and then three-dimensional convolution is performed to obtain the global spatiotemporal response features. The compressed features are subjected to a fast Fourier transform to convert them to the frequency domain. The real part is taken and averaged in the time dimension to extract the main frequency response. Then, the original features are modulated by channel projection to obtain the frequency domain modulation features. The global spatiotemporal response features and the frequency domain modulation features are added together to output the fused features; Channel restoration is performed on the fused features, and it is determined whether the number of channels / resolution of the restored features is the same as the number of channels / resolution of the initial feature map; If they are different, the initial feature map is adjusted by bypass convolution and then fused with the channel-restored features to obtain the enhanced feature tensor. If they are the same, the initial feature map is directly fused with the channel-recovered features to obtain the enhanced feature tensor.

Images