A motion recognition method based on skeleton and image data fusion

Abstract

The application discloses a motion recognition method based on skeleton and image data fusion, comprising: a behavior recognition network model based on skeleton data comprises a coordinate motion information guided sampling module, a multi-scale motion information fusion module and a multi-flow space-time relative Transformer model; a behavior recognition network model based on image data comprises a picture cutting module based on a joint node and a key image block feature extraction model; each action category prediction probability obtained by the behavior recognition network model based on skeleton data and the behavior recognition network model based on image data is fused, and then the final classification prediction probability of the whole model is obtained, so that the motion recognition process for public safety is completed. The recognition network model fully mines the skeleton motion information, establishes dependence for a long-distance joint node, and enhances the recognition ability of detailed actions; and further fuses local image data and skeleton data, so that rich action detail information is supplemented, and high calculation cost is avoided.

Description

Technical Field

[0001] Behavior recognition is a technology that analyzes and determines the types of people's actions using specific algorithms based on data such as video. This technology forms the foundation for many applications, including public safety management, human-computer interaction, smart elderly care, and smart healthcare, and has broad application prospects. Therefore, research on behavior recognition has significant theoretical and practical value. In real-world scenarios, behavior recognition is an extremely challenging task, easily affected by external factors such as lighting, background, and shooting angle. Furthermore, different people perform the same action in different ways, leading to significant inter-class variations. Due to its challenging nature and its involvement in multiple disciplines, behavior recognition has become a research hotspot in the field of computer vision. Background Technology

[0002] Based on the type of input data, deep neural network-based action recognition methods can be divided into image-based action recognition methods and skeleton-based action recognition methods. Image-based action recognition methods analyze RGB image sequences to identify human movements in videos, and are mainly divided into three schools of thought:

[0003] 1) Two-stream network models, represented by the Temporal Segment Network (TSN) model;

[0004] 2) 3D convolutional neural network models, represented by three-dimensional convolutional networks (C3D);

[0005] 3) 2D convolutional neural network models, represented by Temporal Difference Network (TDN).

[0006] In recent years, research based on the above-mentioned approaches has been extensive, and all have achieved advanced performance. The input of the above models is usually an image obtained by scaling and randomly cropping video frames. Although the image size is reduced to some extent, the following drawbacks still exist: 1) Reducing the size will reduce the image accuracy, thereby affecting the model's recognition of subtle movements; 2) Although the image size is reduced, the training data scale is still very large, which requires a lot of video memory and has a long computation time.

[0007] Skeleton-based behavior recognition methods identify human actions by analyzing skeleton sequences. As early as the 1970s, Johansson et al. proved that skeleton data could effectively describe human motion. With the development of human motion estimation techniques, such as advanced human pose estimation algorithms and multimodal sensors, the cost of acquiring skeleton data has decreased. Based on this, researchers have conducted extensive research on skeleton-based behavior recognition methods, mainly divided into three categories: network models based on Recurrent Neural Networks (RNNs), network models based on Convolutional Neural Networks (CNNs), and network models based on Graph Convolutional Networks (GCNs). RNN-based and CNN-based network models treat the skeleton as a sequence or pseudo-image, resulting in the destruction of the skeleton's topological structure. In contrast, GCN-based network models extract skeleton features through graph convolution, maintaining the natural structure of the skeleton, and rapidly improving model performance. In recent years, GCN-based network models have become the mainstream method in the field of skeleton-based behavior recognition. Although GCN-based network models have achieved advanced performance, they also have the following drawbacks: 1) Motion information plays an important role in video classification tasks such as behavior recognition, but existing methods have not fully explored the motion information contained in the skeleton sequence; 2) The receptive field of graph convolutional networks is limited by the size of the convolution kernel, which makes it impossible to establish long-distance connections between joints that are far apart in the skeleton.

[0008] Besides the aforementioned shortcomings, both image data and skeleton data have their own limitations. Image data possesses rich scene and detail information, but it is susceptible to interference from environmental factors such as lighting, and its large size leads to long training times for related models. Skeleton data, on the other hand, describes human movement in a more compact way, with smaller data volume, lower hardware requirements, and is less susceptible to external interference (such as occlusion) compared to image data, exhibiting stronger robustness. While skeleton data has many advantages, it lacks the scene and detail information unique to images, yet these two types of information play a crucial role in behavior recognition, especially when actions are subtle or dependent on the scene. In conclusion, image data and skeleton data are highly complementary, and fusing image-based and skeleton-based behavior recognition network models has greater research significance. Summary of the Invention

[0009] To address the problems of existing technologies, the present invention aims to overcome the shortcomings of existing technologies and provide an action recognition method for public safety. This method is based on a behavior recognition network model using skeleton and image data. According to the difference in data types, the network model is divided into two branches: a skeleton-based behavior recognition network model and an image-based behavior recognition network model. The former extracts skeleton features through a lightweight network, excelling at recognizing large-amplitude movements and playing a major role in action recognition tasks. The latter reduces training costs by cropping images and extracts image features from key image blocks, excelling at recognizing small-amplitude movements concentrated in the hands and feet, and supplementing detailed information in action recognition tasks.

[0010] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0011] A motion recognition method for public safety comprises two networks: one based on skeleton data and the other on image data. The skeleton-based network extracts skeleton features using a lightweight network to identify large-amplitude movements, thus completing the primary motion recognition task. The input data for this model is a skeleton sequence, which is sequentially processed by a coordinate motion information-guided sampling module, a multi-scale motion information fusion module, and a multi-stream spatiotemporal relative Transformer model to obtain motion category prediction probabilities. The image-based network extracts image features from image blocks using image cropping to identify small-amplitude movements concentrated in the hands and feet, supplementing the motion recognition task with detailed information. The input data for this model is an image sequence, which is sequentially processed by a keypoint-based image cropping module and a key block feature extraction (KBN) model to obtain supplementary motion category prediction probabilities. The motion category prediction probabilities obtained from the skeleton-based and image-based networks are then fused to obtain the final classification prediction probability of the entire model, thus completing the motion recognition process for public safety.

[0012] Preferably, in the skeleton-based behavior recognition network model, the frame sampling module guided by coordinate motion information selects representative skeleton sequences from the skeleton sequence based on coordinate motion information metrics; the multi-scale motion information fusion module fuses the static information of the skeleton with multi-scale motion information, and also sets two different types of motion information based on the characteristics of different human movements having different change speeds and durations: fixed motion information and adaptive motion information; wherein, the fixed motion information includes two different scales, enabling the network to adapt to movements with different change speeds; the adaptive motion information enables the recognition network to recognize movements with different durations; the multi-stream spatiotemporal relative Transformer model represents the spatiotemporal domain of each joint. To establish long-distance connections, a multi-stream spatiotemporal relative Transformer model is constructed as follows: In the spatial domain, a skeleton-based spatial topology graph is defined, and a spatial relative Transformer module is constructed to establish long-distance dependencies between key points in the spatial domain. In the temporal domain, a skeleton sequence-based temporal topology graph is constructed, and a temporal relative Transformer module is established to establish long-distance dependencies between key points in the temporal domain. Then, the spatial and temporal relative modules are combined to obtain a spatiotemporal relative Transformer model, and the spatiotemporal features of the skeleton sequence are extracted. A multi-timescale framework is used to fuse different spatiotemporal relative models of at least four input data to construct a multi-stream spatiotemporal relative Transformer model.

[0013] More preferably, the frame sampling module guided by coordinate motion information includes:

[0014] 1.1 Design an index to measure coordinate motion information:

[0015] In skeleton data, joints are represented by 3D coordinates. The displacement distance of a joint between two adjacent frames is used as an indicator of the amount of motion information contained in that joint, and the sum of the displacement distances of all joints in the skeleton is used as an indicator of the amount of motion information contained in the skeleton as a whole, thus determining whether the skeleton is representative. Assume the coordinates of the joint labeled i in frame t are... The coordinates of the keypoint labeled i in frame t-1 are: Then the coordinate motion information M contained in the t-th frame t As shown in formula (1):

[0016]

[0017] Where N represents the number of key points contained in a frame;

[0018] To eliminate the scaling effect caused by different video lengths, the coordinate motion information contained in each frame is normalized, as shown in formula (2):

[0019]

[0020] Where T represents the number of frames in the video;

[0021] 1.2. Sampling the video using the cumulative distribution function:

[0022] Suppose we need to sample N frames from a video of length T, the specific operation is as follows:

[0023] First, the skeleton coordinate motion information is accumulated frame by frame to obtain the cumulative coordinate motion information, C, which is the cumulative coordinate motion information of frame t. t The calculation formula is shown in (3):

[0024]

[0025] in accordance with The sequence is divided into N segments, and a frame is randomly sampled from each of the N segments to form a new sequence. This metric is used to select representative skeleton series from the skeleton sequences.

[0026] More preferably, the multi-scale motion information fusion module includes:

[0027] 2.1 Designing motion information at different scales:

[0028] By sampling from the original backbone sequence I origin =[I 1 , ..., I F Select T frames and combine them in their original order to form a new skeleton sequence I. new =[I 1 , ..., I T F represents the total number of frames in the original skeleton sequence, and I represents the coordinates of all relevant nodes in each frame; motion information is obtained by calculating the coordinates of the same joint. The coordinate displacements between the two frames are obtained as follows: Represents the original skeleton sequence I origin The key point labeled i in frame t is... Represents the skeleton sequence I obtained from sampling new The key point labeled i in the t-th frame;

[0029] Adaptive motion information M a Through skeleton sequence I new The formula is obtained by subtracting the coordinates of two consecutive key points in a video. It has the characteristic of obtaining motion information at different scales from videos of different lengths. The formula is as follows:

[0030]

[0031]

[0032] in, Indicates the new skeleton sequence I new The adaptive motion information of the i-th frame;

[0033] Motion information is divided into two types: short-distance motion information M t and long-distance motion information M i Short-distance motion information M s Through the original skeleton sequence I origin The coordinates of skeleton joints spaced two frames apart are subtracted to obtain the motion information for rapidly changing movements; the calculation formula is as follows:

[0034]

[0035]

[0036] in, f represents the short-range motion information of the i-th frame in the new skeleton sequence, where f is the new skeleton sequence I. new In the i-th frame of the original skeleton sequence I origin The number in; Indicates the original skeleton sequence I origin The key point in frame f is labeled N;

[0037] Long-distance sports information M i Through the original skeleton sequence I origin The coordinates of skeleton joints spaced 5 frames apart are subtracted to obtain the motion information for slow-moving motions. The calculation formula is expressed as follows:

[0038]

[0039]

[0040] in, f represents the long-range motion information of the i-th frame in the new skeleton sequence, and f represents the new skeleton sequence I. new In the t-th frame of the original skeleton sequence I origin The number in;

[0041] 2.2 High-dimensional mapping of motion information at different scales:

[0042] Static information of the skeleton I new Adaptive motion information M a Short-term exercise information M s and long-term exercise information M lThe tensor size is (T, N, C0), where T represents the number of video frames, N represents the number of joints in a skeleton, and C0 represents the coordinate dimension of the joints. As shown in the figure, the above four types of information are mapped to a high-dimensional space through an embedding block to obtain high-dimensional features F1, F2, and F3. ma F ms and F ml The embedding module consists of two convolutional layers and two ReLU activation layers:

[0043] The first convolution maps all types of information to a space of dimension C, and the second convolution maps each type of information to a higher-dimensional space of dimensions C1, C2, C3, and C4 respectively; the convolution kernels corresponding to different motion information are independent of each other and their parameters are not shared; taking static information I... new For example, the formula for the second-order mapping of the embedded module is shown in (10):

[0044] F=σ(W2(σ(W1I new +b1))+b2) (10)

[0045] Where σ represents the activation function, W1 and b1 represent the parameters in the first convolution function, and W2 and b2 represent the parameters in the second convolution function. The parameters in both convolution functions are learned. new Represents static information;

[0046] 2.3 Multi-scale motion information fusion:

[0047] The dynamic representation Z of the skeleton is obtained by fusing various types of information through the stacking operation (concat), as shown in formula (11). This operation enables the dynamic representation Z of the skeleton to contain multi-scale motion information, thereby improving the network's ability to adapt to actions with different speeds of change and different durations.

[0048] Z = concat(F, F ma ,F ms ,F ml (11)

[0049] Z, obtained by fusing four high-dimensional features, is used as the output of the multi-scale motion information fusion module.

[0050] More preferably, the multi-stream spatiotemporal relative Transformer model includes:

[0051] 3.1 Constructing a skeleton-based spatial topology graph:

[0052] In addition to the original joints in the skeleton, this step introduces a virtual node, which together with all the relevant nodes forms a new spatial topology as the model input. The introduced virtual node not only needs to collect and integrate information from each joint, but also plays the role of distributing the integrated global information to each joint. This virtual node is named the spatial relay node.

[0053] Meanwhile, two types of connections are established between the nodes: spatial intrinsic connections and spatial virtual connections, to construct the spatial topology graph of the skeleton; the spatial graph structure containing n nodes has a total of n-1 spatial intrinsic connections.

[0054] 3.2 Design space relative to the Transformer module:

[0055] This module consists of two parts: the Spatial Joint Update Module (SJU) and the Spatial Relay Node Update Module (SRU). By alternately updating the SJU and SRU modules, connections are established between long-distance joints in the spatial domain. The model input is the sequence of joints in the skeleton of frame t. Where N represents the number of key points in this frame. Indicates key points A set of labels for all neighboring nodes; each node has a corresponding query vector. key vector value vector

[0056] In the Spatial Joint Nodes Update Block (SJU), for any given joint... First, the query vector q corresponding to this key point. i t and its neighboring nodes The corresponding key vector Perform a dot product operation to obtain the influence strength of each neighboring node on the joint, as shown in formula (12):

[0057]

[0058] in, This represents the strength of node j's influence on node i; neighboring nodes include their adjacent key points. Space relay node R t and its own r represents the label of the spatial relay node;

[0059] Calculate the influence intensity Then, it is compared with the value vectors corresponding to its neighboring nodes. Multiply the products and sum them up; the resulting value is the key point. The updated value is given by the formula shown in (13):

[0060]

[0061] in, This is the result obtained after one update by the Joint Update Submodule (SJU). This result aggregates both local and global information. k The channel dimension of the key vector serves as a normalization function; softmax. j This indicates that for all adjacent joints The impact strength was normalized.

[0062] To ensure that the space relay node can effectively and comprehensively collect and integrate information from various key points, the Space Relay Node Update Submodule (SRU) also employs dot product operations to calculate the influence of each key point on the relay node; this influence magnitude is used to integrate the information from each key point into global information; influence magnitude Through the query vector corresponding to the relay node key vectors corresponding to each key point Multiplying them together, we get the formula shown in (14):

[0063]

[0064] The update of the spatial relay node is shown in equation (15). Indicates key points For space relay node R t Influence score, This is the value vector for all nodes;

[0065]

[0066] Alternating updates of key points and spatial relay nodes enable the exchange of information between key points, ultimately achieving the goal of each key point simultaneously collecting information from neighboring key points and distant key points.

[0067] 3.3 Constructing a temporal topology graph based on the skeleton sequence:

[0068] When constructing the time topology graph, a time relay node is introduced. Through inherent time connections and virtual time connections, the joints can be connected to each other to form a graph structure in the time domain.

[0069] Along the time dimension, the same key points in consecutive frames form a new sequence. This step also builds connections for the first and last key points, forming a loop structure. A sequence consisting of n key points contains n temporally intrinsic connections.

[0070] 3.4 Design the TRT module:

[0071] The Temporal Relative Transformer (TRT) module comprises two parts: the Temporal Joint Update Module (TJU) and the Temporal Relay Node Update Module (TRU), used to extract temporal features. This module treats each joint in the skeleton as an independent node, extracting the temporal features of that joint from sequences of the same joint within the frame sequence. The input to the TRT module is... A sequence consisting of the same key points across all frames; each key point It has a corresponding query vector key vector and value vector Time relay node R v Corresponding query vector key vector and value vector

[0072] In the TJU submodule, each key point to be updated The system collects information about neighboring nodes through virtual connections and updates itself accordingly; the formula for calculating the influence of neighboring nodes is shown in (16):

[0073]

[0074] in, This indicates the same key point or time relay node R in the j-th frame. v The influence strength on a certain joint point in the i-th frame. Indicates to Perform transposition processing; key points The update is shown in formula (17):

[0075]

[0076] All query vectors Combined into matrix Q v ∈R C×1×t All key vectors Combined into matrix K v ∈R C×B×t All value vectors Combined into matrix V v ∈R C×B×t The matrix form of influence is defined as shown in formula (18):

[0077]

[0078] Where B represents the number of neighboring nodes, and ° represents the Hadema product;

[0079] In the TRU module, the time relay node R v Information is collected from other frames via virtual connections to update the node itself; the specific operation is as follows:

[0080]

[0081]

[0082] in, Represents the key points in frame j. For relay node R v The extent of the impact, This is the scaling factor;

[0083] 3.5. Packaging the ST-RT module:

[0084] The ST-RT module is obtained by connecting and combining the SRT module and the TRT module. The SRT module contains a spatial key point update module and a spatial relay node update module; the TRT module contains a temporal key point update module and a temporal relay node update module; each update module is connected to the feedforward network layer to map the features to a larger-dimensional space to enhance the model's expressive power; L× represents L iterations.

[0085] 3.6 Encapsulating MSST-RT Networks:

[0086] The MSST-RT model is obtained by fusing and encapsulating four different ST-RT models for input data through a multi-stream framework; different sampling frequencies can also provide supplementary information for the model, sampling n1 frames and n2 frames for joint sequences and skeleton sequences respectively; the skeleton data will be processed by the MSST-RT network to obtain the final classification prediction probability based on the skeleton data.

[0087] Preferably, in the image data-based behavior recognition network model, the key point-based image cropping module selects to crop the key points of the human hand and foot; an end-to-end trained image block feature extraction model is adopted, and the end-to-end trained image block feature extraction model is encapsulated into a key image block feature extraction model based on the temporal segmentation network as the basic framework.

[0088] More preferably, the key-based image cropping module includes:

[0089] Image I in frame t t Through matrix P t This indicates the number of joints N that need to be cut. jGiven an image with coordinates (x, y) and a cropped image size of l×l, then in image I... t The middle part surrounds the joints of the hands and feet N j The cropped image block set As shown in the following formula:

[0090]

[0091]

[0092] In addition to cropping the image with the coordinates of the key points as the center, optical flow is also extracted from the corresponding image blocks of two adjacent frames, as shown in formula (23):

[0093]

[0094] TV-L1 is a classic method for calculating optical flow. This represents the optical flow field along the x-axis. This represents the optical flow field along the y-axis.

[0095] More preferably, the key-point-based image cropping module includes: the key image patch-based behavior recognition network includes:

[0096] 5.1 Design the IBCN model:

[0097] Image blocks cropped based on skeleton joints exhibit both independence and correlation. The IBCN model first processes each cropped image block... The features of each image block are input into a convolutional neural network. The calculation formula is shown in (24):

[0098]

[0099] in, This indicates that image blocks are extracted using a convolutional neural network with parameters W. The features are shared by all convolutional neural networks; then the features f of each image block are... t j By concatenating the features, a new feature vector is obtained. As shown in formula (25)

[0100]

[0101] Finally, the eigenvector F is calculated using the dot product method. t any spatial position x i Other positions x j similarity f(x) i ,xj As shown in formula (26):

[0102] f(x i ,x j ) = softmax(θ(x) i ) T ·φ(x j (26)

[0103] Where θ(·) and φ(·) are 1×1 convolution functions;

[0104] The obtained similarity f(x) i ,x j ) will be used as the weight and g(x) j Perform a weighted summation to achieve x i Obtain information from other locations, y i That is, x i The result after global information exchange is shown in formula (27):

[0105]

[0106] Where g(·) is the mapping function, which uses a 1×1 convolution function for mapping; Nl' 2 To select feature maps The size of the input is used as a normalization coefficient to avoid scaling due to different input sizes; when the input is a feature tensor, the formula is as shown in (28):

[0107]

[0108] Where θ(·), φ(·), and g(·) are all 1×1 convolution functions, Nl' 2 These are the normalization coefficients;

[0109] 5.2 Encapsulating the KBN network:

[0110] Using the TSN network as a framework, the IBCN model is encapsulated into a KBN network. This network is divided into spatial and temporal streams, where image patches are the input data for the spatial stream, and optical flow blocks are the input data for the temporal stream. The spatial stream approach first samples several frames from the video through sparse sampling, and then processes each frame using a keypoint-based image cropping module. Finally, a set of key image patches corresponding to each frame is then processed. Each IBCN model is input into a sampled frame, and the class probability is initially predicted based on the sampled frame. The parameters of each IBCN model are shared. Then, the predicted classification results of all sampled frames are fused through a consensus function to obtain the video-level classification prediction. The calculation formula is shown in (29).

[0111]

[0112] Where KBN-S represents the prediction result of the spatial flow of the KBN network, and T K This represents the Kth segment after video segmentation. This represents the set of image blocks corresponding to the Kth sampling frame. This indicates the image block set processed through the IBCN module. The processing method for calculating the temporal flow prediction results is consistent with that for the spatial flow.

[0113] Compared with the prior art, the present invention has the following obvious and prominent substantive features and significant advantages:

[0114] 1. This invention proposes a behavior recognition network model that integrates skeleton data and image data. It fully mines skeleton motion information, establishes dependencies for distant joints, and enhances the ability to recognize detailed movements. Furthermore, it integrates local image data with skeleton data, thereby supplementing rich action detail information while avoiding high computational costs.

[0115] 2. This invention achieved a recognition rate of 98.65% on the NTU60 dataset. It proposes a behavior recognition network model based on skeleton data and image data, and fuses the two models to improve model accuracy.

[0116] 3. This invention establishes information exchange channels between spatial locations through a non-local module, realizing global information exchange between image blocks while taking into account the independence and correlation between image blocks, thereby improving the ability to recognize subtle local human movements; finally, it integrates the behavior recognition network model based on skeleton data with the behavior recognition network model based on image data, giving full play to the complementarity of skeleton data and image data. Attached Figure Description

[0117] Figure 1 This is a schematic diagram of the overall structure of the network model of the method of the present invention.

[0118] Figure 2 This is a graph of the cumulative distribution function of skeleton motion information in the method of the present invention.

[0119] Figure 3 This is a schematic diagram illustrating the calculation of various motion information in the method of the present invention.

[0120] Figure 4 This is a schematic diagram of the skeleton dynamics information representation module of the method of the present invention.

[0121] Figure 5 This is a skeleton-based spatial topology diagram of the method of the present invention.

[0122] Figure 6This is a schematic diagram of the spatial relative Transformer module of the method of the present invention.

[0123] Figure 7 This is a temporal topology diagram based on the skeleton sequence of the method of the present invention.

[0124] Figure 8 This is a schematic diagram of the Temporal Relative Transformer (TRT) module of the method of the present invention.

[0125] Figure 9 This is a schematic diagram of the overall architecture of the ST-RT model of the method of the present invention.

[0126] Figure 10 This is a schematic diagram of the overall architecture of the MSST-RT model of the method of the present invention.

[0127] Figure 11 This is a schematic diagram of image cropping and corresponding optical flow based on the joint position of the present invention.

[0128] Figure 12 This is a schematic diagram of the global image block feature extraction model (IBCN) of the method of the present invention.

[0129] Figure 13 This is a schematic diagram of the Key Image Patch-Based Behavior Recognition Network (KBN) of the present invention. Detailed Implementation

[0130] The above solution will be further described below with reference to specific embodiments. The preferred embodiments of the present invention are described in detail below:

[0131] Example 1:

[0132] In this embodiment, as Figure 1As shown, an action recognition method for public safety establishes two separate action recognition network models: one based on skeleton data and the other on image data. The skeleton-based model uses a lightweight network to extract skeleton features for recognizing large-amplitude actions, thus completing the main action recognition task. The input data for the skeleton-based model is a skeleton sequence, which is sequentially processed by a coordinate motion information-guided sampling module, a multi-scale motion information fusion module, and a multi-stream spatiotemporal relative Transformer model to obtain action category prediction probabilities. The image-based model extracts image features from image blocks using image cropping methods to recognize small-amplitude actions concentrated in the hands and feet, supplementing detailed information in the action recognition task. The input data for the image-based model is an image sequence, which is sequentially processed by a keypoint-based image cropping module and a key block feature extraction (KBN) model to obtain supplementary action category prediction probabilities. The action category prediction probabilities obtained from the skeleton-based and image-based models are fused to obtain the final classification prediction probability of the entire model, thereby completing the action recognition process for public safety.

[0133] Each module will be described in detail below.

[0134] (1) Sampling module guided by coordinate motion information

[0135] The innovation of the frame sampling module guided by coordinate motion information lies in selecting representative skeletons from the skeleton sequence based on coordinate motion information measurement indicators, thereby increasing the motion information contained in the sampling sequence. The specific steps are as follows.

[0136] Step 1.1: Design an index to measure coordinate motion information.

[0137] In skeleton data, joints are typically represented by 3D coordinates. The displacement distance of a joint between two adjacent frames is used as an indicator of the amount of motion information contained in that joint, while the sum of the displacement distances of all joints in the skeleton is used as an indicator of the overall motion information contained in the skeleton, thus determining whether the skeleton is representative. Assume the coordinates of the joint labeled i in frame t are... The coordinates of the keypoint labeled i in frame t-1 are: Then the coordinate motion information M contained in the t-th frame t As shown in formula (1):

[0138]

[0139] Where N represents the number of key points contained in a frame.

[0140] To eliminate the scaling effect caused by different video lengths, the coordinate motion information contained in each frame is normalized, as shown in formula (2):

[0141]

[0142] Where T represents the number of frames contained in the video.

[0143] Step 1.2: Sample the video using the cumulative distribution function.

[0144] Suppose we need to sample N frames from a video of length T. The specific operation is as follows: First, accumulate the skeleton coordinate motion information frame by frame to obtain the accumulated coordinate motion information. The accumulated coordinate motion information C of frame t is... t The calculation formula is shown in (3).

[0145]

[0146] in accordance with Divide the sequence into N segments, such as Figure 2 As shown by the dashed line ( Figure 2 (A total of 10 frames were sampled). Finally, one frame was randomly sampled from each of the above N segments to form a new sequence.

[0147] In summary, this module proposes a metric for measuring skeleton coordinate motion information, and uses this metric to select representative skeletons from the skeleton sequence, thereby increasing the motion information contained in the sampled sequence.

[0148] (2) Multi-scale motion information fusion module

[0149] The innovation of the multi-scale motion information fusion module lies in fusing static skeleton information with multi-scale motion information, enriching the model's input information. Based on the characteristics of different human movements having varying speeds and durations, this module incorporates two different types of motion information: fixed motion information and adaptive motion information. Fixed motion information includes two different scales, enabling the network to adapt to movements with varying speeds; adaptive motion information, on the other hand, allows the network to recognize movements with varying durations. Fusing this multi-scale motion information improves the network's generalization ability, and the specific steps are as follows.

[0150] Step 2.1: Design motion information at different scales

[0151] By sampling from the original backbone sequence I origin =[I 1 ,…,I F Select T frames and combine them in their original order to form a new skeleton sequence I. new =[I 1 ,…,IT ],like Figure 3 As shown, the pink frame is the sampled frame, and I represents the coordinates of all relevant nodes in each frame. Motion information is obtained by calculating the coordinates of the same joint. The coordinate displacements between the two frames are obtained as follows: Represents the original skeleton sequence I origin The key point labeled i in frame t, Jn i t Represents the skeleton sequence I obtained from sampling new The key point labeled i in frame t is...

[0152] Adaptive motion information M a Through skeleton sequence I new The formula is obtained by subtracting the coordinates of two consecutive key points in a video. It has the characteristic of obtaining motion information at different scales from videos of different lengths. The formula is as follows:

[0153]

[0154]

[0155] in, Indicates the new skeleton sequence I new The adaptive motion information of the t-th frame.

[0156] Despite adaptive motion information M a By finding the new skeleton sequence I new It is obtained by the difference between two adjacent frames, but the distance between these two frames depends on their position in I. origin The position in the sequence is closely related to the length of the original skeleton sequence, and each skeleton sequence obtains motion information that matches its length.

[0157] Fixed motion information is divided into two types: short-distance motion information M s and long-distance motion information M l Short-distance motion information M s Through the original skeleton sequence I origin The coordinates of skeleton joints spaced two frames apart are subtracted to obtain the motion information for capturing rapidly changing movements. The calculation formula is as follows:

[0158]

[0159]

[0160] in, f represents the short-range motion information of the t-th frame in the new skeleton sequence, where f is the new skeleton sequence I. new In the t-th frame of the original skeleton sequence I origin The number in the middle.

[0161] Long-distance sports information M l Through the original skeleton sequence I origin The coordinates of skeleton joints spaced 5 frames apart are subtracted to obtain the motion information for slow-moving actions. The calculation formula is expressed as follows:

[0162]

[0163]

[0164] in, f represents the long-range motion information of the t-th frame in the new skeleton sequence, and f represents the new skeleton sequence I. new In the t-th frame of the original skeleton sequence Io rigin The number in the middle.

[0165] Step 2.2: High-dimensional mapping of motion information at different scales

[0166] Static information of the skeleton I new Adaptive motion information M a Short-term exercise information M s and long-term exercise information M l The tensor size is (T, N, C0), where T represents the number of video frames, N represents the number of joints in a skeleton, and C0 represents the coordinate dimension of the joints. As shown in the figure, the above four types of information are mapped to a high-dimensional space through an embedding block to obtain high-dimensional features F1, F2, and F3. ma F ms and F ml The embedding module consists of two convolutional layers and two ReLU activation layers: the first convolution maps all types of information to a space of dimension C, and the second convolution maps each type of information to a higher-dimensional space of dimensions C1, C2, C3, and C4, respectively. The convolutional kernels corresponding to different motion information are independent and their parameters are not shared. Taking static information I... new For example, the formula for the second-order mapping of the embedded module is shown in (10):

[0167] F=σ(W2(σ(W1I new +b1))+b2)# (10)

[0168] Step 2.3: Multi-scale motion information fusion

[0169] The dynamic representation Z of the skeleton is obtained by fusing various types of information through a concat operation, as shown in Equation (11). This operation enables the dynamic representation Z of the skeleton to contain multi-scale motion information, thereby improving the network's ability to adapt to actions with different speeds of change and different durations.

[0170] Z = concat(F, F) ma F ms F ml (11)

[0171] In summary, this module proposes three different scales of motion information: adaptive motion information, short-term motion information, and long-term motion information. Then, an embedding module maps these motion and static information to high-dimensional spaces. Finally, the four high-dimensional features are fused and used as the model input. The method proposed in this section enriches the model input with motion information, and its multi-scale characteristics can improve the generalization of behavior recognition networks.

[0172] (3) Multi-stream spatiotemporal relative Transformer model

[0173] In behavior recognition tasks, many human movements often require the coordination of joints that are relatively far apart. For example, when a person claps, the left and right hands need to work together. The joints of the left and right hands are far apart in the skeleton, but they are strongly correlated in the action. The innovation of the multi-stream spatiotemporal relative Transformer model lies in establishing such long-distance connections between joints in the spatiotemporal domain. The work is as follows: In the spatial domain, a skeleton-based spatial topology graph is designed, and a spatial relative Transformer module is proposed to establish long-distance dependencies between joints in the spatial domain; in the temporal domain, a skeleton sequence-based temporal topology graph is designed, and a temporal relative Transformer module is proposed to establish long-distance dependencies between joints in the temporal domain. Then, the spatial and temporal relative modules are combined to obtain the spatiotemporal relative Transformer model, and then the spatiotemporal features of the skeleton sequence are extracted. Finally, a multi-timescale framework is used to fuse the different spatiotemporal relative models of the four input data to obtain the multi-stream relative spatiotemporal model. The specific steps are as follows.

[0174] Step 3.1: Construct a skeleton-based spatial topology graph

[0175] In addition to the existing joints in the skeleton, this step introduces a virtual node, which, together with all the relevant nodes, forms a new spatial topology as the model input. For example... Figure 5 As shown, the blue nodes are the original related nodes, and the purple nodes are the introduced virtual nodes. The introduced virtual nodes not only need to collect and integrate information from various key points, but also undertake the role of distributing the integrated global information to each key point. These virtual nodes are named spatial relay nodes.

[0176] Simultaneously, this step establishes two types of connections between nodes (including key nodes and spatial relay nodes): inherent spatial connections and virtual spatial connections, to construct a spatial topology graph of the skeleton. For example... Figure 5As shown, by establishing spatial intrinsic connections (blue line segments) for all pairs of joints directly connected by bones in the human skeleton, the original graph topology of the skeleton is maintained. Spatial intrinsic connections contain a wealth of prior knowledge, serving to collect local information from adjacent joints. Furthermore, the existence of these connections allows joints to obtain richer information from their neighbors than from distant joints. A spatial graph structure containing n joints has n-1 spatial intrinsic connections.

[0177] Step 3.2: Design the space relative to the Transformer module

[0178] The Spatial Relative Transformer module is essentially a spatial feature extraction algorithm based on Transformer, such as... Figure 6 As shown, this module comprises two parts: a Spatial Joint Update Module (SJU) and a Spatial Relay Node Update Module (SRU). By alternately updating the SJU and SRU modules, it aims to establish connections between long-distance joints in the spatial domain. Since this module independently updates the joints and spatial relay nodes in each frame, this step will describe the model algorithm using a single frame as an example. The model input is the sequence of joints in the skeleton of frame t. Where N represents the number of key points in this frame. Indicates key points A collection of all neighboring node labels. Each node (including nodes) and space relay node R t Each of these has a corresponding query vector. key vector value vector

[0179] In the Spatial JointNodes Update Block (SJU), for any given joint... First, the query vector q corresponding to this key point. i t and its neighboring nodes The corresponding key vector Perform a dot product operation to obtain the influence strength of each neighboring node on the key point, as shown in formula (3.12):

[0180]

[0181] in, This represents the strength of node j's influence on node i. Neighboring nodes include their adjacent key points. Space relay node R t and its own

[0182] Calculate the influence intensity Then, it is compared with the value vectors corresponding to its neighboring nodes. Multiply the products and sum them up; the resulting value is the key point. The updated value is given by the formula shown in (13):

[0183]

[0184] in, This is the result obtained after one update by the Joint Update Submodule (SJU). This result aggregates both local and global information. k This represents the channel dimension of the key vector and serves a normalization function. For example... Figure 6 As shown in the SJU module, the red nodes are the nodes to be updated, and they collect information from neighboring nodes through orange connections.

[0185] To ensure that the space relay node can effectively and fully collect and integrate information from various key points, the space relay node update submodule (SRU) also employs dot product operations to calculate the influence of each key point on this relay node. For example... Figure 6 As shown in the SRU module, the spatial relay node to be updated (red node) collects information from each node through orange connections, and integrates the information of each node into global information through influence. Influence Through the query vector corresponding to the relay node key vectors corresponding to each key point Multiplying them together, we get the formula shown in (14):

[0186]

[0187] The update of the spatial relay node is shown in equation (15). Indicates key points For space relay node R t Influence score, This is the value vector for all nodes (including all joints in the skeleton and spatial relay nodes).

[0188]

[0189] Alternating updates of keypoints and spatial relay nodes enable the exchange of information between keypoints, ultimately achieving the goal of each keypoint simultaneously collecting information from neighboring keypoints and distant keypoints. The overall update algorithm of the SRT module is Algorithm 1, as shown in Table 1, where the first layer iterates through all frames, and the second layer iterates through all keypoints (including spatial relay nodes) in that frame.

[0190] Table 1: Algorithm 1: SRT Module Update Algorithm Description

[0191]

[0192]

[0193] Step 3.3: Construct a temporal topology graph based on the skeleton sequence.

[0194] This step introduces a time relay node when constructing the time topology graph. Through inherent time connections and virtual time connections, the joints can be connected to each other to form a graph structure in the time domain.

[0195] Along the time dimension, the same key points in consecutive frames form a new sequence. This step also builds connections between the first and last key points, forming a loop structure, such as... Figure 7 As shown. These connections, named temporally intrinsic connections (blue lines) to preserve the frame order, serve to directly exchange information with adjacent frames. A sequence consisting of n keypoints contains n temporally intrinsic connections.

[0196] Similar to the construction in step 3.1, temporal virtual connections (purple line segments) connect the temporal relay nodes (purple nodes) and the key points (blue nodes) in the sequence. These key points exchange information over long distances through these connections. Therefore, a graph containing n key points has n temporal virtual connections, such as... Figure 7 As shown.

[0197] Step 3.4: Design the TRT module

[0198] The Temporal Relative Transformer (TRT) module comprises two parts: the Temporal Joint Update Module (TJU) and the Temporal Relay Node Update Module (TRU), used for extracting temporal features. This module treats each joint in the skeleton as an independent node and extracts the temporal features of that joint from sequences of frames containing the same joint. This step will describe the algorithm using a single joint as an example. The input to the TRT module is... A sequence consisting of the same key points across all frames. Each key point It has a corresponding query vector key vector and value vector Time relay node R v Corresponding query vector key vector and value vector

[0199] In the TJU submodule, each key point to be updated (Red node) collects neighbor nodes (time relay node R) through virtual connections (orange line segments). v The same key point in adjacent frames and the node itself The information is updated by itself, such as Figure 8 The TJU module is shown in the figure. The formula for calculating the influence of neighboring nodes is shown in (16):

[0200]

[0201] in, This indicates the same key point or time relay node R in the j-th frame. v The strength of the influence on a certain joint point in the i-th frame. Joint point The update is shown in formula (17):

[0202]

[0203] All query vectors Combined into matrix Q v ∈R C×1×t All key vectors Combined into matrix K v ∈R C×B×t All value vectors Combined into matrix V v ∈R C×B×t The matrix form of influence is defined as shown in formula (18):

[0204]

[0205] Where B represents the number of neighboring nodes, and ° represents the Hadema product.

[0206] In the TRU module, such as Figure 8 As shown, time relay node R v (Red node) collects information from other frames through virtual connections (orange line segments) to update its own node. The specific operation is as follows:

[0207]

[0208]

[0209] in, Represents the key points in frame j. For relay node R v The extent of the impact, This is the scaling factor.

[0210] By alternately updating the time relay node and the same node in all frames, the TRT module ultimately captures the long-distance dependencies between frames. The overall update algorithm of the TRT module is Algorithm 2, as shown in Table 2. The first layer iterates through all joints in the skeleton, and the second layer iterates through the joints corresponding to that joint in all frames (including the time relay node).

[0211] Table 2: Algorithm 2: TRT Module Update Algorithm Description

[0212]

[0213] Step 3.5: Package the ST-RT module:

[0214] The ST-RT module is obtained by connecting and combining the SRT module and the TRT module, such as Figure 9 As shown, the SRT module contains a Spatial Joint Nodes Update Block (SJU) and a Spatial Relay Node Update Block (SRU). The TRT module contains a Temporal Joint Nodes Update Block (TJU) and a Temporal Relay Node Update Block (TRU). Each update block is backward connected to a Feedforward Neural Network (FNN) layer to map features to a larger-dimensional space, thereby enhancing the model's expressive power. L× represents L iterations.

[0215] Step 3.6: Encapsulate the MSST-RT network

[0216] To further improve model accuracy, this step uses a multi-stream framework to fuse and encapsulate the four different ST-RT models for the input data to obtain an MSST-RT model (Multi-stream ST-RT), such as... Figure 10 As shown. Besides extracting features using first-order information (joints), features can also be extracted using second-order information (bones). Furthermore, different sampling frequencies can provide supplementary information to the model, such as sampling n1 frames and n2 frames for the joint sequence and bone sequence respectively. The skeleton data is then processed by the MSST-RT network to obtain the final classification prediction probability based on the skeleton data.

[0217] In summary, the MSST-RT model improves the Transformer model based on the characteristics of the skeleton graph and the sequence, establishing dependencies for distant joints with less computational cost, while maintaining the integrity of the skeleton structure and sequence order, thereby improving computational efficiency and recognition accuracy.

[0218] (4) Image cropping module based on joints

[0219] Since most subtle human movements are concentrated in the hands or feet, the corresponding image blocks contain a large portion of the missing details of the skeleton. Therefore, the innovation of this module lies in cropping the joints of the human hands and feet, significantly reducing training costs. Figure 11 As shown.

[0220] Specifically, the image I of frame t t Through matrix P t This indicates the number of joints N that need to be cut. j Given an image with coordinates (x, y) and a cropped image size of l×l, then in image I... t The middle part surrounds the joints of the hands and feet N j The cropped image block set As shown in the following formula:

[0221]

[0222]

[0223] In addition to cropping the image based on the coordinates of the key points, this section also extracts optical flow by using the corresponding image blocks of two adjacent frames, as shown in formula (23):

[0224]

[0225] TV-L1 is a classic method for calculating optical flow. This represents the optical flow field along the x-axis. This represents the optical flow field along the y-axis.

[0226] (5) Action recognition network based on key image patches (KBN)

[0227] To extract features from each key image patch obtained through cropping, this embodiment designs an end-to-end trained Image Blocks Convolution Network (IBCN) model, and encapsulates the IBCN model into a KBN network based on the Temporal Segment Network (TSN) framework. The specific steps are as follows.

[0228] Step 5.1: Design the IBCN model

[0229] Image blocks cropped based on skeleton joints exhibit both independence and correlation; therefore, as Figure 12 As shown, the IBCN model first processes each cropped image block... The features of each image block are then fed into a convolutional neural network (CNN) to obtain the features of each block. The calculation formula is shown in (24):

[0230]

[0231] in, This indicates that image blocks are extracted using a convolutional neural network with parameters W. The characteristics of this feature are that the parameters are shared by each convolutional neural network.

[0232] Then the features f of each image block t j By concatenating the features, a new feature vector is obtained. As shown in formula (25)

[0233]

[0234] Finally, the eigenvector F is calculated using the dot product method. t any spatial position x i Other positions x j similarity f(x) i ,x j As shown in formula (26):

[0235] f(x i x j ) = softmax(θ(x) i ) T ·φ(x j ))# (26)

[0236] Where θ(·) and φ(·) are 1×1 convolution functions.

[0237] The obtained similarity f(x) i ,x j ) will be used as the weight and g(x) j Perform a weighted summation to achieve x i Obtain information from other locations, y i That is, x i The result after global information exchange is shown in formula (27):

[0238]

[0239] Where g(·) is the mapping function, and this section uses a 1×1 convolution function for mapping. Nl' 2 To select feature maps Using the size of the input as a normalization coefficient can avoid scaling caused by different input sizes. When the input is a feature tensor, the formula is as shown in (28):

[0240]

[0241] Where θ(·), φ(·), and g(·) are all 1×1 convolution functions, Nl' 2 This is the normalization coefficient.

[0242] Step 5.2: Encapsulate the KBN network

[0243] This step uses the TSN network as a framework, encapsulating the IBCN model into a KBN network. This network is divided into spatial and temporal streams, where image patches are the input data for the spatial stream, and optical flow blocks are the input data for the temporal stream. Taking the spatial stream as an example, several frames are first sampled from the video through sparse sampling, and each frame is processed by a keypoint-based image cropping module. Then, the set of key image patches corresponding to each frame is... Each IBCN model is input into a sampled frame, and the class probability is initially predicted based on the sampled frame. The parameters of each IBCN model are shared. Then, the prediction classification results of all sampled frames are fused through a consensus function to obtain the video-level classification prediction. The calculation formula is shown in (29).

[0244]

[0245] Among them, KBN-S is the prediction result of the spatial flow of the KBN network, and the calculation method of the temporal flow prediction result is the same as that of the spatial flow.

[0246] Finally, the spatial flow prediction results are fused with the temporal flow prediction results to obtain the final classification prediction probability based on image data.

[0247] Behavior recognition, a popular research area in computer vision, has broad application prospects in public safety and human-computer interaction, and is of significant research importance. Behavior recognition methods are mainly divided into two categories: those based on skeleton data and those based on image data. This embodiment proposes a behavior recognition network model that integrates skeleton data and image data. It fully mines skeleton motion information, establishes dependencies for distant keypoints, and enhances the ability to recognize detailed actions. Furthermore, it integrates local image data with skeleton data, thereby supplementing rich action detail information while avoiding high computational costs. This embodiment achieves a recognition level of 98.65% on the NTU60 dataset. This embodiment proposes behavior recognition network models based on skeleton data and image data, and fuses the two network models to form a complete system.

[0248] This embodiment addresses the problem that existing skeleton behavior recognition methods do not fully exploit skeleton motion information. It proposes a motion information-guided sampling module and a multi-scale motion information fusion module. In the motion information-guided sampling module, the sum of the coordinate displacements of each joint point in two adjacent frames is proposed as an indicator to measure the skeleton's coordinate motion information. This indicator guides sampling, resulting in a skeleton with richer motion information, thereby improving recognition accuracy. In the multi-scale motion information fusion module, fixed motion information and adaptive motion information are proposed and fused with static information. This gives the model input rich motion information, enhancing the model's adaptability to actions with different speeds and durations, and improving model accuracy.

[0249] This embodiment addresses the problem that graph convolutional networks cannot establish long-range dependencies for distant joints in a skeleton. It proposes a Transformer-based skeleton behavior recognition network, MSST-RT. This network model introduces a virtual node in both the spatiotemporal domains. This virtual node establishes direct connections (virtual connections) with each joint, collecting and integrating joint information to achieve autonomous updates. Each joint then obtains local information from neighboring nodes through its skeleton (intrinsic connections) and global information from the virtual node through the virtual connections, completing its own updates. Through these two updates, each joint exchanges information with any other joint, establishing long-range dependencies and extracting spatiotemporal features.

[0250] This embodiment addresses the problem of high training costs for related models despite the lack of detailed information in image data, which often contains skeleton data. It proposes a keypoint-based image cropping module and a key image patch-based behavior recognition network (KBN). In the keypoint-based image cropping module, to reduce image data size and training costs, the module crops the positions of a person's hands and feet in the image based on keypoint coordinates, resulting in several image blocks. This set of image blocks is used to replace the original image for feature extraction. In the KBN model, this embodiment establishes information exchange channels between spatial locations through a non-local module, enabling global information exchange between image blocks while maintaining both independence and correlation among them, thereby improving the ability to recognize subtle local human movements. Finally, the skeleton data-based behavior recognition network model and the image data-based behavior recognition network model are fused to fully leverage the complementarity of skeleton and image data.

[0251] Example 2:

[0252] This embodiment is basically the same as Embodiment 1, except that:

[0253] In this embodiment, the action recognition method for public safety will be tested using skeleton data and image data from the NTU60 dataset. The dataset is divided into training and test sets according to the C-Subject rule, and the model performance is measured by Top1 accuracy.

[0254] (1) Behavior recognition network model based on skeleton data

[0255] In the multi-scale motion information fusion module, static information, adaptive motion information, short-term motion information, and long-term motion information are mapped from a 3-dimensional space to a 64-dimensional space through the first 1×1 convolution, and then mapped from the 64-dimensional space to high-dimensional spaces of 256, 256, 128, and 128, respectively, through the second 1×1 convolution.

[0256] In the MSST-RT model, the SRT and TRT modules were run 3 times each, the multi-head attention mechanism had 8 heads, and batch normalization was used. All experiments were performed using the PyTorch framework, and the model was trained using the Adam optimizer with parameters β = [0.9, 0.98] and ∈ = 10. -9 Training is divided into two phases: 1) In the first phase (the first 700 iterations), the learning rate is increased from 4×10⁻⁶ to 10⁻⁶ using a warm-start method. -7 Linearity increased to 5×10 -42) In the second stage, the learning rate is gradually reduced using a natural exponential decay strategy with a weight size of 0.9996. This training method not only accelerates model convergence but also makes training more stable. During training, the batch size is set to 64, and the number of training iterations is 30. Simultaneously, all experiments use ∈ ls Label smoothing strategy with a value of 0.1.

[0257] In terms of data processing, the coordinate displacements of the joints in each frame relative to the same joints in the first frame are used to describe the skeleton of each frame, replacing the original coordinate information of each joint. Some actions in the training set are two-person interactive actions, meaning that the same frame contains two skeletons, such as hugging or shaking hands. In this case, the frame containing two skeletons is split into two frames, so that each frame contains one skeleton. Additionally, random rotation of the 3D skeleton is used to obtain more diverse samples to achieve data augmentation, which enhances the network's generalization ability to some extent.

[0258] (2) Behavior recognition network model based on image data

[0259] All experiments in this chapter are conducted using the PyTorch framework, employing stochastic gradient descent with a momentum value of 0.9 to learn the network parameters. In the spatial flow training of the KBN network, the batch size is set to 24, the number of training iterations is set to 80, and the initial learning rate is set to 0.001. The learning rate is updated at the 25th, 45th, and 70th training iterations, decreasing by half each time. A pre-trained model on the ImageNet dataset is used to initialize the network parameters. In the temporal flow training of the KBN network, the batch size is set to 24, the number of training iterations is set to 300, and the initial learning rate is set to 0.001. The learning rate is updated at the 50th, 100th, 150th, and 200th training iterations, decreasing by half each time. Gradient clipping is performed when the gradient value exceeds 20 during training to effectively prevent gradient explosion. To accelerate model convergence, the spatial flow model parameters of the KBN network are used to initialize the temporal flow network. The experiment used the TV-L1 algorithm provided by the CUDA version of OpenCV to extract the optical flow of image blocks.

[0260] Table 3. Performance of each method on the NTU60 dataset

[0261]

[0262] This embodiment describes a motion recognition method for public safety. Based on skeleton and image data, the motion recognition network model is divided into two branches according to the data type: a skeleton-based motion recognition network model and an image-based motion recognition network model. The former extracts skeleton features through a lightweight network and is good at recognizing large-amplitude movements, playing a major role in motion recognition tasks. The latter reduces training costs by cropping images and extracts image features from key image blocks, making it good at recognizing small-amplitude movements concentrated in the hands and feet, and playing a role in supplementing detailed information in motion recognition tasks.

[0263] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. An action recognition method based on skeleton and image data fusion, characterized in that: A recognition network is constructed by establishing two separate action recognition network models: one based on skeleton data and the other based on image data. The skeleton-based model uses a lightweight network to extract skeleton features to identify large-amplitude movements, thus completing the main action recognition task. The input data for the skeleton-based model is a skeleton sequence, which is sequentially processed by a frame sampling module guided by coordinate motion information, a multi-scale motion information fusion module, and a multi-stream spatiotemporal relative Transformer model to obtain the action category prediction probability. The image-based model extracts image features from image blocks through image cropping to identify small-amplitude movements concentrated in the hands and feet, supplementing the action recognition task with detailed information. The input data for the image-based model is an image sequence, which is sequentially processed by a keypoint-based image cropping module and a key image block feature extraction model to obtain supplementary action category prediction probabilities. The predicted probabilities of each action category obtained by the action recognition network model based on skeleton data and the action recognition network model based on image data are fused together to obtain the final classification prediction probability of the entire model, thereby completing the action recognition process for public safety. In the skeleton-based behavior recognition network model, the frame sampling module guided by coordinate motion information selects representative skeleton sequences based on coordinate motion information metrics. The multi-scale motion information fusion module fuses the static information of the skeleton with multi-scale motion information. Furthermore, considering the different speeds and durations of human movements, it sets two different types of motion information: fixed motion information and adaptive motion information. Fixed motion information includes two different scales, enabling the network to adapt to movements with varying speeds. Adaptive motion information enables the recognition network to recognize movements with different durations. The multi-stream spatiotemporal relative Transformer model establishes long-distance coordinates for each joint in the spatiotemporal domain. The multi-stream spatiotemporal relative Transformer model is as follows: In the spatial domain, a skeleton-based spatial topology graph is defined, and a multi-stream spatiotemporal relative Transformer module is constructed to establish long-distance dependencies of key points in the spatial domain; in the temporal domain, a skeleton sequence-based temporal topology graph is constructed, and a multi-stream spatiotemporal relative Transformer module is established to establish long-distance dependencies of key points in the temporal domain; then, the spatial and temporal relative modules are combined to obtain the multi-stream spatiotemporal relative Transformer model, and the spatiotemporal features of the skeleton sequence are extracted; a multi-timescale framework is used to fuse different spatiotemporal relative models of at least four input data to construct the multi-stream spatiotemporal relative Transformer model.

2. The action recognition method based on skeleton and image data fusion according to claim 1, characterized in that: The frame sampling module guided by coordinate motion information includes: 1.1 Design an index to measure coordinate motion information: In skeleton data, joints are represented by 3D coordinates. The displacement distance of a joint between two adjacent frames is used as an indicator of the amount of motion information contained in that joint, and the sum of the displacement distances of all joints in the skeleton is used as an indicator of the amount of motion information contained in the skeleton as a whole, thus determining whether the skeleton is representative. Assume the coordinates of the joint labeled i in frame t are... The coordinates of the keypoint labeled i in frame t-1 are: Then the coordinate motion information M contained in the t-th frame t As shown in formula (1): Where N represents the number of key points contained in a frame; To eliminate the scaling effect caused by different video lengths, the coordinate motion information contained in each frame is normalized, as shown in formula (2): Where T represents the number of frames in the video; 1.

2. Sampling the video using the cumulative distribution function: Suppose we need to sample N frames from a video of length T, the specific operation is as follows: First, the skeleton coordinate motion information is accumulated frame by frame to obtain the cumulative coordinate motion information, C, which is the cumulative coordinate motion information of frame t. t The calculation formula is shown in (3): in accordance with The sequence is divided into N segments, and a frame is randomly sampled from each of the N segments to form a new sequence. This metric is used to select representative skeleton series from the skeleton sequences.

3. The action recognition method based on skeleton and image data fusion according to claim 1, characterized in that: The multi-scale motion information fusion module includes: 2.1 Designing motion information at different scales: By sampling from the original backbone sequence I origin =[I 1 , ..., I F Select T frames and combine them in their original order to form a new skeleton sequence I. new =[I 1 , ..., I T F represents the total number of frames in the original skeleton sequence, and I represents the coordinates of all relevant nodes in each frame; motion information is obtained by calculating the coordinates of the same joint. The coordinate displacements between the two frames are obtained as follows: Represents the original skeleton sequence I origin The key point labeled i in frame t is... Represents the skeleton sequence I obtained from sampling new The key point labeled i in the t-th frame; Adaptive motion information M a Through skeleton sequence I new The formula is obtained by subtracting the coordinates of two consecutive key points in a video. It has the characteristic of obtaining motion information at different scales from videos of different lengths. The formula is as follows: in, Indicates the new skeleton sequence I new The adaptive motion information of the i-th frame; Motion information is divided into two types: short-distance motion information M s and long-distance motion information M i Short-distance motion information M s Through the original skeleton sequence I origin The coordinates of skeleton joints spaced two frames apart are subtracted to obtain the motion information for rapidly changing movements; the calculation formula is as follows: in, f represents the short-range motion information of the i-th frame in the new skeleton sequence, where f is the new skeleton sequence I. new In the i-th frame of the original skeleton sequence I origin The number in; Friend shows the original skeleton sequence I origin The key point in the f-th frame is labeled N; Long-distance sports information M i Through the original skeleton sequence I origin The coordinates of skeleton joints spaced 5 frames apart are subtracted to obtain the motion information for slow-moving motions. The calculation formula is expressed as follows: in, f represents the long-range motion information of the i-th frame in the new skeleton sequence, and f represents the new skeleton sequence I. new In the t-th frame of the original skeleton sequence I origin The number in; 2.2 High-dimensional mapping of motion information at different scales: Static information of the skeleton I new Adaptive motion information M a Short-term exercise information M s and long-term exercise information M l The tensor size is (T, N, C0), where T represents the number of video frames, N represents the number of joints in a skeleton, and C0 represents the coordinate dimension of the joints. These four types of information are mapped to a high-dimensional space through an embedding module to obtain high-dimensional features F1, F2, and F3. ma F ms and F ml The embedding module consists of two convolutional layers and two activation layers: The first convolution maps all types of information to a space of dimension C, and the second convolution maps all types of information to high-dimensional spaces of dimensions C1, C2, C3, and C4 respectively; the convolution kernels corresponding to different motion information are independent of each other and their parameters are not shared; taking static information I as an example... new For example, the formula for the second-order mapping of the embedded module is shown in (10): F=σ(W2(σ(W1I new +b1))+b2) (10) Where σ represents the activation function, W1 and b1 represent the parameters in the first convolution function, and W2 and b2 represent the parameters in the second convolution function. The parameters in both convolution functions are learned. new Represents static information; 2.3 Multi-scale motion information fusion: The dynamic representation Z of the skeleton is obtained by fusing various types of information through stacking operations, as shown in formula (11). This operation enables the dynamic representation Z of the skeleton to contain multi-scale motion information, thereby improving the network's ability to adapt to actions with different speeds of change and different durations. Z=concat(F,F ma ,F ms ,F ml )(11) Z, obtained by fusing four high-dimensional features, is used as the output of the multi-scale motion information fusion module.

4. The action recognition method based on skeleton and image data fusion according to claim 1, characterized in that: Multi-stream spatiotemporal relative Transformer models include: 3.1 Constructing a skeleton-based spatial topology graph: In addition to the original joints in the skeleton, this step introduces a virtual node, which together with all the relevant nodes forms a new spatial topology as the model input. The introduced virtual node not only needs to collect and integrate information from each joint, but also plays the role of distributing the integrated global information to each joint. This virtual node is named the spatial relay node. Meanwhile, two types of connections are established between the nodes: spatial intrinsic connections and spatial virtual connections, to construct the spatial topology graph of the skeleton; the spatial graph structure containing n nodes has a total of n-1 spatial intrinsic connections. 3.2 Design of the Multi-Stream Spatiotemporal Relative Transformer Module (SRT): This module consists of two parts: the Spatial Joint Update Module (SJU) and the Spatial Relay Node Update Module (SRU). By alternately updating the SJU and SRU modules, connections are established between long-distance joints in the spatial domain. The model input is the sequence of joints in the skeleton of frame t. Where N represents the number of key points in this frame. Indicates key points A set of labels for all neighboring nodes; each node has a corresponding query vector. key vector value vector In the spatial joint update module, for any joint... First, the query vector corresponding to this key point. and its neighboring nodes The corresponding key vector Perform a dot product operation to obtain the influence strength of each neighboring node on the joint, as shown in formula (12): in, This represents the strength of node j's influence on node i; neighboring nodes include their adjacent key points. Space relay node R t and its own r represents the label of the spatial relay node; Calculate the influence intensity Then, it is compared with the value vectors corresponding to its neighboring nodes. Multiply the products and sum them up; the resulting value is the key point. The updated value is given by the formula shown in (13): in, This is the result obtained after one update by the key update submodule. This result aggregates both local and global information. k The channel dimension of the key vector serves as a normalization function; softmax. j This indicates that for all adjacent joints The impact strength was normalized. To ensure that the spatial relay node effectively and comprehensively collects and integrates information from various key points, the spatial relay node update submodule also employs dot product operations to calculate the influence of each key point on the relay node; this influence magnitude is used to integrate the information from each key point into global information; influence magnitude Through the query vector corresponding to the relay node key vectors corresponding to each key point Multiplying them together, we get the formula shown in (14): The update of the spatial relay node is shown in equation (15). Indicates key points For space relay node R t Influence score, This is the value vector for all nodes; Alternating updates of key points and spatial relay nodes enable the exchange of information between key points, ultimately achieving the goal of each key point simultaneously collecting information from neighboring key points and distant key points. 3.3 Constructing a temporal topology graph based on the skeleton sequence: When constructing the time topology graph, a time relay node is introduced. Through inherent time connections and virtual time connections, the joints can be connected to each other to form a graph structure in the time domain. Along the time dimension, the same key points in consecutive frames form a new sequence. This step also builds connections for the first and last key points, forming a loop structure. A sequence consisting of n key points contains n temporally intrinsic connections. 3.4 Design the TRT module: The Temporal Relative Transformer (TRT) module comprises two parts: the Temporal Joint Update Module (TJU) and the Temporal Relay Node Update Module (TRU), used to extract temporal features. This module treats each joint in the skeleton as an independent node and extracts the temporal features of that joint from sequences of the same joint within the frame sequence. The input to the TRT module is... A sequence consisting of the same key points across all frames; each key point It has a corresponding query vector key vector and value vector Time relay node R v Corresponding query vector key vector and value vector In the TJU submodule, each key point to be updated The system collects information about neighboring nodes through virtual connections and updates itself accordingly; the formula for calculating the influence of neighboring nodes is shown in (16): in, This indicates the same key point or time relay node R in the j-th frame. v The influence strength on a certain joint point in the i-th frame. Indicates to Perform transposition processing; key points The update is shown in formula (17): All query vectors Combined into matrix Q v ∈R C×1×t All key vectors Combined into matrix K v ∈R C×B×t All value vectors Combined into matrix V v ∈R C×B×t The matrix form of influence is defined as shown in formula (18): Where B represents the number of neighboring nodes, Represents the Hadema product; In the TRU module, the time relay node R v Information is collected from other frames via virtual connections to update the node itself; the specific operation is as follows: in, Represents the key points in frame j. For relay node R v The extent of the impact This is the scaling factor; 3.

5. Packaging the ST-RT module: The ST-RT module is obtained by connecting and combining the SRT module and the TRT module. The SRT module contains a spatial key point update module and a spatial relay node update module; the TRT module contains a temporal key point update module and a temporal relay node update module; each update module is connected to the feedforward network layer to map the features to a larger-dimensional space to enhance the model's expressive power; L× represents L iterations. 3.6 Encapsulating MSST-RT Networks: The MSST-RT model is obtained by fusing and encapsulating four different ST-RT models for input data through a multi-stream framework; different sampling frequencies can also provide supplementary information for the model, sampling n1 frames and n2 frames for joint sequences and skeleton sequences respectively; the skeleton data will be processed by the MSST-RT network to obtain the final classification prediction probability based on the skeleton data.

5. The action recognition method based on skeleton and image data fusion according to claim 1, characterized in that: In the image data-based behavior recognition network model, the key point-based image cropping module selects to crop the key points of the human hand and foot; an end-to-end trained image block feature extraction model is adopted, and the end-to-end trained image block feature extraction model is encapsulated into a key image block feature extraction model based on the temporal segmentation network as the basic framework.

6. The action recognition method based on skeleton and image data fusion according to claim 5, characterized in that: The key-based image cropping module includes: Image I in frame t t Through matrix P t This indicates the number of joints N that need to be cut. j Given an image with coordinates (x, y) and a cropped image size of l×l, then in image I... t The middle part surrounds the joints of the hands and feet N j The cropped image block set As shown in the following formula: In addition to cropping the image with the coordinates of the key points as the center, optical flow is also extracted from the corresponding image blocks of two adjacent frames, as shown in formula (23): TV-L1 is a classic method for calculating optical flow. This represents the optical flow field along the x-axis. This represents the optical flow field along the y-axis.

7. The action recognition method based on skeleton and image data fusion according to claim 5, characterized in that: The key-point-based image cropping module includes: The key image patch-based behavior recognition network KBN includes: 5.1 Design the IBCN model: Image blocks cropped based on skeleton joints exhibit both independence and correlation. The image block feature extraction model, namely the IBCN model, first extracts each cropped image block... The features of each image block are input into a convolutional neural network. The calculation formula is shown in (24): in, This indicates that image blocks are extracted using a convolutional neural network with parameters W. The features are shared by all convolutional neural networks; then the features f of each image block are... t j By concatenating the features, a new feature vector is obtained. As shown in formula (25) Finally, the eigenvector F is calculated using the dot product method. t any spatial position x i Other positions x j similarity f(x) i ,x j As shown in formula (26): f(x i ,x j )=softmax(θ(x i ) T ·φ(x j )) (26) Where θ(·) and φ(·) are 1×1 convolution functions; The obtained similarity f(x) i ,x j ) will be used as the weight and g(x) j Perform a weighted summation to achieve x i Obtain information from other locations, y i That is, x i The result after global information exchange is shown in formula (27): Where g(·) is the mapping function, which uses a 1×1 convolution function for mapping; Nl' 2 To select feature maps The size of the input is used as a normalization coefficient to avoid scaling due to different input sizes; when the input is a feature tensor, the formula is as shown in (28): Where θ(·), φ(·), and g(·) are all 1×1 convolution functions, Nl' 2 These are the normalization coefficients; 5.2 Encapsulating the KBN network: Using the TSN network as a framework, the IBCN model is encapsulated into a KBN network. This network is divided into spatial and temporal streams, where image patches are the input data for the spatial stream, and optical flow blocks are the input data for the temporal stream. The spatial stream approach first samples several frames from the video through sparse sampling, and then processes each frame using a keypoint-based image cropping module. Finally, a set of key image patches corresponding to each frame is then processed. Each IBCN model is input into a sampled frame, and the class probability is initially predicted based on the sampled frame. The parameters of each IBCN model are shared. Then, the prediction classification results of all sampled frames are fused through a consensus function to obtain the video-level classification prediction. The calculation formula is shown in (29). Where KBN-S represents the prediction result of the spatial flow of the KBN network, and T K This represents the Kth segment after video segmentation. This represents the set of image blocks corresponding to the Kth sampling frame. This indicates the image block set processed through the IBCN module. The processing method for calculating the temporal flow prediction results is consistent with that for the spatial flow.

Images