A lightweight human behavior recognition method and system based on RGB data

Abstract

The application discloses a lightweight human behavior recognition method and system based on RGB data, and the spatial features and the time sequence features in a video are extracted in sequence through a human behavior recognition model, the model can retain original spatial information when extracting spatial features, the calculation amount of the model can be reduced and the calculation speed of the model can be accelerated by introducing a channel separation mechanism in the process, the requirement for the performance of equipment is low, meanwhile, the extracted features are classified, the model pays more attention to frames with more important classification results, the accuracy of model recognition is improved, and the problem between the recognition accuracy and the recognition speed is balanced.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a lightweight human behavior recognition method and system based on RGB data. Background Technology

[0002] Actions in videos exhibit complex temporal relationships, requiring behavior recognition models to capture not only the spatial information of each frame but also the temporal relationships between multiple frames. With continuous technological advancements, behavior recognition algorithms based on RGB data have evolved into three main categories: methods based on 3D convolutional networks, 2D convolutional networks, and Transformers.

[0003] Convolutional neural networks (CNNs) possess excellent spatial feature extraction capabilities, and the introduction of 3D convolutional kernels extends their application to the temporal dimension. Action recognition models based on 3D convolutional networks directly capture information in both temporal and spatial dimensions using these kernels. In recent years, with the excellent performance of the Transformer architecture in natural language processing, more and more researchers have applied it to action recognition, developing a series of high-performance models. Although these two types of algorithms have achieved good results on multiple public datasets, they have a large number of parameters, which can lead to difficulties in network convergence and overfitting. Furthermore, these models require high computational power, making them difficult to deploy on low-performance devices and unsuitable for real-time applications.

[0004] Two-dimensional convolutional network-based models are one of the mainstream methods in the field of behavior recognition. Such models use two-dimensional convolutional kernels for feature extraction. Their advantages are small number of parameters, low requirements for device performance, and fast training and inference speed. However, existing mainstream 2D ​​CNN behavior recognition models, such as the TSN model, directly use the 2D CNN backbone network BN-Inception

[47] to model the input data, which makes it impossible for the model to effectively extract temporal information. To address this problem, the TSM model proposed a plug-and-play shift module. Its core idea is to shift some information in the time dimension so that adjacent frames can obtain some information from other frames. However, the TSM model's offset in the time dimension is only 3, that is, the t-th frame can only obtain some information from the t-1 and t+1 frames, so it can only complete the modeling within a very small time window and cannot capture enough temporal features. Subsequently, inspired by the TSM model, relevant scholars have proposed more plug-and-play models. However, these models have two problems: 1) Some models attempt to mix video data from a temporal dimension, while others attempt to mix video data from a spatial and channel dimension. However, none of these models consider motion information between frames, nor do they adequately mix video data from multiple dimensions. 2) When performing video data mixing operations, these modules often operate on all parts of the original data, which destroys the spatial information of the original data, preventing the network from extracting sufficient spatial features. Therefore, these plug-and-play models need further improvement to better extract spatiotemporal features from video data. Overall, current mainstream behavior recognition algorithms have two main problems: On the one hand, models with high recognition accuracy usually have a large number of parameters, requiring high device performance; on the other hand, models with good real-time performance often cannot adequately mix video data from multiple dimensions, and process all data simultaneously during the mixing operation, resulting in an inability to accurately extract the temporal and spatial features of the video data, leading to poor recognition accuracy. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of existing technologies where RGB data has a large amount of information, numerous parameters, complex environments, is easily affected by changes in background or lighting conditions, requires high-performance equipment, and cannot fully mix videos from multiple dimensions. When processing all data simultaneously, it is impossible to achieve a balance between recognition accuracy and recognition speed. This invention provides a lightweight human behavior recognition method and system based on RGB data.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] A lightweight human behavior recognition method based on RGB data includes the following steps:

[0008] The acquired video data to be identified is processed by frame extraction, and the extracted data is then deformed to obtain the deformed data.

[0009] A human behavior recognition model is constructed, and the deformed data is used as the input of the human behavior recognition model to extract the spatial features and temporal features of the video data respectively.

[0010] The extracted spatial and temporal features are classified to obtain the classification and recognition results corresponding to the actions in the video data to be identified.

[0011] A further improvement of the present invention is that:

[0012] The human behavior recognition model includes an MBSTFE module and an MPLTFE module;

[0013] The MBSTFE module is used to extract spatial features from the input data;

[0014] The MPLTFE module is used to extract temporal features from the input data.

[0015] The MBSTFE module is:

[0016] Input features X∈R NT×C×H×W Separate into X according to channel dimension spatial and X temporal Two parts;

[0017] X temporal The multi-dimensional blending block and motion blending module are fed into the system to obtain the blended feature representation.

[0018] A splicing operation is used to splice together some features used for spatial modeling and short-term temporal features obtained after a mixing operation.

[0019] The MBSTFE module is:

[0020]

[0021] In the formula: X temporal X represents the partial features used for time modeling; spatial Represents partial features used for spatial modeling; ChannelSplit represents the channel splitting operation; Concate represents the channel concatenation operation; Y temporal This represents the feature map after the blending operation; MDH() represents the multidimensional blending module operation; MH() represents the motion blending module operation; out MBSTFE This indicates the output characteristics of the MBSTFE module.

[0022] The MPLTFE module is

[0023] The MPLTFE module includes an LSTM layer and a timing nonlocal module based on an improved non-local module.

[0024] The MPLTFE module is:

[0025]

[0026] In the formula, LSTM() represents an LSTM layer; sample i This represents the feature vector obtained after the sample operation; Softmax() represents the Softmax activation function.

[0027] When classifying the extracted spatial and temporal features, a Dropout

[48] layer and a fully connected layer are used.

[0028] A lightweight human behavior recognition system based on RGB data includes the following steps:

[0029] The acquired video data to be identified is processed by frame extraction, and the extracted data is deformed to obtain the deformed input data.

[0030] A human behavior recognition model is constructed, and the deformed input data is used as the input to the human behavior recognition model to extract the spatial features and temporal features of the video data respectively.

[0031] The extracted spatial and temporal features are classified to obtain the corresponding classification and recognition results of actions in the video data to be identified.

[0032] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in this invention.

[0033] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in this invention.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This invention discloses a lightweight human behavior recognition method based on RGB data. By constructing a human behavior recognition model, spatial and temporal features are extracted sequentially from the video. When extracting spatial features, the model can retain the original spatial information. Furthermore, the introduction of a channel separation mechanism in this process can reduce the computational load of the model, accelerate the model's calculation speed, and reduce the requirements for device performance. At the same time, the extracted features are classified, allowing the model to focus on frames that are more important to the classification results, thereby improving the model's recognition accuracy and balancing the issue between recognition accuracy and recognition speed.

[0036] Furthermore, in this invention, the MBSTFE module will input feature X∈R NT×C×H×W Separate into X according to channel dimension spatial and X temporal The MBSTFE module is divided into two parts; it does not support X for spatial modeling. spatial Any processing is performed to ensure that the input features retain their original spatial information. Simultaneously, the introduction of the channel separation mechanism significantly reduces the computational cost of the model. Furthermore, X... temporal The multi-dimensional blending block and motion blending module are fed in. The multi-dimensional blending module can solve the gradient vanishing problem and break the symmetry of the model, thereby improving the model's representation ability and enabling the model to learn more effective feature representations. The motion blending module can obtain motion information of the input RGB video frames to achieve real-time inference. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the overall architecture of the model of the present invention;

[0039] Figure 2 This is a schematic diagram of the MBSTFE module structure of the present invention;

[0040] Figure 3 This is a schematic diagram of the multi-dimensional hybrid module structure of the present invention;

[0041] Figure 4 This is a schematic diagram of the motion hybrid module structure of the present invention;

[0042] Figure 5 This is a schematic diagram of the MPLTFE module structure of the present invention;

[0043] Figure 6 This is a schematic diagram of the sample operation of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0046] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0047] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0048] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0049] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0050] The present invention will now be described in further detail with reference to the accompanying drawings:

[0051] See Figure 1 This invention discloses a lightweight human behavior recognition method based on RGB data to address the problems of large RGB data information volume, complex environment, susceptibility to interference from background or lighting conditions, weak feature extraction capability of existing algorithms for video data, huge computational load, and inability to achieve a balance between recognition accuracy and recognition speed.

[0052] Specifically, the following steps are included:

[0053] Step 1: For a given dataset, preprocess the data.

[0054] Given a dataset, data preprocessing includes three stages.

[0055] The first stage is video frame extraction, which converts the video into input data consisting of multiple frames of images.

[0056] The second stage: In order to improve the recognition accuracy, data augmentation operations need to be performed on the input data generated by frame extraction.

[0057] The third stage: Since the RBHAR model is based on a two-dimensional convolutional network, the input data needs to be deformed to meet the requirements of the RBHAR model for input data;

[0058] Furthermore, the dataset is divided into training and testing sets, with a sample size ratio of 7:3. During training, for each input video, the same frame sampling strategy as the TSM model is used. That is, the input video is first divided into 8 disjoint sub-videos, and one frame is randomly extracted from each sub-video. This results in input data with dimensions of 8×3×320×240. The batch size is set to C, so the shape of the input data fed into the model each time is C×8×3×320×240. Simultaneously, to meet the input requirements of a two-dimensional convolutional network, the input data needs to be stacked in the time dimension. After stacking, the shape of the input data becomes 8C×3×320×240.

[0059] For each frame extracted from the video, data augmentation is performed to improve model performance. First, the images are resized to a resolution of 224×224. Second, random horizontal flipping with a probability of 0.5 is applied. Finally, the input data is normalized to have a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225]. After data augmentation, the model's input data has a shape of 8C×3×324×224.

[0060] Step 2: Perform feature extraction on the preprocessed data using the MBSTFE and MPLTFE modules.

[0061] See Figure 2 The MBSTFE module borrows the design concept of the TSM model, which is to only perform a shift operation on a part of the input features so that they can be used for time modeling; the other part of the features that are not shifted are left untouched and used for spatial modeling.

[0062] Channel splitting is the first operation in the MBSTFE module, and its main purpose is to split the input features X∈R. NT×C×H×W (Where N is the number of samples required for each loss calculation, T is the number of video frames acquired, C is the number of channels, H is the height of a single video frame, and W is the width of a single video frame) Separate into X according to the channel dimension. spatial and X temporal Two parts, of which X temporal Used for time modeling X spatial Used for spatial modeling. The MBSTFE module does not support X for spatial modeling. spatial Any processing is performed to ensure that the input features retain their original spatial information. Simultaneously, the introduction of the channel separation mechanism significantly reduces the computational cost of the model. According to Equation 1, the computational cost of a single convolutional layer increases with the number of channels in the convolution operation. After channel separation, if the number of channels requiring convolution is reduced to half of its original value, the computational cost of the convolution operation will also be reduced to one-quarter of its original value.

[0063] FLOPs conv =[(C in ×k 2 )+(C in ×k 2 -1)+1]×C out ×M 2 (1)

[0064] Where: FLOPs conv Indicates the computational cost of a single convolution operation; C in Indicates the number of input channels; C out represents the number of output channels; k represents the kernel size; M represents the side length of the output feature, assuming that the length and width of the output feature are equal.

[0065] After the channel separation operation, the MBSTFE module will process X used for time modeling. temporal The mixed operation is performed, specifically, this mixed operation includes two parallel sub-modules:

[0066] Multi-dimensional Hybrid (MDH) module and Motion Hybrid (MH) module. X temporal The data is fed into these two modules to obtain a hybrid feature representation.

[0067] Further, see Figure 3 The Multidimensional Hybridization Module (MDH) includes channel branches, spatiotemporal branches, and residual connection branches. The channel branch mixes data along the channel dimension, while the spatiotemporal branch mixes data along the spatiotemporal dimension. By adding the residual branch, the vanishing gradient problem that may occur during model training is resolved, and the model's symmetry is broken, thereby improving the model's representational ability and enabling it to learn more effective feature representations.

[0068] Furthermore, the channel branch includes a global average pooling layer and multiple convolutional layers. First, the channel branch uses the global average pooling layer to compress the input features, obtaining a feature representation of size NT×C×1×1. Then, this branch uses three convolutional layers to process the compressed features to obtain the output feature `out`. channel At the beginning and end of the channel branch, 1×1 convolutional layers are used to perform channel compression and decompression operations, with a scaling factor of 16. Between these two convolutional layers, a one-dimensional convolutional layer with a kernel size of 3 is inserted to mix the data from the channels. Before mixing, the data needs to be deformed to meet the input shape requirements of the one-dimensional convolutional layer. Compared to the channel branch, the spatiotemporal branch processes the input features differently, performing average pooling only in the channel dimension to obtain a feature representation of size N×1×T×H×W. Then, spatiotemporal separation convolution is performed using two three-dimensional convolutions with kernel sizes of 3×1×1 and 1×3×3, respectively. This means that information in the temporal dimension is mixed first, and then information in the spatial dimension is mixed, achieving the integration of information in both the temporal and spatial dimensions. Compared to directly using a three-dimensional convolution with a kernel size of 3×3×3, the spatiotemporal separation convolution used in the spatiotemporal branch has fewer parameters. Meanwhile, the dual 3D convolutional structure of the spatiotemporal branch increases the number of nonlinear operations, enabling the model to better fit complex functions and improving its expressive power. On the other hand, when using a single 3D convolution for feature extraction, temporal and spatial features are mixed into a single feature during the extraction process, which is detrimental to network optimization.

[0069] Furthermore, after completing the channel branching and spatiotemporal branching operations, the output features obtained from the two branches are multiplied by a matrix, and the result is activated by Sigmoid to obtain the fused features of the two branches.

[0070] Furthermore, the fused features are element-wise multiplied and element-wise added to the original input features to obtain the final output features, which have the same size as the original input features. This design can adaptively adjust the contribution of features from different channels and spatiotemporal locations to the final output through the learned fused features, thereby improving the model's expressive power and recognition accuracy. The specific internal calculation steps can be represented by the following formula:

[0071]

[0072] In the formula: GAPool() represents global average pooling operation; ChannelPool() represents channel pooling operation; The superscript x represents the kernel size, and the subscript y represents the dimension of the convolution; σ() represents the Sigmoid activation function; × represents matrix multiplication; * represents element-wise multiplication; + represents element-wise addition; out channel Indicates the output characteristics of the channel branch; out temporal-spatial Represents the output characteristics of the spatiotemporal branch; out fusion Indicates fusion features; out MDH This represents the output characteristics of the multi-dimensional hybrid module.

[0073] Further, see Figure 4 The Motion Mixing Module (MH) can acquire motion information from input RGB video frames and perform real-time inference. The specific structure of the Motion Mixing Module (MH) is as follows: Figure 3-5 As shown, this module employs the same channel compression and decompression strategies as the channel branches in the Multidimensional Hybrid Module (MDH), with a compression factor of 16. First, a 1×1 convolution is used for channel compression, yielding feature I of size NT×C / 16×H×W. squeeze Meanwhile, before separating the last t-1 frames, for I... squeeze Performing two convolutions with kernel sizes of 1×3 and 3×1 allows for information fusion in the spatial dimension, enabling the extraction of richer motion information. Similar to optical flow, motion information is obtained by frame-by-frame subtraction between the later (t-1)-frame and the previous (t-1)-frame. This process can be represented by the following formula:

[0074]

[0075] In the formula: I motion Represents motion information; Split() (i,j) This indicates the operation of separating video frames, where the subscript (i,j) indicates taking the i-th to j-th frames of the RGB frame sequence.

[0076] Obtain I motionNext, the motion blending module (MH) performs zero-padding on the feature along the time dimension, resulting in a feature of size NT×C / 16×H×W. Then, MH compresses the feature using global average pooling, followed by channel decompression using 1×1 convolutions, yielding a feature I of size NT×C×1×1. unsqueeze Finally, the Motion Hybridization Module (MDH) employs Sigmoid activation and performs element-wise multiplication and addition with the original input features to obtain the final output feature `out`. MH The detailed calculation steps of the Motion Hybrid Module (MDH) can be described by the following formula:

[0077]

[0078] In the formula: Pad represents the zero-padding operation.

[0079] After the fusion operation, to generate the final output features, the MBSTFE module uses a concatenation operation to combine the features used for spatial modeling with the short-term temporal features obtained after the fusion operation. This step is significant because the concatenation operation preserves the feature dimensions, allowing the MBSTFE module to seamlessly connect with subsequent convolutional networks. Furthermore, the concatenation operation effectively integrates information from both the temporal and spatial dimensions, improving the model's recognition accuracy. In summary, the MBSTFE module can use the following formula to perform short-term temporal and spatial modeling on the input data:

[0080]

[0081] In the formula: X temporal X represents the partial features used for time modeling; spatial Represents partial features used for spatial modeling; ChannelSplit represents the channel splitting operation; Concate represents the channel concatenation operation; Y temporal This represents the feature map after the blending operation; MDH() represents the multidimensional blending module operation; MH() represents the motion blending module operation; out MBSTFE This indicates the output characteristics of the MBSTFE module.

[0082] Further, see Figure 5 The MPLTFE module consists of an LSTM layer and a Temporal Non-local (TNL) module, which is an improvement on the Non-local module.

[0083] The input to the MPLTFE module is the feature vector out output by the MBSTFE module. MBSTFE ∈RNT×C×H×W To meet the input requirements of the LSTM layer, average pooling and flattening operations are needed to obtain the feature vector X. LSTM ∈R NT×C Furthermore, X LSTM The input is fed into an LSTM layer for computation to obtain the output. Essentially, this module utilizes an LSTM layer to process X... LSTM Long-term time series modeling is performed to extract more abstract feature vectors.

[0084] Non-local modules are self-attention-based modules that can significantly improve model performance. To reduce the excessive computational demands of non-local modules, this chapter proposes a lightweight improved non-local module, the sequential non-local module TNL, which improves the model's recognition accuracy while focusing on more important frames.

[0085] Further, see Figure 6 The Temporally Nonlocal (TNL) module computes only one global weight attention map, which is shared across all locations. Before computing the weight attention map, TNL uses multi-scale pyramid pooling to reduce the number of input feature channels, significantly reducing the number of model parameters. The internal computation steps can be represented by the following formula.

[0086]

[0087] In the formula: LSTM() represents the LSTM layer; sample i This represents the feature vector obtained after the sample operation; Softmax() represents the Softmax activation function.

[0088] Step 3: Model Implementation Details

[0089] The specific details of the network are shown in Table 1. The "Output size" column in the table indicates the shape of the output feature map for each stage. In the second column, taking the Conv1 stage as an example, 7×7 represents the kernel size, the number of output channels of the convolutional layer is 64, and stride 2 indicates the stride of this convolutional layer is 2. The remaining stages are similar. The RBHAR model uses a ResNet50 network as its backbone network, and the network can be divided into three parts:

[0090] 1) Extract short-term temporal and spatial features, i.e., Conv1 to Conv5_x in the table. MBSTFE in the table is as follows: Figure 2The MBSTFE module shown is inserted into each Bottleneck module of the Resnet50 network. The number of input channels of the MBSTFE module is equal to the number of output channels of the previous stage, and the number of output channels is equal to the number of input channels of the immediately following residual unit.

[0091] 2) The long-term temporal feature enhancement part, namely the MPLTFE module. The dimension of the LTSM hidden layer is set to 256. For the one-dimensional convolutions in the MPLTFE module, the input and output channels of the first two one-dimensional convolutions are set to 256 and 128, respectively. The input and output channels of the last one-dimensional convolution are set to 8 and 1, respectively. This aims to reduce the computational cost of the model, thereby improving its efficiency.

[0092] The classification part of the network consists of a Dropout layer and a fully connected layer. The Dropout layer is introduced to avoid overfitting, and its dropout probability is set to 0.5. The input dimension of the fully connected layer is 256, and the output dimension of the fully connected layer varies for different datasets.

[0093] 3) Table 1 Implementation details of the RBHAR model

[0094]

[0095]

[0096] The lightweight human behavior recognition method based on RGB data provided in this invention provides a general pre-trained model for downstream behavior recognition tasks, which can replace the backbone with this model in any downstream task of the recognition task.

[0097] This invention discloses a lightweight human action recognition model based on RGB data—RBHAR (RGB Based Human Action Recognition)—which balances recognition accuracy and running speed. RBHAR is based on a two-dimensional convolutional network, thus having a small number of parameters and good running speed. The model includes two feature extraction modules: a multi-branches based short-term temporal-spatial feature extraction (MBSTFE) module and a multi-scale pyramid-pooling based long-term temporal feature enhancement (MPLTFE) module. The MBSTFE module is used to extract short-term temporal and spatial features from video data. The MPLTFE module aims to enhance the model's ability to extract long-term temporal features and make the model focus on frames that are more important for the classification result. The insertion of these two modules effectively improves the model's recognition accuracy. It achieves state-of-the-art performance on three highly challenging behavior recognition datasets, addressing the challenges of large amounts of information in RGB data, complex environments, susceptibility to interference from changes in background or lighting conditions, weak feature extraction capabilities of existing algorithms for video data, and enormous computational demands, making it impossible to achieve a balance between recognition accuracy and speed.

[0098] This embodiment also discloses a lightweight human behavior recognition system based on RGB data, including the following steps:

[0099] The acquired video data to be identified is processed by frame extraction, and the extracted data is deformed to obtain the deformed input data.

[0100] A human behavior recognition model is constructed, and the deformed input data is used as the input to the human behavior recognition model to extract the spatial features and temporal features of the video data respectively.

[0101] The extracted spatial and temporal features are classified to obtain the corresponding classification and recognition results of actions in the video data to be identified.

[0102] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0103] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0104] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0105] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0107] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0108] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight human behavior recognition method based on RGB data, characterized in that, Includes the following steps: The acquired video data to be identified is processed by frame extraction, and the extracted data is then deformed to obtain the deformed data. A human behavior recognition model is constructed, and the deformed data is used as the input of the human behavior recognition model to extract the spatial features and temporal features of the video data respectively. The extracted spatial and temporal features are classified to obtain the classification and recognition results of the actions in the video data to be identified. The human behavior recognition model includes an MBSTFE module and an MPLTFE module; The MBSTFE module is used to extract spatial features from the input data; The MPLTFE module is used to extract temporal features from the input data; The MBSTFE module is: Input features Separated according to channel dimension and Two parts; Will The multi-dimensional blending block and motion blending module are fed into the system to obtain the blended feature representation. A splicing operation is used to splice together some features used for spatial modeling and short-term temporal features obtained after a mixing operation; The MBSTFE module is: In the formula: This represents a subset of features used for time modeling. This represents the partial features used for spatial modeling; ChannelSplit Indicates channel separation operation; Concate This indicates a channel splicing operation; This represents the feature map after the blending operation; MDH () indicates multi-dimensional mixed module operations; MH () indicates operations performed by the motion mixing module; This indicates the output characteristics of the MBSTFE module; The MPLTFE module is The MPLTFE module includes an LSTM layer and a timing nonlocal module based on an improved non-local module. The MPLTFE module is: In the formula, Indicates an LSTM layer; This represents the feature vector obtained after the sample operation; Softmax() represents the Softmax activation function.

2. The lightweight human behavior recognition method based on RGB data according to claim 1, characterized in that, When classifying the extracted spatial and temporal features, a Dropout[48] layer and a fully connected layer are used.

3. A lightweight human behavior recognition system based on RGB data that implements the method of claim 1, characterized in that, Includes the following steps: The acquired video data to be identified is processed by frame extraction, and the extracted data is deformed to obtain the deformed input data. A human behavior recognition model is constructed, and the deformed input data is used as the input to the human behavior recognition model to extract the spatial features and temporal features of the video data respectively. The extracted spatial and temporal features are classified to obtain the corresponding classification and recognition results of actions in the video data to be identified.

4. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-2.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-2.

Images