A behavior recognition method, device, equipment and computer readable storage medium

Abstract

This application discloses a behavior recognition method, apparatus, device, and computer-readable storage medium, comprising: acquiring video data to be recognized; extracting multiple video frames from the video data; dividing the multiple video frames according to different scales to obtain division results corresponding to each scale, the division results including first image spatial features and first skeleton data; extracting first image spatiotemporal features corresponding to the first image spatial features in each division result; determining second image spatiotemporal features of the video data based on each first image spatiotemporal feature, and determining skeleton spatiotemporal features of the video data based on each first skeleton data; fusing the second image spatiotemporal features and skeleton spatiotemporal features based on a trained adaptive attention fusion network to obtain fused spatiotemporal features of the video data; and performing behavior classification processing on the video data based on the fused spatiotemporal features to obtain behavior recognition results, thereby achieving accurate behavior recognition through multi-scale division and feature fusion.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and to, but is not limited to, a behavior recognition method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of computer hardware and software, machine vision technology has become one of the popular research methods in video research. The main research content of machine vision is to analyze and obtain the information contained in the video. The goal is to extract the appearance and motion features of the human body from the video, and classify the obtained features to identify human behavior.

[0003] In related technologies, on the one hand, video processing involves identifying the entire video sequence to obtain the overall features of the actions. However, in long-duration videos, there are many actions unrelated to key actions, so analyzing the whole video can easily lead to misclassification due to overlooking key actions. On the other hand, using fixed weights to fuse features from different modalities without considering the relative importance of different modalities can result in inaccurate classification results. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a behavior recognition method, apparatus, device, and computer-readable storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a behavior recognition method, the method comprising:

[0007] Acquire the video data to be identified, and extract multiple video frames from the video data;

[0008] The multiple video frames are divided according to different scales to obtain the division results corresponding to each scale. The division results include first image spatial features and first skeleton data. The first image spatiotemporal features corresponding to the first image spatial features in each division result are extracted.

[0009] The second image spatiotemporal features of the video data are determined based on the spatiotemporal features of each first image, and the spatiotemporal features of the skeleton of the video data are determined based on each first skeleton data.

[0010] The second image spatiotemporal features and the skeleton spatiotemporal features are fused based on the trained adaptive attention fusion network to obtain the fused spatiotemporal features of the video data.

[0011] Based on the fused spatiotemporal features, the video data is subjected to behavior classification processing to obtain behavior recognition results.

[0012] This application provides a behavior recognition device, the behavior recognition device comprising:

[0013] The acquisition module is used to acquire the video data to be identified and extract multiple video frames from the video data;

[0014] The segmentation module is used to segment the multiple video frames according to different scales to obtain segmentation results corresponding to each scale. The segmentation results include first image spatial features and first skeleton data. The module also extracts the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result.

[0015] The determining module is used to determine the second image spatiotemporal features of the video data based on each first image spatiotemporal feature, and to determine the skeletal spatiotemporal features of the video data based on each first skeletal data.

[0016] The fusion module is used to fuse the spatiotemporal features of the second image and the spatiotemporal features of the skeleton based on a trained adaptive attention fusion network to obtain the fused spatiotemporal features of the video data.

[0017] The classification module is used to perform behavior classification processing on the video data based on the fused spatiotemporal features to obtain behavior recognition results.

[0018] This application provides a behavior recognition device, the behavior recognition device comprising:

[0019] Processor; and

[0020] Memory for storing computer programs that can run on the processor;

[0021] The computer program implements the above-mentioned behavior recognition method when executed by a processor.

[0022] This application provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described behavior recognition method.

[0023] This application provides a behavior recognition method, apparatus, device, and computer-readable storage medium. The behavior recognition method includes: extracting multiple video frames from the acquired video data to be recognized at certain intervals; then, dividing the multiple video frames using multiple different scales to obtain a division result corresponding to each scale, the division result including first image spatial features and first skeletal data; then, extracting first image spatiotemporal features corresponding to the first image spatial features in each division result; determining second image spatiotemporal features of the video data based on each first image spatiotemporal feature, and determining skeletal spatiotemporal features of the video data based on each first skeletal data; finally, fusing the second image spatiotemporal features and skeletal spatiotemporal features based on a trained adaptive attention fusion network to obtain fused spatiotemporal features of the video data, and performing behavior classification processing on the video data based on the fused spatiotemporal features to finally obtain a behavior recognition result. During the segmentation process, multiple video frames are divided at different scales to obtain different segmentation results containing multi-dimensional information, providing a foundation for correct classification. In the process of fusing different modal features, the trained adaptive attention fusion network can fully consider the importance of different modal features, thereby improving the accuracy of classification results. Attached Figure Description

[0024] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. The drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0025] Figure 1 A schematic diagram of the structure of human skeleton data in four scenarios provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram illustrating an implementation process of the behavior recognition method provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating one implementation flow of the partitioning process provided in an embodiment of this application;

[0028] Figure 4A This is a schematic diagram illustrating an implementation process for determining the spatiotemporal features of an image, as provided in an embodiment of this application.

[0029] Figure 4B This is a schematic diagram illustrating an implementation process for determining spatiotemporal features of a skeleton, as provided in an embodiment of this application.

[0030] Figure 5 This is a schematic diagram illustrating an implementation process for determining the third and fourth weights, provided in an embodiment of this application.

[0031] Figure 6 This is a schematic diagram illustrating an implementation process for determining fused spatiotemporal features, provided in an embodiment of this application.

[0032] Figure 7 This is a schematic diagram illustrating an implementation process of the behavior recognition model block diagram provided in an embodiment of this application.

[0033] Figure 8 A schematic diagram of a multi-scale temporal feature extraction block diagram based on RGB images provided in an embodiment of this application;

[0034] Figure 9 A schematic diagram of a multi-scale temporal feature extraction block diagram based on skeletal coordinates provided in an embodiment of this application;

[0035] Figure 10 A schematic diagram of a fusion classification block diagram provided in an embodiment of this application;

[0036] Figure 11 This is a schematic diagram of the composition structure of the behavior recognition device provided in the embodiments of this application;

[0037] Figure 12 This is a schematic diagram of the composition structure of the behavior recognition device provided in the embodiments of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0042] Machine vision technology is a discipline that studies the extraction, processing, and recognition of information from images or videos. Object tracking, person re-identification, and human pose estimation are among its main research directions. Simultaneously, with the rapid proliferation of data acquisition devices and the exponential growth in the amount of video data, analyzing and extracting the information contained within videos has become a key research area in machine vision. Therefore, video-based human behavior recognition has gradually become a research hotspot in machine vision, aiming to extract the appearance and motion features of the human body from videos and classify these features to identify human behavior.

[0043] In traditional action recognition research based on red-green-blue (RGB) images, the focus is usually on extracting more discriminative spatiotemporal features from videos. Classifying video samples using hand-designed features extracted from video samples can be collectively referred to as model-driven action recognition algorithms. With the rapid development of deep learning technology, data-driven deep learning algorithms have begun to be applied to action recognition research based on RGB images. Currently, there are three main types of deep learning-based action recognition algorithms: Long Short-Term Memory (LSTM), Two-Stream Convolutional Neural Networks (TwoStream CNN), and Three-Dimensional Convolutional Neural Networks (3D CNN). Among these three algorithms, Two-Stream CNN is too time-consuming in acquiring optical flow images, 3D CNN suffers from difficulties in parameter tuning due to its large number of parameters, while LSTM strikes a balance between the two.

[0044] Currently, the data modalities used in behavior recognition, besides RGB images and optical flow images in two-stream networks, also include depth images and skeletal data. Unlike RGB images, optical flow images, and depth images, skeletal data does not contain appearance information about the human body and background; it only contains skeletal information, such as... Figure 1 As shown, although human skeletal data is robust to changes in lighting and texture, it lacks information about interactions between people and objects. Therefore, relying solely on human skeletal data for recognition becomes too simplistic when dealing with similar and complex interactions. Consequently, many early behavior recognition algorithms incorporated both RGB image features and skeletal features.

[0045] The relevant implementation schemes have the following two problems:

[0046] Question 1: Most related action recognition algorithms identify the entire video sequence, and analyzing the entire sequence can obtain the overall features of the actions. However, in long-duration actions, there are many actions unrelated to the key actions. Therefore, analyzing the whole sequence can easily overlook key sub-actions, leading to misclassification. For example, in the action of "long jump," there is the action of "approach run." If the "approach run" accounts for a large proportion of the overall video sequence, and the entire video sequence is analyzed and identified, it is easy to identify it as "running." Therefore, it is necessary to identify actions in video sequences more precisely, paying attention to key sub-actions in the video sequence while identifying the entire video sequence; that is, focusing on key sequences within the video.

[0047] Question 2: Most related action recognition methods that fuse RGB image features and skeletal features utilize fixed-weight fusion to combine features from different modalities into a single high-dimensional feature. The drawbacks of this approach are: it ignores the fact that different actions are suited to different modal features for classification, it cannot adjust for each sample, and it fails to effectively utilize the complementary information between the two modalities. Furthermore, different modalities exhibit highly consistent human motion semantics; simple fusion can easily lead to information redundancy, affecting the network's feature representation and classification. Fixed-weight fusion cannot solve this information redundancy problem. Simultaneously, among the extracted features at multiple scales, the importance of features from different segments varies for recognition. Therefore, fixed-weight fusion does not consider the importance of features from different video segments at different scales. In reality, to improve recognition accuracy, it is necessary to differentiate between different video segments, increasing the weight of more important segments during classification and decreasing the weight of less important segments.

[0048] To address the problems existing in related technologies, this application provides a behavior recognition method. This method can be implemented by a computer program, which, when executed, performs the behavior recognition method provided in this application. In some embodiments, the computer program can be executed by a processor in a behavior recognition device. Figure 2 One implementation flow of the behavior recognition method provided in the embodiments of this application is as follows: Figure 2 As shown, the behavior recognition method includes:

[0049] Step S201: Obtain the video data to be identified and extract multiple video frames from the video data.

[0050] Here, the video data to be identified can be a video object targeting an individual, such as a company employee, a sports team member, a family member, or a school student. In practice, this video data can be acquired through camera equipment, surveillance equipment, or similar devices.

[0051] In this embodiment, multiple video frames can be extracted from video data at certain intervals, where each video frame can be a grayscale image or a color image. During the extraction process, a time interval threshold can be set first, and then the video data can be periodically extracted based on this time interval threshold to obtain multiple video frame data; alternatively, a number threshold can be set for the number of video frames, in which case multiple video frames are randomly extracted from the video data such that the number of these multiple video frames equals the number threshold.

[0052] Step S202: Divide multiple video frames into segments according to different scales to obtain the segmentation results corresponding to each scale; and extract the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result.

[0053] Here, different scales include at least two scales. These different scales can include two scales, three scales, four scales, etc. Taking two scales as an example, in this case, the division result representing the global features of multiple video frames can be obtained by dividing at one scale, while the division result representing the local features of multiple videos can be obtained by dividing at another scale.

[0054] Taking different scales, including three scales, as examples, the first scale can divide multiple video frames into one video segment, and the resulting segment is the partitioning result corresponding to that one video segment. This one segment is equivalent to multiple video frames, and it can represent the video data. Therefore, the partitioning result corresponding to this one video segment can represent the global features of the video data. The second scale can divide multiple video frames into two video segments, and the resulting segment is the two partitioning results corresponding to those two video segments. These two video segments together form multiple video frames, meaning that the two video segments together represent the video data. Thus, the partitioning result of each video segment represents the local features of the video data. Similarly, the third scale can divide multiple video frames into four video segments, and the resulting segment is the four partitioning results corresponding to those four video segments. The partitioning result of each video segment represents the local features of the video data.

[0055] In this embodiment, the partitioning result includes a first image spatial feature. During partitioning, on the one hand, spatial features can be extracted from multiple video frames to obtain second image spatial features for the multiple video frames; then, the second image spatial features are partitioned based on different scales to obtain the partitioned first image spatial feature. Continuing the example above, if the scale is the first scale, the first image spatial feature consists of the image spatial features corresponding to multiple video frames. That is, at this scale, it is equivalent to not partitioning the second image spatial feature; the first image spatial feature is equivalent to the second image spatial feature. If the scale is the second scale, the first image spatial feature includes two image spatial features corresponding to two video segments.

[0056] On the other hand, the segmentation result also includes first bone data, which can be represented by bone coordinates or joint coordinates. During segmentation, pose estimation is performed on multiple video frames based on model methods or learning methods to obtain second bone data for multiple video frames. Then, the second bone data is segmented based on different scales to obtain the segmented first bone data. Continuing the example above, if the scale is the first scale, the first bone data consists of bone data corresponding to multiple video frames. That is, at this scale, it is equivalent to not segmenting the second bone data; the first bone data is equivalent to the second bone data. If the scale is the second scale, the first bone data includes two bone data corresponding to two video segments.

[0057] In this embodiment of the application, temporal features are also extracted from the spatial features of the first image in the segmentation result to obtain the spatiotemporal features of the first image corresponding to the spatial features of the first image. The spatiotemporal features of the first image include not only spatial features but also temporal features, which can comprehensively reflect the features of the segmented image.

[0058] Step S203: Determine the second image spatiotemporal features of the video data based on the spatiotemporal features of each first image, and determine the skeletal spatiotemporal features of the video data based on each first skeletal data.

[0059] In practice, the partitioning results can fall into two categories:

[0060] In the first scenario, the segmentation result includes a first image spatial feature.

[0061] In this case, by directly extracting temporal features from the second image spatial features of multiple video frames, the fourth image spatiotemporal features corresponding to multiple video frames can be obtained. Temporal feature extraction can be performed using methods such as LSTM, bidirectional LSTM, and autoregressive moving average models.

[0062] Scenario 2: The segmentation result includes multiple first image spatial features or multiple first skeleton data.

[0063] In this case, on the one hand, the first adaptive attention model corresponding to the first target scale of the division result is obtained. The first adaptive attention model can determine the first weights corresponding to each spatiotemporal feature of the first image. Then, the spatiotemporal features of the first image are input into the first adaptive attention model to obtain the first weights corresponding to each spatiotemporal feature of the first image, so that each spatiotemporal feature of the first image corresponds to its own first weight. Then, based on the first weights and the spatiotemporal features of the first image, the spatiotemporal features of the fused image of multiple video frames under the first target scale can be determined through weighted fusion processing.

[0064] On the other hand, a second adaptive attention model corresponding to the second target scale of the division result is also obtained. This second adaptive attention model can determine the second weights corresponding to each first skeleton data. Then, each first skeleton data is input into the second adaptive attention model to obtain the second weights corresponding to each first skeleton data, so that each first skeleton data has its own second weight. Next, the first skeleton data is weighted based on the second weights to obtain weighted skeleton data of multiple video frames.

[0065] Based on the above two scenarios, on the one hand, after determining the spatiotemporal features of the fourth image and the fused image, these two spatiotemporal features can be fused by adding them element-wise to obtain the second spatiotemporal features of the video data. On the other hand, the weighted skeleton data and the second skeleton data are also spliced ​​and feature extracted to obtain the skeleton spatiotemporal features.

[0066] Step S204: Based on the trained adaptive attention fusion network, the spatiotemporal features of the second image and the spatiotemporal features of the skeleton are fused to obtain the fused spatiotemporal features of the video data.

[0067] In this embodiment, the trained adaptive attention fusion network is obtained by training an initial adaptive attention fusion network using sample data. The trained adaptive attention fusion network can determine the weights of different modal features based on the importance of different modalities. Therefore, by inputting the second image spatiotemporal features and the skeletal spatiotemporal features into the trained adaptive attention fusion network, the third weight corresponding to the second image spatiotemporal features and the fourth weight corresponding to the skeletal spatiotemporal features can be determined. Then, based on the second image spatiotemporal features, the skeletal spatiotemporal features, the third weight, and the fourth weight, the fused spatiotemporal features of the video data can be determined. These fused spatiotemporal features can accurately and comprehensively represent the features of the video data.

[0068] Step S205: Perform behavior classification processing on the video data based on the fused spatiotemporal features to obtain behavior recognition results.

[0069] Here, classification methods such as decision trees, Naive Bayes, and random forests can be used to classify the fused spatiotemporal features, thereby classifying the video data and obtaining the corresponding behavior recognition results.

[0070] This application provides a behavior recognition method. For acquired video data to be recognized, multiple video frames are extracted from the video data at certain intervals. Then, the multiple video frames are divided using multiple different scales to obtain division results corresponding to each scale. These division results include first image spatial features and first skeletal data. Next, first image spatiotemporal features corresponding to the first image spatial features in each division result are extracted. Second image spatiotemporal features of the video data are determined based on each first image spatiotemporal feature, and skeletal spatiotemporal features of the video data are determined based on each first skeletal data. Finally, the second image spatiotemporal features and skeletal spatiotemporal features are fused using a trained adaptive attention fusion network to obtain fused spatiotemporal features of the video data. Behavior classification is then performed on the video data based on these fused spatiotemporal features to obtain the final behavior recognition result. During the division process, multiple video frames are divided using different scales, resulting in different division results containing multi-dimensional information, providing a foundation for correct classification. During the fusion of different modal features, the trained adaptive attention fusion network fully considers the importance of different modal features, thereby improving the accuracy of the classification results.

[0071] In some embodiments, the segmentation result includes first image spatial features and first skeleton data. When actually implementing step S202, "segmenting multiple video frames according to different scales to obtain segmentation results corresponding to each scale, and extracting the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result," as follows... Figure 3 As shown, this can be achieved through the following steps S2021 to S2026:

[0072] Step S2021: Spatial feature extraction is performed on multiple video frames to obtain the second image spatial features of multiple video frames.

[0073] Here, one video frame is one image, and multiple video frames are multiple images. Spatial features are extracted from each video frame to obtain the image spatial features corresponding to each video frame, which are denoted as the second image spatial features.

[0074] When performing spatial feature extraction, convolutional neural networks such as Inception v3, LeNet-5, and AlexNet can be used. Taking Inception v3 network for feature extraction as an example, after feature extraction, the output dimension of each video frame is 1*1024.

[0075] Step S2022: Perform pose estimation processing on multiple video frames to obtain the second skeleton data of multiple video frames.

[0076] Here, pose estimation methods such as Convolutional Pose Machine (CPM) and Cascaded Pyramid Network (CPN) can be used to perform pose estimation processing on multiple video frames separately, thereby obtaining the second bone data corresponding to each video frame. The second bone data can be the joint coordinates of the object to be identified in the video frame.

[0077] In this embodiment, there is no specific order in which steps S2021 and S2022 are executed. Step S2021 can be executed first, or step S2022 can be executed first, or steps S2021 and S2022 can be executed simultaneously. This embodiment does not limit this.

[0078] Step S2023: Divide the second image spatial features according to different scales to obtain the first image spatial features of each group of video segments at different scales.

[0079] Here, "different scales" includes at least two scales, and these different scales can include two, three, four, etc. Dividing the second image features according to different scales is equivalent to dividing them using different windows. By dividing them according to different scales, we can obtain image spatial features that represent the global features of the video data, as well as image spatial features that represent the local features of the video data.

[0080] Continuing with the example above, let's take three scales as examples again. The first scale can divide multiple video frames into one video segment, and the result is the first image spatial feature corresponding to this one video segment. The first image spatial feature corresponding to this one video segment can represent the global image spatial feature of the video data. The second scale can divide multiple video frames into two video segments, and the result is two first image spatial features corresponding to these two video segments. The first image spatial feature of each video segment represents the local image spatial feature of the video data. Similarly, the third scale can divide multiple video frames into four video segments, and the result is four first image spatial features corresponding to these four video segments. The first image spatial feature of each video segment represents the local image spatial feature of the video data.

[0081] In the example above, one first image spatial feature is obtained at the first scale, two first image spatial features are obtained at the second scale, and four first image spatial features are obtained at the third scale. A total of seven first image spatial features are obtained through these three different scales.

[0082] Step S2024: Divide the second skeleton data according to different scales to obtain the first skeleton data of each group of video segments at different scales.

[0083] Here, the different scales are similar to those in step S2023, including at least two scales. However, the number of different scales here can be the same as or different from the number in step S2023. That is, if step S2023 includes three scales, this step can include three scales, or it can include four scales. By dividing the data into different scales, skeletal data representing the global features of the video data or skeletal data representing the local features of the video data can be obtained.

[0084] In this embodiment, there is no specific order in which steps S2023 and S2024 are executed. Step S2023 can be executed first, or step S2024 can be executed first, or steps S2023 and S2024 can be executed simultaneously. This embodiment does not limit this.

[0085] Step S2025: Extract temporal features from the first image spatial features of each group of video segments to obtain the third image spatiotemporal features of each group of video segments.

[0086] Here, temporal features can be extracted from the spatial features of the first image of each group of video segments using methods such as LSTM, bidirectional LSTM, and autoregressive moving average model, to obtain the spatiotemporal features of each third image, which include both spatial and temporal features.

[0087] Taking temporal feature extraction using bidirectional LSTM as an example, if the partitioning result is the first image spatial feature corresponding to one video segment, then after temporal feature extraction, the image spatiotemporal feature with a dimension of 1*1024 will be obtained; if the partitioning result is the two first image spatial features corresponding to two video segments, then after temporal feature extraction, the image spatiotemporal feature with a dimension of 2*1*1024 will be obtained.

[0088] Step S2026: Determine the third image spatiotemporal features of each group of video segments as the first image spatiotemporal features.

[0089] Here, the spatiotemporal features of the third image corresponding to each group of video segments obtained in step S2025 are all determined as the spatiotemporal features of the first image.

[0090] Through steps S2021 to S2026, on the one hand, spatial features are extracted and scales are divided for multiple video frames to obtain the first image spatial features of each group of video segments at different scales, i.e., global image spatial features and local image spatial features. On the other hand, pose estimation is performed on multiple video frames to obtain the second skeleton data of multiple video frames, and the second skeleton data is divided at different scales to obtain the first skeleton data of each group of video segments at different scales, i.e., global skeleton data and local skeleton data. Finally, the first image spatial features and the first skeleton data constitute the division result. In addition, temporal features are extracted from the first image spatial features to obtain the first image spatiotemporal features corresponding to the first image spatial features. Based on this, spatiotemporal features of different modalities are extracted, and each modal spatiotemporal feature includes global spatiotemporal features and local spatiotemporal features, enriching the features of the video data and improving the speed and accuracy of subsequent behavior recognition.

[0091] In some embodiments, step S203, "determining the second image spatiotemporal features of the video data based on the spatiotemporal features of each first image, and determining the spatiotemporal skeletal features of the video data based on each first skeletal data," can be implemented in practice through the following two steps:

[0092] Step S2031: Determine the second image spatiotemporal features of the video data based on the spatiotemporal features of each first image;

[0093] Step S2032: Determine the spatiotemporal skeletal features of the video data based on each first skeleton data.

[0094] In practice, refer to Figure 4A The above step S2031, "determining the second image spatiotemporal features of the video data based on the spatiotemporal features of each first image," can be achieved through the following steps S20311 to S20316; see reference. Figure 4BThe above step S2032, "determine the spatiotemporal features of the skeleton in the video data based on each first skeleton data", can be achieved through the following steps S20321 to S20326.

[0095] Among them, reference Figure 4A Step S20311, "Determining the second image spatiotemporal features of the video data based on the spatiotemporal features of each first image," can be achieved through the following steps S20311 to S20316:

[0096] Step S20311: Temporal feature extraction is performed on the second image spatial features of multiple video frames to obtain the fourth image spatiotemporal features of multiple video frames.

[0097] Here, temporal features are extracted from the second image spatial features of multiple video frames to obtain the fourth image spatiotemporal features corresponding to the multiple video frames, which are used to characterize the features of the multiple video frames. The method for temporal feature extraction can refer to the temporal feature extraction method in step S2025.

[0098] Step S20312: Determine whether the segmentation result includes multiple first image spatial features.

[0099] Here, the number of first image spatial features in the segmentation result at each scale can be determined by reading the instruction. If it is determined that the segmentation result at the scale includes multiple first image spatial features, then proceed to step S20313; if it is determined that the segmentation result at the scale includes one first image spatial feature, then return to step S20311.

[0100] Step S20313: Obtain the first adaptive attention model corresponding to the first target scale of the partitioning result.

[0101] At this point, the partitioning result at this scale includes multiple first image spatial features. For example, the partitioning result at this scale includes 2, 4, 8, etc., firstly, the first target scale corresponding to the partitioning result is determined. The first target scale includes at least one scale, and may also include two scales, three scales, etc. Next, the first adaptive attention model corresponding to the first target scale is obtained. The adaptive attention model is a model pre-trained with samples, which can assign different first weights to different first image spatial features according to the importance of different first image spatial features.

[0102] Step S20314: Based on multiple first image spatiotemporal features and a first adaptive attention model, determine each first weight corresponding to each first image spatiotemporal feature.

[0103] Here, multiple spatiotemporal features of the first image can be input into the first adaptive attention model. The first adaptive attention model can obtain the first weights corresponding to each spatiotemporal feature of the first image. In this embodiment, the first adaptive attention model can assign different first weights to different spatiotemporal features of the first image according to their importance. Specifically, higher weights are assigned to the first spatiotemporal features with higher importance to highlight the local features that are more critical to classification, while lower weights are assigned to the first spatiotemporal features with lower importance. Based on this principle, each spatiotemporal feature of the first image is assigned its own corresponding first weight.

[0104] Step S20315: Based on each first weight and each first image spatiotemporal feature, determine the fused image spatiotemporal features of multiple video frames at the first target scale.

[0105] Here, the spatiotemporal features of the corresponding first image are weighted based on each first weight to obtain the spatiotemporal features of the fused image of multiple video frames at the first target scale.

[0106] Step S20316: The spatiotemporal features of the fused image and the spatiotemporal features of the fourth image are fused to obtain the spatiotemporal features of the second image.

[0107] Here, the fused spatiotemporal features and the fourth image spatiotemporal features can be fused using methods such as element-wise addition and cross-fusion to obtain the second image spatiotemporal features of the video data. The fused image spatiotemporal features include local features of the video data, as well as features of key parts of the object to be identified in the video data, while the fourth image spatiotemporal features represent the global features of the video data.

[0108] Through steps S20311 to S20316, on the one hand, by extracting temporal features from the second image spatial features of multiple video frames, fourth image spatiotemporal features of multiple video frames are obtained; on the other hand, if the segmentation result includes multiple first image spatial features, a first adaptive attention model is also obtained, and each first weight corresponding to each first image spatiotemporal feature is determined based on the multiple first image spatiotemporal features and the first adaptive attention model. When determining each first weight through the first adaptive attention model, each first weight is determined according to the importance of each first image spatial feature, linking the weight with the importance. Furthermore, the fused image spatiotemporal features are determined based on each first weight and each first image spatiotemporal feature; finally, the fused image spatiotemporal features are... The first image spatiotemporal features are fused with the fourth image spatiotemporal features to obtain the second image spatiotemporal features of the video data. Since the fused image spatiotemporal features can better reflect the local features of multiple video frames, while the fourth image spatiotemporal features can better reflect the global features of multiple video frames, the fused second image spatiotemporal features can comprehensively reflect the features of the video data. When determining the first weights corresponding to each first image spatiotemporal feature, the first weights are not fixed weights, but are determined according to the importance of each first image spatiotemporal feature. This can also be considered as determining the first weights according to the degree of influence of each first image spatiotemporal feature on the classification result, thereby more accurately representing the image modal features of the video data and ultimately improving the accuracy of the classification result.

[0109] refer to Figure 4B The above step S2032, "determining the spatiotemporal features of the video data based on each first skeleton data", can be achieved through the following steps S20321 to S20326:

[0110] Step S20321: Determine whether the segmentation result includes multiple first bone data.

[0111] Here, the number of first bone data in the segmentation result at each scale can be determined by reading the instruction. If it is determined that the segmentation result at the scale includes multiple first bone data, then proceed to step S20322; if it is determined that the segmentation result at the scale includes one first bone data, then execute step S20326.

[0112] Step S20322: Obtain the second adaptive attention model corresponding to the second target scale of the partitioning result.

[0113] At this point, the segmentation result includes multiple first skeleton data. For example, the segmentation result at this scale includes 2, 4, 8, etc., firstly, the second target scale corresponding to the segmentation result is determined. The second target scale includes at least one scale, and may also include two scales, three scales, etc. Next, the second adaptive attention model corresponding to the second target scale is obtained. This adaptive attention model is a model pre-trained with samples, which can assign different second weights to different second skeleton data according to the importance of different second skeleton data.

[0114] Step S20323: Based on multiple first skeleton data and a second adaptive attention model, determine each second weight of each first skeleton data.

[0115] Here, the implementation process of step S20323 is similar to that of step S20314. Therefore, the implementation process of step S20323 can refer to the implementation process of step S20314.

[0116] Step S20324: Weight the first skeleton data based on each second weight to obtain weighted skeleton data of multiple video frames.

[0117] Here, the corresponding first skeleton data are weighted according to each second weight; that is, each first skeleton data is multiplied by its corresponding second weight to obtain weighted skeleton data. This weighting process highlights the skeleton data of key parts of the object to be identified in the video data.

[0118] Step S20325: The weighted skeletal data and the second skeletal data are sequentially spliced ​​and feature extracted to obtain the spatiotemporal features of the skeleton.

[0119] Here, the weighted skeleton data can represent the local features of multiple video frames, while the second skeleton data can represent the global features of multiple video frames. Then, the weighted skeleton data is stitched together to obtain the stitched skeleton data. Then, the stitched skeleton data is further processed by template-based and edge-based methods to extract features, thereby obtaining the spatiotemporal features of the skeleton that represent the features of the video data.

[0120] Step S20326, End.

[0121] Through steps S20321 to S20326 above, when the segmentation result includes multiple first skeleton data, the corresponding second target scale and the second adaptive attention model corresponding to the second target scale are obtained; then, the second weights of each first skeleton data are determined through the second adaptive attention model; then, the first skeleton data are weighted based on each second weight to obtain weighted skeleton data; finally, the weighted skeleton data are spliced ​​and feature extracted to obtain the spatiotemporal features of the skeleton representing the features of the video data. When determining the second weights corresponding to each first skeleton data, the second weights are not fixed weights, but the importance of each first skeleton data is associated with the second weights through the second adaptive attention model. It can also be considered that the corresponding second weights are determined according to the degree of influence of each first skeleton data on the classification result, thereby more accurately representing the skeletal modal features of the video data and improving the accuracy of the classification result.

[0122] In some embodiments, step S204, "fusing the spatiotemporal features of the second image and the spatiotemporal features of the skeleton based on the trained adaptive attention fusion network to obtain the fused spatiotemporal features of the video data," can be achieved through the following steps S2041 to S2042:

[0123] Step S2041: Based on the trained adaptive attention fusion network, determine the third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton.

[0124] In actual implementation, such as Figure 5 As shown, step S2041 can be achieved through the following steps S411 to S413:

[0125] Step S411: The spatiotemporal features of the second image and the spatiotemporal features of the skeleton are spliced ​​together to obtain spliced ​​spatiotemporal features.

[0126] Here, the spatiotemporal features of the skeleton can be stitched after the spatiotemporal features of the second image to achieve the stitching process of the spatiotemporal features of the second image and the spatiotemporal features of the skeleton, or the spatiotemporal features of the second image can be stitched after the spatiotemporal features of the skeleton to achieve the stitching process of the spatiotemporal features of the second image and the spatiotemporal features of the skeleton, thereby obtaining the stitched spatiotemporal features.

[0127] Step S412: Obtain the initialized weight matrix and the initialized bias scalar.

[0128] Here, the dimensions of the initialized weight matrix and the initialized bias scalar are matched with the dimensions of the spatiotemporal features. For example, if the dimensions of the spatiotemporal features are 1024*2, then the dimensions of the initialized weight matrix are 1*1024, and the dimensions of the initialized bias scalar are 1*2.

[0129] In actual implementation, the initialized weight matrix and the initialized bias scalar can be either default values ​​or custom values.

[0130] Step S413: Based on the splicing spatiotemporal features, the initialized weight matrix, and the initialized bias scalar, determine the third and fourth weights.

[0131] Here, the initialized weight matrix and the spliced ​​spatiotemporal features can be multiplied to obtain the matrix product result; then, the matrix product result and the initialized bias scalar can be added to obtain the matrix sum; finally, the matrix sum is input into the logistic regression model to obtain the third weight corresponding to the second image spatiotemporal features and the fourth weight corresponding to the skeletal spatiotemporal features.

[0132] Step S2042: Determine the fusion spatiotemporal features of the video data based on the second image spatiotemporal features, the skeleton spatiotemporal features, the third weight, and the fourth weight.

[0133] In actual implementation, step S2042 is as follows: Figure 6 As shown, this can be achieved through the following steps S421 to S423:

[0134] Step S421: The second image spatiotemporal features are weighted using the third weight to obtain the weighted image spatiotemporal features.

[0135] Here, the second image spatiotemporal features are weighted using the third weight determined in step S413, that is, the second image spatiotemporal features are multiplied by their own importance to obtain the weighted image spatiotemporal features.

[0136] Step S422: Use the fourth weight to weight the spatiotemporal features of the skeleton to obtain the weighted spatiotemporal features of the skeleton.

[0137] Here, the skeletal spatiotemporal features are weighted using the fourth weight determined in step S413. That is, the skeletal spatiotemporal features are multiplied by their respective importance to obtain the weighted skeletal spatiotemporal features.

[0138] Step S423: Determine the fused spatiotemporal features based on the weighted image spatiotemporal features and the weighted skeleton spatiotemporal features.

[0139] Here, the weighted image spatiotemporal features and the weighted skeletal spatiotemporal features can be matrix-added to obtain the fused spatiotemporal features.

[0140] Through steps S2041 and S2042, the adaptive attention fusion network determines the third weight of the spatiotemporal features of the second image and the fourth weight of the skeletal data according to their importance. Then, based on the third and fourth weights, a weighted fusion process is performed on the spatiotemporal features of the second image and the skeletal data to obtain the fused spatiotemporal features. The adaptive attention fusion network can determine the third weight of the spatiotemporal features of the second image and the fourth weight of the skeletal data according to the actual scene, making the obtained third and fourth weights more consistent with the actual situation, thereby improving recognition accuracy.

[0141] Based on the above embodiments, this application further provides an action recognition method. First, for an RGB image, it is segmented into video subsequences of different lengths at three scales, and then a bidirectional LSTM network is used to extract the temporal features of the video subsequences respectively. Next, an adaptive attention mechanism assigns weights to each video subsequence and fuses the spatiotemporal features of the three scales. Here, the adaptive attention mechanism corresponds to the first adaptive attention model in the above embodiments. Then, for skeletal coordinates, which correspond to the skeletal data in the above embodiments, based on the traditional convolutional network feature extraction, the skeletal coordinates are segmented into different subsequences at three scales. An adaptive attention mechanism assigns weights to different subsequences. The adaptive attention mechanism for processing skeletal coordinates corresponds to the second adaptive attention model in the above embodiments. The weighted skeletal coordinate sequences at each scale are concatenated, and spatiotemporal features are extracted using a convolutional neural network. Finally, the adaptive attention network is used to calculate the fusion weights to adjust for different samples to achieve better recognition performance. Subsequently, the RGB and skeletal multi-scale spatiotemporal features are fused and classified.

[0142] This application proposes an adaptive attention fusion behavior recognition algorithm based on multi-scale temporal feature extraction, which is based on LSTM and aims to improve recognition accuracy. It focuses on enhancing the spatiotemporal feature representation capability of videos and improving the fusion efficiency of the two features.

[0143] In this embodiment, an adaptive attention fusion behavior recognition network based on multi-scale temporal feature extraction is designed for behavior recognition, using LSTM as the foundation. The overall block diagram of this model is as follows: Figure 7 As shown, in Figure 7In this model, the overall recognition process is based on multi-scale temporal feature extraction from RGB images (701), multi-scale temporal feature extraction based on skeletal coordinates (702), and fusion classification (703). Using RGB images (71) and human skeletal coordinates (72) as inputs, spatiotemporal features are extracted from both inputs, and adaptive fusion is performed on the extracted spatiotemporal features. RGB images (71) are used to extract spatiotemporal features representing the appearance of human movement, while skeletal coordinates (72) are used to reduce the impact of changes in clothing and background on recognition. The two complement and fuse with each other to achieve better performance. The core of this model lies in multi-scale feature extraction and adaptive attention fusion. Multi-scale feature extraction captures video segments that are more critical for classification, while adaptive attention fusion distinguishes features with stronger discriminative power from the input.

[0144] In the RGB image feature extraction part, after extracting spatial features from the image using the Inception V3 network 7011, the video is divided into 1 segment, 2 segments, and 4 segments at three scales 7012. Then, a bidirectional LSTM network 7013 is used to extract the spatiotemporal features of each video segment. At scales two and three, adaptive attention 7014 is used to capture video segments that are more beneficial for classification and increase their weights during fusion. Finally, the spatiotemporal features from the three scales are fused using an element-wise addition method.

[0145] In the skeletal coordinates section, the skeletal coordinates are also divided into 1 segment, 2 segments, and 4 segments using three scales (7021). Then, adaptive attention (7022) is used to calculate the weights of different segments for classification, and these weights are used to weight the skeletal coordinates. Finally, the skeletal coordinates from the three scales are concatenated, and spatiotemporal features are extracted using CNN (7023).

[0146] In the fusion process, the RGB spatiotemporal features and skeletal spatiotemporal features obtained are fused using an adaptive attention network 7031, increasing the weight of the more beneficial parts for classification. This results in superior fused spatiotemporal features, thereby improving overall recognition performance.

[0147] In this embodiment, the behavior recognition method can be implemented through the following six steps:

[0148] Step 1: Acquire RGB video and extract video frames from the video frames at uniform intervals to obtain an RGB video sequence. Where N is the number of frames extracted, f o Let represent the o-th frame, and i represent the i-th video.

[0149] Step two: Obtain human skeleton coordinates. Use a pose prediction network to process video frames to obtain human skeleton coordinates, using f... oTo calculate the human skeleton coordinates e, use the input. o Obtain the human skeleton coordinates from video data

[0150] Step 3: Input the video sequence obtained in Step 1 into the multi-scale temporal feature extraction based on RGB images to extract the spatiotemporal features FeaRGB of the RGB images. i Similarly, i represents the i-th sample.

[0151] Step four: Input the human skeleton coordinate sequence obtained in step two into the multi-scale temporal feature extraction based on skeleton coordinates to extract the spatiotemporal features of the human skeleton coordinates. i .

[0152] Step 5: Using the features of the two modalities obtained in Steps 3 and 4 as input, construct an adaptive attention fusion network to fuse the features of the two modalities, and obtain the fused feature FeaWeight of the two modalities. i .

[0153] Step 6: After obtaining the fused spatiotemporal features from Step 5, construct a fully connected network and use Softmax as the activation function to classify the fused features.

[0154] In some embodiments, step three is implemented as follows:

[0155] By extracting temporal features from coarse to fine at different scales and employing adaptive attention to assign weights to different video subsequences, the overall performance of the network is improved through multi-scale temporal features and the weight allocation of the adaptive attention mechanism. The network first divides the video data into multiple video subsequence segments at multiple scales, and then uses a bidirectional LSTM network to extract temporal features from these segments at multiple scales. Next, an adaptive attention mechanism is used to assign different weights to these segments at different scales, with higher weights assigned to segments that have a greater impact on the recognition result. By combining the adaptive attention mechanism and the multi-scale temporal feature extraction network, the network focuses on video segments that are more important for classification. Finally, the temporal features from multiple scales are fused to obtain the final multi-scale temporal features. Figure 8 The topmost time scales 81, 82, and 83 indicate that the video sequence is divided into 1, 2, and 4 video subsequences using three time scales, respectively.

[0156] refer to Figure 8 First, Formula 1 can be used to extract the spatial features of video frames through the Inception V3 network 801, with each frame outputting a dimension of 1×1024. After extracting the spatial features of the video, Formula 2 can be used to divide the video sequence into 1, 2, and 4 video subsequence segments according to three time scales.

[0157] FSpa i =Inncep(S i (1);

[0158]

[0159] Among them, S i Let S be the sequence of the i-th sample video frame in the video; Inncep(·) is the sequence of S extracted by Inception V3. i Spatial features obtained FSpa i Sp(·) represents a video sequence that divides a video frame sequence into three scales.

[0160] Time scale one uses a bidirectional LSTM 802 to extract temporal features from the entire video, resulting in spatiotemporal features with a dimension of 1*1024. Time scale two divides the video into two video sub-sequence segments. Using formula 3, a bidirectional LSTM 803 is used to extract temporal features from the two segments, resulting in spatiotemporal features with a dimension of 2*1*1024. Then, using formula 4, the spatiotemporal features of the two sub-sequence segments are fused through an adaptive attention mechanism 804, and the fusion weight of the segment that is more important for classification is increased, resulting in fused spatiotemporal features of time scale two with a dimension of 1*1024. Time scale 3 divides the video into four video sub-sequence segments. Formula 3 is used to extract the temporal features of the four segments using a bidirectional LSTM 805, resulting in a spatiotemporal feature with a dimension of 4*1*1024. Then, Formula 4 is used to calculate the weight of the temporal features of the four sub-sequence segments in the classification using an adaptive attention mechanism 806. Finally, the obtained weights are used to fuse the temporal features of the four sub-sequence segments in time scale 3, resulting in a fused spatiotemporal feature of time scale 3 with a dimension of 1*1024.

[0161]

[0162]

[0163] Wherein, LSTM(·) is obtained by extracting the spatiotemporal features of video sequences at three time scales through bidirectional LSTM. i represents the i-th sample video sequence, j represents the segmentation time scale, with values ​​of 1, 2, and 3, and k represents the k-th video segment at time scale j. The adaptive attention network AdaAtten(·) is used to highlight and fuse the key parts in time scale two and time scale three respectively.

[0164] Where the input is XTemp t For example, Formula 4 can be expressed as Formula 4-1, Formula 4-2, and Formula 4-3:

[0165] u t =w AdaAtten ×XTemp t +b AdaAtten (4-1);

[0166] α t =σ(u) t (4-2);

[0167]

[0168] Among them, XTemp t This represents the input of AdaAtten(·), where t represents XTemp. t The t-th channel; W AdaAtten The dimension transformation matrix R is initialized. 1×1024 b AdaAtten The initial bias scalar is σ(·), which is the nonlinear activation function Sigmoid; m represents the input XTemp. t The total number of channels, FeaAdaAtten, is AdaAtten(XTemp) t The output of ).

[0169] Thus, spatiotemporal features at three different time scales were obtained. By assigning different weights to different video subsequence segments, local temporal features that are more critical for classification are highlighted. Finally, Formula 5 can be used to fuse the temporal features at the three time scales by adding them element-wise, resulting in the final multi-scale temporal features:

[0170]

[0171] in, The global spatiotemporal features extracted by the bidirectional LSTM for the i-th sample. The local spatiotemporal features of the i-th sample at time scale two and time scale three after being weighted by the adaptive attention mechanism; This is for performing element-by-element addition on the data.

[0172] In some embodiments, the implementation block diagram of step four is as follows: Figure 9 As shown, the implementation process includes:

[0173] For each video frame, a pose prediction algorithm is used to calculate the joint coordinates (skeleton joint coordinates) of the person in the video. A total of 15 joints are predicted for the person. These joints are then concatenated in frame order to obtain the input for the skeletal feature extraction network. Where T is the number of frames extracted from the video, N is the number of character joints (15), and 2 represents the two dimensions of the skeletal joint coordinates. Referring to Formula 6, the skeletal sequence is divided into 1, 2, and 4 skeletal sequence segments using three scales.

[0174]

[0175] in, The initial skeletal coordinate sequence of the i-th sample; Sp(·) is divided into three scales. Divide into multiple subsequence segments, For the global skeletal sequence below the time scale, It is a two-segment subsequence divided by a time scale. It is a 4-segment subsequence divided into three time scales.

[0176] like Figure 9 As shown, the partitioning result of time scale one (91) represents the overall spatiotemporal features globally. Time scale two (92) divides the skeletal sequence into two sub-sequence segments. Formula 7 can be used to input these two sub-sequence segments into the adaptive attention mechanism (901), assigning different weights to each segment. A larger weight is used to highlight the segment more important for classification. Formula 8 can be used to apply a weighted average to the two sub-sequence segments using the obtained weights. Time scale three (93) divides the skeletal sequence into four sub-sequence segments, capturing key spatiotemporal features at a smaller scale. These four sub-sequence segments are input into the adaptive attention mechanism (902). Formula 9 can be used to assign different weights to the four sub-sequence segments using the adaptive attention mechanism, highlighting the segment more important for classification with a larger weight. Formula 10 can be used to apply a weighted average to the four sub-sequence segments using the obtained weights.

[0177]

[0178]

[0179]

[0180]

[0181] in, Let AdaAtten2(·) be the attention weights of each subsequence in timescale 2 for the i-th sample, calculated by AdaAtten2(·). To compute the attention weights of each subsequence in time scale three for the i-th sample using AdaAtten3(·); To utilize Weighted timescale two-weighted skeletal sequences To utilize Weighted time-scale three-weighted skeletal sequences This involves element-wise multiplication of two data points. Adaptive attention is used in this process. and The calculation formulas can be used as follows: Formula 7-1 and Formula 7-2:

[0182]

[0183] β t =σ(r) t (7-2);

[0184] in, W is the input to AdaAtten2(·) and AdaAtten3(·), where t is the t-th channel of the input (i.e., the t-th skeletal subsequence); SkeAtten The dimension transformation matrix is ​​initialized, with dimension in AdaAtten2(·). In AdaAtten3(·), the dimension is b SkeAtten The initial bias scalar; β t Let σ(·) be the weight of the t-th channel (i.e., the t-th skeletal subsequence), and let σ(·) be the activation function Sigmoid.

[0185] Thus, through time scale one, time scale two, and time scale three, we have obtained 1, 2, and 4 weighted subsequence segments, respectively. Formula 11 can be used to concatenate these 7 weighted subsequence segments along dimension T to obtain... The multi-scale weighted skeletal coordinates are 94; finally, formula 12 can be used to extract the spatiotemporal features contained in the multi-scale weighted skeletal sequence using a convolutional neural network 903.

[0186]

[0187] FeaSke i =Conv(XMultiS i (12);

[0188] Among them, XMultiS i This involves concatenating seven skeletal subsequences of the i-th sample across time scales one, two, and three in the frame-count dimension (i.e., the time dimension); FeaSke i These are the spatiotemporal features extracted using Conv(·) (i.e., convolutional neural networks).

[0189] In some embodiments, the implementation block diagram of step five above is as follows: Figure 10As shown, after obtaining RGB spatiotemporal features 11 and skeletal spatiotemporal features 12, the fusion weights of the two features for each sample are calculated using an adaptive attention fusion network 1001. The fusion weights are then used to weight the RGB spatiotemporal features and skeletal spatiotemporal features respectively, and the weighted RGB spatiotemporal features and skeletal spatiotemporal features are added together to obtain the final fused spatiotemporal features 13.

[0190] By extracting multi-scale temporal features from RGB images and multi-scale spatiotemporal features from skeletal coordinates, the spatiotemporal features of RGB and skeletons, FeaRGB, are captured. i and FeaSke i Both features have dimensionality. Formula 13 can be used to combine the two features into exist Figure 10 In this process, the adaptive attention in Formula 14 is used to calculate the fusion weights suitable for each sample. If it is more suitable to use skeletal feature classification, the weight of skeletal features is increased; if it is more suitable to use RGB features, the weight of RGB part is increased.

[0191] FeaFina i =concat(FeaRGB) i ,FeaSke i (13);

[0192] WeightFina i =softmax(W Adaptive ×FeaFina i +b Adaptive (14);

[0193] Among them, FeaRGB i and FeaSke i For the first i RGB and skeletal multi-scale features of each sample; concat(·) is used to convert FeaRGB i and FeaSke i splicing; W Adaptive The initial adaptive weight matrix b Adaptive WeightFina is the initial bias scalar. i That is, the adaptive attention fusion weights are calculated by the adaptive attention network for the i-th sample (different samples).

[0194] For multi-scale features of RGB and skeleton, Equation 15 can be used to obtain adaptive weights, which are then weighted and summed to obtain the final fused RGB and skeleton features. Classification is then performed based on the fused features.

[0195]

[0196] in, and Let be the weights of the RGB and skeletal multi-scale features of the i-th sample in the fused features. To perform element-wise addition of two data points; FeaWeight i Let be the fusion feature of the i-th sample.

[0197] In this embodiment, the constructed adaptive attention fusion network can calculate the weights of different input channels during classification and increase the weights of beneficial parts during classification. Furthermore, multi-scale temporal feature extraction from RGB images and skeletal coordinates enables the model to focus on action segments that are truly beneficial for classification within long-term action sequences, thereby improving recognition accuracy.

[0198] In related technologies, most networks use global temporal features to identify behaviors in videos, but their ability to extract local key temporal features is weak. However, video sequences for behavior recognition often contain many actions unrelated to the category, so it's necessary to focus on the most critical actions in the video sequence. For example, in a long jump video, if the run-up time before the jump is longer than the jump itself, it's easily misclassified as running. By combining adaptive attention mechanisms and multi-scale temporal features to focus on video segments more important for classification, the discriminative power of the extracted spatiotemporal features is improved.

[0199] Furthermore, most networks use fixed weights to fuse multiple features from all videos, failing to leverage the complementarity of these features. When classifying videos, features suitable for each video should be used. For example, "playing golf" and "playing baseball" achieve higher accuracy using RGB features; therefore, RGB features should have a larger weight in the final fused feature set. Similarly, "running" and "walking" achieve higher accuracy using skeletal features; thus, skeletal features should have a larger weight in the final fused feature set. Different samples face different situations, and using a traditional fixed fusion ratio for all samples cannot effectively solve this problem. The adaptive attention fusion network proposed in this application can adaptively adjust its fusion weights for different video samples to obtain more suitable fusion features, improving the recognition accuracy of the fused features.

[0200] Based on the foregoing embodiments, this application provides a behavior recognition device. The modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by corresponding logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0201] This application embodiment further provides a behavior recognition device. Figure 11 This is a schematic diagram of the composition structure of the behavior recognition device provided in the embodiments of this application, as shown below. Figure 11 As shown, the behavior recognition device 1100 includes:

[0202] The acquisition module 1101 is used to acquire video data to be identified and extract multiple video frames from the video data;

[0203] The segmentation module 1102 is used to segment the multiple video frames according to different scales to obtain segmentation results corresponding to each scale. The segmentation results include first image spatial features and first skeleton data. The module also extracts the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result.

[0204] The determining module 1103 is used to determine the second image spatiotemporal features of the video data based on each first image spatiotemporal feature, and to determine the skeletal spatiotemporal features of the video data based on each first skeletal data.

[0205] The fusion module 1104 is used to fuse the spatiotemporal features of the second image and the spatiotemporal features of the skeleton based on the trained adaptive attention fusion network to obtain the fused spatiotemporal features of the video data.

[0206] The classification module 1105 is used to perform behavior classification processing on the video data based on the fused spatiotemporal features to obtain behavior recognition results.

[0207] In some embodiments, the partitioning module 1102 includes:

[0208] A spatial feature extraction submodule is used to extract spatial features from the plurality of video frames to obtain the second image spatial features of the plurality of video frames.

[0209] The pose estimation submodule is used to perform pose estimation processing on the multiple video frames to obtain the second skeleton data of the multiple video frames.

[0210] The first segmentation submodule is used to segment the second image spatial features according to different scales to obtain the first image spatial features of each group of video segments at different scales;

[0211] The second segmentation submodule is used to segment the second skeleton data according to different scales to obtain the first skeleton data of each group of video segments at different scales.

[0212] In some embodiments, the partitioning module 1102 further includes:

[0213] The first temporal feature extraction submodule is used to extract temporal features from the first image spatial features of each group of video segments to obtain the third image spatiotemporal features of each group of video segments.

[0214] The first determining submodule is used to determine the third image spatiotemporal features of each group of video segments as the first image spatiotemporal features.

[0215] In some embodiments, the determining module 1103 includes:

[0216] The second temporal feature extraction submodule is used to extract temporal features from the second image spatial features of the multiple video frames to obtain the fourth image spatiotemporal features of the multiple video frames.

[0217] The first acquisition submodule is used to acquire a first adaptive attention model corresponding to a first target scale if the segmentation result includes multiple first image spatial features;

[0218] The second determining submodule is used to determine each first weight corresponding to each first image spatiotemporal feature based on the plurality of first image spatiotemporal features and the first adaptive attention model;

[0219] The third determining submodule is used to determine the fused image spatiotemporal features of multiple video frames at the first target scale based on the first weights and the first image spatiotemporal features.

[0220] The fusion submodule is used to perform fusion processing on the spatiotemporal features of the fused image and the spatiotemporal features of the fourth image to obtain the spatiotemporal features of the second image.

[0221] In some embodiments, the determining module 1103 further includes:

[0222] The second acquisition submodule is used to acquire a second adaptive attention model corresponding to the second target scale of the segmentation result if the segmentation result includes multiple first skeleton data.

[0223] The fourth determining submodule is used to determine each second weight of each first skeleton data based on the plurality of first skeleton data and the second adaptive attention model;

[0224] The weighting submodule is used to perform weighting processing on each of the first skeleton data based on each of the second weights to obtain the weighted skeleton data of the multiple video frames;

[0225] The splicing submodule is used to sequentially splice the weighted skeletal data and the second skeletal data and perform feature extraction processing to obtain the spatiotemporal features of the skeleton.

[0226] In some embodiments, the fusion module 1104 includes:

[0227] The fifth determination submodule is used to determine the third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton based on the trained adaptive attention fusion network;

[0228] The sixth determining submodule is used to determine the fusion spatiotemporal features of the video data based on the second image spatiotemporal features, the skeleton spatiotemporal features, the third weight, and the fourth weight.

[0229] In some embodiments, the fifth determining submodule includes:

[0230] The stitching unit is used to stitch together the spatiotemporal features of the second image and the spatiotemporal features of the skeleton to obtain stitched spatiotemporal features.

[0231] The acquisition unit is used to acquire the initialized weight matrix and the initialized bias scalar.

[0232] The first determining unit is used to determine the third weight and the fourth weight based on the splicing spatiotemporal features, the initialized weight matrix, and the initialized bias scalar.

[0233] In some embodiments, the sixth determining submodule further includes:

[0234] The first weighting unit is used to weight the second image spatiotemporal features using the third weight to obtain the weighted image spatiotemporal features;

[0235] The second weighting unit is used to weight the spatiotemporal features of the skeleton using the fourth weight to obtain the weighted spatiotemporal features of the skeleton.

[0236] The second determining unit is used to determine the fused spatiotemporal features based on the weighted image spatiotemporal features and the weighted skeletal spatiotemporal features.

[0237] It should be noted that the description of the behavior recognition device in this application is similar to the description of the method embodiment described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment in this application for understanding.

[0238] It should be noted that, in the embodiments of this application, if the above-described behavior recognition method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0239] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the behavior recognition method provided in the above embodiments.

[0240] This application provides a behavior recognition device. Figure 12 This is a schematic diagram of the composition structure of the behavior recognition device provided in the embodiments of this application, such as... Figure 12 As shown, the behavior recognition device 1200 includes: a processor 1201, at least one communication bus 1202, a user interface 1203, at least one external communication interface 1204, and a memory 1205. The communication bus 1202 is configured to enable communication between these components. The user interface 1203 may include a display screen, and the external communication interface 1204 may include standard wired and wireless interfaces. The processor 1201 is configured to execute a program of a behavior recognition method stored in the memory to implement the behavior recognition method provided in the above embodiment.

[0241] The descriptions of the above embodiments of the behavior recognition device and storage medium are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the behavior recognition device and storage medium of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0242] It should be noted that the descriptions of the storage medium and behavior recognition device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and behavior recognition device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0243] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0246] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0247] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0248] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0249] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an AC to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0250] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A behavior recognition method, characterized in that, The method includes: Acquire the video data to be identified, and extract multiple video frames from the video data; The multiple video frames are divided according to different scales to obtain the division results corresponding to each scale. The division results include first image spatial features and first skeleton data. The first image spatiotemporal features corresponding to the first image spatial features in each division result are extracted. Using the first adaptive attention model of the first target scale corresponding to the division result, determine the first weights corresponding to each first image spatiotemporal feature, and perform weighted fusion processing on each first image spatiotemporal feature based on each first weight to obtain the second image spatiotemporal feature corresponding to the division result. Using the second adaptive attention model corresponding to the second target scale of the segmentation result, determine each second weight based on each first skeleton data, and perform weighted processing on each first skeleton data based on each second weight to obtain the spatiotemporal skeletal features of the video data; Based on the trained adaptive attention fusion network, the third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton are determined. The spatiotemporal features of the second image, the spatiotemporal features of the skeleton, the third weight and the fourth weight are fused to obtain the fused spatiotemporal features of the video data. Based on the fused spatiotemporal features, the video data is subjected to behavior classification processing to obtain behavior recognition results.

2. The method according to claim 1, characterized in that, The multiple video frames are divided according to different scales to obtain the division results corresponding to each scale, including: Spatial features are extracted from the plurality of video frames to obtain the second image spatial features of the plurality of video frames; Pose estimation processing is performed on the multiple video frames to obtain the second skeleton data of the multiple video frames; The second image spatial features are divided according to different scales to obtain the first image spatial features of each group of video segments at different scales; The second skeleton data is divided according to different scales to obtain the first skeleton data of each group of video segments at different scales.

3. The method according to claim 2, characterized in that, The extraction of the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result includes: Temporal features are extracted from the first image spatial features of each group of video segments to obtain the third image spatiotemporal features of each group of video segments; The spatiotemporal features of the third image of each group of video segments are determined as the spatiotemporal features of the first image.

4. The method according to claim 1, characterized in that, The step involves determining each first weight corresponding to each first image spatiotemporal feature using a first adaptive attention model corresponding to the first target scale of the segmentation result, and performing weighted fusion processing on each first image spatiotemporal feature based on each first weight to obtain the second image spatiotemporal feature corresponding to the segmentation result, including: Temporal features are extracted from the second image spatial features of the multiple video frames to obtain the fourth image spatiotemporal features of the multiple video frames; If the partitioning result includes multiple first image spatial features, obtain the first adaptive attention model corresponding to the first target scale of the partitioning result; Based on the multiple spatiotemporal features of the first image and the first adaptive attention model, determine the first weights corresponding to each spatiotemporal feature of the first image. Based on the first weights and the first image spatiotemporal features, the fused image spatiotemporal features of multiple video frames at the first target scale are determined. The spatiotemporal features of the fused image and the spatiotemporal features of the fourth image are fused to obtain the spatiotemporal features of the second image.

5. The method according to claim 2, characterized in that, The second adaptive attention model, corresponding to the second target scale of the segmentation result, determines each second weight based on each first skeleton data, and performs weighted processing on each first skeleton data based on each second weight to obtain the spatiotemporal skeletal features of the video data, including: If the segmentation result includes multiple first skeleton data, obtain the second adaptive attention model corresponding to the second target scale of the segmentation result; Based on the multiple first skeleton data and the second adaptive attention model, determine each second weight of each first skeleton data; The weighted data of the first skeleton is obtained by weighting the first skeleton data based on the second weights. The weighted skeletal data and the second skeletal data are sequentially spliced ​​and feature extracted to obtain the spatiotemporal features of the skeleton.

6. The method according to claim 1, characterized in that, The trained adaptive attention fusion network determines a third weight corresponding to the second image spatiotemporal features and a fourth weight corresponding to the skeletal spatiotemporal features. It then performs fusion processing based on the second image spatiotemporal features, the skeletal spatiotemporal features, the third weight, and the fourth weight to obtain the fused spatiotemporal features of the video data, including: The third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton are determined based on the trained adaptive attention fusion network. The fusion spatiotemporal features of the video data are determined based on the second image spatiotemporal features, the skeleton spatiotemporal features, the third weight, and the fourth weight.

7. The method according to claim 6, characterized in that, The method for determining the third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton based on the trained adaptive attention fusion network includes: The spatiotemporal features of the second image and the spatiotemporal features of the skeleton are spliced ​​together to obtain spliced ​​spatiotemporal features; Obtain the initialized weight matrix and the initialized bias scalar; Based on the splicing spatiotemporal features, the initialized weight matrix, and the initialized bias scalar, the third weight and the fourth weight are determined.

8. The method according to claim 6, characterized in that, The step of determining the fused spatiotemporal features of the video data based on the second image spatiotemporal features, the skeleton spatiotemporal features, the third weight, and the fourth weight includes: The second image spatiotemporal features are weighted using the third weight to obtain the weighted image spatiotemporal features; The spatiotemporal features of the skeleton are weighted using the fourth weight to obtain the weighted spatiotemporal features of the skeleton. The fused spatiotemporal features are determined based on the weighted image spatiotemporal features and the weighted skeleton spatiotemporal features.

9. A behavior recognition device, characterized in that, The behavior recognition device includes: The acquisition module is used to acquire the video data to be identified and extract multiple video frames from the video data; The segmentation module is used to segment the multiple video frames according to different scales to obtain segmentation results corresponding to each scale. The segmentation results include first image spatial features and first skeleton data. The module also extracts the first image spatiotemporal features corresponding to the first image spatial features in each segmentation result. The determining module is configured to: determine each first weight corresponding to each first image spatiotemporal feature using a first adaptive attention model at a first target scale corresponding to the segmentation result; perform weighted fusion processing on each first image spatiotemporal feature based on each first weight to obtain a second image spatiotemporal feature corresponding to the segmentation result; and determine each second weight based on each first skeleton data using a second adaptive attention model at a second target scale corresponding to the segmentation result; perform weighted processing on each first skeleton data based on each second weight to obtain the skeleton spatiotemporal feature of the video data. The fusion module is used to determine the third weight corresponding to the spatiotemporal features of the second image and the fourth weight corresponding to the spatiotemporal features of the skeleton based on the trained adaptive attention fusion network, and to perform fusion processing on the spatiotemporal features of the second image, the spatiotemporal features of the skeleton, the third weight and the fourth weight to obtain the fused spatiotemporal features of the video data; The classification module is used to perform behavior classification processing on the video data based on the fused spatiotemporal features to obtain behavior recognition results.

10. A behavior recognition device, characterized in that, The behavior recognition device includes: Processor; and Memory for storing computer programs that can run on the processor; When the computer program is executed by a processor, it implements the behavior recognition method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions configured to perform the behavior recognition method according to any one of claims 1 to 8.

Images