A human action recognition method and device based on a boundary-sensitive network

Abstract

The application relates to a human action recognition method and device based on a boundary-sensitive network and a computer device. The method comprises the following steps: acquiring video data, performing frame extraction processing on the video data, and obtaining target image data; the target image data is image data of a target object performing an action; time sequence position feature data of the target object is constructed based on a human posture estimation model, the target image data and action category data; the time sequence position feature data is classified and processed according to a sequence model, so that an initial classification result corresponding to the target image data is obtained; and the target action category of the target object is determined according to the initial classification results. The method can improve the efficiency of the human action recognition method.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for human action recognition based on boundary-sensitive networks. Background Technology

[0002] In various applications such as human-computer interaction, robot data, and game control, it is necessary to determine the categories of human actions. By using human action recognition methods in the field of computer vision to identify and process videos, the categories of human actions can be obtained, providing a data foundation for multiple applications.

[0003] Current methods for human motion recognition utilize video data consisting of consecutive frames. A 3D convolutional network is used to extract features from each frame, resulting in feature data that is then concatenated into a feature dataset. The 3D convolutional network then performs prediction processing on this feature dataset to obtain a classification result for the motion category.

[0004] However, current human motion recognition methods are inefficient because video data contains a large amount of image data, and the feature extraction and prediction processing of this large amount of image data using 3D convolutional networks consumes a significant amount of time. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, computer equipment, computer-readable storage medium, and computer program product for human motion recognition based on boundary-sensitive networks to address the aforementioned technical problems.

[0006] Firstly, this application provides a human action recognition method based on a boundary-sensitive network, including:

[0007] Acquire video data and perform frame extraction on the video data to obtain target image data; the target image data is image data of the target object performing actions.

[0008] Based on the human pose estimation model, the target image data, and the action category data, the temporal location feature data of the target object is constructed.

[0009] The temporal location feature data is classified according to the sequence model to obtain the initial classification result corresponding to the target image data;

[0010] The target action category of the target object is determined based on the initial classification results.

[0011] In one embodiment, the step of performing frame extraction processing on the video data to obtain target image data includes:

[0012] Image recognition is performed on the video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object; the video data includes image data of each frame;

[0013] Based on the action start sequence and the action end sequence, candidate target image data are filtered from the image data in each frame;

[0014] Target image data is filtered from each candidate target image data based on the action probability sequence and the preset action probability threshold.

[0015] In one embodiment, constructing the temporal location feature data of the target object based on the human pose estimation model, the target image data, and action category data includes:

[0016] Based on the human pose estimation model and the target image data, determine the heat map of each key point corresponding to the target object;

[0017] A class token is constructed based on the heat map of each key point, the target image data, and the action category data;

[0018] By concatenating the class token and each of the key point heatmaps, the temporal location feature data corresponding to the target object is obtained.

[0019] In one embodiment, determining the heatmap of each key point corresponding to the target object based on the human pose estimation model and the target image data includes:

[0020] Based on the human pose estimation model, the key point data of the target object in the target image data are extracted to obtain the key point heatmap corresponding to the target object;

[0021] The heatmap of the key points is divided to obtain each initial heatmap of the key points;

[0022] The initial key point heatmaps are positionally encoded to obtain the key point heatmaps corresponding to the target object.

[0023] In one embodiment, constructing a class token based on each of the keypoint heatmap submaps, the target image data, and the action category data includes:

[0024] Based on the positional relationships between the heatmaps of each key point, determine the positional relationship data;

[0025] Determine the position of the target image data in the video data to obtain the temporal position data of the target image data;

[0026] A class token is constructed based on the location relationship data, the temporal location data, and the action category data; the action category data is the action feature data of the action category to be identified.

[0027] In one embodiment, the step of classifying the temporal location feature data according to a sequence model to obtain an initial classification result corresponding to the target image data includes:

[0028] The temporal location feature data is classified using a sequence model to obtain an output result; the output result includes the probability of action type.

[0029] Determine whether the probability of the action category is greater than or equal to a preset probability threshold, and obtain the determination result;

[0030] Based on the judgment result, the initial classification result corresponding to the target image data is determined.

[0031] In one embodiment, determining the target action category of the target object based on each of the initial classification results includes:

[0032] Each initial classification result is filtered to obtain an initial target classification result; the initial target classification result represents the action category;

[0033] Based on the initial target classification results, determine the proportion of each action category;

[0034] The action category with the highest proportion is identified as the target action category.

[0035] Secondly, this application also provides a human motion recognition device based on a boundary-sensitive network, comprising:

[0036] The first processing module is used to acquire video data and perform frame extraction processing on the video data to obtain target image data; the target image data is image data of the target object performing an action.

[0037] The construction module is used to construct the temporal position feature data of the target object based on the human pose estimation model, the target image data, and the action category data;

[0038] The second processing module is used to classify the temporal location feature data according to the sequence model to obtain the initial classification result corresponding to the target image data;

[0039] The determination module is used to determine the target action category of the target object based on the initial classification results.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0041] Acquire video data and perform frame extraction on the video data to obtain target image data; the target image data is image data of the target object performing actions.

[0042] Based on the human pose estimation model, the target image data, and the action category data, the temporal location feature data of the target object is constructed.

[0043] The temporal location feature data is classified according to the sequence model to obtain the initial classification result corresponding to the target image data;

[0044] The target action category of the target object is determined based on the initial classification results.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0046] Acquire video data and perform frame extraction on the video data to obtain target image data; the target image data is image data of the target object performing actions.

[0047] Based on the human pose estimation model, the target image data, and the action category data, the temporal location feature data of the target object is constructed.

[0048] The temporal location feature data is classified according to the sequence model to obtain the initial classification result corresponding to the target image data;

[0049] The target action category of the target object is determined based on the initial classification results.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0051] Acquire video data and perform frame extraction on the video data to obtain target image data; the target image data is image data of the target object performing actions.

[0052] Based on the human pose estimation model, the target image data, and the action category data, the temporal location feature data of the target object is constructed.

[0053] The temporal location feature data is classified according to the sequence model to obtain the initial classification result corresponding to the target image data;

[0054] The target action category of the target object is determined based on the initial classification results.

[0055] The aforementioned human action recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on boundary-sensitive networks acquire video data and perform frame extraction processing on the video data to obtain target image data; the target image data is image data of a target object performing an action; based on a human pose estimation model, the target image data, and action category data, temporal position feature data of the target object is constructed; the temporal position feature data is classified according to a sequence model to obtain initial classification results corresponding to the target image data; and the target action category of the target object is determined based on each of the initial classification results. Using this method, by performing frame extraction processing on video data to obtain target image data of a target object performing an action, and constructing temporal position feature data of the target object based on a human pose estimation model and action category data, the temporal and positional features of the target object are obtained, reducing the amount of target image data that needs to be processed. Then, the temporal position feature data is classified using a sequence model to obtain initial classification results, and the target action category of the target object is determined based on each of the initial classification results, thus improving the efficiency of the human action recognition method. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating a human motion recognition method based on a boundary-sensitive network in one embodiment.

[0058] Figure 2 This is a schematic diagram of the process for determining target image data in one embodiment;

[0059] Figure 3 This is a flowchart illustrating the process of constructing temporal location feature data in one embodiment;

[0060] Figure 4 This is a flowchart illustrating the process of determining the heat map of each key point in one embodiment;

[0061] Figure 5 This is a schematic diagram illustrating the process of constructing a token in one embodiment;

[0062] Figure 6This is a flowchart illustrating the process of determining the initial classification result in one embodiment;

[0063] Figure 7 This is a flowchart illustrating the process of determining the target action category in one embodiment;

[0064] Figure 8 This is a schematic diagram of the process for determining target image data in one embodiment;

[0065] Figure 9 This is a structural block diagram of a human motion recognition device based on a boundary-sensitive network in one embodiment;

[0066] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In various applications such as human-computer interaction, robot data, and game control, it is necessary to determine the categories of human actions. By using human action recognition methods in the field of computer vision to identify and process video data, the categories of human actions can be obtained, providing a data foundation for multiple applications.

[0069] In traditional technology, computer devices acquire video data. This video data is constructed from a series of consecutive image frames. The computer device inputs all the image data from the video data into a 3D convolutional network. The 3D convolutional network models the temporal information of the video data, obtaining a time series dimension. Based on this time series dimension, features are extracted from each frame of the video data, yielding various feature data. Then, the computer device uses the 3D convolutional network to concatenate these feature data into a feature dataset and performs predictive processing on the feature dataset to obtain the classification result of the action category.

[0070] However, current human motion recognition methods are inefficient because video data contains a large amount of image data, and extracting and predicting features from all the image data in the video using 3D convolutional networks is very time-consuming.

[0071] Based on the aforementioned traditional techniques, this application provides a human action recognition method based on a boundary-sensitive network. By extracting frames from video data, target image data of the target object's actions is obtained, reducing the amount of data that the human pose estimation model and sequence model need to process. Then, the initial classification results of the target image data are determined based on the human pose estimation model and sequence model, and the target action category of the target object is determined based on each initial classification result, thereby improving the efficiency of the human action recognition method.

[0072] In one embodiment, such as Figure 1 As shown, a human motion recognition method based on boundary-sensitive networks is provided. This application uses the application of this method to a computer device as an example for illustration. This application does not limit the execution device of the human motion recognition method based on boundary-sensitive networks, and includes the following steps 102 to 108:

[0073] Step 102: Acquire video data and perform frame extraction on the video data to obtain target image data.

[0074] The target image data consists of image data of the target object during its actions. The video data contains consecutive frame images.

[0075] In practice, the computer equipment acquires the video data that needs to be image recognized. Then, the computer equipment performs frame extraction processing on each frame of image data to obtain the target image data.

[0076] Specifically, the computer equipment and the image acquisition equipment are connected via a network. The image acquisition equipment acquires video data and sends it to the computer equipment via the network. The computer equipment receives the video data. Then, the computer equipment performs image recognition on each frame of the video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object in the image data. Then, based on the action start sequence, action end sequence, and action probability sequence of the target object, the computer equipment determines the image data in each frame of the image data in which the target object performs an action, and identifies the image data in which the target object performs an action as the target image data.

[0077] In one exemplary embodiment, the computer device needs to identify the action category of a target object in video data, thereby providing a data basis for determining whether the target object's behavior is abnormal. The monitoring device collects video data of the target object and sends it to the computer device, which receives the video data. Then, the computer device performs frame extraction processing on the video data based on a BSN (Boundary Sensitive Network) to obtain target image data. Specifically, the computer device inputs the video data into the BSN network, and performs image recognition on each frame of image data in the video data through the BSN network to obtain the action start sequence, action end sequence, and action probability sequence of the target object in the image data. Then, the BSN network in the computer device determines the image data in each frame of image data where the target object is performing an action based on the action start sequence, action end sequence, and action probability sequence, and identifies the image data where the target object is performing an action as the target image data.

[0078] Optionally, the computer device may acquire video data from an image acquisition device or a computer-readable storage medium, and the computer device may acquire video data via a network, Bluetooth, or other means, but this application embodiment does not limit the source and means of the video data.

[0079] Optionally, the target object is generally a person, which can be a child, a man, a woman, etc. The number of target objects can be, but is not limited to, one. This application embodiment does not limit the type or number of target objects.

[0080] Step 104: Construct temporal location feature data of the target object based on the human pose estimation model, target image data, and action category data.

[0081] Among them, the action category data is the action feature data of the action category that needs to be identified.

[0082] In implementation, the computer device determines heatmaps of key points of the target object based on a human pose estimation model and target image data, and constructs class tokens based on these heatmaps, the target image data, and action category data. These class tokens are derived from the class tokens in the deep learning model. The computer device then concatenates the class tokens and the heatmaps of each key point to obtain the temporal location feature data of the target object. This temporal location feature data contains both the location and temporal characteristics of the target object within the video data.

[0083] Specifically, the computer device extracts key point data of the target object from the target image data model based on a human pose estimation model, obtaining heatmaps of each key point. Then, the computer device constructs a class token based on the positional relationships between the heatmaps of each key point, the target image data, and the action category data. The computer device concatenates the class token and the heatmaps of each key point to obtain the temporal positional feature data of the target object.

[0084] Step 106: Classify the temporal location feature data according to the sequence model to obtain the initial classification result corresponding to the target image data.

[0085] In practice, the computer equipment performs image recognition on temporal location feature data using a sequence model to obtain initial output results. Then, the computer equipment preprocesses the initial output results to obtain initial classification results corresponding to the image data.

[0086] Specifically, the computer device performs image recognition processing on the temporal location feature data based on a sequence model to obtain an initial output result. Then, the computer device preprocesses the initial output result using a normalization algorithm and an activation function to obtain an output result. Finally, the computer device determines the initial classification result corresponding to the target image data based on the output result.

[0087] In one exemplary embodiment, the computer device is equipped with multiple trained sequence models, each recognizing an action category. The computer device simultaneously performs image recognition processing on temporal location feature data based on the multiple sequence models, obtaining multiple initial output results. Then, the computer device preprocesses each initial output result according to a normalization algorithm and an activation function to obtain an output result. Finally, the computer device determines the initial classification result corresponding to the target image data based on each output result.

[0088] Step 108: Determine the target action category of the target object based on the initial classification results.

[0089] In practice, the computer device filters the initial target classification results from each initial classification result. Then, the computer device determines the target action category of the target object based on each initial target classification result.

[0090] Specifically, the initial classification results include action categories and non-action categories. The computer device filters the initial classification results to identify the action categories as initial target categories. Then, based on the initial target classification results, the computer device determines the action category with the highest percentage as the target action category.

[0091] In the aforementioned human action recognition method based on boundary-sensitive networks, frame extraction is performed on video data to obtain target image data of the target object's actions. Then, based on a human pose estimation model and action category data, temporal location feature data of the target object is constructed, resulting in both temporal and location features, thus reducing the amount of target image data that needs to be processed. Next, a sequence model is used to classify the temporal location feature data, obtaining initial classification results. Based on these initial classification results, the target action category of the target object is determined, improving the efficiency of the boundary-sensitive network-based human action recognition method.

[0092] In one exemplary embodiment, such as Figure 2 As shown, the specific processing steps for extracting frames from the video data in step 102 to obtain the target image data include steps 202 to 206. Wherein:

[0093] Step 202: Perform image recognition on the video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object.

[0094] The video data includes image data for each frame.

[0095] In implementation, computer equipment performs image recognition on video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object. The action start sequence represents the probability that an action will begin in each frame of the video data, i.e., the action start probability. The action end sequence represents the probability that an action will end in each frame of the video data. The action probability sequence represents the probability that an action exists in each frame of the video data, i.e., the action end probability.

[0096] Specifically, the BSN network includes a temporal evaluation module, a nomination generation module, and a nomination evaluation module. The temporal evaluation module contains three temporal convolutional layers. The computer device inputs video data into the BSN network, and the temporal evaluation module extracts image feature sequences (temporal action regions) from the video data. These image feature sequences include action start sequences, action end sequences, and action probability sequences.

[0097] In one exemplary embodiment, the computer device inputs video data into a BSN network. The three temporal convolutional layers in the BSN network simultaneously model the probability of an action starting, the probability of an action ending, and the probability of an action being performed in each frame of the video data, generating an action start probability sequence, an action end probability sequence, and an action probability sequence, providing a data foundation for the subsequent nomination generation module.

[0098] Step 204: Based on the action start sequence and action end sequence, filter candidate target image data from each frame of image data.

[0099] In implementation, the computer device determines the start probability and end probability of a target action in the action start sequence and action end sequence based on a preset node action probability threshold. Then, the computer device filters candidate target image data from each frame of image data based on the target action start probability and target action end probability.

[0100] Specifically, the BSN network includes a nomination generation module. The computer device has pre-set node action probability thresholds. Based on the nomination generation model, the computer device determines the target action start probability in the action start sequence where the action start probability is greater than the preset node action probability threshold. Simultaneously, based on the nomination generation module, the computer device determines the target action end probability in the action end sequence where the action end probability is greater than the preset node action probability threshold. The computer device determines the first image data corresponding to the target action start probability and the second image data corresponding to the target action end probability in the video data. Then, the computer device uses the image data between the first and second image data in the video data (including the first and second image data) as candidate target image data.

[0101] In one exemplary embodiment, the computer device determines a target action start probability greater than a preset node action probability threshold in the action start sequence based on a nomination generation model. Simultaneously, the computer device determines a target action end probability greater than a preset node action probability threshold in the action end sequence based on a nomination generation module. The nomination generation model in the computer device combines the target action start probability and the target action end probability pairwise to obtain a probability combination. Then, the nomination generation module in the computer device determines candidate target image data in the video data based on the probability combination and generates corresponding feature descriptions for the candidate target image data, namely Boundary-Sensitive Proposal (BSP) features.

[0102] Optionally, the node action probability threshold is determined according to the screening requirements, and may be, but is not limited to, 90% (percentage), etc. This application embodiment does not limit the node action probability threshold.

[0103] Step 206: Filter target image data from each candidate target image data based on the action probability sequence and the preset action probability threshold.

[0104] In implementation, the computer equipment has a pre-set action probability threshold. Based on this threshold, the computer equipment determines the target action probability within the action probability sequence. Then, based on the target action probability, the computer equipment determines the target image data from among the candidate target image data.

[0105] Specifically, the computer device has a pre-set action probability threshold. The computer device uses a BSN network to determine the probability of a target action that is greater than or equal to the action probability threshold from the action probability sequence. Then, the computer device uses the BSN network to determine the target image data corresponding to the target action probability from each candidate target image data.

[0106] In an exemplary embodiment, a computer device inputs video data into a BSN network, performs image recognition on the video sequence in the video data through the BSN network, and simultaneously models the probability of action start, action end, and action in progress at each position in the video sequence, thereby generating an action start probability sequence, an action end sequence, and an action probability sequence. Then, the BSN network in the computer device constructs a temporal action region based on the action start probability sequence, action end sequence, and action probability sequence. The BSN network in the computer device determines a target time period in the time period corresponding to the video data according to a preset rule. The rule is: (1) the action probability in the action probability sequence corresponding to the target time period is greater than the action probability threshold; (2) the action start probability at the beginning of the target time period is higher than the probability of the previous and next moments (i.e., a probability peak); (3) the action end probability at the end of the target time period is higher than the probability of the previous and next moments (i.e., a probability peak). In computer equipment, the BSN network identifies the target video data corresponding to the target time period within the video data and crops the target video data into individual RGB (Red, Green, Blue) frames to obtain the target video data. The target video data represents the region where the target object is most likely to exhibit action.

[0107] Optionally, the action probability threshold is determined based on the filtering requirements of the target image data. The action probability threshold can be, but is not limited to, 85%, etc., and this application embodiment does not limit the action probability threshold.

[0108] In this embodiment, image recognition is performed on video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object. Based on these sequences, each image in the video data is filtered, allowing the human pose estimation model to directly process the target image data where actions occur. This avoids the human pose estimation model processing the entire image data in the video, thus reducing the amount of target image data that the model needs to process and improving the efficiency of the image recognition method.

[0109] In one exemplary embodiment, such as Figure 3 As shown, the specific processing procedure of step 104 includes steps 302 to 306. Wherein:

[0110] Step 302: Based on the human pose estimation model and target image data, determine the heat map of each key point corresponding to the target object.

[0111] In practice, the computer equipment extracts features from the target image data based on a human pose estimation model to obtain a key point heatmap. Then, the computer equipment divides the key point heatmap into sub-maps and processes the data of the initial key point heatmaps to obtain individual key point heatmaps.

[0112] Specifically, the computer device extracts key point data of the target object from the target image data using a human pose estimation model, obtaining a key point heatmap. Then, the computer device divides the key point heatmap into initial key point sub-maps. The computer device performs position encoding processing on each initial key point sub-map to obtain a final key point heatmap.

[0113] In one exemplary embodiment, the human pose estimation model is a ViT Pose (ViTPose: Simple VisionTransformer Baselines for Human Pose Estimation, where Transformer is a neural network structure) model. The computer device extracts keypoint data of the target object from the target image data based on the ViT Pose model. Then, the computer device divides the keypoint heatmap into initial keypoint heatmaps and performs position encoding on each initial keypoint heatmap to obtain the keypoint heatmap.

[0114] Step 304: Construct class tokens based on the heat map of each key point, target image data, and action category data.

[0115] In implementation, the computer device determines positional relationship data based on the positional relationships between the heatmaps of each key point. Then, the computer device constructs a class token based on the target image data's location within the video data, positional relationship data, and action category data. This class token contains the temporal and positional characteristics of the target object.

[0116] Specifically, the computer device determines the positional relationship between each keypoint heatmap and other keypoint heatmaps, and generates positional relationship data based on this relationship. Then, based on the position of the target image data in the video data, the computer device generates temporal positional data of the target image data. The computer device constructs class tokens based on each keypoint heatmap, the target image data, and the action category data, thereby obtaining global information for each keypoint heatmap.

[0117] Step 306: Piece together the tokens and the heatmaps of each key point to obtain the temporal location feature data corresponding to the target object.

[0118] In practice, the computer device stitches together the class token and the heat map of each key point to obtain the time-series feature data corresponding to the target object.

[0119] In one exemplary embodiment, the class token is a class token in a deep learning model that represents global information of the input vector. Therefore, the computer device concatenates the class tokens to the first dimension of the heatmap of each key point to obtain the temporal feature data of the target object.

[0120] In this embodiment, a human pose estimation model is used to extract features from the target image data, resulting in heatmaps of key points of the target object, thus clarifying the positional features of the target object. Then, by constructing class tokens and concatenating the class tokens with the heatmaps of key points, temporal positional feature data corresponding to the target object is obtained, clarifying both the positional and temporal features of the target object. This adds temporal features, and based on these temporal positional features, the target action category is determined, improving the accuracy of the target action category.

[0121] In one exemplary embodiment, such as Figure 4 As shown, the specific processing procedure of step 302 includes steps 402 to 406. Wherein:

[0122] Step 402: Extract key point data of the target object from the target image data based on the human pose estimation model to obtain the key point heatmap corresponding to the target object.

[0123] The human pose estimation model is the ViT Pose model. Keypoint data includes the location and confidence scores of keypoints. In human pose estimation, keypoints refer to specific parts of the human body used to describe pose, such as the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0124] In practice, the computer equipment inputs the target image data into the ViT Pose model. The ViT Pose model then performs image recognition on the target object in the target image data and extracts the key point data of the target object to obtain a key point heatmap. The key point heatmap represents the posture information of the human body.

[0125] Specifically, the computer device inputs the target image data into the ViT Pose model. The ViT Pose model performs image recognition on the target object in the target image data and extracts 17 keypoint data points of the target object, thereby constructing a keypoint heatmap corresponding to the target object based on these 17 keypoint data points. The ViT Pose model is a pre-trained ViT Pose model containing a multi-layer Transformer structure. The pre-trained ViT Pose model using a multi-layer Transformer structure accurately and efficiently predicts the human keypoint heatmap of the target image.

[0126] In one exemplary embodiment, the computer device contains multiple frames of target image data. For each frame of target image data, the computer device inputs the target image data into the ViT Pose model, performs image recognition on the target object in the target image data through the ViT Pose model, and extracts 17 key point data of the target object, thereby constructing a key point heatmap corresponding to the target object based on the 17 key point data.

[0127] Step 404: Divide the heat map of key points to obtain the initial heat map of each key point.

[0128] In implementation, the computer equipment is pre-set with a heatmap format. The computer equipment divides the key point heatmap according to the heatmap format to obtain each initial key point heatmap.

[0129] In an exemplary embodiment, the computer device contains multiple key point heatmaps. For each key point heatmap, the computer device divides the key point heatmap according to a heatmap submap format to obtain initial key point heatmaps corresponding to the key point heatmap.

[0130] Optionally, the heatmap format is determined based on the size of the sequence model and the target image data. The heatmap format can be, but is not limited to, 16px (pixel) * 16px. This embodiment of the application does not limit the heatmap format.

[0131] Step 406: Perform position encoding on each initial key point heat map to obtain the key point heat map corresponding to the target object.

[0132] In implementation, the computer equipment is pre-configured with a location encoding algorithm. Based on this algorithm, the computer equipment encodes the location of each initial keypoint heatmap, thus obtaining the keypoint heatmap corresponding to the target object.

[0133] In one exemplary embodiment, the positional encoding algorithm is an absolute positional encoding algorithm. The computer device performs positional encoding processing on each initial keypoint heatmap corresponding to each keypoint heatmap using the absolute positional encoding algorithm, thereby obtaining the keypoint heatmap corresponding to the target object. Through positional encoding, each keypoint heatmap gains a learnable positional embedding, facilitating the subsequent sequence model's use of the sequence information from the 17 keypoint data in the keypoint heatmap.

[0134] Optionally, the position encoding algorithm can be an absolute position encoding algorithm or a relative position encoding algorithm, etc. The embodiments of this application do not limit the position encoding algorithm.

[0135] In this embodiment, key point data of the target object in the target image data are extracted by the human pose estimation model to obtain a key point heatmap. The key point heatmap is then divided and its location is encoded to obtain each key point heatmap sub-map, clarifying the positional relationship between each key point, which facilitates image recognition by the subsequent sequence model.

[0136] In one exemplary embodiment, such as Figure 5 As shown, the specific processing procedure of step 304 includes steps 502 to 506. Wherein:

[0137] Step 502: Determine the positional relationship data based on the positional relationship between the heat maps of each key point.

[0138] In implementation, the computer equipment determines the positional relationship between each key point heatmap and other key point heatmaps in each key point heatmap, and generates positional relationship data based on the positional relationship.

[0139] In an exemplary embodiment, the positional relationship characterizes the relative position and direction between two keypoint heatmaps. The computer device includes multiple keypoint heatmaps corresponding to keypoint heatmaps. For each keypoint heatmap corresponding to a keypoint heatmap, the computer device determines the positional relationship between each keypoint heatmap and other keypoint heatmaps. This positional relationship is the relative direction and relative position of the keypoint heatmap and other keypoint heatmaps within the keypoint heatmap. The computer device generates positional relationship data based on these positional relationships.

[0140] Step 504: Determine the position of the target image data in the video data to obtain the temporal position data of the target image data.

[0141] In practice, computer equipment determines the temporal position of the target image data in the video data, thus obtaining the temporal position data of the target image data.

[0142] Specifically, the computer device determines the time of the target image data within the video data, thereby generating temporal location data of the target image data. This time refers to the minute and second of the target image data within the video data, or the specific frame of the target image data within the target video data. For example, the target image data may be at the 2nd minute and 30th second of the video data, or at the 240th frame of the video data.

[0143] Step 506: Construct class tokens based on location relationship data, temporal location data, and action category data.

[0144] Among them, the action category data is the action feature data of the action category that needs to be identified.

[0145] In practice, the computer device acquires action category data. Then, the computer device combines the location relationship data, temporal location data, and action category data to obtain a class token.

[0146] Specifically, when a computer device needs to identify the target action category of a target object in video data, the user determines the action category to be identified and inputs the action category data into the computer device. The computer device acquires the action category data. Then, the computer device concatenates the positional relationship data, temporal position data, and action category data to obtain a class token.

[0147] For example, a user determines that the action category to be identified for the target object is falling, and inputs the fall data into a computer device. The computer device acquires the fall data and concatenates it with positional relationship data, temporal position data, and fall data to obtain a class token.

[0148] In an exemplary embodiment, when a user needs to identify multiple action categories of a target user, the user inputs action category data for all multiple action categories into a computer device. The computer device acquires the action category data for each category. Then, for each action category, the computer device concatenates the action category data, location relationship data, and temporal feature data together to obtain a class token. Next, the computer device concatenates each class token with each keypoint heatmap to obtain the temporal location feature data for each action category. Based on the action category, the computer device determines the sequence model corresponding to each temporal location feature data and, based on the sequence model and the temporal location feature data, determines the initial classification result of the target image data for that action category.

[0149] In this embodiment, class tokens are constructed using location relationship data, temporal location data, and action category data, clarifying the global information of each key point heatmap submap. Furthermore, by adding temporal location information to the class tokens, the global information is enriched. By constructing temporal location feature data based on class tokens, the accuracy of the temporal location feature data is improved, thereby enhancing the accuracy of the human action recognition method based on boundary-sensitive networks.

[0150] In one exemplary embodiment, such as Figure 6 As shown, the specific processing procedure of step 106 includes steps 602 to 606. Wherein:

[0151] Step 602: Classify the temporal location feature data using a sequence model to obtain the output results.

[0152] The output includes the probability of the action type.

[0153] In implementation, the computer equipment inputs temporal location feature data into a sequence model. The sequence model then performs image recognition of the target object based on the temporal location feature data, obtaining an initial output classification result. Next, the computer equipment standardizes the initial output result using a normalization algorithm and maps the standardized initial output result using an activation function to obtain the final output result.

[0154] Specifically, the sequence model is a Transformer model (a deep neural network model based on a self-attention mechanism), which contains a six-layer Transformer structure. The normalization algorithm is the Normalization layer of the neural network, and the mapping function is the Softmax layer. The computer device inputs temporal location feature data into the Transformer model. The Transformer model extracts the positions and connections between each key point of the target object from the temporal location feature data, and performs image recognition based on the positions and connections to obtain the initial output result. The six-layer Transformer structure (Transformer self-attention mechanism) here can better extract the interrelationships between the 17 key points of the target object and achieve a more accurate recognition effect. The computer device inputs the initial output result into the Normalization layer, which normalizes the initial output result to obtain a standardized initial output result. Then, the computer device inputs the standardized initial output result into the Softmax layer, which maps the standardized initial output result to obtain the output result.

[0155] For example, the initial output has a ratio of 200:800 between action categories and non-action categories. The computer device inputs this initial output into a Normalization layer, which normalizes the output, resulting in a standardized output with a ratio of 2:8 between action categories and non-action categories. Then, the computer device inputs this standardized output into a Softmax layer, which maps the output to a sum of 1 between the ratios of action categories and non-action categories, resulting in an output with a ratio of 0.2:0.8 between action categories and non-action categories.

[0156] In one exemplary embodiment, a computer device is provided with a sequence model for recognizing multiple action categories, and the computer device contains temporal location feature data for multiple action categories. For each temporal location feature data, the computer device determines the corresponding sequence model based on the action category of the temporal location feature data. Then, the computer device inputs the temporal location feature data into the corresponding sequence model, and performs image recognition of the target object using the sequence model to obtain an initial output classification result. This initial output classification result characterizes whether the action performed by the target object belongs to that action category.

[0157] For example, a computer device contains sequence models for recognizing falls and normal walking, and also contains temporal positional feature data for both falls and normal walking. The computer device determines that the sequence model corresponding to the temporal positional feature data of a fall is the fall recognition sequence model, and inputs the fall temporal positional feature data into the fall recognition sequence model. The fall recognition sequence model then performs image recognition on this temporal positional feature data to obtain an initial output result. This initial output result characterizes whether the action performed by the target object in the video data constitutes a fall. Similarly, the computer device determines that the sequence model corresponding to the temporal positional feature data of normal walking is the normal walking sequence model, and inputs the normal walking temporal positional feature data into the normal walking sequence model. The normal walking sequence model then performs image recognition on this temporal positional feature data to obtain an initial output result.

[0158] Step 604: Determine whether the probability of the action category is greater than or equal to a preset probability threshold, and obtain the determination result.

[0159] In the implementation of the red-light district rule, a probability threshold is preset in the computer equipment. The computer equipment determines whether the probability of an action category is greater than or equal to the preset probability threshold, and obtains a judgment result. If the probability of the action category is greater than or equal to the probability threshold, the computer equipment determines the judgment result as the first judgment result. If the probability of the action category is less than the probability threshold, the computer equipment determines the judgment result as the second judgment result.

[0160] In one exemplary embodiment, the action category is falling. The computer device determines whether the probability of falling is greater than or equal to a preset probability threshold. If the probability of falling is greater than or equal to the probability threshold, the computer device determines the determination result as a first determination result. If the probability of falling is less than the probability threshold, the computer device determines the determination result as a second determination result.

[0161] In an optional embodiment, a probability threshold can be set for each action category, or a single probability threshold can be uniformly set for all action categories. If a probability threshold is set for each action category, the computer device determines the probability threshold corresponding to the action category and judges whether the probability of the action category is greater than or equal to the probability threshold.

[0162] Optionally, the probability threshold can be set to, but is not limited to, 80%, and is determined according to the image recognition requirements. This application embodiment does not limit the probability threshold.

[0163] Step 606: Determine the initial classification result corresponding to the target image data based on the judgment result.

[0164] In practice, if the judgment result is the first judgment result, it indicates that the probability of the target object performing an action of that type in the target image data is relatively high. Therefore, the computer device determines the initial classification result corresponding to the target image data as the action category. If the judgment result is the second judgment result, it indicates that the probability of the target object performing an action of that type in the target image data is relatively low. Therefore, the computer device determines the initial classification result corresponding to the target image data as not belonging to that action category, meaning that the target object did not perform an action of that type in the target image data.

[0165] In an exemplary embodiment, the action category is falling. If the judgment result is a first judgment result, it indicates that the probability of the target object falling in the target image data is relatively high. Therefore, the computer device determines the initial classification result corresponding to the target image data as falling. If the judgment result is a second judgment result, it indicates that the probability of the target object falling in the target image data is relatively low. Therefore, the computer device determines the initial classification result corresponding to the target image data as not falling.

[0166] In this embodiment, the temporal location feature data is classified using a sequence model to obtain the output result. Based on the output result, the initial classification result of the target image data is determined, which preliminarily clarifies the action of the target object in the target image data, making it easier to determine the target action type in the subsequent process.

[0167] In one exemplary embodiment, such as Figure 7 As shown, the specific processing procedure of step 108 includes steps 702 to 706. Wherein:

[0168] Step 702: Filter each initial classification result to obtain each initial target classification result.

[0169] The initial target classification result represents the action category.

[0170] In practice, the computer device determines the initial classification result that represents the action category as the initial target classification result among the initial classification results.

[0171] Specifically, for each initial classification result, the computer device determines whether the initial classification result represents an action category. If the initial classification result represents an action category, the computer device determines the initial classification result as the initial target classification result.

[0172] Step 704: Determine the proportion of each action category based on the initial target classification results.

[0173] In implementation, the computer device determines the action category represented by each initial target classification result. Then, the computer device determines the proportion of each action category based on each action category.

[0174] In one exemplary embodiment, the computer device contains five initial target classification results: Initial Target Classification Result A, Initial Target Classification Result B, Initial Target Classification Result C, Initial Target Classification Result D, and Initial Target Classification Result E. Initial Target Classification Result A represents the action of falling; Initial Target Classification Result B represents the action of falling; Initial Target Classification Result C represents the action of falling; Initial Target Classification Result D represents the action of falling; and Initial Target Classification Result E represents the action of normal walking. The computer device determines that the proportion of normal walking is 20% and the proportion of falling is 80%.

[0175] Step 706: Determine the action category with the highest proportion as the target action category.

[0176] During implementation, the computer device determines the action category with the highest proportion among all proportions. Then, the computer device designates that action category as the target action category.

[0177] In one exemplary embodiment, the proportion of normal walking is 20%, and the proportion of falling is 80%. The computer device determines that the action category with the highest proportion is falling. Then, the computer device determines falling as the target action category corresponding to the target object.

[0178] In one exemplary embodiment, the computer device votes on the initial classification results of each target image data to obtain the proportion of each action category. Then, the computer device determines the target action category based on the action category with the highest proportion. This target action category characterizes the type of action performed by the target object in the video data, and is also the action classification of the video data.

[0179] In an optional embodiment, the computer device determines whether the proportion of each action category exceeds a preset action category proportion threshold. If the proportion of an action category exceeds the action category proportion threshold, the computer device identifies that action category as an initial target action category. If the number of initial target action categories is 1, the computer device identifies that initial target action category as the target action category. If the number of initial target action categories is greater than 1 or is 0, the computer device identifies the action category with the highest proportion as the target action category.

[0180] Optionally, the action category ratio threshold can be set to 50%, but is not limited to 50%, and is determined according to the image recognition requirements. This application embodiment does not limit the action category ratio threshold.

[0181] In this embodiment, the target action type of the target object is determined by the proportion of action types represented by each initial classification result, thus clarifying the action performed by the target object in the video data. Furthermore, determining the target action type based on the initial classification results corresponding to continuous target image data is more accurate than determining the target action type based on single frames or discontinuous frames, thereby improving the accuracy of the human action recognition method based on boundary-sensitive networks.

[0182] In one exemplary embodiment, a human action recognition method based on a boundary-sensitive network is provided, such as... Figure 8 As shown. Figure 8 This is a schematic diagram illustrating the processing flow of a human action recognition method based on a boundary-sensitive network in another embodiment. The human action recognition method based on a boundary-sensitive network includes:

[0183] Step 801: Frame extraction is performed on the video data through the BSN network to obtain the target image data.

[0184] Step 802: Based on the ViT Pose model, feature extraction is performed on the target image data to obtain a key point heatmap.

[0185] Step 803: Divide the key point heatmap and perform position embedding on the divided key point heatmap to obtain each key point heatmap submap.

[0186] Step 804: Construct a class token and combine the class token with the heatmaps of each key point to obtain temporal location feature data.

[0187] Step 805: Perform image recognition on the temporal location feature data based on the Transformer model to obtain the initial output results.

[0188] Step 806: Process the initial output results based on the Normalization layer and the Softmax layer to obtain the output results, and determine the initial classification result (class) based on the output results.

[0189] Step 806: Vote on each initial classification result to obtain the target action category (final class) of the target object.

[0190] This boundary-sensitive network-based human action recognition method effectively utilizes multi-frame target image data for human action recognition and detection, achieving high precision and recall. Compared to other multi-frame methods that may suffer from a loss of precision due to incomplete consecutive frames being used for judgment, this boundary-sensitive network-based human action recognition method can utilize all video frames containing the specified action category for action recognition, thus improving the accuracy and efficiency of boundary-sensitive network-based human action recognition.

[0191] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0192] Based on the same inventive concept, this application also provides a boundary-sensitive network-based human motion recognition device for implementing the aforementioned boundary-sensitive network-based human motion recognition method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more image recognition device embodiments provided below can be found in the above-described limitations of the boundary-sensitive network-based human motion recognition method, and will not be repeated here.

[0193] In one exemplary embodiment, such as Figure 9 As shown, a human motion recognition device 900 based on a boundary-sensitive network is provided, comprising: a first processing module 901, a construction module 902, a second processing module 903, and a determination module 904, wherein:

[0194] The first processing module 901 is used to acquire video data and perform frame extraction processing on the video data to obtain target image data; the target image data is the image data of the target object performing actions.

[0195] Module 902 is used to construct temporal location feature data of the target object based on the human pose estimation model, target image data and action category data.

[0196] The second processing module 903 is used to classify the temporal location feature data according to the sequence model to obtain the initial classification result corresponding to the target image data.

[0197] The determination module 904 is used to determine the target action category of the target object based on the initial classification results.

[0198] In an exemplary embodiment, the first processing module 901 includes a first acquisition submodule and a first processing submodule. The first processing submodule is used for:

[0199] The first recognition submodule is used to perform image recognition on video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object; the video data contains image data of each frame.

[0200] The first filtering submodule is used to filter candidate target image data from each frame of image data based on the action start sequence and the action end sequence.

[0201] The second filtering submodule is used to filter target image data from each candidate target image data based on the action probability sequence and the preset action probability threshold.

[0202] In one exemplary embodiment, the construction module 902 includes:

[0203] The first determination submodule is used to determine the heat map of each key point of the target object based on the human pose estimation model and the target image data.

[0204] The first construction submodule is used to construct class tokens based on the heat map of each key point, target image data, and action category data.

[0205] The first splicing submodule is used to splice tokens and heatmaps of key points to obtain temporal location feature data corresponding to the target object.

[0206] In one exemplary embodiment, the first determining submodule includes:

[0207] The first extraction submodule is used to extract key point data of the target object in the target image data based on the human pose estimation model, and obtain the key point heatmap corresponding to the target object.

[0208] The first partitioning submodule is used to partition the heatmap of key points to obtain each initial heatmap of key points.

[0209] The first encoding submodule is used to encode the position of each initial key point heat map to obtain the key point heat map corresponding to the target object.

[0210] In one exemplary embodiment, the first construction submodule includes:

[0211] The second determination submodule is used to determine the positional relationship data based on the positional relationship between the heat maps of each key point.

[0212] The third determination submodule is used to determine the position of the target image data in the video data and obtain the temporal position data of the target image data.

[0213] The second construction submodule is used to construct class tokens based on location relationship data, temporal location data, and action category data; the action category data is the action feature data of the action category to be identified.

[0214] In one exemplary embodiment, the second processing module 903 includes:

[0215] The second processing submodule is used to classify the temporal location feature data using a sequence model to obtain an output result; the output result includes the probability of the action type.

[0216] The first judgment submodule is used to determine whether the probability of the action category is greater than or equal to a preset probability threshold, and obtain the judgment result.

[0217] The fourth determination submodule is used to determine the initial classification result corresponding to the target image data based on the judgment result.

[0218] In one exemplary embodiment, the determining module 904 includes:

[0219] The third filtering submodule is used to filter each initial classification result to obtain each initial target classification result; the initial target classification result represents the action category.

[0220] The fifth determination submodule is used to determine the proportion of each action category based on the initial target classification results.

[0221] The sixth determination submodule is used to determine the action category with the highest proportion as the target action category.

[0222] The modules in the aforementioned human motion recognition device based on boundary-sensitive networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0223] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a human motion recognition method based on a boundary-sensitive network. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0224] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0225] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0226] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0227] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0228] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0230] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A human action recognition method based on boundary-sensitive networks, characterized in that, The method includes: Acquire video data and perform frame extraction on the video data to obtain target image data; the target image data is image data of the target object performing actions. Based on the human pose estimation model, the target image data, and the action category data, the temporal location feature data of the target object is constructed; the temporal location feature data includes class tokens and heat maps of each key point; the action category data is the action feature data of the action category to be identified. The temporal location feature data is classified according to the sequence model to obtain the initial classification result corresponding to the target image data; The target action category of the target object is determined based on the initial classification results. The construction of temporal location feature data of the target object based on the human pose estimation model, the target image data, and action category data includes: Based on the positional relationships between the heatmaps of each key point, positional relationship data is determined; the positional relationship data represents the relative position and relative direction between the heatmaps of each key point; the heatmaps of each key point are determined based on the human posture estimation model and the target image data. Determine the position of the target image data in the video data to obtain the temporal position data of the target image data; the temporal position data is the frame number or timestamp of the target image data in the video data. For each action category to be identified, the action category data, the positional relationship data, and the temporal position data corresponding to the action category to be identified are concatenated to construct a class token corresponding to the action category to be identified; the class token represents the global information of the input vector. By concatenating each of the aforementioned token types with the respective keypoint heatmaps, the temporal location feature data corresponding to each action category is obtained.

2. The method according to claim 1, characterized in that, The step of extracting frames from the video data to obtain the target image data includes: Image recognition is performed on the video data to obtain the action start sequence, action end sequence, and action probability sequence of the target object; the video data includes image data of each frame; Based on the action start sequence and the action end sequence, candidate target image data are filtered from the image data in each frame; Target image data is filtered from each candidate target image data based on the action probability sequence and the preset action probability threshold.

3. The method according to claim 1, characterized in that, The construction of temporal location feature data of the target object based on the human pose estimation model, the target image data, and action category data includes: Based on the human pose estimation model and the target image data, determine the heat map of each key point corresponding to the target object; A class token is constructed based on the heat map of each key point, the target image data, and the action category data; By concatenating the class token and each of the key point heatmaps, the temporal location feature data corresponding to the target object is obtained.

4. The method according to claim 3, characterized in that, The step of determining the heatmap of each key point corresponding to the target object based on the human pose estimation model and the target image data includes: Based on the human pose estimation model, the key point data of the target object in the target image data are extracted to obtain the key point heatmap corresponding to the target object; The heatmap of the key points is divided to obtain each initial heatmap of the key points; The initial key point heatmaps are positionally encoded to obtain the key point heatmaps corresponding to the target object.

5. The method according to claim 3, characterized in that, The construction of class tokens based on the heatmaps of each key point, the target image data, and the action category data includes: Based on the positional relationships between the heatmaps of each key point, determine the positional relationship data; Determine the position of the target image data in the video data to obtain the temporal position data of the target image data; A class token is constructed based on the location relationship data, the temporal location data, and the action category data; the action category data is the action feature data of the action category to be identified.

6. The method according to claim 1, characterized in that, The step of classifying the temporal location feature data according to the sequence model to obtain the initial classification result corresponding to the target image data includes: The temporal location feature data is classified using a sequence model to obtain an output result; the output result includes the probability of action type. Determine whether the probability of the action category is greater than or equal to a preset probability threshold, and obtain the determination result; Based on the judgment result, the initial classification result corresponding to the target image data is determined.

7. The method according to claim 1, characterized in that, Determining the target action category of the target object based on each of the initial classification results includes: Each initial classification result is filtered to obtain an initial target classification result; the initial target classification result represents the action category; Based on the initial target classification results, determine the proportion of each action category; The action category with the highest proportion is identified as the target action category.

8. A human motion recognition device based on a boundary-sensitive network, characterized in that, The device includes: The first processing module is used to acquire video data and perform frame extraction processing on the video data to obtain target image data; the target image data is image data of the target object performing an action. The construction module is used to construct the temporal position feature data of the target object based on the human pose estimation model, the target image data, and the action category data; the temporal position feature data includes class tokens and heat maps of each key point; the action category data is the action feature data of the action category to be identified; The second processing module is used to classify the temporal location feature data according to the sequence model to obtain the initial classification result corresponding to the target image data; A determination module is used to determine the target action category of the target object based on the initial classification results. The construction module is specifically used for: determining positional relationship data based on the positional relationships between the heatmaps of each key point; the positional relationship data characterizes the relative position and relative direction between the heatmaps of each key point; the heatmaps of each key point are determined based on the human pose estimation model and the target image data; determining the position of the target image data in the video data to obtain the temporal position data of the target image data; the temporal position data is the frame number or timestamp of the target image data in the video data; for each action category to be identified, concatenating the action category data, the positional relationship data, and the temporal position data corresponding to the action category to be identified to construct a class token corresponding to the action category to be identified; the class token characterizes the global information of the input vector; concatenating each class token with each heatmap of each key point to obtain the temporal position feature data corresponding to each action category.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images