A method, device and storage medium for detecting a specific action

Abstract

The present disclosure provides a specific action detection method and device, equipment and storage medium, relates to computer vision, action recognition and other artificial intelligence technical fields, especially to the behavior detection field. The specific implementation scheme is: obtaining a video to be detected; performing target tracking on image frames in the video to be detected to obtain a human tracking box and a human tracking ID of the human tracking box; performing skeleton point data extraction on a human image corresponding to the human tracking box to obtain skeleton point data of the human tracking ID; for a human tracking ID requiring specific action detection, performing specific action recognition on continuous multiple frame skeleton point data of the human tracking ID based on a specific action recognition model to obtain a recognition result. On the basis of being applicable to specific action detection in a multi-target scene, the obtained recognition result is also more accurate, the accuracy and universality of specific action detection are improved, and the specific action detection can be realized by deploying a camera, which has high deployment friendliness.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence technologies such as computer vision and action recognition, and in particular to the field of behavior detection technology. Background Technology

[0002] In daily life or work, due to reasons such as bodily dysfunction or loss of balance, people may make certain movements, such as falling, lying flat, jumping, or spinning.

[0003] Taking falls as an example, if a person falls, it may indicate a personal safety problem or an accident. In order to provide timely warnings of such personal safety problems and dangerous accidents and reduce personal injury, an efficient and accurate method for detecting specific movements is of great significance. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for detecting specific actions.

[0005] According to a first aspect of this disclosure, a method for detecting a specific action is provided, comprising:

[0006] Obtain the video to be tested;

[0007] Target tracking is performed on the image frames in the video to be detected to obtain a human body tracking box and the human body tracking ID of the human body tracking box;

[0008] Skeletal point data is extracted from the human image corresponding to the human tracking frame to obtain the skeletal point data of each human tracking ID;

[0009] For human tracking IDs that require specific action detection, a pre-trained specific action recognition model is used to perform specific action recognition on multiple consecutive frames of skeletal point data of the human tracking ID to obtain the recognition results.

[0010] According to a second aspect of this disclosure, a method for training a specific action recognition model is provided, comprising:

[0011] Obtain sample videos;

[0012] Target tracking is performed on the image frames in the sample video to obtain the human body tracking box and the human body tracking ID of the human body tracking box;

[0013] Skeletal point data is extracted from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID;

[0014] The specific action recognition model is trained based on the skeletal point data.

[0015] According to a third aspect of this disclosure, a specific action recognition model provides a device for detecting a specific action, comprising:

[0016] The acquisition module is used to acquire the video to be detected;

[0017] The tracking module is used to perform target tracking on image frames in the video to be detected, and obtain a human body tracking box and the human body tracking ID of the human body tracking box;

[0018] The extraction module is used to extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0019] The recognition module is used to identify specific actions of human tracking IDs that require specific action detection. Based on a pre-trained specific action recognition model, it performs specific action recognition on multiple consecutive frames of skeletal point data of the human tracking ID and obtains the recognition result.

[0020] According to a fourth aspect of this disclosure, a training apparatus for a specific action recognition model is provided, comprising:

[0021] The acquisition module is used to acquire sample videos;

[0022] The tracking module is used to perform target tracking on image frames in the sample video to obtain a human body tracking box and the human body tracking ID of the human body tracking box;

[0023] The extraction module is used to extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0024] The training module is used to train the specific action recognition model based on the skeletal point data.

[0025] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0026] At least one processor; and

[0027] A memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a detection method for a specific action.

[0029] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform a specific action, and a detection method thereof is provided.

[0030] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, performs a method for detecting specific actions.

[0031] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0032] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0033] Figure 1 This is a flowchart illustrating a specific action detection method provided in an embodiment of this disclosure;

[0034] Figure 2 This is a schematic diagram of the process for specific action recognition provided in the embodiments of this disclosure;

[0035] Figure 3 This is a flowchart illustrating the training method for a specific action recognition model provided in this embodiment of the disclosure;

[0036] Figure 4 This is a block diagram of an apparatus for implementing a detection method for a specific action according to embodiments of the present disclosure;

[0037] Figure 5 This is a block diagram of an apparatus for implementing a training method for a specific action recognition model according to embodiments of the present disclosure;

[0038] Figure 6 This is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] In daily life or work, due to reasons such as bodily dysfunction or loss of balance, people may make certain movements, such as falling, lying flat, or jumping.

[0041] Taking falls as an example, if a person falls in an industrial production setting, it may indicate personal safety issues such as gas poisoning or dizziness, and may also lead to accidents in dangerous operating environments.

[0042] In order to provide timely warnings of such dangerous accidents, a method for efficiently and accurately detecting specific actions is of great significance.

[0043] Taking falls as an example, current specific action detection algorithms can be mainly divided into the following three categories:

[0044] Firstly, methods based on multimodal environmental sensors.

[0045] The method based on multimodal environmental sensors mainly relies on multiple sensors installed indoors to perceive multimodal information such as sound and pressure, and monitors fall actions by fusing information from multiple sensors.

[0046] However, methods based on multimodal environmental sensors rely on sensors placed in specific locations for fall detection, which lacks deployment friendliness and versatility.

[0047] Secondly, methods based on wearable sensors.

[0048] Wearable sensor-based methods place small sensors such as accelerometers and gyroscopes at human joints to acquire motion information from multiple joints.

[0049] Thresholding and machine learning are two commonly used methods for processing joint motion information in wearable devices.

[0050] Among them, the threshold method performs simple preprocessing of the motion information of wearable devices and then directly judges the fall action by setting a threshold. This method has low computational load and fast operation speed, and is easy to deploy directly to wearable devices. However, the setting of the threshold is extremely difficult and it is hard to generalize to different scenarios.

[0051] Machine learning-based methods extract features from wearable devices to determine fall actions, using common machine learning algorithms such as support vector machines and K-nearest neighbors for binary classification reasoning.

[0052] Methods based on wearable sensors also rely on sensors placed in specific locations for fall detection, and also suffer from problems of deployment friendliness and versatility.

[0053] Third, computer vision-based methods.

[0054] Computer vision-based methods collect visual data solely through a camera, extract features using deep learning algorithms, and train a classification network to recognize fall actions.

[0055] Taking image-based computer vision methods as an example, image-based methods ignore the temporal information of human actions and only use a single frame image to determine whether the human body is in a fallen state. They usually directly train target detection algorithms based on RGB (R (red), G (green), B (blue), a color mode) images or keypoint pose classification algorithms based on pose estimation.

[0056] Computer vision-based methods are highly deployment-friendly because they rely solely on camera data. However, algorithms based on single-frame information ignore temporal dynamics, leading to decreased accuracy. Furthermore, methods based on RGB image data are susceptible to changes in environment and lighting, significantly limiting their versatility.

[0057] To address the aforementioned problems, this disclosure provides a method, apparatus, electronic device, and storage medium for detecting a specific action.

[0058] Figure 1 This is a flowchart illustrating a specific action detection method provided in an embodiment of this disclosure, such as... Figure 1 As shown, the method includes the following steps:

[0059] Step S101: Obtain the video to be tested.

[0060] The specific action detection method provided in this disclosure can specifically detect specific actions of the human body in the video to be detected.

[0061] The specific action can be an action that is different from the normal standing or walking of a human body. For example, the specific action can include: falling, lying flat, jumping, running, spinning, etc. The embodiments of this disclosure do not limit the scope of the specific action.

[0062] In this embodiment of the invention, by training a specific action recognition model, a specific action can be recognized, and the recognition result is whether the specific action is performed.

[0063] As an example, when a specific action is falling, a fall detection model is trained to identify the human body, and the result is: whether the human body has fallen.

[0064] As another example, when the specific action is rotation, a rotation action detection model is trained to identify the human body, and the identification result is: whether the human body performs a rotation action.

[0065] In one or more embodiments of this disclosure, a method for detecting a specific action will be specifically described using falling as an example of a specific action.

[0066] Specifically, the video to be detected can be captured by cameras deployed in various scenarios. For example, video captured by surveillance cameras deployed in industrial production scenarios can be used to detect specific actions of workers during production, or video captured by indoor cameras can be used to detect specific actions of elderly people indoors.

[0067] Step S102: Perform target tracking on the image frames in the video to be detected to obtain the human body tracking box and the human body tracking ID of the human body tracking box.

[0068] It should be understood that the video to be detected includes multiple temporally consecutive image frames, and a certain human body in the video to be detected may appear in multiple image frames.

[0069] As an example, image frames to be detected can be obtained from the video to be detected at certain time intervals, for example, one image frame can be taken every 10 frames.

[0070] The target detection box of a human body in one or more image frames is the human body tracking box of that human body, and each human body tracking box corresponds to the human body tracking ID of that human body. In other words, each human body tracking ID corresponds to a specific human body in the video to be detected.

[0071] As can be seen, the human tracking ID can identify which specific human body in the video to which the human tracking box belongs, and the human tracking box of the human tracking ID can represent the temporal motion trajectory of the corresponding human body.

[0072] Taking three image frames as an example, if human body A corresponds to target detection boxes 1, 2, and 3 in the three image frames respectively, then 1, 2, and 3 are the human body tracking boxes of human body A, and the human body tracking ID can be denoted as a.

[0073] Step S103: Extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0074] Specifically, in each image frame, the human tracking bounding box identifies the area where a human body appears in that frame, thus marking the location of the human body in the image. Therefore, skeletal point data can be extracted from the human body image in the corresponding image frame based on the human tracking bounding box, and the extracted skeletal point data corresponds to the human tracking ID of the human tracking bounding box.

[0075] Step S104: For a human tracking ID that requires specific action detection, perform specific action recognition on the continuous multi-frame skeletal point data of the human tracking ID based on a pre-trained specific action recognition model to obtain the recognition result.

[0076] The human tracking ID that needs to be detected for specific actions can be the human tracking ID of any human body in the video to be detected. This embodiment does not limit the specific scope.

[0077] In the aforementioned step S103, the skeletal point data of each human tracking ID in the image frame corresponding to the human tracking box were extracted. Therefore, multi-frame skeletal point data with temporal order can be obtained, so that the spatial and temporal features of the skeletal point data can be combined to perform specific action recognition for the human tracking ID.

[0078] The identification results can include whether a specific action was performed or not.

[0079] As an example, if the tracking bounding boxes of the human body tracking ID denoted as 'a' in the three image frames are 1, 2, and 3 respectively, then the skeletal point data can be extracted from each of the three image frames. Based on the temporal sequence of the skeletal point data in the three frames, the recognition result can be obtained, for example, 'a' did not perform a specific action.

[0080] Specifically, multiple consecutive frames of skeletal point data can be input into a pre-trained specific action recognition model, which will then output the recognition result. This disclosure does not specifically limit the scope of the specific action recognition model.

[0081] Compared to methods for detecting specific actions based on single-frame skeletal point data, this embodiment processes multiple consecutive frames of skeletal point data, resulting in more accurate recognition results. Taking a fall as an example, if a person is in a prone position, detection methods based on single-frame skeletal point data can easily misidentify it as a fall. However, this embodiment, by considering both the spatial and temporal characteristics of the skeletal point data, can avoid misidentifying a static prone position as a dynamic fall.

[0082] The specific action detection method provided in this disclosure involves target tracking of image frames in a video to be detected, obtaining human body tracking boxes and human body tracking IDs for each tracking box, and extracting skeletal point data from the human image corresponding to each tracking box to obtain skeletal point data for each tracking ID. This method is applicable to extracting skeletal point data for each target in a multi-target scene and distinguishes different targets based on the human body tracking ID. In the process of specific action recognition based on skeletal point data, for the human body tracking ID requiring specific action detection, specific action recognition is performed based on multiple consecutive frames of skeletal point data for that human body tracking ID, yielding a recognition result. The use of temporally sequential multi-frame skeletal point data in specific action recognition captures the dynamic features of human movements. In addition to being applicable to specific action detection in multi-target scenes, the obtained recognition results are more accurate, improving the accuracy and versatility of specific action detection.

[0083] Furthermore, the specific action detection method provided in this disclosure uses video to be detected captured by a camera to perform specific action detection, and a single camera can also achieve specific action detection, which has high deployment friendliness.

[0084] In one embodiment of this disclosure, the aforementioned step S102 may specifically include the following refined steps:

[0085] Perform target detection on the current image frame in the video to be detected to obtain the target detection box of the current image frame;

[0086] The target detection box is matched with the human tracking trajectory of the human tracking ID in the previous image frame to determine the human tracking box in the current image frame and the human tracking ID of the human tracking box.

[0087] Specifically, target detection can be performed on the current image frame using a target detection algorithm. This disclosure does not limit the specific algorithm used.

[0088] As an example, object detection can be performed on the current frame image using a pre-trained human detection model PPYOLOE (a neural network model).

[0089] PPYOLOE is a single-stage object detection model that extracts features through the Backbone module, fuses shallow geometric localization information and deep semantic information through the Neck module, and outputs detection boxes of three different sizes and category labels through the Head module.

[0090] Specifically, PPYOLOE's Backbone module is composed of stacked RepResBlock (a neural network structure). This network structure utilizes the idea of ​​structural reparameterization. During training, it uses a multi-branch and skip connection multi-feature map multi-receptor field fusion structure to improve the detection accuracy of training. During inference, RepResBlock can degenerate into a single-path structure with the same effect to save memory and significantly improve inference speed.

[0091] PPYOLOE's Neck network module employs both FPN (a neural network architecture) and PAN (a neural network architecture) structures. Its purpose is to better integrate the features provided by the backbone, thereby improving network performance. FPN is a top-down downsampling network that transmits strong semantic features; it has a large receptive field and abstract features, which is beneficial for classification. PAN is a bottom-up upsampling network that transmits strong localization information, which is beneficial for bounding box localization.

[0092] During training, PPYOLOE's Head module uses the TAL (Task Alignment Learning) algorithm for dynamic label allocation. It uses a hybrid metric of classification confidence and localization IoU (Intersection over Union, a performance parameter) as the benchmark for positive and negative sample allocation. For each bounding box, the top-k predicted boxes are selected as candidates, and it is guaranteed that these top-k candidate points are within the bounding box, thus solving the problem of misalignment between classification and localization branches.

[0093] As mentioned earlier, the video to be detected includes multiple temporally consecutive image frames. The target detection boxes in any two consecutive frames can be matched to obtain the tracking trajectory of the human tracking ID across those two frames. Then, the target detection box in the next image frame is matched against the tracking trajectory to determine the human tracking box and human tracking ID in that next image frame. The same principle applies to target tracking in subsequent image frames. Therefore, this embodiment only illustrates the target tracking steps using the determination of the human tracking box and human tracking ID in the current image frame as an example.

[0094] Taking three consecutive image frames as an example, the first image frame contains target detection boxes 1 and 2, the second image frame contains target detection boxes 3 and 4, and the third image frame, i.e. the current image frame, contains target detection boxes 5 and 6.

[0095] Wherein, the human tracking ID denoted as a has human tracking trajectories of 1 and 3 in the first two image frames, i.e., the preceding image frames. If the target detection box 5 in the current image frame matches 3, then the human tracking ID of the target detection box 5 is a, and the human tracking trajectory of a is 1, 3, 5.

[0096] The human tracking ID denoted as b has a human tracking trajectory of 2,4 in the previous image frame. If the target detection box 6 in the current image frame matches 4, then the human tracking ID of the target detection box 6 is b, and the human tracking trajectory of b is 2,4,6.

[0097] For example, in the process of matching the target detection box with the human body tracking trajectory in the preceding image frame, the matching can be based on the similarity between the target detection box and the human body tracking trajectory.

[0098] Following the foregoing example in the embodiments of this disclosure, if the similarity between target detection boxes 5 and 3 is greater than a predetermined threshold, then target detection boxes 5 and 3 can be considered a match, thereby determining the human tracking ID of target detection box 5 as a, and the human tracking trajectory of a as 1,3,5. The embodiments of this disclosure do not limit the specific calculation of similarity; for example, it can be the similarity between features within the target detection box.

[0099] This embodiment matches the target detection box in the current image frame with the human body tracking trajectory in the previous image frame to determine the human body tracking box in the current image frame and the human body tracking ID of each human body tracking box, which has high accuracy in target tracking.

[0100] In one embodiment of this disclosure, the aforementioned step of matching the target detection box with the human tracking trajectory of the human tracking ID in the previous image frame to determine the human tracking box and the human tracking ID of the human tracking box in the current image frame may specifically include the following sub-steps:

[0101] Step 11: Based on the confidence scores of the target detection boxes, classify the target detection boxes into high-scoring boxes and low-scoring boxes;

[0102] Step 12: Filter the human tracking box in the previous image frame to obtain the tracking trajectory prediction box;

[0103] Step 13: For the high-resolution bounding box, match the high-resolution bounding box with the tracking trajectory prediction box to determine the human tracking box in the high-resolution bounding box and the human tracking ID of the human tracking box.

[0104] Step 14: For the low-resolution bounding box, match the low-resolution bounding box with the remaining tracking trajectory prediction boxes to determine the human tracking box in the low-resolution bounding box and the human tracking ID of the human tracking box.

[0105] Step 15: Identify the remaining high-resolution bounding boxes as human tracking boxes and create human tracking IDs for them.

[0106] Here, confidence score is a parameter of each target detection box in the target detection process. When the confidence score of a target detection box is greater than a preset confidence threshold, such as 0.5, the object in the target detection box is considered a positive sample, i.e., the human body in this embodiment of the present disclosure. When the confidence score of a target detection box is not greater than the preset confidence threshold, the object in the target detection box is considered a negative sample, i.e., the background.

[0107] In other words, the confidence score of a bounding box can be understood as the probability that the object within the bounding box is a human body.

[0108] However, the target detection box with a confidence score lower than the confidence threshold may still contain a human body. For example, the human body in the target detection box may be occluded by an obstacle in the corresponding image frame, resulting in a low confidence score.

[0109] When matching the target detection box in the current image frame with the human tracking trajectory in the previous image frame, if the target detection box with a low confidence score is directly discarded, the human tracking trajectory obtained after matching may be incomplete. This may reduce the accuracy of the recognition result when extracting skeletal point data and performing specific action recognition based on the skeletal point data.

[0110] To address this issue, in this embodiment of the disclosure, the target detection boxes are divided into high-scoring boxes and low-scoring boxes based on their confidence scores, and the high-scoring boxes and low-scoring boxes are matched separately.

[0111] This disclosure does not specifically limit the method of dividing high-scoring and low-scoring boxes. For example, a high-scoring box can be a target detection box with a confidence score greater than 0.5, while a low-scoring box is a target detection box with a confidence score not greater than 0.5.

[0112] In this embodiment of the disclosure, the human body tracking box in the previous preceding image frame is filtered to obtain the tracking trajectory prediction box.

[0113] After obtaining the tracking trajectory prediction box, the high-resolution box and the tracking trajectory prediction box are matched. The human tracking ID is determined for the successfully matched high-resolution box and the corresponding tracking trajectory is updated.

[0114] After matching the high-resolution bounding boxes with the perceived trajectory prediction boxes, the low-resolution bounding boxes and the remaining tracking trajectory prediction boxes are matched, that is, the low-resolution bounding boxes and the tracking trajectory prediction boxes that have not been matched with the high-resolution bounding boxes. For the successfully matched low-resolution bounding boxes, the human tracking ID is determined and the corresponding tracking trajectory is updated.

[0115] For high-scoring bounding boxes that do not match the tracking trajectory, a new human tracking ID and tracking trajectory are created for them.

[0116] Taking three consecutive image frames as an example, the first image frame contains human tracking boxes 1 and 2 with confidence scores of 0.9 and 0.8 respectively, and human tracking IDs a and b respectively. The second image frame contains human tracking boxes 3 and 4 with confidence scores of 0.9 and 0.4 respectively, and human tracking IDs a and b respectively. The third image frame, i.e. the current frame, contains target detection boxes 5, 6, and 7 with confidence scores of 0.9, 0.1, and 0.8 respectively.

[0117] After filtering the human tracking boxes 3 and 4, the predicted tracking trajectories for 3 and 4 can be obtained respectively.

[0118] First, match the high-resolution bounding boxes 5 and 7 with the predicted tracking trajectory bounding boxes. If the predicted tracking trajectory bounding boxes 5 and 3 are successfully matched, the human tracking ID of the high-resolution bounding box 5 is obtained as a, and the tracking trajectory is (1,3,5).

[0119] Then, the low-resolution bounding box 6 is matched with the remaining tracking trajectory prediction boxes. If the tracking trajectory prediction boxes of 6 and 4 are successfully matched, the human tracking ID of the low-resolution bounding box is obtained. The human tracking ID is b, and the tracking trajectory is (2,4,6).

[0120] If the high-resolution bounding box 7 is not matched successfully, a new human body tracking ID with the ID c can be created.

[0121] When matching object detection boxes and tracking trajectory prediction boxes, matching can be based on similarity. For example, the similarity between object detection boxes and tracking trajectory prediction boxes in terms of features, or the similarity between object detection boxes and tracking trajectory prediction boxes in terms of location.

[0122] This example shows that if low-scoring boxes are discarded directly during target tracking, target detection boxes 4 and 6 will be identified as human tracking boxes containing the background instead of 'b', resulting in missing tracking trajectories.

[0123] During target tracking, when a target is occluded, the confidence score will slowly decrease from high to low. Before the target is occluded, the confidence score of the target detection box is high, which can successfully determine the human tracking ID and tracking trajectory.

[0124] This embodiment processes low-resolution bounding boxes and high-resolution bounding boxes separately. It uses the matching between low-resolution bounding boxes and tracking trajectory prediction boxes to determine low-resolution detection boxes containing human bodies. Even after the target is occluded, a connection can still be established between the low-resolution bounding box and the previously determined human body tracking ID and tracking trajectory, ensuring the integrity of the human body tracking trajectory. This results in higher accuracy when performing subsequent skeletal point extraction and specific action recognition.

[0125] In one embodiment of this disclosure, the aforementioned step of filtering the human tracking box in the previous preceding image frame to obtain the tracking trajectory prediction box may specifically include the following refined steps:

[0126] Step 21: Obtain the state variables and confidence scores of the human body tracking box in the previous preceding image frame; wherein, the state variables of the human body tracking box include the center point coordinates, length value, and width value of the human body tracking box;

[0127] Step 22: For the state variables of the human body tracking box, determine the covariance matrix of the state variables, and smooth the covariance matrix based on the confidence score of the human body tracking box to obtain a smoothing matrix;

[0128] Step 23: Based on the prediction equation, process the state variables of the human tracking box and the corresponding smoothing matrix in the previous preceding image frame to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and obtain the tracking trajectory prediction box.

[0129] Specifically, this embodiment uses a Kalman filter to filter the human body tracking box in the previous preceding image frame, and improves the filtering algorithm used by the traditional Kalman filter.

[0130] Among them, filtering process is to predict the state variables of the target, such as predicting the position or velocity of the target in the next frame.

[0131] During the filtering process, the target's state variables at a certain time, such as time t, are known. These state variables may include multiple dimensions, such as position and velocity, and there is a correlation between state variables of different dimensions. This correlation can be characterized based on the covariance matrix of each dimension of the state variables. The filtering process can then predict the target's state variables at time t+1 based on the state variables at time t and the covariance matrix.

[0132] In this embodiment of the disclosure, the state quantity of the human body tracking box in the previous image frame in the next frame, that is, the state quantity of the tracking trajectory prediction box, is predicted, and the tracking trajectory prediction box can be determined based on the state quantity.

[0133] The state variables of the target at time t+1 can be calculated using the prediction equation. For details on the prediction equation, please refer to the relevant techniques on Kalman filters.

[0134] It is worth noting that, in this embodiment of the disclosure, the matrix used in the calculation of the prediction equation is not the original covariance matrix, but a smoothed matrix after smoothing.

[0135] In this embodiment, when filtering the human tracking bounding box in the previous frame to obtain the tracking trajectory prediction box, (x, y, w, h), i.e., the center point coordinates x, y, width w, and length h of the human tracking bounding box and / or the tracking trajectory prediction box, are used as state variables. This allows for adaptation to drastic changes in the aspect ratio of the target detection box when the human performs specific actions. Furthermore, the confidence score of the human tracking bounding box is used to smooth the covariance matrix, reducing noise interference during the prediction process. This improves the accuracy of the tracking trajectory prediction box obtained through filtering, resulting in more accurate target tracking when matching the tracking trajectory prediction box and the target detection box.

[0136] In one embodiment of this disclosure, the aforementioned step S103 may specifically include:

[0137] For a human tracking ID, a human image is obtained based on the human tracking bounding box of that human tracking ID, and the human image is processed using a pose estimation model to obtain a key point heatmap. The key point heatmap is then integrated to obtain the skeletal point data of that human tracking ID.

[0138] As mentioned above, the human tracking bounding box for each human tracking ID can include the human tracking bounding box of that human tracking ID in one or more image frames, and the position of each human tracking bounding box represents the position of the human body in the corresponding image frame.

[0139] Therefore, based on the human tracking bounding box of each human tracking ID, the corresponding original image frame can be cropped to obtain the human image of that human tracking ID, and then the pose estimation model can be used to process the human image.

[0140] As an example, during the cropping process of the original image frame, it can be determined whether the image frame tracks the human body. If not, the step of processing the corresponding human body image using a pose estimation model can be skipped.

[0141] As an example, the pose estimation model could be an HRNet (High-Resolution Network, a type of neural network model).

[0142] When processing human images using the HRNet model, a fixed-size human image can be input into the HRNet model. The output of the HRNet model is multiple keypoint heatmaps, each corresponding to a human skeletal point. For example, there could be 17 keypoint heatmaps, and the human skeletal points could include joints such as the knee and shoulder. By integrating and processing these keypoint heatmaps, the skeletal point data corresponding to the human image can be obtained.

[0143] As an example, the image frames obtained from the video to be detected can be uniformly processed into 640×640 size input images. After cropping the image frames to obtain human images, the cropped human images are preprocessed to a size of 256×192 while maintaining the aspect ratio, and then input into the HRNet model for processing.

[0144] Specifically, the HRNet model connects high-resolution to low-resolution subnets in parallel, which can maintain the high-resolution representation and perform repeated multi-scale fusion of low-resolution and high-resolution at the same depth and level, thus obtaining more accurate key point heatmaps, i.e., more accurate skeleton point data.

[0145] In this embodiment of the disclosure, a human image is first obtained based on a human tracking box, and then skeletal point data is extracted based on the human image. The extraction efficiency is higher, and the obtained skeletal point data is more accurate. Therefore, when performing specific action recognition based on the skeletal point data in the future, the recognition results are more accurate.

[0146] In one embodiment of this disclosure, after obtaining the skeletal point data of the human tracking ID, the method further includes:

[0147] If any human tracking ID is lost in a consecutive preset number of image frames, that human tracking ID is identified as the human tracking ID that needs to be detected for a specific action.

[0148] The loss of a human tracking ID in an image frame can be understood as the absence of a human tracking bounding box for that human tracking ID in that image frame, meaning that the human tracking ID does not have skeletal point data in that image frame.

[0149] As an example, the preset number can be 2. If any human tracking ID is lost in two consecutive image frames, the human tracking ID is determined as the human tracking ID that needs to be detected for a specific action. Based on the skeletal point data of the human tracking ID in consecutive frames, the specific action is recognized.

[0150] As an example, in a multi-target scenario, the missing human tracking IDs in a predetermined number of consecutive image frames can be determined through the following steps:

[0151] The human tracking ID is cached for skeletal point data in each image frame;

[0152] For the current image frame, determine whether the human tracking ID in the temporary ID set is lost in the current frame, and identify the lost human tracking ID as the human tracking ID that needs to be detected for specific actions;

[0153] Determine whether the human tracking ID from the previous frame is lost in the current frame, and store the lost human tracking ID into a temporary ID set.

[0154] Specifically, before caching the skeletal point data, the temporary ID set can be created as an empty set.

[0155] Once a lost human tracking ID is identified as one that requires specific action detection, and the specific action detection for that human tracking ID is completed, the human tracking ID can be deleted from the temporary ID set.

[0156] If any human tracking ID is lost in multiple consecutive image frames, the human body corresponding to that human tracking ID may have already performed a specific action or left the area where the camera is deployed. Therefore, if any human tracking ID is lost in a preset number of consecutive image frames, that human tracking ID is identified as the human tracking ID that needs to be detected for a specific action. This allows for timely detection of specific actions of human bodies that may be performing specific actions, thus improving the real-time performance of specific action detection.

[0157] In one embodiment of this disclosure, after obtaining the skeletal point data of the human tracking ID, the method further includes:

[0158] Cache the skeletal point data of the human tracking ID;

[0159] When the skeleton point data cache of any human tracking ID reaches the first preset frame number, the human tracking ID is determined as the human tracking ID that needs to be detected for a specific action.

[0160] For human tracking IDs that require specific action detection, specific action recognition is performed based on multiple consecutive frames of skeletal point data of the human tracking ID. After obtaining the recognition result, the process also includes:

[0161] Delete the cached skeletal point data of the first second preset number of frames for the human tracking ID; the second preset number of frames is not greater than the first preset number of frames.

[0162] As an example, the first preset frame count is 5, and the second preset frame count is 2. When the skeletal point data of any human tracking ID reaches 5 frames, specific action recognition is performed based on the 5 frames of skeletal point data of that human tracking ID. After the recognition is completed, the skeletal point data of the first 2 frames of that human tracking ID is displayed. Thereafter, the skeletal point data cache of that human tracking ID is updated in the form of a sliding window.

[0163] In this embodiment of the disclosure, when the skeleton point data cache of any human tracking ID reaches a first preset number of frames, the human tracking ID is determined as the human tracking ID that needs to be detected for a specific action, and the skeleton point data cache of the human tracking ID is updated in a sliding window manner, which can improve the real-time performance of the specific action detection algorithm and save cache.

[0164] In one embodiment of this disclosure, each frame of skeletal point data includes data from multiple joint nodes;

[0165] Figure 2 This is a schematic diagram of the specific action recognition process provided in the embodiments of this disclosure, such as... Figure 2 As shown, the aforementioned steps for identifying specific actions in human tracking IDs that require specific action detection, based on a pre-trained specific action recognition model, by performing specific action recognition on multiple consecutive frames of skeletal point data of the human tracking ID to obtain the recognition result, can specifically include the following detailed steps:

[0166] Step S201: For a human tracking ID that requires specific motion detection, determine a spatiotemporal feature map based on the continuous multi-frame skeletal point data of the human tracking ID; the spatiotemporal feature map includes the skeletal point data of each frame of the human tracking ID.

[0167] Step S202: For each root node in the skeletal point data of each frame of the human tracking ID, the set of adjacent nodes of the root node is partitioned according to the distance between the joint node and the centroid, to obtain a root node subgraph, a centripetal subgraph composed of centripetal nodes, and an eccentric subgraph composed of eccentric nodes. The root node subgraph, centripetal subgraph, and eccentric subgraph each correspond to a learnable weight matrix; wherein, the distance between the centripetal node and the centroid is less than the distance between the root node and the centroid; the distance between the eccentric node and the centroid is greater than the distance between the root node and the centroid; the centroid represents the average coordinates of all joint nodes in the skeletal point data of that frame.

[0168] Step S203: Based on the node features, adjacency matrix, weight matrix, and global mask of the convolutional layer of the root node subgraph, perform graph convolution processing on the root node subgraph to obtain the first spatial features of the root node.

[0169] Step S204: Based on the node features, adjacency matrix, weight matrix and global mask of the centripetal subgraph, perform graph convolution processing on the centripetal subgraph to obtain the second spatial features of the root node;

[0170] Step S205: Based on the node features, adjacency matrix, weight matrix and global mask of the centrifugal subgraph, perform graph convolution processing on the centrifugal subgraph to obtain the third spatial features of the root node;

[0171] Step S206: Integrate the first spatial feature, the second spatial feature, and the third spatial feature to obtain the weighted spatial feature of the root node;

[0172] Step S207: For each node to be processed in the spatiotemporal feature map, determine the set of temporally adjacent nodes of the node to be processed in time sequence;

[0173] Step S208: Based on the weighted spatial features of each temporally adjacent node, perform temporal convolution processing on the set of temporally adjacent nodes to obtain the spatiotemporal features of the node to be processed.

[0174] Step S209: Perform full connection processing on the spatiotemporal features of each node to be processed in the spatiotemporal feature map to obtain the recognition result.

[0175] Specifically, each frame of skeletal point data includes data from multiple joint nodes, such as data from the knee joint node and data from the elbow joint node.

[0176] As an example, the data for each joint node can be the node's position coordinates (x, y) and confidence score.

[0177] The spatiotemporal feature map obtained from continuous multi-frame skeletal point data can be represented as G = (V, E), where V is the node set, and the element vti in the node set can be understood as the i-th node in the t-th image frame. Each node corresponds to node data. The edge set E includes two subsets, ES and EF. Elements in ES are used to represent the spatial topology of joint nodes, such as elbow and wrist joint nodes being connected in the same frame. Elements in EF are used to represent the temporal topology of joint nodes, such as elbow joint nodes being connected in two consecutive frames.

[0178] When performing fall recognition, the skeletal point data is first processed by graph convolution.

[0179] According to the traditional graph convolution method, the formula for calculating graph convolution in space can be expressed as:

[0180]

[0181] Where P() is the sampling function, representing the neighboring nodes of the sampling center, including the sampling center itself, in a convolutional neural network. Its size is determined by the convolutional kernel. As an example, the neighboring nodes are the sampling center v. i The node whose distance to the sampling center is less than D, and D = 1, is the nearest neighbor node of the sampling center. The sampling center can be understood as the root node.

[0182] w() represents the weight matrix, specifically the parameter matrix of the convolution kernel, initialized based on the kernel parameters. During graph convolution processing, nodes with the same label share a single weight matrix.

[0183] f in This can be understood as the node features input to the model for graph convolution processing; specifically, it can be node data within a set of neighboring nodes, f out The output is the model output, which can be understood as the feature obtained after the above graph convolution operation.

[0184] In this embodiment of the disclosure, a method of dividing the graph into subgraphs is used to determine the labels of the nodes, so the weight matrix is ​​transformed into the weight matrix of each subgraph.

[0185] Specifically, for each root node in each frame, the set of adjacent nodes is divided into a root node subgraph, an eccentric node subgraph, and a centripetal node subgraph, and labels are determined for each. For example, the labels for the root node subgraph, the eccentric node subgraph, and the centripetal node subgraph are 0, 1, and 2, respectively. Therefore, each of the three subgraphs corresponds to a learnable weight matrix.

[0186] Among them, the distance between the centripetal node and the centroid is less than the distance between the root node and the centroid; the distance between the centrifugal node and the centroid is greater than the distance between the root node and the centroid.

[0187] As an example, with the knee joint node as the root node, its neighboring node set includes the knee joint node, the ankle joint node, and the thigh root joint node. The thigh root joint node is closer to the center of gravity and is a centripetal node, while the ankle joint node is farther from the center of gravity and is an eccentric node.

[0188] When using a partitioning strategy, the formula for graph convolution operation can be expressed as follows:

[0189]

[0190] Here, j can be understood as the label corresponding to each subgraph, for example, j = 0, 1, 2, W j Let A be the weight matrix corresponding to each subgraph. j Let be the adjacency matrix for each subgraph, where the adjacency matrix of the root node subgraph can be initialized as the identity matrix.

[0191] M is the global mask for each convolutional layer, which can be understood as a learnable weight attention matrix. Indicates that A j Multiply by M bitwise. It is the normalized version

[0192]

[0193] By introducing a global mask, we can assign greater weights to important edges or nodes in the adjacency matrix and suppress the weights of unimportant edges or nodes, thereby further improving the accuracy of the recognition results.

[0194] It can be seen that the final weighted spatial feature f out , which is the weighted sum of the first spatial features, second spatial features, and third spatial features corresponding to the root node subgraph, centripetal subgraph, and centrifugal subgraph.

[0195] The weighted spatial features obtained after graph convolution can characterize the spatial features of human movements. After obtaining the weighted spatial features, temporal convolution processing can be further performed in this embodiment.

[0196] Specifically, for each node to be processed in the spatiotemporal feature map, determine the set of temporally adjacent nodes of that node in the time sequence.

[0197] The temporally adjacent nodes of the node to be processed can be understood as nodes whose time difference with the node to be processed is within a preset range.

[0198] As an example, if the node to be processed is the knee joint node in frame t, then its temporally adjacent node set can include the knee joint nodes in frames t-1, t, and t+1.

[0199] In the process of graph convolution, the weighted spatial features of each node have been obtained, and the features have been fused in space. On this basis, temporal convolution can be performed to further fuse the features in time, and finally obtain the spatiotemporal features of the node to be processed.

[0200] By performing fully connected processing on the spatiotemporal features of the node to be processed, the recognition result can be output. For example, an output of 0 indicates that a specific action has been performed, while an output of 1 indicates that a specific action has not been performed.

[0201] In real-world scenarios, human motion can be categorized into three states: stillness, centrifugal motion, and centripetal motion. For example, centrifugal motion can be seen as standing up, while centripetal motion can be seen as sitting down.

[0202] In this embodiment, weight matrices are assigned to the root node subgraph, the centripetal subgraph, and the centrifugal subgraph respectively, so as to realize differentiated learning of the features of different nodes, which can more accurately distinguish between centrifugal motion and centripetal motion, thereby obtaining more accurate recognition results.

[0203] In one embodiment of this disclosure, the specific action recognition model includes a GCN (Graph Convolutional Networks) module and a TCN (Temporal Convolutional Networks) module; the GCN module is used to perform graph convolution processing, and the TCN module is used to perform temporal convolution processing.

[0204] The TCN module includes a 1×1 convolution branch, a Maxpooling branch, and a 3×1 dilated convolution branch with 4 dilated convolutions at intervals of 1 to 4.

[0205] Specifically, TCN performs one-dimensional convolution on the same nodes of consecutive frames in the spatiotemporal feature map. If the size of the convolution kernel is K, then K×1 convolution is performed in the time dimension to fuse the information of K image frames, with a step size of s frames each time.

[0206] Taking the knee joint node as an example, assuming the kernel size K is 3 and the stride s is 1, the TCN can perform temporal convolution on the knee joint nodes in the first three image frames, fusing the weighted spatial features of the knee joint nodes in the first three image frames, and then move 1 frame to perform temporal convolution on the knee joint nodes in the second to fourth image frames, and so on.

[0207] Therefore, based on the spatial features of the skeletal point data learned by the GCN module, the TCN module further learns the spatial features of the skeletal point data in the temporal sequence. The specific action recognition model can finally fuse the spatial and temporal features of the skeletal point data, and the recognition results obtained have higher accuracy.

[0208] In this embodiment, the TCN module is designed as a multi-path convolutional structure comprising six branches: a 1×1 convolution branch, a Maxpooling branch, and four 3×1 dilated convolution branches with spacings of 1 to 4. This TCN module design not only significantly reduces computational cost but also enhances the ability to fuse receptive field information from different time periods.

[0209] In one embodiment of this disclosure, the specific action recognition model is STGCN++ (Spatial Temporal Graph Convolutional Networks++).

[0210] In one embodiment of this disclosure, the aforementioned step of acquiring the video to be detected may specifically include:

[0211] Acquire the video to be tested from multiple cameras; each camera corresponds to a camera ID.

[0212] When the video to be detected includes videos captured by multiple cameras, the embodiments of this disclosure can achieve large-scale detection of specific actions. After obtaining the video to be detected, the specific steps may further include:

[0213] For each camera ID corresponding to the video to be detected, target tracking is performed on the image frames in the video to be detected to obtain the human body tracking box under the camera ID and the human body tracking ID of the human body tracking box;

[0214] For each camera ID, the skeletal point data of the human image corresponding to the human tracking frame is extracted to obtain the skeletal point data of the human tracking ID under that camera ID.

[0215] For each camera ID, a human tracking ID is assigned to it. For human tracking IDs that require specific action detection, specific action recognition is performed based on multiple consecutive frames of skeletal point data of that human tracking ID to obtain the recognition result.

[0216] When the recognition result of a specific action of any human tracking ID indicates that the specific action has been identified, an alarm prompt message is generated based on the camera ID to which the human tracking ID belongs.

[0217] As an example, the multiple cameras can specifically be multiple surveillance cameras in an industrial production scenario. Based on the specific action detection method provided in this disclosure, large-scale specific action detection can be achieved.

[0218] After obtaining the video to be detected, multiple image frames can be acquired from the video captured by each camera, and the corresponding camera IDs can be recorded. For example, for the video captured by each camera, one image frame can be acquired every 10 frames, and the acquired image frames can be used for subsequent specific action detection.

[0219] For each camera capturing the video to be tested, specific human actions in the video can be detected. Details of this can be found in steps S102-S104 above, and will not be repeated here. In this embodiment, the human tracking ID involved in the specific action detection process corresponds to its respective camera ID.

[0220] When the identification result of any human tracking ID indicates the execution of a specific action, an alarm message is generated based on the camera ID to which that human tracking ID belongs. Taking a fall as an example, this helps relevant personnel to review whether a fall accident has occurred based on the alarmed camera. If so, they can promptly go to the scene to provide assistance. With the ability to achieve large-scale fall detection, this helps ensure timely assistance in fall accidents.

[0221] This disclosure also provides a method for training a specific action recognition model. Figure 3 This is a flowchart illustrating the training method for a specific action recognition model provided in this embodiment of the disclosure, as shown below. Figure 3 As shown, the method specifically includes the following steps:

[0222] Step S301: Obtain sample video.

[0223] This disclosure does not limit the source of the sample video. Specifically, the human body in the sample video corresponds to the real recognition result, that is, whether a specific action is performed or not.

[0224] Step S302: Perform target tracking on the image frames in the sample video to obtain the human body tracking box and the human body tracking ID of the human body tracking box.

[0225] Step S303: Extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0226] Steps S302-S303 are based on similar principles to steps S102-S103 in the previous text, and you can refer to the description in the previous text for details.

[0227] As an example, the skeletal point data obtained in step S303 can be high frame rate skeletal point data. For example, the average number of frames of skeletal point data for each human tracking ID can be 50 frames.

[0228] As an example, to obtain high frame rate skeletal point data, the period for acquiring image frames in the sample video can be set to a smaller value.

[0229] Step S304: Train a specific action recognition model based on skeletal point data.

[0230] As can be seen, by performing target tracking on image frames in the sample video, human body tracking boxes and human body tracking IDs are obtained. Skeletal point data is extracted from the human image corresponding to the human body tracking box to obtain the skeletal point data of the human body tracking ID. This method is suitable for extracting the skeletal point data of each target in multi-target scenarios and distinguishes different targets based on the human body tracking ID. Therefore, the model trained based on the skeletal point data of the human body tracking ID has more accurate recognition results, improving the accuracy and versatility of specific action detection.

[0231] In one embodiment of this disclosure, after obtaining the skeletal point data of the human tracking ID, the method further includes:

[0232] Uniform sampling of skeletal point data is performed to obtain low-frame skeletal point data.

[0233] As an example, one frame can be sampled every 10 frames. If the original frame count of the skeleton data for a certain human tracking ID is 50 frames, after uniform sampling, the skeleton data for that human tracking ID will be reduced to 5 frames, thus obtaining low-frame skeleton data.

[0234] The specific action recognition model obtained by the training method of the specific action recognition model provided in the embodiments of this disclosure can be specifically applied to the specific action detection method provided in any of the embodiments of this disclosure above.

[0235] The aforementioned steps for training the specific action recognition model based on skeletal point data specifically include the following detailed steps:

[0236] Step 31: Input the low-frame skeleton point data into the specific action recognition model to obtain the specific action recognition result of the human tracking ID.

[0237] As an example, a specific action recognition model could be STGCN++.

[0238] After inputting low-frame skeleton point data into the specific action recognition model, the specific action recognition model will output the specific action recognition result. For example, an output of 0 indicates that the specific action is performed, and an output of 1 indicates that the specific action is not performed.

[0239] Step 32: Determine a first loss value based on the specific action recognition result and the real result corresponding to the human body tracking ID; the real result indicates whether the human body corresponding to the human body tracking ID performs the specific action.

[0240] Step 33: Adjust the model parameters of the specific action recognition model according to the first loss value until the training termination condition is met.

[0241] Specifically, based on the specific action recognition results of each human tracking ID obtained in step 31, and the real results corresponding to each human tracking ID, the first loss value can be calculated, and on this basis, the training of the specific action recognition model can be realized.

[0242] The specific action recognition model training method provided in this disclosure is based on low-frame skeleton point data for model training. The obtained specific action recognition model has high accuracy when performing specific action detection based on low-frame-rate skeleton point data, and can generalize the detection of specific actions at different action speeds. When applied to the specific action detection method provided in any embodiment of this disclosure, it can improve the real-time performance and action speed generalization of the specific action detection method in practical scenarios.

[0243] Tests have shown that the specific action recognition model trained using the training method provided in this embodiment can still maintain over 99% of the validation set prediction accuracy even at low frame rates.

[0244] Figure 4 This is a block diagram of an apparatus for implementing a specific action detection method according to embodiments of the present disclosure, such as... Figure 4 As shown, the device includes:

[0245] The first acquisition module 401 is used to acquire the video to be detected;

[0246] The first tracking module 402 is used to perform target tracking on image frames in the video to be detected, and to obtain the human body tracking box and the human body tracking ID of the human body tracking box.

[0247] The first extraction module 403 is used to extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0248] The recognition module 404 is used to perform specific action recognition on the human tracking ID that requires specific action detection, based on a pre-trained specific action recognition model, on the continuous multi-frame skeletal point data of the human tracking ID, and obtain the recognition result.

[0249] The specific action detection apparatus provided in the disclosed embodiments obtains human body tracking boxes and human body tracking IDs for each human body tracking box by tracking image frames in the video to be detected. It then extracts skeletal point data from the human image corresponding to each human body tracking box to obtain skeletal point data for each human body tracking ID. This apparatus is suitable for extracting skeletal point data for each target in multi-target scenarios and distinguishes different targets based on the human body tracking ID. In the process of specific action recognition based on skeletal point data, for the human body tracking ID requiring specific action detection, specific action recognition is performed based on multiple consecutive frames of skeletal point data for that human body tracking ID to obtain the recognition result. The use of temporally sequential multi-frame skeletal point data in the specific action recognition process captures the dynamic features of human actions. In addition to being applicable to specific action detection in multi-target scenarios, the obtained recognition results are more accurate, improving the accuracy and versatility of specific action detection.

[0250] Furthermore, the specific action detection device provided in this disclosure performs specific action detection through the video to be detected captured by the camera, and a single camera can also achieve specific action detection, which has high deployment friendliness.

[0251] In one embodiment of this disclosure, the first tracking module 402 specifically includes:

[0252] The detection unit is used to perform target detection in the current image frame of the video to be detected, and obtain the target detection box of the current image frame.

[0253] The matching unit is used to match the target detection box with the human tracking trajectory of the human tracking ID in the previous image frame to determine the human tracking box and the human tracking ID of the human tracking box in the current image frame.

[0254] In one embodiment of this disclosure, the matching unit specifically includes:

[0255] Sub-units are used to divide the target detection boxes into high-scoring boxes and low-scoring boxes based on the confidence scores of the target detection boxes;

[0256] The filtering subunit is used to filter the human tracking box in the previous image frame to obtain the tracking trajectory prediction box.

[0257] The first matching subunit is used to match the high-resolution bounding box with the tracking trajectory prediction box to determine the human tracking box in the high-resolution bounding box and the human tracking ID of the human tracking box.

[0258] The second matching subunit is used to match the low-segment bounding box with the remaining tracking trajectory prediction boxes to determine the human tracking box in the low-segment bounding box and the human tracking ID of the human tracking box.

[0259] Create a sub-unit to identify the remaining high-resolution bounding boxes as human tracking boxes and create a human tracking ID for each human tracking box.

[0260] In one embodiment of this disclosure, the filtering subunit is specifically used for:

[0261] Obtain the state variables and confidence scores of the human body tracking box in the previous image frame; wherein, the state variables of each human body tracking box include the center point coordinates, length value and width value of the human body tracking box;

[0262] For the state variables of the human body tracking box, the covariance matrix of the state variables is determined, and the covariance matrix is ​​smoothed based on the confidence score of the human body tracking box to obtain a smoothed matrix.

[0263] Based on the prediction equation, the state variables of the human tracking box and the corresponding smoothing matrix in the previous image frame are processed to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and the tracking trajectory prediction box is obtained.

[0264] In one embodiment of this disclosure, the first extraction module 403 is specifically used for:

[0265] For a human tracking ID, a human image is obtained based on the human tracking bounding box of that human tracking ID, and the human image is processed using a pose estimation model to obtain a key point heatmap. The key point heatmap is then integrated to obtain the skeletal point data of that human tracking ID.

[0266] In one embodiment of this disclosure, the device further includes:

[0267] The first determining module is used to determine any human tracking ID as a human tracking ID that needs to be detected for a specific action when any human tracking ID is lost in a consecutive preset number of image frames.

[0268] In one embodiment of this disclosure, the device further includes:

[0269] The caching module is used to cache the skeletal point data of the human tracking ID;

[0270] The second determining module is used to determine the human tracking ID as the human tracking ID that needs to be detected for a specific action when the skeleton point data cache of any human tracking ID reaches the first preset frame number.

[0271] The deletion module is used to delete the cached skeletal point data of the first second preset number of frames for the human body tracking ID; the second preset number of frames is not greater than the first preset number of frames.

[0272] In one embodiment of this disclosure, each frame of skeletal point data includes data from multiple joint nodes;

[0273] The recognition module 404 is specifically used for:

[0274] For a human tracking ID that requires specific motion detection, a spatiotemporal feature map is determined based on multiple consecutive frames of skeletal point data of the human tracking ID; the spatiotemporal feature map includes the skeletal point data of each frame of the human tracking ID.

[0275] For each root node in the skeletal point data of this human tracking ID, the set of neighboring nodes of the root node is partitioned according to the distance between the joint node and the centroid, resulting in a root node subgraph, a centripetal subgraph composed of centripetal nodes, and an eccentric subgraph composed of eccentric nodes. Each of the root node subgraph, centripetal subgraph, and eccentric subgraph corresponds to a learnable weight matrix. The distance between the centripetal node and the centroid is less than the distance between the root node and the centroid; the distance between the eccentric node and the centroid is greater than the distance between the root node and the centroid. The centroid represents the average coordinates of all joint nodes in the skeletal point data of this frame.

[0276] Based on the node features, adjacency matrix, weight matrix, and global mask of the convolutional layer of the root node subgraph, graph convolution processing is performed on the root node subgraph to obtain the first spatial features of the root node.

[0277] Based on the node features, adjacency matrix, weight matrix, and global mask of the centripetal subgraph, graph convolution processing is performed on the centripetal subgraph to obtain the second spatial features of the root node.

[0278] Based on the node features, adjacency matrix, weight matrix, and global mask of the centrifugal subgraph, graph convolution processing is performed on the centrifugal subgraph to obtain the third spatial features of the root node.

[0279] The first spatial feature, the second spatial feature, and the third spatial feature are integrated to obtain the weighted spatial feature of the root node;

[0280] For each node to be processed in the spatiotemporal feature map, determine the set of temporally adjacent nodes of that node in time sequence;

[0281] Based on the weighted spatial features of each temporally adjacent node, temporal convolution is performed on the set of temporally adjacent nodes to obtain the spatiotemporal features of the node to be processed.

[0282] The spatiotemporal features of each node to be processed in the spatiotemporal feature map are fully connected to obtain the recognition result.

[0283] In one embodiment of this disclosure, the specific action recognition model includes a graph convolutional (GCN) module and a temporal convolutional (TCN) module; the GCN module is used to perform graph convolution processing, and the TCN module is used to perform temporal convolution processing.

[0284] The TCN module includes a 1×1 convolution branch, a Maxpooling branch, and a 3×1 dilated convolution branch with 4-way dilated convolutions at intervals of 1 to 4.

[0285] In one embodiment of this disclosure, the specific action recognition model is STGCN++.

[0286] In one embodiment of this disclosure, the first acquisition module 401 is specifically used for:

[0287] Acquire the video to be tested from multiple cameras; each camera corresponds to a camera ID.

[0288] The first tracking module 402 is specifically used to perform target tracking on the image frames in the video to be detected for each camera ID, and to obtain the human body tracking box under the camera ID and the human body tracking ID of each human body tracking box.

[0289] The first extraction module 403 is specifically used to extract skeletal point data for the human image corresponding to each human tracking frame under each camera ID, so as to obtain the skeletal point data of each human tracking ID under that camera ID.

[0290] The recognition module 404 is specifically used to identify specific actions based on the continuous multi-frame skeleton point data of the human tracking ID under each camera ID, and to obtain the recognition result for the human tracking ID that needs to be detected for specific actions.

[0291] The device also includes:

[0292] The generation module is used to generate an alarm message based on the camera ID to which the human tracking ID belongs when the recognition result of any of the human tracking IDs identifies the specific action.

[0293] Figure 5 This is a block diagram of an apparatus for implementing a method for a specific action recognition model according to embodiments of the present disclosure, such as... Figure 5 As shown, the device includes:

[0294] The second acquisition module 501 is used to acquire sample videos;

[0295] The second tracking module 502 is used to perform target tracking on image frames in the sample video to obtain the human body tracking box and the human body tracking ID of the human body tracking box.

[0296] The second extraction module 503 is used to extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID.

[0297] Training module 504 is used to train a specific action recognition model based on skeletal point data.

[0298] In one embodiment of this disclosure, the apparatus for implementing the method of a specific action recognition model of this disclosure further includes:

[0299] The sampling module 505 is used to uniformly sample the bone point data to obtain low-frame bone point data.

[0300] Training module 504 specifically includes:

[0301] The input module is used to input low-frame skeleton point data into a specific action recognition model to obtain the specific action recognition result of the human body tracking ID;

[0302] The determination module is used to determine a first loss value based on the specific action recognition result and the real result corresponding to the human body tracking ID; the real result indicates whether the human body corresponding to the human body tracking ID performs the specific action.

[0303] The adjustment module is used to adjust the model parameters of a specific action recognition model based on the first loss value until the training termination condition is met.

[0304] The specific action recognition model training device provided in this embodiment trains the model based on low-frame skeleton point data. The obtained specific action recognition model has high accuracy when performing specific action detection based on low-frame-rate skeleton point data, and can generalize the detection of specific actions at different speeds. When applied to specific action detection, it can improve the real-time performance and action speed generalization of specific action detection in practical scenarios.

[0305] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0306] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0307] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0308] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0309] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for detecting a specific action. For example, in some embodiments, the method for detecting a specific action may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method for detecting a specific action described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method for detecting a specific action by any other suitable means (e.g., by means of firmware).

[0310] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0311] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0312] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0313] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0314] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0315] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0316] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0317] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting a specific action, comprising: Obtain the video to be tested; Target tracking is performed on the image frames in the video to be detected to obtain a human body tracking box and the human body tracking ID of the human body tracking box; Skeletal point data is extracted from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID; For human tracking IDs that require specific action detection, specific action recognition is performed on the continuous multi-frame skeletal point data of the human tracking ID based on a pre-trained specific action recognition model to obtain the recognition result; The step of performing target tracking on image frames in the video to be detected to obtain a human body tracking box and a human body tracking ID of the human body tracking box includes: Target detection is performed on the current image frame in the video to be detected to obtain the target detection box of the current image frame; Based on the confidence scores of the target detection boxes, the target detection boxes are divided into high-scoring boxes and low-scoring boxes; Obtain the state variables and confidence scores of the human tracking box in the previous preceding image frame; wherein, the state variables of the human tracking box include the center point coordinates, length value, and width value of the human tracking box; for the state variables of the human tracking box, determine the covariance matrix of the state variables, and smooth the covariance matrix based on the confidence score of the human tracking box to obtain a smoothing matrix; based on the prediction equation, process the state variables of the human tracking box and the corresponding smoothing matrix in the previous preceding image frame to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and obtain the tracking trajectory prediction box; For the high-resolution bounding box, the high-resolution bounding box is matched with the tracking trajectory prediction box to determine the human body tracking box in the high-resolution bounding box and the human body tracking ID of the human body tracking box; For the low-scoring bounding box, the low-scoring bounding box and the remaining tracking trajectory prediction bounding boxes are matched to determine the human body tracking box in the low-scoring bounding box and the human body tracking ID of the human body tracking box; The remaining high-resolution bounding boxes are identified as human tracking bounding boxes, and human tracking IDs are created for these human tracking bounding boxes.

2. The method according to claim 1, wherein the step of extracting skeletal point data from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID includes: For the human tracking ID, a human image is obtained based on the human tracking bounding box of the human tracking ID, and the human image is processed using a pose estimation model to obtain a key point heatmap. The key point heatmap is then integrated to obtain the skeletal point data of the human tracking ID.

3. The method according to claim 1, further comprising, after obtaining the skeletal point data of the human body tracking ID: If any human tracking ID is lost in a consecutive preset number of image frames, that human tracking ID is identified as the human tracking ID that needs to be detected for a specific action.

4. The method according to claim 1, further comprising, after obtaining the skeletal point data of the human body tracking ID: The skeletal point data of the human tracking ID is cached; When the skeleton point data cache of any human tracking ID reaches the first preset frame number, the human tracking ID is determined as the human tracking ID that needs to be detected for a specific action. For human tracking IDs requiring specific action detection, the method involves performing specific action recognition on multiple consecutive frames of skeletal point data of the human tracking ID based on a pre-trained specific action recognition model, and after obtaining the recognition result, the method further includes: Delete the cached skeletal point data of the first two preset frames for this human tracking ID; The second preset frame number is not greater than the first preset frame number.

5. The method according to claim 1, wherein, The skeletal point data in each frame includes data from multiple joint nodes; The step of performing specific action recognition on multiple consecutive frames of skeletal point data of a human tracking ID that requires specific action detection, based on a pre-trained specific action recognition model, to obtain the recognition result, includes: For a human tracking ID that requires specific motion detection, a spatiotemporal feature map is determined based on multiple consecutive frames of skeletal point data of the human tracking ID; the spatiotemporal feature map includes the skeletal point data of each frame of the human tracking ID. For each root node in the skeletal point data of each frame of the human tracking ID, the set of neighboring nodes of the root node is partitioned according to the distance between the joint node and the center of gravity, resulting in a root node subgraph, a centripetal subgraph composed of centripetal nodes, and an eccentric subgraph composed of eccentric nodes. Each of the root node subgraph, the centripetal subgraph, and the eccentric subgraph corresponds to a learnable weight matrix. The distance between the centripetal node and the center of gravity is less than the distance between the root node and the center of gravity; the distance between the eccentric node and the center of gravity is greater than the distance between the root node and the center of gravity. The center of gravity represents the average coordinates of all joint nodes in the frame of skeletal point data. Based on the node features, adjacency matrix, weight matrix, and global mask of the convolutional layer of the root node subgraph, graph convolution processing is performed on the root node subgraph to obtain the first spatial features of the root node. Based on the node features, adjacency matrix, weight matrix, and global mask of the centripetal subgraph, graph convolution is performed on the centripetal subgraph to obtain the second spatial features of the root node. Based on the node features, adjacency matrix, weight matrix, and global mask of the centrifugal subgraph, graph convolution processing is performed on the centrifugal subgraph to obtain the third spatial features of the root node. The first spatial feature, the second spatial feature, and the third spatial feature are integrated to obtain the weighted spatial feature of the root node; For each node to be processed in the spatiotemporal feature map, determine the set of temporally adjacent nodes of the node to be processed in time sequence; Based on the weighted spatial features of each temporally adjacent node, the set of temporally adjacent nodes is subjected to temporal convolution to obtain the spatiotemporal features of the node to be processed. The spatiotemporal features of each node to be processed in the spatiotemporal feature map are fully connected to obtain the recognition result.

6. The method according to claim 5, wherein, The specific action recognition model includes a graph convolutional (GCN) module and a temporal convolutional (TCN) module; the GCN module is used for graph convolution processing, and the TCN module is used for temporal convolution processing. The TCN module includes a 1×1 convolution branch, a maxpooling branch, and a 3×1 dilated convolution branch with 4 dilated convolutions at intervals of 1 to 4.

7. The method according to claim 1, wherein the step of acquiring the video to be detected comprises: Acquire the video to be tested from multiple cameras; each camera corresponds to a camera ID. The step of performing target tracking on image frames in the video to be detected to obtain a human body tracking box and a human body tracking ID of the human body tracking box includes: For each camera ID corresponding to the video to be detected, target tracking is performed on the image frames in the video to be detected to obtain the human body tracking box under the camera ID and the human body tracking ID of the human body tracking box; The step of extracting skeletal point data from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID includes: For each camera ID, the skeletal point data of the human image corresponding to the human tracking frame is extracted to obtain the skeletal point data of the human tracking ID under that camera ID. The step of performing specific action recognition on multiple consecutive frames of skeletal point data of a human tracking ID that requires specific action detection, based on a pre-trained specific action recognition model, to obtain the recognition result, includes: For each camera ID, a human tracking ID is assigned to it. For human tracking IDs that require specific action detection, specific action recognition is performed based on multiple consecutive frames of skeletal point data of that human tracking ID to obtain the recognition result. When the identification result of any of the human body tracking IDs indicates that the specific action has been identified, an alarm prompt message is generated based on the camera ID to which the human body tracking ID belongs.

8. A training method for a specific action recognition model, comprising: Obtain sample videos; Target tracking is performed on the image frames in the sample video to obtain the human body tracking box and the human body tracking ID of the human body tracking box; Skeletal point data is extracted from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID; The specific action recognition model is trained based on the skeletal point data; The step of performing target tracking on image frames in the sample video to obtain a human tracking bounding box and a human tracking ID of the human tracking bounding box includes: Target detection is performed on the current image frame in the sample video to obtain the target detection box of the current image frame; Based on the confidence scores of the target detection boxes, the target detection boxes are divided into high-scoring boxes and low-scoring boxes; Obtain the state variables and confidence scores of the human tracking box in the previous preceding image frame; wherein, the state variables of the human tracking box include the center point coordinates, length value, and width value of the human tracking box; for the state variables of the human tracking box, determine the covariance matrix of the state variables, and smooth the covariance matrix based on the confidence score of the human tracking box to obtain a smoothing matrix; based on the prediction equation, process the state variables of the human tracking box and the corresponding smoothing matrix in the previous preceding image frame to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and obtain the tracking trajectory prediction box; For the high-resolution bounding box, the high-resolution bounding box is matched with the tracking trajectory prediction box to determine the human body tracking box in the high-resolution bounding box and the human body tracking ID of the human body tracking box; For the low-scoring bounding box, the low-scoring bounding box and the remaining tracking trajectory prediction bounding boxes are matched to determine the human body tracking box in the low-scoring bounding box and the human body tracking ID of the human body tracking box; The remaining high-resolution bounding boxes are identified as human tracking bounding boxes, and human tracking IDs are created for these human tracking bounding boxes.

9. The method according to claim 8, further comprising, after obtaining the skeletal point data of the human body tracking ID: The bone point data is uniformly sampled to obtain low-frame bone point data; The step of training the specific action recognition model based on the skeletal point data includes: The low-frame skeleton point data is input into a specific action recognition model to obtain the specific action recognition result of the human body tracking ID; Based on the specific action recognition result and the real result corresponding to the human body tracking ID, a first loss value is determined; the real result indicates whether the human body corresponding to the human body tracking ID performs the specific action. Based on the first loss value, adjust the model parameters of the specific action recognition model until the training termination condition is met.

10. A detection device for a specific action, comprising: The first acquisition module is used to acquire the video to be detected; The first tracking module is used to perform target tracking on image frames in the video to be detected, and obtain a human body tracking box and the human body tracking ID of the human body tracking box; The first extraction module is used to extract skeletal point data from the human image corresponding to the human tracking frame to obtain the skeletal point data of the human tracking ID. The recognition module is used to identify specific actions of human tracking IDs that require specific action detection, based on a pre-trained specific action recognition model, by performing specific action recognition on multiple consecutive frames of skeletal point data of the human tracking ID, and obtain the recognition result. The first tracking module includes: The detection unit is used to perform target detection on the current image frame in the video to be detected, and obtain the target detection box of the current image frame; The sub-unit is used to divide the target detection box into high-scoring boxes and low-scoring boxes based on the confidence score of the target detection box; The filtering subunit is specifically used to obtain the state variables and confidence scores of the human tracking box in the previous image frame; wherein, the state variables of the human tracking box include the center point coordinates, length value, and width value of the human tracking box; for the state variables of the human tracking box, the covariance matrix of the state variables is determined, and the covariance matrix is ​​smoothed based on the confidence score of the human tracking box to obtain a smoothing matrix; based on the prediction equation, the state variables of the human tracking box and the corresponding smoothing matrix in the previous image frame are processed to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and the tracking trajectory prediction box is obtained; The first matching subunit is used to match the high-resolution bounding box with the tracking trajectory prediction box to determine the human tracking box in the high-resolution bounding box and the human tracking ID of the human tracking box. The second matching subunit is used to match the low-score box with the remaining tracking trajectory prediction boxes to determine the human tracking box in the low-score box and the human tracking ID of the human tracking box. Create a sub-unit to identify the remaining high-resolution bounding boxes as human tracking boxes and create a human tracking ID for the human tracking boxes.

11. The apparatus according to claim 10, wherein the first extraction module is specifically used for: For a human tracking ID, a human image is obtained based on the human tracking bounding box of that human tracking ID, and the human image is processed using a pose estimation model to obtain a key point heatmap. The key point heatmap is then integrated to obtain the skeletal point data of that human tracking ID.

12. The apparatus of claim 10, further comprising: The first determining module is used to determine any human tracking ID as a human tracking ID that needs to be detected for a specific action when any human tracking ID is lost in a consecutive preset number of image frames.

13. The apparatus of claim 10, further comprising: The caching module is used to cache the skeletal point data of the human body tracking ID; The second determining module is used to determine the human tracking ID as the human tracking ID that needs to be detected for a specific action when the skeleton point data cache of any human tracking ID reaches the first preset frame number. The deletion module is used to delete the cached skeletal point data of the first two preset frames for the human body tracking ID; The second preset frame number is not greater than the first preset frame number.

14. The apparatus according to claim 10, wherein, The skeletal point data in each frame includes data from multiple joint nodes; The identification module is specifically used for: For a human tracking ID that requires specific motion detection, a spatiotemporal feature map is determined based on multiple consecutive frames of skeletal point data of the human tracking ID; the spatiotemporal feature map includes the skeletal point data of each frame of the human tracking ID. For each root node in the skeletal point data of each frame of the human tracking ID, the set of neighboring nodes of the root node is partitioned according to the distance between the joint node and the center of gravity, resulting in a root node subgraph, a centripetal subgraph composed of centripetal nodes, and an eccentric subgraph composed of eccentric nodes. Each of the root node subgraph, the centripetal subgraph, and the eccentric subgraph corresponds to a learnable weight matrix. The distance between the centripetal node and the center of gravity is less than the distance between the root node and the center of gravity; the distance between the eccentric node and the center of gravity is greater than the distance between the root node and the center of gravity. The center of gravity represents the average coordinates of all joint nodes in the frame of skeletal point data. Based on the node features, adjacency matrix, weight matrix, and global mask of the convolutional layer of the root node subgraph, graph convolution processing is performed on the root node subgraph to obtain the first spatial features of the root node. Based on the node features, adjacency matrix, weight matrix, and global mask of the centripetal subgraph, graph convolution is performed on the centripetal subgraph to obtain the second spatial features of the root node. Based on the node features, adjacency matrix, weight matrix, and global mask of the centrifugal subgraph, graph convolution processing is performed on the centrifugal subgraph to obtain the third spatial features of the root node. The first spatial feature, the second spatial feature, and the third spatial feature are integrated to obtain the weighted spatial feature of the root node; For each node to be processed in the spatiotemporal feature map, determine the set of temporally adjacent nodes of the node to be processed in time sequence; Based on the weighted spatial features of each temporally adjacent node, the set of temporally adjacent nodes is subjected to temporal convolution to obtain the spatiotemporal features of the node to be processed. The spatiotemporal features of each node to be processed in the spatiotemporal feature map are fully connected to obtain the recognition result.

15. The apparatus according to claim 14, wherein, The specific action recognition model includes a graph convolutional (GCN) module and a temporal convolutional (TCN) module; the GCN module is used for graph convolution processing, and the TCN module is used for temporal convolution processing. The TCN module includes a 1×1 convolution branch, a maxpooling branch, and a 3×1 dilated convolution branch with 4 dilated convolutions at intervals of 1 to 4.

16. The apparatus according to claim 10, wherein the first acquisition module is specifically used for: Acquire the video footage to be tested captured by multiple cameras; among them, Each camera corresponds to a camera ID; The first tracking module is specifically used to perform target tracking on the image frames in the video to be detected for each camera ID, so as to obtain the human body tracking box under the camera ID and the human body tracking ID of the human body tracking box. The first extraction module is specifically used to extract skeletal point data for the human image corresponding to the human tracking frame under each camera ID, so as to obtain the skeletal point data of the human tracking ID under that camera ID. The recognition module is specifically used to identify specific actions based on the continuous multi-frame skeleton point data of the human tracking ID under each camera ID, and to obtain the recognition result for the human tracking ID that needs to be detected for specific actions. The device further includes: The generation module is used to generate an alarm message based on the camera ID to which the human tracking ID belongs when the recognition result of any of the human tracking IDs identifies the specific action.

17. A training device for a specific action recognition model, comprising: The second acquisition module is used to acquire sample videos; The second tracking module is used to perform target tracking on image frames in the sample video to obtain a human body tracking box and the human body tracking ID of the human body tracking box; The second extraction module is used to extract skeletal point data from the human image corresponding to the human tracking box to obtain the skeletal point data of the human tracking ID. The training module is used to train the specific action recognition model based on the skeletal point data; The second tracking module is specifically used for: Target detection is performed on the current image frame in the sample video to obtain the target detection box of the current image frame; Based on the confidence scores of the target detection boxes, the target detection boxes are divided into high-scoring boxes and low-scoring boxes; Obtain the state variables and confidence scores of the human tracking box in the previous preceding image frame; wherein, the state variables of the human tracking box include the center point coordinates, length value, and width value of the human tracking box; for the state variables of the human tracking box, determine the covariance matrix of the state variables, and smooth the covariance matrix based on the confidence score of the human tracking box to obtain a smoothing matrix; based on the prediction equation, process the state variables of the human tracking box and the corresponding smoothing matrix in the previous preceding image frame to determine the state variables of the tracking trajectory prediction box corresponding to the human tracking box, and obtain the tracking trajectory prediction box; For the high-resolution bounding box, the high-resolution bounding box is matched with the tracking trajectory prediction box to determine the human body tracking box in the high-resolution bounding box and the human body tracking ID of the human body tracking box; For the low-scoring bounding box, the low-scoring bounding box and the remaining tracking trajectory prediction bounding boxes are matched to determine the human body tracking box in the low-scoring bounding box and the human body tracking ID of the human body tracking box; The remaining high-resolution bounding boxes are identified as human tracking bounding boxes, and human tracking IDs are created for these human tracking bounding boxes.

18. The apparatus of claim 17, further comprising: The sampling module is used to uniformly sample the bone point data to obtain low-frame bone point data. The training module includes: The input module is used to input the low-frame skeleton point data into the specific action recognition model to obtain the specific action recognition result of the human body tracking ID; The determination module is used to determine a first loss value based on the specific action recognition result and the real result corresponding to the human body tracking ID; the real result indicates whether the human body corresponding to the human body tracking ID performed the specific action; An adjustment module is used to adjust the model parameters of the specific action recognition model according to the first loss value until the training termination condition is met.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images