User action classification method and computer program product

Abstract

The invention discloses a user action classification method and a computer program product, and the method comprises the steps: carrying out the 3D reconstruction based on a human body region of a target user in each image frame of a collected video, so as to obtain the 3D human body posture information of the target user in each image frame; based on the 3D human body posture information of the target user in each picture frame, determining 3D hand posture information of the target user in each picture frame; splicing the 3D hand posture information and the 3D human body posture information of the target user in the same picture frame to obtain 3D spliced human body posture information of the target user in each picture frame; continuous picture frames corresponding to the 3D spliced human body posture information containing the target user are input into an action classification model to obtain an action classification result of the target user, and the action classification model is obtained by taking the 3D spliced human body posture information of the user as a sample and taking the action classification result of the user as a label for training.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a user action classification method and a computer program product. Background Art

[0002] In monitoring scenarios, it is necessary to detect the actions or behaviors of one or more users appearing in the monitoring video to determine whether the corresponding user's behavior is a dangerous event or emergency based on the user's actions, such as intrusion, fighting, theft, falling, fainting, etc., so as to quickly discover, respond and intervene, which will help prevent and reduce the occurrence of related incidents, or reduce the losses and impacts caused by the incidents.

[0003] At present, user action detection is mainly carried out through automatic monitoring, which reduces the dependence on manual monitoring, reduces labor costs, and improves overall monitoring efficiency. Usually, automatic monitoring is based on abnormal behavior detection of human key points. It first detects the 2D joint key points of the human body in the video, and then uses these joint key points to construct skeleton features. The constructed features are input into the machine learning or deep learning model to output the corresponding user's action classification results.

[0004] However, the accuracy of action classification results is extremely dependent on the accuracy of key point detection. If the key point detection is wrong, the action will be recognized incorrectly. The key point distribution of people of different heights, weights and body shapes varies greatly, and the generalization of the model is limited, resulting in low accuracy in user action recognition. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a user action classification method and a computer program product to solve the problem of low accuracy in user action recognition.

[0006] In order to solve the above technical problems, this specification is implemented as follows: In a first aspect, a user action classification method is provided, comprising: Performing 3D reconstruction based on the human body area of ​​the target user in each picture frame of the captured video to obtain 3D human body posture information of the target user in each picture frame; Determine the 3D hand gesture information of the target user in each picture frame based on the 3D human body gesture information of the target user in each picture frame; splicing the 3D hand gesture information and the 3D human body gesture information of the target user in the same picture frame to obtain the 3D spliced ​​human body gesture information of the target user in each picture frame; The continuous image frames corresponding to the 3D spliced ​​human posture information of the target user are input into the action classification model to obtain the action classification result of the target user. The action classification model is trained with the user's 3D spliced ​​human posture information as a sample and the user's action classification result as a label.

[0007] Optionally, the performing 3D reconstruction based on the human body region of the target user in each picture frame of the captured video to obtain 3D human body posture information of the target user in each picture frame includes: Tracking and detecting the target user in the collected video by using a tracking algorithm to obtain a human body frame corresponding to the human body region of the target user in each picture frame of the collected video; Obtaining a human body frame image corresponding to a human body frame in each picture frame; Each human frame image is predicted by a human posture shape estimation algorithm to obtain the 3D human posture information of the target user in each picture frame.

[0008] Optionally, the 3D human body posture information includes 3D human body posture parameters, human body shape parameters and camera parameters, and the determining the 3D hand posture information of the target user in each picture frame based on the 3D human body posture information of the target user in each picture frame includes: Inputting the 3D human body posture parameters and human body shape parameters of the target user in the target picture frame into a parameterized human body model to output each 3D human body key point of the target user in the target picture frame, including the coordinates and categories of the key points; Based on the camera parameters of the target user in the target picture frame, the coordinates of each 3D human body key point of the target user are converted into the coordinates of 2D human body key points; Determine a hand region of the target user in the target picture frame based on coordinates and categories of each 2D human body key point of the target user in the target picture frame; 3D reconstruction is performed based on the hand area of ​​the target user in the target picture frame to obtain 3D hand posture information of the target user in the target picture frame, and the 3D hand posture information includes 3D hand posture parameters and 3D hand shape parameters.

[0009] Optionally, the converting the coordinates of each 3D human body key point of the target user into the coordinates of 2D human body key points based on the camera parameters of the target user in the target picture frame includes: Using the external parameters in the camera parameters, converting the position coordinates of each 3D human body key point of the target user in the target image frame into camera coordinates; Using perspective projection, transform the camera coordinates into normalized image coordinates; The normalized image coordinates are converted into pixel coordinates corresponding to the coordinates of 2D human body key points using the intrinsic parameters of the camera that captured the acquisition video.

[0010] Optionally, determining the hand region of the target user in the target picture frame based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame includes: Based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame, respectively determine the coordinates of the wrist key point and the hand key point in the target picture frame; Taking a target wrist key point as a reference point, and based on the coordinates of the target wrist key point and the coordinates of the corresponding hand key point, calculating the distance between the target wrist key point and the corresponding hand key point, wherein the target wrist key point is a left wrist key point or a right wrist key point; Generate a first square area perpendicular to a line connecting the target wrist key point and the corresponding hand key point, wherein a side length of the first square area is greater than the distance; A second square area circumscribed to the first square area is determined as a left hand area or a right hand area corresponding to a target wrist key point of the target user in the target picture frame.

[0011] Optionally, the performing 3D reconstruction based on the hand region of the target user in the target picture frame to obtain 3D hand posture information of the target user in the target picture frame includes: Determine a left hand frame corresponding to a left hand region and a right hand frame corresponding to a right hand region included in a hand region of the target user in the target picture frame; Acquire a left-hand frame image corresponding to the left-hand frame and a right-hand frame image corresponding to the right-hand frame in the target picture frame; The left hand frame image and the right hand frame image are predicted respectively by a hand reconstruction algorithm to obtain 3D left hand posture information and 3D right hand posture information of the target user in the target picture frame.

[0012] Optionally, the step of splicing the 3D hand gesture information and the 3D human body gesture information of the target user in the same picture frame to obtain the 3D spliced ​​human body gesture information of the target user in each picture frame includes: Removing the 3D left hand posture information and the 3D right hand posture information included in the 3D human body posture information of the target user in the same picture frame to obtain updated 3D human body posture information of the target user; The 3D left hand posture information and the 3D right hand posture information included in the 3D hand posture information of the target user in the same picture frame are spliced ​​with the updated 3D human body posture information to obtain the 3D spliced ​​human body posture information of the target user in the same picture frame.

[0013] Optionally, before inputting the continuous picture frames corresponding to the 3D spliced ​​human body posture information containing the target user into the action classification model, the method further includes: The 3D stitched human body posture information of the target user is smoothed by a SmoothNet algorithm to remove noise in the 3D stitched human body posture information.

[0014] Optionally, the human body posture shape estimation algorithm includes a video inference VIBE algorithm for human body posture shape estimation, and the hand reconstruction algorithm includes a FrankMocap algorithm.

[0015] In a second aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute the steps of the method described in the first aspect.

[0016] In an embodiment of the present application, 3D reconstruction is performed based on the human body area of ​​the target user in each picture frame of the captured video to obtain the 3D human body posture information of the target user in each picture frame; based on the 3D human body posture information of the target user in each picture frame, the 3D hand posture information of the target user in each picture frame is determined; the 3D hand posture information and the 3D human body posture information of the target user in the same picture frame are spliced ​​to obtain the 3D spliced ​​human body posture information of the target user in each picture frame; the continuous picture frames corresponding to the 3D spliced ​​human body posture information of the target user are input into the action classification model to obtain the action classification result of the target user, and the action classification model is based on the 3D spliced ​​human body posture information of the user. It is trained by taking human posture information as samples and user action classification results as labels. In this way, human actions, environmental scenes and human shapes are decoupled through 3D reconstruction, and relatively small human hand areas are accurately detected based on 3D human posture information. The 3D spliced ​​human posture information of multiple key points including the human body area and the hand area is input into the action classification model. The action classification model can focus only on human actions and has significant anti-interference ability against local occlusion, blur, harsh environment and human shape differences in the picture. It is suitable for various complex scenes, greatly reducing the difficulty of learning the action classification model and improving the recognition accuracy of the action classification model. The action classification results of the target user are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a flowchart of the user action classification method of an embodiment of the present application.

[0018] Figure 2 It is a schematic diagram of the process of 3D reconstruction of a human body region according to an embodiment of the present application.

[0019] Figure 3 It is a schematic diagram of the process of 2D hand area prediction in an embodiment of the present application.

[0020] Figure 4 It is a flowchart of 3D reconstruction of the hand area according to an embodiment of the present application.

[0021] Figure 5 It is a schematic diagram of the network architecture of the action classification model of an embodiment of the present application.

[0022] Figure 6 It is a flow chart of the smoothing process of 3D stitching human body posture information in an embodiment of the present application.

[0023] Figure 7 It is a schematic diagram of the overall process of the user action classification method according to an embodiment of the present application.

[0024] Figure 8 It is a structural block diagram of the user action classification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. The numbering of the drawings in this application is only used to distinguish the various steps in the scheme, and is not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0026] In order to solve the problems existing in the prior art, the present application embodiment provides a user action classification method, such as Figure 1 As shown, the following steps 102 to 108 are included.

[0027] Step 102 , performing 3D reconstruction based on the human body region of the target user in each picture frame of the captured video, so as to obtain 3D human body posture information of the target user in each picture frame.

[0028] The captured video is a surveillance video captured in a surveillance scene. One or more users may appear in the surveillance video to perform corresponding actions. The captured video includes multiple continuous image frames. In this step, it is necessary to perform 3D reconstruction of the human body area of ​​the target user that appears based on each image frame of the captured video. The target user can be one user or multiple users.

[0029] The human body region of 3D reconstruction is the human body region of a single user. Through 3D reconstruction, the 3D posture information of the human body region corresponding to a single user in each picture frame can be obtained. The 3D posture information includes 3D posture parameters that describe the movements of each key point of the human body, and 3D shape parameters that describe the shape of the human body, such as height, weight, etc. The key points of the human body can be represented by the joints of the human body, and the 3D human body posture parameters can be represented by the axis angles of the 3D space X / Y / Z of each key point of the target user's body. The human body shape parameters can be represented by vector values, and different vector values ​​describe different human body shapes.

[0030] Optionally, the 3D reconstruction is performed based on the human body region of the target user in each picture frame of the captured video to obtain 3D human body posture information of the target user in each picture frame, including: tracking and detecting the target user in the captured video by a tracking algorithm to obtain a human body frame corresponding to the human body region of the target user in each picture frame of the captured video; acquiring a human body frame image corresponding to the human body frame in each picture frame; and predicting each human body frame image by a human body posture shape estimation algorithm to obtain the 3D human body posture information of the target user in each picture frame.

[0031] During the time when the target user appears and disappears from the captured video, the target user in the corresponding picture frame sequence of the video is tracked by the tracking algorithm to detect the human body frame corresponding to the human body area of ​​the target user in each picture frame. The tracking algorithm can adopt the BoT-SORT (Robust Associations Multi-Pedestrian Tracking) multi-target tracking algorithm, which can detect and track all objects in the captured video and retain a unique identifier for each object.

[0032] After obtaining the human frame through pedestrian detection and tracking, the human frame of the human body region can be cropped out from the original picture frame. The cropped human frame image can be a square with a side length equal to the maximum of the width and height of the human frame. The cropped human frame image is predicted by a human posture shape estimation algorithm to obtain the 3D human posture information of the target user in each picture frame, thereby realizing 3D reconstruction of the human body region of the target user.

[0033] In one embodiment, the human body pose shape estimation algorithm includes a video inference VIBE algorithm for human body pose shape estimation.

[0034] The VIBE model corresponding to the Video Inference for Human Body Pose and Shape Estimation (VIBE) algorithm is as follows: Figure 2 As shown. For example, the number of picture frames input into the VIBE model each time is 16, and the picture size of the square human frame image is resized to 224*224*3, where 224*224 represents the side length of the square, and 3 represents the three channels of R, G, and B. Then, multiple consecutive frames of pictures 16*224*224*3 are input into the VIBE model, where 16 represents the number of picture frames. The input picture frame is first subjected to the convolutional neural network (CNN) feature extraction of Resnet50 (step 202) to obtain the extracted 16*7*7*2048 feature vector, and then the extracted feature vector is globally averaged pooled (step 204) to obtain the 16*2048 feature vector. Next, the feature vector is temporally encoded by GRU temporal modeling (step 206) to obtain the corresponding 16*1024 temporal features to establish the connection between adjacent picture frames, correct the erroneous prediction results of the previous picture frame, and improve the accuracy of the prediction. Then, the parameterized model (Skinned Multi-Person Linear Model, SMPL) is used to regress the temporal features and output the 3D human posture information of the target user in each image frame, including the key points of the human body area.

[0035] The SMPL model is a parametric model of human body shape and posture. For example, it can detect the key points of 24 human joints, and the corresponding 3D human posture information is 3D human posture parameters (24*3), shape parameters (10) and camera parameters (3). The vector of 3D human posture parameters (24*3) describes the axis angles of 24 human joints in 3D space, the vector of shape parameters (10) describes 10 different human shapes, and the vector of camera parameters (3) describes the translation vector t of 3D space X / Y / Z in the camera's external parameters. x ,t y ,t z For example, combining Figure 2 , we get the posture parameters corresponding to 16 picture frames: 16*24*3, posture parameters (also known as shape parameters) 16*10, and camera parameters 16*3.

[0036] Step 104 : determining the 3D hand gesture information of the target user in each picture frame based on the 3D human body gesture information of the target user in each picture frame.

[0037] After obtaining the 3D human body posture information of the target user in multiple picture frames of the captured video in step 102, the 3D hand posture information of the target user in each picture frame may be further obtained based on the 3D human body posture information.

[0038] Optionally, the 3D human body posture information includes 3D human body posture parameters, human body shape parameters and camera parameters, and determining the 3D hand posture information of the target user in each picture frame based on the 3D human body posture information of the target user in each picture frame includes: inputting the 3D human body posture parameters and human body shape parameters of the target user in the target picture frame into a parameterized human body model to output each 3D human body key point of the target user in the target picture frame, including the coordinates and categories of the key points; based on the camera parameters of the target user in the target picture frame, converting the coordinates of each 3D human body key point of the target user into the coordinates of 2D human body key points; based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame, determining the hand area of ​​the target user in the target picture frame; performing 3D reconstruction based on the hand area of ​​the target user in the target picture frame to obtain the 3D hand posture information of the target user in the target picture frame, the 3D hand posture information including 3D hand posture parameters and 3D hand shape parameters.

[0039] In this embodiment, the 3D human body key points are regressed based on the 3D human body posture information obtained by 3D human body reconstruction, and then the world coordinate system of the 3D human body key points is projected to the 2D key point pixel coordinates to detect the hand area of ​​the target user in each picture frame. Finally, the hand area is 3D reconstructed to obtain the 3D hand posture information of the target user in each picture frame.

[0040] Specifically, the 3D human posture parameters and human shape parameters output by the VIBE model are input into a parametric human model, such as the SMPL model, to obtain all vertex coordinates (6890*3) and key point coordinates (24*3) of the target user's human body area in the 3D space and the corresponding key point categories. Figure 3As shown, it includes 24 key points corresponding to 0 to 23, where the corresponding relationship is 0: pelvis (pelvis); 1: r_hip (right hip); 2: l_hip (left hip); 3: spine1 (spine 1); 4: r_knee (right knee); 5: l_knee (left knee); 6: spine2 (spine 2); 7: r_ankle (right ankle); 8: l_ankle (left ankle); 9: spine3 (spine 3); 10: 7: r_foot (right foot); 11: l_foot (left foot); 12: neck (neck); 13: r_collor (right clavicle); 14: l_collar (left clavicle); 15:head (head); 16:r_shoulder (right shoulder); 17:l_shoulder (left shoulder); 18:r_elbow (right elbow); 19:l_elbow (left elbow); 20:r_wrist (right wrist); 21:l_wrist (left wrist); 22: r_hand (right hand); 23:l_hand (left hand).

[0041] The vector of vertex coordinates (6890*3) represents 6890 vertices, each with the 3D coordinates of x, y, and z in 3D space; the vector of key point coordinates (24*3) represents 24 key points of the human body, each with the 3D coordinates of x, y, and z in 3D space.

[0042] Then, the 3D human key points are converted to 2D key points. Specifically, based on the camera parameters of the target user in the target picture frame, the coordinates of each 3D human key point of the target user are converted to the coordinates of the 2D human key point, including: using the external parameters in the camera parameters to convert the position coordinates of each 3D human key point of the target user in the target picture frame to camera coordinates; using perspective projection to convert the camera coordinates to normalized image coordinates; using the internal parameters of the camera that shoots the acquisition video, converting the normalized image coordinates to the pixel coordinates of the coordinates of the corresponding 2D human key point.

[0043] Projecting 3D human key points to 2D key points in the image includes the following steps: (1) Use the external parameters in the camera parameters of the 3D human posture information to convert the 3D world coordinates Convert to camera coordinates

[0044]

[0045] in, is the rotation matrix, t is the translation vector, including t x ,t y ,tz .

[0046] (2) Use perspective projection to transform the camera X coordinate Convert to normalized image coordinates

[0047]

[0048]

[0049] in, is the camera Y coordinate, is the camera Z coordinate.

[0050] (3) Use the camera internal parameters of the captured video to normalize the image coordinates Convert to pixel coordinates

[0051]

[0052]

[0053] in and is the focal length, and It is the optical center.

[0054] Thus, the coordinates of each 3D human body key point of the target user in each picture frame can be converted into the pixel coordinates of the 2D human body key point one by one.

[0055] According to the pixel coordinates of the 2D human body key points, the hand area of ​​the target user in each picture frame can be determined. Specifically, the method of determining the hand area of ​​the target user in the target picture frame based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame includes: determining the coordinates of the wrist key point and the hand key point in the target picture frame based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame; taking the target wrist key point as a reference point, based on the coordinates of the target wrist key point and the coordinates of the corresponding hand key point, calculating the distance between the target wrist key point and the corresponding hand key point, the target wrist key point being the left wrist key point or the right wrist key point; generating a first square area perpendicular to the line between the target wrist key point and the corresponding hand key point, wherein the side length of the first square area is greater than the distance; determining the second square area circumscribed by the first square area as the left hand area or the right hand area corresponding to the target wrist key point of the target user in the target picture frame.

[0056] After obtaining the 2D human body key points, the coordinates of the wrist key points and hand key points can be determined, for example Figure 3 The right wrist 20, the left wrist 21, the right hand 22, and the left hand 23.

[0057] Taking the key point of the wrist as the reference point, first find the distance between the wrist point and the corresponding hand point. For example, taking the key point of the right wrist 20 as the reference point, a square area is generated perpendicular to the line connecting the key point of the right wrist 20 and the key point of the right hand 22, such as Figure 3 The dotted square 100 is shown in the figure. The side length of the square 100 is greater than the distance between the key points of the right wrist 20 and the key points of the hand 22, for example, 1.2 times of the distance, and the dotted square 100 is obtained, and then the circumscribed square area 200 of the square 100 is obtained. This square area 200 is the estimated right hand area, and the left hand area can be determined similarly.

[0058] After obtaining the 2D hand region of the target user in each picture frame, the hand region in each picture frame may be reconstructed in 3D to obtain the 3D hand posture information of the target user in the picture frame.

[0059] Optionally, the 3D reconstruction is performed based on the hand area of ​​the target user in the target picture frame to obtain 3D hand posture information of the target user in the target picture frame, including: determining a left hand frame corresponding to the left hand area and a right hand frame corresponding to the right hand area included in the hand area of ​​the target user in the target picture frame; obtaining a left hand frame image corresponding to the left hand frame and a right hand frame image corresponding to the right hand frame in the target picture frame; and respectively predicting the left hand frame image and the right hand frame image by a hand reconstruction algorithm to obtain 3D left hand posture information and 3D right hand posture information of the target user in the target picture frame.

[0060] Similar to the 3D reconstruction of the human body area, first determine the detection frame corresponding to the hand area in each image frame, including the left hand frame corresponding to the left hand area and the right hand frame corresponding to the right hand area. According to the hand frame, the left and right hand images are cut out from each image frame respectively to obtain the left hand frame image corresponding to the left hand frame and the right hand frame image corresponding to the right hand frame.

[0061] The left hand frame image and the right hand frame image are predicted respectively by the hand reconstruction algorithm, and the 3D left hand posture information and the 3D right hand posture information are obtained one by one, so as to realize the 3D reconstruction of the hand area of ​​the target user in each picture frame.

[0062] Optionally, the hand reconstruction algorithm includes a FrankMocap algorithm. The FrankMocap algorithm is used to reconstruct the hand branch of the human body region. The FrankMocap algorithm is a regression algorithm based on an image. Because the hand is a small non-rigid target with a large range of motion changes, the regression algorithm based on an image will improve the accuracy of 3D reconstruction. The network structure of the FrankMocap algorithm is an encoder-decoder structure, such as Figure 4 As shown, the input of the FrankMocap algorithm is a 224*224 left hand frame image or a right hand frame image, which is first subjected to feature encoding by the encoder (step 302), wherein the encoder uses Restnet50 feature extraction and global pooling operations. Then, the features extracted in step 302 are decoded by the decoder (step 304), wherein the Dncoder uses a fully connected layer (FC layer) to directly regress the 3D posture information of the left hand region or the right hand region, including 3D posture parameters, shape parameters, and camera parameters.

[0063] Under normal circumstances, the vectors of the 3D posture parameters of the left and right hands are both 15*3, where 15 means that a single hand has 5 fingers, each finger has 3 joints, and each hand has 15 3D key points. 3 means the 3D coordinates of the X / Y / Z dimensions of the 3D space.

[0064] Thus, the 3D hand posture information of the target user in each picture frame can be obtained.

[0065] The target in the human hand area is relatively small, and the accuracy of directly detecting the hand area cannot be guaranteed. It is difficult to obtain accurate detection results through ordinary target detection. The embodiment of the present application obtains the hand area based on the human body area, which can improve the accuracy of the detected hand area.

[0066] Step 106 , splicing the 3D hand gesture information and 3D body gesture information of the target user in the same picture frame to obtain the 3D spliced ​​body gesture information of the target user in each picture frame.

[0067] For multiple picture frames, the 3D hand posture information and 3D human body posture information obtained by 3D reconstruction based on the same picture frame are spliced ​​respectively, so as to splice the 3D human body posture information of, for example, 24 key points and the 3D hand posture information of 30 key points in the same picture frame.

[0068] Optionally, the 3D left hand posture information and the 3D right hand posture information included in the 3D human body posture information of the target user in the same picture frame are removed to obtain updated 3D human body posture information of the target user; the 3D left hand posture information and the 3D right hand posture information included in the 3D hand posture information of the target user in the same picture frame are spliced ​​with the updated 3D human body posture information to obtain the 3D spliced ​​human body posture information of the target user in the same picture frame.

[0069] In this embodiment, the 3D posture information corresponding to the left and right hands is first removed from the 3D human posture information of the human body area. For example, the posture information of the key points of the left and right hands is removed from the 24 human body key points, and the posture information of 21 human body key points is left. Add the 3D posture information of 15 key points in the left hand area and the 3D posture information of 15 key points in the right hand area reconstructed based on the 3D human posture information in the same picture frame, and a total of 22+15+15=52 key points are obtained. Therefore, the dimension of the spliced ​​human posture parameters is 52*3, wherein the human body shape parameters and camera parameters in the 3D spliced ​​human body posture information can adopt the human body shape parameters and camera parameters in the original 3D human body posture information.

[0070] Step 108, inputting the continuous image frames corresponding to the 3D spliced ​​human posture information of the target user into the action classification model to obtain the action classification result of the target user, wherein the action classification model is trained using the user's 3D spliced ​​human posture information as a sample and the user's action classification result as a label.

[0071] Based on the continuous image frames corresponding to the target user's 3D stitching human posture information, the target user's large-scale movements can be more prominent, and the target user's movements can be displayed through more key points of 3D posture information, thus obtaining richer human 3D posture information. By adding hand movement analysis, more action categories can be supported, covering almost all actions that a person can do.

[0072] In one embodiment, the action classification model includes a Transformer autoregressive model. Figure 5 It is a schematic diagram of the network architecture of the action classification model of an embodiment of the present application.

[0073] The present application embodiment proposes a Transformer autoregressive model. The Transformer's self-attention layer can fully explore the relationship between 52 human body key points and also capture the connection between each frame of the video. The algorithm flow of the Transformer autoregressive model is as follows: Figure 5As shown, 3D posture information B*200*52*3 is input (step 402), where B is the batch size, 200 is the number of picture frames selected each time, and 52*3 is the human posture parameter vector of 52 key points spliced ​​together. Then, the input feature vector is converted to B*200*156 by reshaping (step 404).

[0074] In addition, since the dimension of the input feature itself is not large, a word embedding dimension that is too large will cause the Transformer autoregressive model to overfit. Therefore, the word embedding dimension can be selected as 128, 256 or 512, for example. Taking the word embedding dimension of 128 as an example, the B*200*156 feature is converted to B*200*128 through a linear layer (step 406). Then, the feature order is transformed by reordering (Permute) to obtain a feature of 200*B*128 (step 408). Next, the feature is successively positionally encoded (positional encoding) (step 410) and encoded by the Transformer encoder (Transformer Encoder) (step 412), and the obtained 200*B*128-dimensional feature is input into the average pooling layer for average pooling (step 414) to obtain a B*128-dimensional feature. Finally, the feature is mapped to the corresponding action classification result B*C through a linear layer, where C represents the action classification result, that is, the action category of the target user.

[0075] For example, the set action category C may include moving things, fighting, playing with mobile phones, sleeping on the post, smoking, falling, intrusion, stealing, fainting, and other actions. When the action classification model recognizes that the target user of the current video has abnormal actions, an alarm can be sent to the platform.

[0076] After obtaining the 3D posture parameters of the preliminary fusion of the human body area and the hand area in step 106, the 3D posture parameters may have inter-frame jitter or abnormal posture problems, which is not conducive to subsequent action classification and recognition. In order to eliminate the influence of jitter, optionally, before the continuous image frames corresponding to the 3D spliced ​​human posture information of the target user are input into the action classification model, it also includes: smoothing the 3D spliced ​​human posture information of the target user by a SmoothNet algorithm to remove noise in the 3D spliced ​​human posture information.

[0077] In one embodiment, noise removal may be performed by a low pass filter.

[0078] In another embodiment, a smoothing scheme SmoothNet algorithm based on a deep learning model may be selected.

[0079] Combination Figure 6,The SmoothNet algorithm, for example, inputs 3D pose parameters of 32 picture frames each time (step 502), takes picture frames in the picture frames of the video with a step length of 16 frames, and the current 32 frames and the next 32 frames have an overlapping area of ​​16 frames. The SmoothNet algorithm can stretch the 32*52*3 3D pose parameters into 32*156 frames of position data (step 504). Then, based on the position data, the velocity (first-order difference) and acceleration (second-order difference) information are obtained, using a multi-layer fully connected layer network, such as Figure 6 The FC+Leaky ReLU layer, N Blocks layer, and FC layer are used to extract the feature vectors of position, velocity, and acceleration respectively. The extracted features of the three are fused in the fusion layer (step 504) to obtain the fusion feature, that is, the smoothed result 32*156. The final posture parameter 32*52*3 is then predicted based on the fusion feature. Since the added velocity features and acceleration features contain rich inter-frame motion relationships, the noise in the 3D posture parameters can be effectively removed to ensure the consistency and accuracy of the 3D posture parameters. Then, after the continuous picture frames corresponding to the smoothed 3D spliced ​​human posture information are input into the action classification model, a more accurate action classification result can be obtained.

[0080] Next, combine Figure 7 , the overall process of the user action classification method of the embodiment of the present application is described. Figure 7 As shown, including: For the monitored video picture frame sequence, the pedestrian detection and tracking algorithm is first called to detect the human body area of ​​the target user in each picture frame sequence (step 602), and then the human body area is 3D reconstructed (step 604). Next, the 3D human body key points are directly regressed from the 3D reconstructed human body area posture information (step 606), and then the 3D human body key point world coordinates are projected to the image pixel coordinates of the 2D human body key point in combination with the camera parameters predicted by the 3D reconstruction (step 608). In the 2D image pixel coordinate system, the 2D key points of the left and right wrists are used as reference points to estimate the position of the hand frame according to the rules (step 610). After obtaining the position of the hand frame, the hand area can be 3D reconstructed (step 612), and after obtaining the posture information of the 3D hand area, the posture information of the human body area and the posture information of the hand area are aligned and spliced ​​in the same coordinate system (step 614). After the alignment and stitching is completed, the inter-frame jitter or abnormal noise problem in the 3D stitching posture information is processed by a smoothing algorithm to obtain continuous and stable 3D stitching human posture information (step 616). The 3D stitching human posture information is input into the Transformer autoregressive model for action classification to obtain the final action category (step 618).

[0081] In an embodiment of the present application, 3D reconstruction is performed based on the human body area of ​​the target user in each picture frame of the captured video to obtain the 3D human body posture information of the target user in each picture frame; based on the 3D human body posture information of the target user in each picture frame, the 3D hand posture information of the target user in each picture frame is determined; the 3D hand posture information and the 3D human body posture information of the target user in the same picture frame are spliced ​​to obtain the 3D spliced ​​human body posture information of the target user in each picture frame; the continuous picture frames corresponding to the 3D spliced ​​human body posture information of the target user are input into the action classification model to obtain the action classification result of the target user, and the action classification model is based on the 3D spliced ​​human body posture information of the user. It is trained by taking human posture information as samples and user action classification results as labels. In this way, human actions, environmental scenes and human shapes are decoupled through 3D reconstruction, and relatively small human hand areas are accurately detected based on 3D human posture information. The 3D spliced ​​human posture information of multiple key points including the human body area and the hand area is input into the action classification model. The action classification model can focus only on human actions and has significant anti-interference ability against local occlusion, blur, harsh environment and human shape differences in the picture. It is suitable for various complex scenes, greatly reducing the difficulty of learning the action classification model and improving the recognition accuracy of the action classification model. The action classification results of the target user are more accurate.

[0082] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides a user action classification device 2000, including a processor 2400 and a memory 2200, and the memory 2200 stores programs or instructions that can be run on the processor 2400. When the program or instructions are executed by the processor 2400, the various steps of the above-mentioned user action classification method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0083] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of any of the above-mentioned user action classification method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The readable storage medium includes a computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0084] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to enable a computer to implement the various processes of any of the above-mentioned user action classification method embodiments when executed, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0087] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A user action classification method, characterized in that: include: Performing 3D reconstruction based on the human body area of ​​the target user in each picture frame of the captured video to obtain 3D human body posture information of the target user in each picture frame; Determine the 3D hand gesture information of the target user in each picture frame based on the 3D human body gesture information of the target user in each picture frame; splicing the 3D hand gesture information and the 3D human body gesture information of the target user in the same picture frame to obtain the 3D spliced ​​human body gesture information of the target user in each picture frame; The continuous image frames corresponding to the 3D spliced ​​human posture information of the target user are input into the action classification model to obtain the action classification result of the target user. The action classification model is trained with the user's 3D spliced ​​human posture information as a sample and the user's action classification result as a label.

2. The method according to claim 1, characterized in that The 3D reconstruction is performed based on the human body area of ​​the target user in each picture frame of the captured video to obtain the 3D human body posture information of the target user in each picture frame, including: Tracking and detecting the target user in the collected video by using a tracking algorithm to obtain a human body frame corresponding to the human body region of the target user in each picture frame of the collected video; Obtaining a human body frame image corresponding to a human body frame in each picture frame; Each human frame image is predicted by a human posture shape estimation algorithm to obtain the 3D human posture information of the target user in each picture frame.

3. The method according to claim 2, characterized in that The 3D human body posture information includes 3D human body posture parameters, human body shape parameters and camera parameters, and the determining of the 3D hand posture information of the target user in each picture frame based on the 3D human body posture information of the target user in each picture frame includes: Inputting the 3D human body posture parameters and human body shape parameters of the target user in the target picture frame into a parameterized human body model to output each 3D human body key point of the target user in the target picture frame, including the coordinates and categories of the key points; Based on the camera parameters of the target user in the target picture frame, the coordinates of each 3D human body key point of the target user are converted into the coordinates of 2D human body key points; Determine a hand region of the target user in the target picture frame based on coordinates and categories of each 2D human body key point of the target user in the target picture frame; 3D reconstruction is performed based on the hand area of ​​the target user in the target picture frame to obtain 3D hand posture information of the target user in the target picture frame, and the 3D hand posture information includes 3D hand posture parameters and 3D hand shape parameters.

4. The method according to claim 3, characterized in that The step of converting the coordinates of each 3D human body key point of the target user into the coordinates of each 2D human body key point based on the camera parameters of the target user in the target picture frame includes: Using the external parameters in the camera parameters, converting the position coordinates of each 3D human body key point of the target user in the target image frame into camera coordinates; Using perspective projection, transform the camera coordinates into normalized image coordinates; The normalized image coordinates are converted into pixel coordinates corresponding to the coordinates of 2D human body key points using the intrinsic parameters of the camera that captured the acquisition video.

5. The method according to claim 3, characterized in that: The determining, based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame, a hand area of ​​the target user in the target picture frame includes: Based on the coordinates and categories of each 2D human body key point of the target user in the target picture frame, respectively determine the coordinates of the wrist key point and the hand key point in the target picture frame; Taking a target wrist key point as a reference point, and based on the coordinates of the target wrist key point and the coordinates of the corresponding hand key point, calculating the distance between the target wrist key point and the corresponding hand key point, wherein the target wrist key point is a left wrist key point or a right wrist key point; Generate a first square area perpendicular to a line connecting the target wrist key point and the corresponding hand key point, wherein a side length of the first square area is greater than the distance; A second square area circumscribed to the first square area is determined as a left hand area or a right hand area corresponding to a target wrist key point of the target user in the target picture frame.

6. The method according to claim 3, characterized in that The performing 3D reconstruction based on the hand region of the target user in the target picture frame to obtain 3D hand posture information of the target user in the target picture frame includes: Determine a left hand frame corresponding to a left hand region and a right hand frame corresponding to a right hand region included in a hand region of the target user in the target picture frame; Acquire a left-hand frame image corresponding to the left-hand frame and a right-hand frame image corresponding to the right-hand frame in the target picture frame; The left hand frame image and the right hand frame image are predicted respectively by a hand reconstruction algorithm to obtain 3D left hand posture information and 3D right hand posture information of the target user in the target picture frame.

7. The method according to claim 6, characterized in that The step of splicing the 3D hand gesture information and the 3D human body gesture information of the target user in the same picture frame to obtain the 3D spliced ​​human body gesture information of the target user in each picture frame includes: Removing the 3D left hand posture information and the 3D right hand posture information included in the 3D human body posture information of the target user in the same picture frame to obtain updated 3D human body posture information of the target user; The 3D left hand posture information and the 3D right hand posture information included in the 3D hand posture information of the target user in the same picture frame are spliced ​​with the updated 3D human body posture information to obtain the 3D spliced ​​human body posture information of the target user in the same picture frame.

8. The method according to claim 1, characterized in that Before inputting the continuous picture frames corresponding to the 3D spliced ​​human body posture information containing the target user into the action classification model, the method further includes: The 3D stitched human body posture information of the target user is smoothed by a SmoothNet algorithm to remove noise in the 3D stitched human body posture information.

9. The method according to claim 6, characterized in that The human body posture shape estimation algorithm includes a video inference VIBE algorithm for human body posture shape estimation, and the hand reconstruction algorithm includes a FrankMocap algorithm.

10. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to perform the steps of the method according to any one of claims 1 to 9.

Images