Multi-person action recognition method and device, electronic equipment and readable storage medium

Abstract

The invention relates to a multi-person action recognition method and device, electronic equipment and a readable storage medium. The multi-person action recognition method comprises the following steps: acquiring video data; key feature point detection is carried out on the video data, and human body key feature points associated with a figure object in each image frame of the video data are obtained; performing character object matching on the adjacent image frames based on the obtained human body key feature points associated with the character object; processing the human body key feature points corresponding to the next image frame in the adjacent image frames according to the character object matching result; inputting feature point information of human body key feature points of the character object in continuous K processed image frames into a preset multi-dimensional space-time diagram convolution model every R image frames to obtain an action recognition result of the character object in the video data; wherein K is larger than R, and the number of the character objects is one or more. According to the invention, the multi-person action recognition process can be simplified, and the multi-person action recognition efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a method, device, electronic device, and readable storage medium for multi-person motion recognition. Background Art

[0002] As a development direction of deep learning, action recognition has developed rapidly in recent years and has great application value.

[0003] In the related technology, multi-person action recognition is mainly based on the action recognition of a single person, and involves multiple models in series at the same time; for example, the existing action recognition solution is to first detect each person, then track each person, and finally identify the person through a time-series action of the specified person. The specified person here may be one or more people; however, the solution does not support the action recognition of multiple groups of one or more people at the same time, and recognizes the actions one by one.

[0004] It can be seen that the multi-person action recognition method in the related art has a complicated process and low recognition efficiency. Summary of the invention

[0005] In view of this, the present disclosure provides a multi-person motion recognition method, device, electronic device and readable storage medium, which can simplify the multi-person motion recognition process and improve the efficiency of multi-person motion recognition.

[0006] The technical solution of the present disclosure is achieved as follows:

[0007] In one embodiment, a method for multi-person action recognition is provided, the method comprising:

[0008] Get video data;

[0009] Performing key feature point detection on the video data to obtain key feature points of a human body associated with a human object in each image frame of the video data;

[0010] Performing character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object;

[0011] Processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result;

[0012] Every R image frames, the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames is input into a preset multi-dimensional spatiotemporal graph convolution model to obtain the action recognition result of the human object in the video data; wherein K is greater than R, and the human object is one or more.

[0013] Further, the processing of the key human feature points corresponding to the next image frame in the adjacent image frames according to the human object matching result includes:

[0014] In response to the matching result that the human body key feature points corresponding to the human object do not match, and the human body key feature points are detected at corresponding positions in the previous image frame in the adjacent image frames;

[0015] If the preset feature point discarding condition is currently met, the key human feature points of the human body corresponding to the human object in the next image frame in the adjacent image frames are deleted; or,

[0016] If the preset feature point discarding condition is not met at present, the motion flow information of the key feature points of the human body of the person object corresponding to the previous image frame in the adjacent image frames is used to fill the key feature points of the human body corresponding to the next image frame in the adjacent image frames;

[0017] Among them, the preset feature point discarding condition is: the number of consecutive mismatches reaches a preset threshold, and the number of key human feature points detected for the person object in the next image frame in the current adjacent image frame is less than the preset point threshold; the preset point threshold is less than the total number of preset human key feature points.

[0018] Further, the processing of the key human feature points corresponding to the next image frame in the adjacent image frames according to the human object matching result includes:

[0019] In response to the matching result being a match of key human feature points corresponding to the person object, it is determined that the person object corresponding to the subsequent image frame in adjacent image frames and the person object corresponding to the previous image frame are the same person object.

[0020] Further, the processing of the key human feature points corresponding to the next image frame in the adjacent image frames according to the human object matching result includes:

[0021] In response to the matching result being that the key human feature points of the person object do not match, and no key human feature points are detected at corresponding positions in a previous image frame in adjacent image frames, it is determined that the person object in a subsequent image frame in the adjacent image frames is a newly added person object.

[0022] Furthermore, the training method of the preset multi-dimensional spatiotemporal graph convolution model includes:

[0023] Establishing an initial multi-dimensional spatio-temporal graph convolution model; wherein the input of the initial multi-dimensional spatio-temporal graph convolution model is five-dimensional [N, M, T, V, C]; and the output is three-dimensional [N, M, S];

[0024] Get training samples;

[0025] Training the initial multidimensional space-time graph convolution model based on the acquired training samples to obtain a preset multidimensional space-time graph convolution model;

[0026] Among them, N represents the number of video data used for model training, M represents the maximum number of human objects that can be recognized by the preset multi-dimensional spatiotemporal graph convolution model, T represents the time dimension of an action, V represents the total number of human key feature points used; C represents the dimension of prediction for each human key feature point, and S represents the category of the action.

[0027] further,

[0028] The setting of K is related to the action to be recognized;

[0029] The setting of R is related to the acquisition speed of acquiring the video data.

[0030] Further, performing character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object includes:

[0031] Based on the obtained key feature points of the human body associated with the human object, a key point similarity algorithm is used to perform human object matching on adjacent image frames.

[0032] In one embodiment, a multi-person action recognition device is provided, the device comprising:

[0033] An acquisition unit, configured to acquire video data;

[0034] A detection unit is configured to perform key feature point detection on the video data to obtain key feature points of a human body associated with a human object in each image frame of the video data;

[0035] A matching unit, configured to perform character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object;

[0036] A processing unit configured to process key human feature points corresponding to a subsequent image frame in adjacent image frames according to a human object matching result;

[0037] The recognition unit is configured to execute every R image frames, input the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames into a preset multi-dimensional spatiotemporal graph convolution model, and obtain the action recognition result of the human object in the video data; wherein K is greater than R, and the human object is one or more.

[0038] In another embodiment, an electronic device is provided, comprising:

[0039] processor;

[0040] a memory for storing executable instructions for the processor;

[0041] Wherein, the processor is configured to execute the executable instructions to implement the multi-person action recognition method as described in any one of the above items.

[0042] In another embodiment, a computer-readable storage medium, when at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to implement any of the above multi-person motion recognition methods.

[0043] It can be seen from the above technical solutions that the multi-person action recognition method, device, electronic device and readable storage medium of the disclosed embodiment detect key points of video data, obtain key human feature points associated with the person object in each image frame of the video data, and process the key human feature points of the corresponding person object in the next image frame in the adjacent image frames by matching the key human feature points of the person objects in the adjacent frames; every R image frames, the feature point information of the key human feature points of the person objects in the consecutive K processed image frames is input into the preset multi-dimensional spatio-temporal graph convolution model to obtain the action recognition results of all the person objects in the video data. This solution can directly perform action recognition on multiple people without splitting them into individual people for action recognition, which simplifies the multi-person action recognition process and improves the efficiency of multi-person action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flow chart of a training method for a preset multi-dimensional spatiotemporal graph convolution model according to an exemplary embodiment;

[0046] Figure 2 is a schematic diagram of key feature points of a human body of a person object according to an exemplary embodiment;

[0047] Figure 3 is a schematic diagram of a network structure of a multi-dimensional spatiotemporal graph convolution model according to an exemplary embodiment;

[0048] Figure 4 is a flow chart of a method for multi-person action recognition according to an exemplary embodiment;

[0049] Figure 5 is a flow chart showing a method of processing key feature points of a human body of a human object detected in an image frame according to an exemplary embodiment;

[0050] Figure 6 is a flow chart of another multi-person action recognition method according to an exemplary embodiment;

[0051] Figure 7 is a schematic diagram of key feature points of a human body obtained by performing key point detection on a certain image frame in video data according to an exemplary embodiment;

[0052] Figure 8 is a schematic diagram showing the relationship between a character object and key feature points of a human body according to an exemplary embodiment;

[0053] Fig. 9 is a schematic diagram of a human body key feature point of a human object after processing according to an exemplary embodiment;

[0054] Fig.10 is a schematic diagram of a multi-person motion recognition device according to an exemplary embodiment;

[0055] Fig.11 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and examples.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] In the related technology, multi-person action recognition is mainly based on the action recognition of a single person, and involves multiple models in series at the same time; for example, the existing action recognition solution is to first detect each person, then track each person, and finally identify the person through a time-series action of the specified person. The specified person here may be one or more people; however, the solution does not support the action recognition of multiple groups of one or more people at the same time, and recognizes the actions one by one; there is also a solution that directly uses an entire single-person video for 3D convolution for action recognition.

[0059] In these related technologies, the overall process of action recognition for either a single person or multiple people is relatively cumbersome, and the recognition efficiency is low; if multiple levels are connected in series, errors will propagate to a certain extent, which will also lead to low recognition accuracy.

[0060] Based on this, the embodiments of the present disclosure provide a method, device, electronic device and readable storage medium for multi-person action recognition, which can simplify the multi-person action recognition process and improve the multi-person action recognition effect; compared with the multi-stage series-connected related action recognition method, it can also improve the accuracy of multi-person action recognition;

[0061] In addition, since the present disclosure simplifies the overall multi-person action recognition process, the computing power overhead of the entire solution is greatly reduced, so that the solution can be deployed not only on the server side but also on the terminal side, which is more cost-effective.

[0062] In the disclosed embodiment, before performing multi-person action recognition, it is necessary to obtain a preset multi-dimensional spatiotemporal graph convolution model in advance.

[0063] The preset multi-dimensional spatiotemporal graph convolution model may be pre-established and stored and directly acquired for use, or it may be established and trained for subsequent multi-person action recognition in the implementation of the present disclosure, which is not limited in the embodiments of the present disclosure.

[0064] Figure 1 FIG. 1 is a flow chart of a training method for a preset multi-dimensional spatiotemporal graph convolution model according to an exemplary embodiment. Figure 1 As shown, the training method of the preset multi-dimensional spatiotemporal graph convolution model of the embodiment of the present disclosure mainly includes the following steps 101 to 103.

[0065] Step 101, establish an initial multi-dimensional spatio-temporal graph convolution model; wherein the input of the initial multi-dimensional spatio-temporal graph convolution model is five-dimensional [N, M, T, V, C]; and the output is three-dimensional [N, M, S].

[0066] The initial multi-dimensional spatiotemporal graph convolution model is built based on the spatiotemporal convolutional graph neural network (STGCN).

[0067] Where N represents the number of video data used for model training, that is, the batch_size of model training;

[0068] M represents the maximum number of human objects that can be recognized by the preset multidimensional spatiotemporal graph convolution model. For example, it can be set to 5, that is, the maximum number of human objects recognized at one time is 5; M here is the dimension corresponding to the multidimensional spatiotemporal convolutional graph neural network;

[0069] T represents the time dimension of an action. For example, if an action takes 50 frames, then T=50.

[0070] V represents the total number of key human feature points used; Figure 2 is a schematic diagram showing key feature points of a human body of a person object according to an exemplary embodiment, Figure 2 In the figure, x represents the horizontal axis of the image, and y represents the vertical axis of the image. Taking the 17 key feature points of the human body in the COCO data set as an example, they are: nose 0, left eye 1, right eye 2, left ear 3, right ear 4, left shoulder 5, right shoulder 6, left elbow 7, right elbow 8, left wrist 9, right wrist 10, left hip 11, right hip 12, left knee 13, right knee 14, left ankle 15, right ankle 16. When the embodiment of the present application is specifically implemented, the selection method of the key feature points of the human body is not limited, such as 18 key feature points of the human body can also be selected to achieve it.

[0071] C represents the dimension of prediction for each key feature point of the human body. In the embodiment of the present disclosure, C=2 can be set to represent the coordinates of the key feature points of the human body, such as the coordinates (x coordinate, y coordinate) of the current key feature point of the human body;

[0072] S represents the category of the action. The category of the action is set according to the specific action. In the specific implementation, multiple action categories can be identified, or only one action category can be identified. For example, if the identified action is fall detection, the action category is fall and non-fall.

[0073] Figure 3 FIG. 1 is a schematic diagram of a network structure of a multi-dimensional spatiotemporal graph convolution model according to an exemplary embodiment. Figure 3 As shown in the figure, the network structure diagram of the multi-dimensional spatiotemporal graph convolution model is mainly divided into three parts: normalization, changes in time and space dimensions, and feature classification. The specific implementation is as follows:

[0074] The first part: Normalization, normalize the input matrix, through Figure 3 The IN and BN units in the code are implemented as follows:

[0075] Normalization is performed in the time and space dimensions (V×C). That is, the position characteristics (x and y) of a key feature of the human body in different image frames are normalized. The positions of key feature points of the human body in different image frames vary greatly. If normalization is not performed, it will be detrimental to the convergence of the algorithm.

[0076] The positions of the key feature points of the human body in different image frames of different batches basically obey random distribution, which will not cause the normalization results of different batches to differ too much and cause fluctuations in accuracy.

[0077] The second part, time and space dimension changes, through multiple ST-GCN units, alternately use GCN and TCN to transform the time and space dimensions, as follows:

[0078] The spatial dimension is the dimension of the key feature points of the human body, and the temporal dimension is the number of image frames. After the spatiotemporal convolution of all ST-GCN units, the feature dimension of the key feature points of the human body increases, and the dimension of the key frame decreases.

[0079] The third part, feature classification, uses average pooling and fully connected layers (or FCN) to classify features. Figure 3 It is implemented using three units: POOL, FC and OUT; the specific implementation is as follows:

[0080] Average pooling on a graph can be understood as reading out the graph, that is, the process of summarizing the features of the key feature points of the human body to represent the features of the entire graph. The read out here is the process of summarizing the features of the key feature points of the human body to represent the features of the action. You can use statistical methods, such as finding max, sum, meanmax, sum, mean, etc. for the key feature points of the human body. In specific implementation, you can use mean, which has better robustness.

[0081] Step 102: Obtain training samples.

[0082] The training samples are feature information of key feature points of human bodies of human objects detected corresponding to the video data, and the action category corresponding to each human object.

[0083] Step 103: Train the initial multidimensional space-time graph convolution model based on the acquired training samples to obtain a preset multidimensional space-time graph convolution model.

[0084] At this point, the preset multi-dimensional spatiotemporal graph convolution model is completed and can be used for multi-person action recognition.

[0085] In the embodiment of the present application, there is no restriction on the specific training of the model and the convergence process, and the training principle of the stgcn model can be used to train M-stgcn.

[0086] Figure 4 is a flow chart of a method for multi-person action recognition according to an exemplary embodiment. Figure 4 As shown, the multi-person action recognition method of the embodiment of the present disclosure mainly includes the following steps 401 to 404.

[0087] Step 401, obtaining video data.

[0088] In the disclosed embodiment, the video data may be derived from video data collected and stored in advance, and based on the stored video data, multi-person action recognition of historical data can be realized. The stored video data may be data stored in any form in any storage device.

[0089] Video data can also come from real-time collected data. Based on real-time acquired video data, it is possible to quickly identify and respond to multiple people's actions in real-time collected scenes, such as home scenes. Real-time collected video data can come from various video collection terminal devices.

[0090] Step 402: perform key feature point detection on the video data, and obtain key feature points of the human body associated with the human object in each image frame of the video data.

[0091] In the disclosed example, step 402 can be implemented by first performing human key point detection on each image frame in the video data using a human key point detection model established based on the Bottom_up human key point detection algorithm to obtain human key feature points in each image frame;

[0092] Then, the clustering algorithm and the internal connection between the key human feature points are used to determine which key human feature points correspond to the same person object, that is, the key human feature points associated with the person object in each image frame of the video data are obtained; the person object is identified using the ID, and the key human feature points of the same person object belong to the ID of the same person object.

[0093] Figure 2 The situation shown in the figure is 17 key human feature points of the human object. In the specific implementation process, according to the situation of the human object existing in the image frame, the number of key human feature points of any human image detected is any value between 1 and 17 key human feature points.

[0094] Step 403 : performing person object matching on adjacent image frames based on the obtained key human feature points associated with the person object.

[0095] In specific implementation, the spatial overlap of two key features can be calculated by calculating the similarity of the key feature points of the two people's bodies, and then it can be determined whether the person object in the previous image frame of two adjacent image frames is matched in the next image frame.

[0096] Step 404 , processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result.

[0097] Figure 5 FIG. 1 is a flow chart showing a method for processing key feature points of a human body of a human object detected in an image frame according to an exemplary embodiment. Figure 5 As shown, the method for processing key human feature points of a human object detected in an image frame in an embodiment of the present disclosure mainly includes the following steps 501 to 505.

[0098] Step 501, obtaining matching results.

[0099] Step 502: In response to the matching result that the key feature points of the human body corresponding to the human body object match, determine that the human body object corresponding to the next image frame in the adjacent image frames is the same as the human body object corresponding to the previous image frame. End this process.

[0100] When a person object in a subsequent image frame of adjacent frames matches a person object in a previous image frame, they are determined to be the same person object, and the person object in the subsequent image frame uses the identification ID of the corresponding person object in the previous image frame.

[0101] Step 503: In response to the matching result that the key human feature points of the human object do not match and no key human feature points are detected at the corresponding position in the previous image frame of the adjacent image frames, the human object in the next image frame of the adjacent image frames is determined to be a newly added human object. This process ends.

[0102] In step 503, if the key human feature points of a person object in the latter image frame of adjacent image frames are not matched with the key human feature points of the corresponding person object in the previous image frame, it is considered that the person object in the latter image frame of the adjacent image frames is a newly added person object. Therefore, the person object is directly treated as a newly added person object and a new identification ID is assigned.

[0103] Step 504, in response to the matching result that the human body key feature points corresponding to the human object do not match, and the human body key feature points are detected at the corresponding position in the previous image frame in the adjacent image frame, determine whether the preset feature point discarding conditions are currently met. If so, execute step 505; otherwise, execute step 506.

[0104] Among them, the preset feature point discarding conditions are: the number of consecutive mismatches reaches the preset threshold, and the number of key human feature points detected for the human object in the next image frame in the current adjacent image frame is less than the preset point threshold; the preset point threshold is less than the total number of preset human key feature points.

[0105] In the specific implementation, there are no specific restrictions on the preset number threshold and the preset point threshold. For example, the preset number threshold can be set to 3, and the preset point threshold can be set to half of the total number of preset human body key feature points rounded down. For example, if the total number of preset human body key feature points is 17, it can be set to 8; if the total number of preset human body key feature points is 18, it can be set to 9.

[0106] If the matching has failed for three or more times in a row, and the number of key human feature points detected for the person object in the next image frame in the current adjacent image frame is less than 8, it is determined that the person object corresponding to the image frame has been lost or has left the screen of the currently collected video data.

[0107] Step 505: Delete the key feature points of the human body of the corresponding human object in the next image frame among the adjacent image frames.

[0108] This situation indicates that the person objects corresponding to these key points of human body features no longer exist, and these key feature points do not need to be retained and can be directly deleted.

[0109] Step 506: Use the motion flow information of the key feature points of the human body of the person object corresponding to the previous image frame in the adjacent image frames to complete the key feature points of the human body corresponding to the next image frame in the adjacent image frames.

[0110] If only a certain person object in a certain image frame is missed or misdetected, the motion flow information of the corresponding key feature points of the person object in the adjacent previous image frame is used to complete it, and the identification ID of the person object is used.

[0111] Step 405 , every R image frames, input the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames into a preset multi-dimensional spatiotemporal graph convolution model to obtain the action recognition result of the human object in the video data.

[0112] Among them, K is greater than R, and the number of character objects is 1 or more.

[0113] In this step, R and K are integers, wherein the setting of K is related to the action to be recognized; and the setting of R is related to the acquisition speed of the video data.

[0114] In specific implementation, K and T can be set to the same value or different values.

[0115] Assuming that the acquisition speed of the image frames in the video data is 25 frames per second, and the action to be identified is fall detection, R can be set to 25 and K to 50, but it is not limited thereto.

[0116] In the disclosed embodiment, key point detection is performed on the video data to obtain the key human feature points associated with the human object in each image frame of the video data, and the key human feature points of the human object in the adjacent frame are matched to process the key human feature points of the human object in the next image frame of the adjacent image frames; every R image frames, the feature point information of the key human feature points of the human object in the consecutive K processed image frames is input into the preset multi-dimensional spatio-temporal graph convolution model to obtain the action recognition results of all the human objects in the video data. This solution can directly perform action recognition on multiple people without splitting them into individual people for action recognition, which simplifies the multi-person action recognition process and improves the efficiency of multi-person action recognition.

[0117] Figure 6 is a flow chart of another method for multi-person action recognition according to an exemplary embodiment. Figure 6 As shown, the multi-person action recognition method of the embodiment of the present disclosure mainly includes the following steps 601 to 608.

[0118] Step 601, obtaining video data.

[0119] In the disclosed embodiment, the video data may be derived from video data collected and stored in advance, and based on the stored video data, multi-person action recognition of historical data can be realized. The stored video data may be data stored in any form in any storage device.

[0120] Video data can also come from real-time collected data. Based on real-time acquired video data, it is possible to quickly identify and respond to the actions of multiple people in the real-time collected scene. The real-time collected data can be real-time video data from various video collection terminal devices.

[0121] Step 602: perform key feature point detection on the video data, and obtain key feature points of the human body associated with the human object in each image frame of the video data.

[0122] In the disclosed example, step 602 can be implemented by first performing human key point detection on each image frame in the video data through a human key point detection model established based on the Bottom_up human key point detection algorithm, and obtaining the human key feature points in each image frame;

[0123] Then, the clustering algorithm and the internal connection between the key human feature points are used to determine which key human feature points correspond to the same person object, that is, the key human feature points associated with the person object in each image frame of the video data are obtained; the person object is identified using the ID, and the key human feature points of the same person object belong to the ID of the same person object.

[0124] Step 603: Based on the obtained key feature points of the human body associated with the human object, a key point similarity algorithm is used to perform human object matching on adjacent image frames.

[0125] Taking OKS as an example of the key point similarity algorithm, OKS calculates the similarity of two key feature points of the human body. The higher the similarity, the more similar the two key feature points of the human body are. In spatial terms, the higher the overlap of the two key feature points of the human body (bone points) is.

[0126] In specific implementation, the similarity of key feature points of the human body is calculated by the following formula:

[0127]

[0128] Among them, p represents the ID of a person object in ground truth;

[0129] OKS P Represents the similarity of the key feature points of the human body of the person object p in adjacent image frames;

[0130] P i Represents the ID of the i-th key point of a certain person object;

[0131] Indicates that the visibility of the i-th key feature point of the human body is 1, that is, it is visible in the image frame;

[0132] S P Represents the square root of the area occupied by the person, calculated based on the bounding box of the person object;

[0133] σ i Represents the normalization factor of the i-th human key feature point. This factor is obtained by calculating the standard deviation of all ground truths in the existing data set, reflecting the impact of the current human key feature point on the whole. The larger the value, the worse the annotation effect of this point in the entire data set; the smaller the value, the better the annotation effect of this point in the entire data set.

[0134] The numerator in the above OKS formula is equivalent to a Gaussian distribution centered on the true value of the key point. It is used for scale normalization. is the standard deviation of the data, i.e., the standard deviation of the annotation. This makes OKS a perceptually meaningful and easy-to-interpret similarity measure. This standard deviation is obtained by statistically analyzing the information of 5,000 different human object annotations in MSCOCO, which represents the standard deviation generated when humans annotate the key feature points of the same human body. There are great differences in the key feature points of different human bodies; as the final standard deviation.

[0135] OKS is a similarity calculation of the key feature points of the human body between two people. When there are multiple people, assuming that there are M people in the image frame to be predicted, and N people are predicted, then each of the M people will be similar to the key feature points of the predicted N people, so in the end you will get an M×N matrix. For one of the character objects, the person with the largest OKS will be selected as the predicted label for calculation.

[0136] According to the obtained OKS matrix, the OKS scores of all people in a certain image frame can be obtained. There are many image frames in the video data, and each image frame may have multiple human objects. The quality of the OKS algorithm can be measured by AP. AP is measured by giving a threshold. If the current OKS is greater than the given threshold, it means that the key feature points of the human body of the human object have been successfully detected and the detection is correct. If it is not greater than the given threshold, it means that the detection has failed or missed detection, etc. Therefore, for all OKS, the number of those greater than the threshold is counted, and its ratio to all OKS is calculated. That is, assuming that there are 100 OKS in total, and 30 of them are greater than the threshold, then the AP value is 30 / 100=0.3.

[0137] In the embodiment of the present application, the threshold value for OKS matching is set according to the specific application scenario without any specific restriction, for example, it can be set to 0.6.

[0138] Step 604 , in response to the matching result that the key human feature points corresponding to the person object match, determine that the person object corresponding to the next image frame in the adjacent image frames is the same person object as the person object corresponding to the previous image frame. Execute step 609 .

[0139] When a person object in a subsequent image frame of adjacent frames matches a person object in a previous image frame, they are determined to be the same person object, and the person object in the subsequent image frame uses the identification ID of the corresponding person object in the previous image frame.

[0140] Step 605 , in response to the matching result that the key human feature points of the human object do not match and no key human feature points are detected at the corresponding position in the previous image frame in the adjacent image frames, determining that the human object in the next image frame in the adjacent image frames is a newly added human object, and executing step 609 .

[0141] In step 605, if the key human feature points of a person object in the latter image frame of adjacent image frames are not matched with the key human feature points of the corresponding person object in the previous image frame, it is considered that the person object in the latter image frame of the adjacent image frames is a newly added person object. Therefore, the person object is directly treated as a newly added person object and assigned a new identification ID.

[0142] Step 606, in response to the matching result that the human key feature points corresponding to the human object do not match, and the human key feature points are detected at the corresponding position in the previous image frame in the adjacent image frame, determine whether the preset feature point discarding conditions are currently met. If so, execute step 607; otherwise, execute step 608.

[0143] Among them, the preset feature point discarding conditions are: the number of consecutive mismatches reaches the preset threshold, and the number of key human feature points detected for the human object in the next image frame in the current adjacent image frame is less than the preset point threshold; the preset point threshold is less than the total number of preset human key feature points.

[0144] In the specific implementation, there are no specific restrictions on the preset number threshold and the preset point threshold. For example, the preset number threshold can be set to 3, and the preset point threshold can be set to half of the total number of preset human body key feature points rounded down. For example, if the total number of preset human body key feature points is 17, it can be set to 8; if the total number of preset human body key feature points is 18, it can be set to 9.

[0145] If the matching has failed for three or more times in a row, and the number of key human feature points detected for the person object in the next image frame in the current adjacent image frame is less than 8, it is determined that the person object corresponding to the image frame has been lost or has left the screen of the currently collected video data.

[0146] Step 607 , deleting the key feature points of the human body of the corresponding human object in the next image frame of the adjacent image frames. Execute step 609 .

[0147] This situation indicates that the person objects corresponding to these key points of human body features no longer exist, and these key feature points do not need to be retained and can be directly deleted.

[0148] Step 608: Use the motion flow information of the key feature points of the human body of the person object corresponding to the previous image frame in the adjacent image frames to complete the key feature points of the human body corresponding to the next image frame in the adjacent image frames.

[0149] If only a certain person object in a certain image frame is missed or misdetected, the motion flow information of the corresponding key feature points of the person object in the adjacent previous image frame is used to complete it, and the identification ID of the person object is used.

[0150] Step 609 , every R image frames, input the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames into a preset multi-dimensional spatiotemporal graph convolution model to obtain the action recognition result of the human object in the video data.

[0151] Among them, K is greater than R, and the number of character objects is 1 or more.

[0152] In this step, R and K are integers, wherein the setting of K is related to the action to be recognized; and the setting of R is related to the acquisition speed of the video data.

[0153] Assuming that the acquisition speed of the image frames in the video data is 25 frames per second, and the action to be identified is fall detection, R can be set to 25 and K to 50, but it is not limited thereto.

[0154] The feature information of the key feature points of the human body here includes the coordinate information of the key feature points of the human body.

[0155] In the disclosed embodiment, the key points of the video data are detected to obtain the key human feature points associated with the human object in each image frame of the video data, and the key human feature points of the human objects in adjacent frames are matched by the key point similarity algorithm to process the key human feature points of the human objects corresponding to the latter image frame in the adjacent image frames, so that multiple people in the image frames of the video data can be recognized at the same time; every R image frames, the feature point information of the key human feature points of the human objects in the consecutive K processed image frames is input into the preset multi-dimensional spatio-temporal graph convolution model to obtain the action recognition results of all the human objects in the video data. This solution can directly recognize the actions of multiple people without splitting them into individual people for action recognition, simplifying the multi-person action recognition process and improving the efficiency of multi-person action recognition.

[0156] The multi-person action recognition method provided in the embodiments of the present disclosure can simultaneously perform action recognition on multiple human objects in multiple actions. In the embodiments of the present disclosure, the application scenario of monitoring real-time scenes (such as smart home scenes) and identifying falls of multiple people is taken as an example to explain the multi-person action recognition process in detail.

[0157] Assuming that the acquisition speed of the image frames in the video data is 25 frames per second, and the action to be recognized is fall detection, set R to 25 and K to 50.

[0158] The specific steps are as follows:

[0159] The first step is to obtain real-time video data at a rate of 25 frames per second.

[0160] The second step is to detect the human key points of multiple human objects in each image frame in the video data through the Bottom up human key point detection algorithm.

[0161] Figure 7 FIG. 1 is a schematic diagram of key feature points of a human body obtained by performing key point detection on a certain image frame in video data according to an exemplary embodiment. Figure 7 As shown, Figure 7 Many key feature points of the human body are detected for an image frame. In this embodiment, 17 key feature points of the human body are taken as an example. Obviously, the image frame corresponds to multiple character images.

[0162] The third step is to determine which key human feature points correspond to the same person object through clustering algorithms and the internal connections between key human feature points.

[0163] Figure 8 FIG. 1 is a schematic diagram showing the relationship between a character object and key feature points of a human body according to an exemplary embodiment. Figure 8 As shown, Figure 8 is based on Figure 7 The key feature points of the human body detected in the figure are associated with the human body objects, and a globally unique identifier is assigned to each human object, such as Figure 8 The key feature points of the human body correspond to three human objects. Starting from the left side of the image, the first human object can be assigned human object identification 1, the second human object can be assigned human object identification 2, and the third human object can be assigned human object identification 3.

[0164] Step 4: Match the human objects in adjacent image frames using the key point similarity algorithm.

[0165] Step 5: Assumptions Figure 8 The corresponding image frame is the next image frame in the adjacent image frames currently being matched. The key feature points of the human body corresponding to the next image frame in the adjacent image frames are processed according to the human object matching result. The specific implementation is as follows:

[0166] Assume that the person object identified by ID 1 matches the key feature points of the person object A in the previous image frame, then Figure 8 The person object in the corresponding image frame uses the identifier A of the corresponding person object in the previous image frame;

[0167] Assuming that the person object identified by identification 2 matches the key feature points of the person object B in the previous image frame, then Figure 8 The person object in the corresponding image frame uses the identifier B of the corresponding person object in the previous image frame;

[0168] Assume that the human key feature points of the person object identified as 3 do not match the human key feature points of the person object C in the previous image frame, and this is the third time that they do not match. Furthermore, it is determined that the number of human key points detected is 4, which is also less than 8. Therefore, the human key feature points of the person object identified as 3 are directly deleted, and the person object identification is also released.

[0169] Fig. 9 FIG. 1 is a schematic diagram showing the key feature points of a human body of a human object after processing according to an exemplary embodiment. Fig. 9 As shown, Fig. 9 yes Figure 8 Schematic diagram of human feature points of the human object after the human feature points are processed according to the matching results. Fig. 9 In Figure 8The key feature points of the third person from the left are deleted, indicating that the person has been lost or has left the video data acquisition area.

[0170] In the disclosed example, a temporary character object identifier is assigned to each character object in an image frame that has not been matched. This is a convenient description. In actual implementation, the identifier may not be assigned, and the character object identifier may be continued to be used or added after matching with the previous adjacent image frame.

[0171] The above gives the detection, matching, and processing of key feature points of the human body for each image frame. After processing 50 frames of images continuously, the feature information of the processed key feature points of the human body can be input into the preset multi-dimensional spatiotemporal graph convolution model in batches for action recognition.

[0172] Based on the data processed based on the matching results, every 25 image frames, the feature point information of the key feature points of the human body of the person object in 50 consecutive processed image frames is input into the preset multi-dimensional spatiotemporal graph convolution model to obtain the action recognition results of the person object in the video data.

[0173] based on Fig. 9 The key feature points of the human body shown in the figure, as well as the feature information of the key feature points of the human body objects in 50 consecutive processed image frames that are input into the model together, the preset multi-dimensional spatiotemporal graph convolution model outputs 2 human objects, and the corresponding action recognition results are: human object A: did not fall; human object B: fell.

[0174] At this point, real-time video monitoring can be completed, and whether multiple human objects in the video screen have fallen can be automatically identified.

[0175] Fig.10 is a schematic diagram of a multi-person motion recognition device according to an exemplary embodiment. Fig.10 As shown, the multi-person action recognition device mainly includes:

[0176] The acquisition unit 1001 is configured to acquire video data;

[0177] The detection unit 1002 is configured to perform key feature point detection on the video data to obtain key feature points of a human body associated with a human object in each image frame of the video data;

[0178] A matching unit 1003 is configured to perform character object matching on adjacent image frames based on the obtained human key feature points associated with the character object;

[0179] The processing unit 1004 is configured to process the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result;

[0180] The recognition unit 1005 is configured to execute every R image frames, input the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames into a preset multi-dimensional spatiotemporal graph convolution model, and obtain the action recognition result of the human object in the video data; wherein K is greater than R, and the human object is one or more.

[0181] In some embodiments,

[0182] The processing unit 1004 is configured to execute processing of the key human feature points corresponding to the next image frame in the adjacent image frames according to the matching result of the human object, in response to the matching result that the key human feature points corresponding to the human object do not match, and the key human feature points are detected at the corresponding position in the previous image frame in the adjacent image frames, if the preset feature point discarding condition is currently met, the key human feature points corresponding to the human object in the next image frame in the adjacent image frames are deleted; or, if the preset feature point discarding condition is not currently met, the motion flow information of the key human feature points of the human object corresponding to the previous image frame in the adjacent image frames is used to complete the key human feature points corresponding to the next image frame in the adjacent image frames; wherein the preset feature point discarding condition is: the number of consecutive mismatches reaches a preset number threshold, and the number of key human feature points detected for the human object in the next image frame in the current adjacent image frames is less than a preset point number threshold; the preset point number threshold is less than the total number of preset human key feature points.

[0183] In some embodiments,

[0184] The processing unit 1004 is configured to process the key feature points of the human body corresponding to the subsequent image frame in the adjacent image frames according to the character object matching result, and in response to the matching result being a match of the key feature points of the human body corresponding to the character object, determine that the character object corresponding to the subsequent image frame in the adjacent image frames and the character object corresponding to the previous image frame are the same character object.

[0185] In some embodiments,

[0186] The processing unit 1004 is configured to execute processing of the human body key feature points corresponding to the latter image frame in the adjacent image frames according to the human body object matching result, and in response to the matching result being that the human body key feature points of the human body object do not match, and the human body key feature points are not detected at the corresponding position in the previous image frame in the adjacent image frames, determine that the human body object in the latter image frame in the adjacent image frames is a newly added human body object.

[0187] In some embodiments, the apparatus further comprises:

[0188] The training unit 1006 is configured to execute training of a preset multidimensional space-time graph convolution model, specifically: establishing an initial multidimensional space-time graph convolution model; wherein the input of the initial multidimensional space-time graph convolution model is five-dimensional [N, M, T, V, C]; the output is three-dimensional [N, M, S]; obtaining training samples; training the initial multidimensional space-time graph convolution model based on the obtained training samples to obtain a preset multidimensional space-time graph convolution model; wherein N represents the number of video data used for model training, M represents the maximum number of human objects that can be recognized by the preset multidimensional space-time graph convolution model, T represents the time dimension of an action, V represents the total number of human key feature points used; C represents the dimension predicted for each human key feature point, and S represents the category of the action.

[0189] In some embodiments,

[0190] The setting of K is related to the action to be recognized;

[0191] The setting of R is related to the acquisition speed of acquiring the video data.

[0192] In some embodiments,

[0193] The matching unit 1003 is configured to perform character object matching on adjacent image frames based on the obtained human key feature points associated with the human object, and to use a key point similarity algorithm to perform character object matching on adjacent image frames based on the obtained human key feature points associated with the human object.

[0194] Regarding the multi-person motion recognition device in each of the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiment of the multi-person motion recognition method, and will not be elaborated here.

[0195] It should be noted that: the above embodiment only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0196] Fig.11It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a server. The electronic device 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein at least one program code is stored in the memory 1102, and the at least one program code is loaded and executed by the processor 1101 to implement the multi-person motion recognition method provided by each of the above embodiments. Of course, the electronic device 700 may also have components such as image acquisition components, wired or wireless network interfaces, keyboards, and input and output interfaces for input and output, and the electronic device 1100 may also include other components for implementing device functions, which will not be repeated here.

[0197] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction. The at least one instruction can be executed by a processor in a computer device to complete the multi-person motion recognition method in the above embodiment.

[0198] Optionally, the above-mentioned computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may include ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.

[0199] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of a computer device to complete the multi-person motion recognition method provided by the above-mentioned embodiments.

[0200] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for multi-person action recognition, characterized in that: The method comprises: Get video data; Performing key feature point detection on the video data to obtain key feature points of a human body associated with a human object in each image frame of the video data; Performing character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object; Processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result; Every R image frames, the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames is input into a preset multi-dimensional spatiotemporal graph convolution model to obtain the action recognition result of the human object in the video data; wherein K is greater than R, and the human object is one or more.

2. The method according to claim 1, characterized in that The step of processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result includes: In response to the matching result that the human body key feature points corresponding to the human object do not match, and the human body key feature points are detected at corresponding positions in the previous image frame in the adjacent image frames; If the preset feature point discarding condition is currently met, the key human feature points of the human body corresponding to the human object in the next image frame in the adjacent image frames are deleted; or, If the preset feature point discarding condition is not met at present, the motion flow information of the key feature points of the human body of the person object corresponding to the previous image frame in the adjacent image frames is used to fill the key feature points of the human body corresponding to the next image frame in the adjacent image frames; Among them, the preset feature point discarding condition is: the number of consecutive mismatches reaches a preset threshold, and the number of key human feature points detected for the person object in the next image frame in the current adjacent image frame is less than the preset point threshold; the preset point threshold is less than the total number of preset human key feature points.

3. The method according to claim 1, characterized in that The step of processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result includes: In response to the matching result being a match of key human feature points corresponding to the person object, it is determined that the person object corresponding to the subsequent image frame in adjacent image frames and the person object corresponding to the previous image frame are the same person object.

4. The method according to claim 1, characterized in that: The step of processing the key feature points of the human body corresponding to the next image frame in the adjacent image frames according to the human object matching result includes: In response to the matching result being that the key human feature points of the person object do not match, and no key human feature points are detected at corresponding positions in a previous image frame in adjacent image frames, it is determined that the person object in a subsequent image frame in the adjacent image frames is a newly added person object.

5. The method according to claim 1, characterized in that The training method of the preset multi-dimensional spatiotemporal graph convolution model includes: Establishing an initial multi-dimensional spatio-temporal graph convolution model; wherein the input of the initial multi-dimensional spatio-temporal graph convolution model is five-dimensional [N, M, T, V, C]; and the output is three-dimensional [N, M, S]; Get training samples; Training the initial multidimensional space-time graph convolution model based on the acquired training samples to obtain a preset multidimensional space-time graph convolution model; Among them, N represents the number of video data used for model training, M represents the maximum number of human objects that can be recognized by the preset multi-dimensional spatiotemporal graph convolution model, T represents the time dimension of an action, V represents the total number of human key feature points used; C represents the dimension of prediction for each human key feature point, and S represents the category of the action.

6. The method according to any one of claims 1 to 5, characterized in that: The setting of K is related to the action to be recognized; The setting of R is related to the acquisition speed of acquiring the video data.

7. The method according to any one of claims 1 to 5, characterized in that: The performing character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object comprises: Based on the obtained key feature points of the human body associated with the human object, a key point similarity algorithm is used to perform human object matching on adjacent image frames.

8. A multi-person action recognition device, characterized in that: The device comprises: An acquisition unit, configured to acquire video data; A detection unit is configured to perform key feature point detection on the video data to obtain key feature points of a human body associated with a human object in each image frame of the video data; A matching unit, configured to perform character object matching on adjacent image frames based on the obtained key feature points of the human body associated with the character object; A processing unit configured to process key human feature points corresponding to a subsequent image frame in adjacent image frames according to a human object matching result; The recognition unit is configured to execute every R image frames, input the feature point information of the key feature points of the human body of the human object in K consecutive processed image frames into a preset multi-dimensional spatiotemporal graph convolution model, and obtain the action recognition result of the human object in the video data; wherein K is greater than R, and the human object is one or more.

9. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; Wherein, the processor is configured to execute the executable instructions to implement the multi-person action recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to implement the multi-person motion recognition method as described in any one of claims 1 to 7.

Images