A method, apparatus, and device for skeleton human behavior recognition based on derived data distillation

Abstract

A method, apparatus, and device for skeleton-based human behavior recognition based on derived data distillation belong to the field of human behavior recognition technology. The method includes: acquiring skeleton data of a target object; obtaining joint data through data preprocessing; calculating skeletal data based on the joint data; calculating joint difference data based on the joint data; calculating skeletal difference data based on the skeletal data; training a teacher network based on the skeletal data and skeletal difference data; training a student network based on the skeletal data, skeletal difference data, and soft labels calculated by the teacher network; and inputting the data to be recognized into the student network to obtain prediction results. This invention can maximize the utilization of skeleton-derived data while controlling the number of parameters in the network and improving the prediction accuracy and computation speed of the network.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, and in particular to a skeleton human behavior recognition method, apparatus and equipment based on derived data distillation. Background Technology

[0002] Human behavior recognition is an important research topic in computer vision. Its purpose is to extract effective information or discriminative features from human action data and use this information and features to classify human actions. This technology has significant application prospects in human-computer interaction, video surveillance, medical assistance, and gaming. Traditional human behavior recognition methods use preprocessed RGB video data as input. This type of data often contains a large amount of invalid background and human appearance information, which can significantly impact the recognition performance. In contrast, human skeleton data has advantages such as insensitivity to environmental interference, high effective information density, and small data storage space. Traditional skeleton-based human behavior recognition methods only use joint coordinate data as input, ignoring other derived skeleton data calculated from joint data. In recent years, human behavior recognition methods have used multiple branch networks to extract effective information from both the original and derived data, fusing the network prediction results to obtain the final prediction. This method requires retaining all branch networks of the entire model during use. While this significantly improves the performance of human behavior recognition, the marginal effect is quite obvious compared to the increase in the number of network model parameters under various derived data conditions. At the same time, the real-time performance of network computation is also greatly affected. Therefore, it is necessary to limit the number of model parameters when using skeleton derived data. Summary of the Invention

[0003] The purpose of this invention is to solve the problems existing in the prior art and provide a skeleton human behavior recognition method, device and equipment based on derived data distillation, which can control the number of parameters in the network and improve the prediction accuracy and calculation speed of the network while maximizing the use of skeleton derived data.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] According to a first aspect, the present invention provides a skeleton human behavior recognition method based on derived data distillation, comprising: acquiring skeleton data of a target object, and obtaining joint data through data preprocessing; calculating bone data based on the joint data; calculating joint difference data based on the joint data, and calculating bone difference data based on the bone data; training a teacher network based on the bone data and bone difference data; training a student network based on the bone data, bone difference data, and soft labels calculated by the teacher network; and inputting the data to be recognized into the student network to obtain a prediction result.

[0006] Optionally, the step of calculating the skeletal data based on the joint data includes: establishing a joint coordinate system based on the joint data; and calculating the coordinates of the skeletal data based on the coordinates of the two endpoints of the skeletal bone.

[0007] Optionally, the step of calculating joint difference data based on the joint data includes: establishing a joint coordinate system based on the joint data; extracting joint change data within a preset time period based on the joint coordinate system; and calculating the difference within the preset time period based on the joint change data and the joint data to obtain joint difference data.

[0008] Optionally, the step of calculating bone difference data based on the bone data includes: establishing a bone coordinate system based on the bone data; extracting bone change data within a preset time period based on the bone coordinate system; and calculating the difference within the preset time period based on the bone change data and the bone data, thereby obtaining bone difference data.

[0009] Optionally, training the teacher network based on the skeletal data and skeletal difference data includes: inputting skeletal data into the spatial branch network of the teacher network for training; inputting skeletal difference data into the temporal branch network of the teacher network for training; updating the teacher network weights through backpropagation, and obtaining the trained teacher network weights after a predetermined number of training rounds.

[0010] Optionally, training the student network based on the skeletal data, skeletal difference data, and soft labels calculated by the teacher network includes: inputting skeletal data into the spatial branch network of the teacher network or inputting skeletal difference data into the temporal branch network of the teacher network, at a parameter distillation temperature. Soft labels are calculated under the given conditions; joint data is input into the spatial branch network of the student network or joint difference data is input into the temporal branch network of the student network to calculate hard predictions, and at the parameter distillation temperature... Under the given conditions, a soft prediction is calculated; based on the soft label and the soft prediction, a soft loss is calculated; based on the hard label and the hard prediction, a hard loss is calculated; based on the soft loss and the hard loss, a final loss is calculated.

[0011] Optionally, the algorithm for calculating the soft loss based on the soft label and soft prediction is as follows:

[0012]

[0013] in, Represents the total number of action categories; The teacher network at the distillation temperature Under the conditions The function in the first Classification scores based on action type For An exponential function with base 0; This represents the student network at the distillation temperature. Under the conditions The function in the first Classification scores based on action type; and Representing the teacher network and student network respectively, through Before the function, in the first Category score on the class label.

[0014] Optionally, the algorithm for calculating the hard loss based on the hard label and the hard prediction is as follows:

[0015]

[0016] in, This represents the true label corresponding to the first The value of the action class, The student network represents the distillation temperature Under the conditions of After the function, the second Classification scores based on action type.

[0017] Optionally, the algorithm for calculating the final loss based on the soft loss and hard loss is as follows:

[0018]

[0019] in, For hard losses, It is a soft loss. and This is a hyperparameter.

[0020] Optionally, during the training of the student network based on the skeletal data, skeletal difference data, and soft labels calculated by the teacher network: when inputting skeletal data into the spatial branch network of the teacher network or inputting skeletal difference data into the temporal branch network of the teacher network, the weight parameters are frozen; when inputting joint data into the spatial branch network of the student network or inputting joint difference data into the temporal branch network of the student network, the weight parameters are updated with backpropagation.

[0021] Optionally, the step of inputting the data to be identified into the student network to obtain the prediction result includes: inputting the data to be identified into the student network to obtain the spatial branch network prediction score and the temporal branch network prediction score of the student network; performing weighted score fusion on the spatial branch network prediction score and the temporal branch network prediction score of the student network to obtain the final prediction score, wherein the class with the highest score in the final prediction score is the predicted action classification result.

[0022] According to a second aspect, the present invention provides a skeleton human behavior recognition device based on derived data distillation, comprising: an acquisition module for acquiring joint data of a target object; a calculation module for calculating skeletal data based on the joint data; calculating joint difference data based on the joint data; and calculating skeletal difference data based on the skeletal data; a network training module for training a teacher network based on the skeletal data and the skeletal difference data; and training a student network based on the skeletal data, the skeletal difference data, and soft labels calculated by the teacher network; and a prediction module for inputting the data to be recognized into the student network to obtain a prediction result.

[0023] According to a third aspect, a skeleton human behavior recognition device based on derived data distillation includes: a communication unit, a memory, and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the steps of the method described in the first aspect or any of the optional methods by executing the computer instructions.

[0024] According to a fourth aspect, a computer-readable storage medium is characterized in that the computer-readable storage medium stores computer instructions for causing the computer to perform the skeleton human behavior recognition method described in the first aspect or any alternative method.

[0025] The advantages of this invention are:

[0026] 1. Unlike general knowledge distillation methods, this method uses second-order derived data of the skeleton, i.e., skeletal data, to train the teacher network and joint data to train the student network. The data used in both stages comes from the same source, and the second-order derived data is easy to calculate, without the need to collect additional data for training the teacher network.

[0027] 2. Applying knowledge distillation to skeleton-based human behavior recognition, compared to networks trained with single skeleton data, networks trained using knowledge distillation extract higher quality information and can make fuller use of the effective information in the skeleton data. This is reflected in the fact that skeleton-based human behavior recognition methods based on derived data distillation have higher recognition accuracy while having the same computational cost. Attached Figure Description

[0028] Figure 1 This is a flowchart of a skeleton human behavior recognition method based on derived data distillation according to the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the calculation of skeletal data from joint data in this invention;

[0030] Figure 3 This is a schematic diagram illustrating the calculation of joint difference data in the joint data of this invention;

[0031] Figure 4 This is a schematic diagram of teacher network training in this invention (spatial branch network part);

[0032] Figure 5 This is a schematic diagram of the student network training in this invention (spatial branch network part);

[0033] Figure 6 This is a schematic diagram of student network identification in this invention;

[0034] Figure 7 This is a schematic diagram of a skeleton human behavior recognition device based on derived data distillation according to the present invention;

[0035] Figure 8 This is a schematic diagram of a skeleton human behavior recognition device based on derived data distillation according to the present invention. Detailed Implementation

[0036] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] This invention extracts relevant joint and bone data from the skeleton data of the target object, and then trains a teacher network and a student network based on the joint and bone data, thereby using the student network to obtain action classification prediction results. In the following embodiments, the human skeleton is used as an example; however, this invention can also be applied to action recognition of other skeletons, and this application is not limited thereto.

[0039] Figure 1 A flowchart of a skeleton human behavior recognition method based on derived data distillation according to an embodiment of the present invention is shown. The method specifically includes the following steps:

[0040] S100: Obtain the skeleton data of the target object and obtain the joint data through data preprocessing.

[0041] Specifically, skeleton data can be acquired using the Microsoft Kinect sensor or extracted from RGB human motion videos using pose estimation algorithms, and joint data can be obtained through data preprocessing. And divided into time dimension Frame joint data frame ,in , , These represent the channel dimension, time dimension, and spatial dimension of the skeleton data, respectively.

[0042] S200: Calculate skeletal derived data, including: calculating bone data based on the joint data; calculating joint difference data based on the joint data; and calculating bone difference data based on the bone data.

[0043] Specifically, the method for calculating skeletal data from joint data is as follows: taking the central joint point of the body as the origin, and setting the coordinates to... The coordinate difference of the joint data is calculated by following the natural connection of the human skeleton, such as... Figure 2 As shown. For example, two points on the natural connection of the human skeleton have a centripetal point and an egregious point. The centripetal point is the data point closer to the central joint of the body, and its coordinates are set as follows. The centrifugal point is a data point that is relatively farther away from the central joint point, and its coordinates are given as follows: The coordinates of the corresponding centrifugal point skeletal data are: .

[0044] The method for calculating temporal difference data from spatial data is as follows: Taking joint data as an example, a node in the joint data... The data at time is recorded as The corresponding joint difference data can be represented as the difference in node coordinates between two adjacent frames in the time dimension, which can be expressed as: ,like Figure 3 As shown, for data alignment, the data at time 0 is set to the average of all data, i.e. The method for calculating skeletal differential data from skeletal data is the same as the method for calculating joint differential data from joint data.

[0045] S300: Train the teacher network based on the skeletal data and skeletal difference data.

[0046] Specifically, such as Figure 4 As shown, the base network can be replaced with any network suitable for processing skeletal data and whose predicted output is in one-hot format; here, an adaptive graph convolutional network is used as an example. The spatial branch network takes skeletal data as input, and the temporal branch network takes skeletal difference data as input. Both networks are trained separately. The network weights are updated through backpropagation, and after a predetermined number of training epochs, the trained teacher network weights are obtained.

[0047] S400: Train the student network based on the skeletal data, skeletal difference data, and soft labels calculated by the teacher network.

[0048] Specifically, the spatial branch network and temporal branch network of the student network are trained separately. For example... Figure 5 As shown, taking the spatial branch network as an example, the trained teacher network corresponding to the branch is first transferred to the branch network training model shown in the figure. The weight parameters of the teacher network are frozen in this step and will not be updated with backpropagation. By inputting skeletal data, the parameters are then adjusted at the distillation temperature. Under certain conditions, soft labels are calculated. These soft labels differ from the true labels; in fact, non-predicted categories also receive scores in the soft labels, while the true labels only score 1 for correct action categories and 0 for other categories. The student network's base network and teacher network have the same structure, but their weight parameters are updated during backpropagation in this step. The spatial branch network of the student network takes joint data as input and outputs two data points: one is the distillation temperature. The first is the soft prediction, and the second is the prediction score that is usually output; both are referred to as hard prediction below.

[0049] Based on the soft label and soft prediction, the soft loss is calculated using the following formula:

[0050]

[0051] in, Represents the total number of action categories; The teacher network at the distillation temperature Under the conditions The function in the first Classification scores based on action type For An exponential function with base 0; This represents the student network at the distillation temperature. Under the conditions The function in the first Classification scores based on action type; and Representing the teacher network and student network respectively, through Before the function, in the first Category score on the class label.

[0052] Based on the hard labels and the hard predictions, the hard loss is calculated using the following formula:

[0053]

[0054] in, This represents the true label corresponding to the first The value of the action class, The student network represents the distillation temperature Under the conditions of After the function, the second Classification scores based on action type.

[0055] Ultimately, the loss function of the entire network can be expressed as:

[0056]

[0057] in, For hard losses, For soft loss, and These are hyperparameters; by default, .

[0058] The training method for the temporal branch network of the student network is the same as that for the spatial branch network. When training the temporal branch network, the teacher network's temporal branch network needs to input skeletal difference data, while the student network's temporal branch network needs to input joint difference data.

[0059] S500: Input the data to be identified into the student network to obtain the prediction result.

[0060] Specifically, such as Figure 6 As shown, after processing the key data to be identified according to the network's input format, and then inputting it into the trained student network, the spatial branch network prediction score of the student network will be obtained. And time branch network prediction score The final predicted score is obtained by weighting and averaging the prediction results from both methods. It can be expressed by the following formula:

[0061]

[0062] By default That is, the average score is merged, and finally The class with the highest score is the predicted action classification result.

[0063] like Figure 7 As shown, this embodiment of the invention provides a skeleton human behavior recognition device based on derived data distillation, including an acquisition module 1, a calculation module 2, a network training module 3, and a prediction module 4, wherein...

[0064] Module 1 is used to acquire joint data of the target object; for details, please refer to the relevant description of step S100 in the above method embodiment.

[0065] Calculation module 2 is used to calculate bone data based on the joint data; calculate joint difference data based on the joint data; and calculate bone difference data based on the bone data; for details, please refer to the relevant description of step S200 in the above method embodiment;

[0066] Network training module 3 is used to train the teacher network based on the skeletal data and skeletal difference data; and to train the student network based on the skeletal data, skeletal difference data, and soft labels calculated by the teacher network; for details, please refer to the relevant descriptions of steps S300 and S400 in the above method embodiment.

[0067] Prediction module 4 is used to input the data to be identified into the student network and obtain the prediction result; for details, please refer to the relevant description of step S500 in the above method embodiment.

[0068] like Figure 8 As shown, this embodiment of the invention also provides a skeleton human behavior recognition device based on derived data distillation, including at least one processor 41, at least one communication interface 42, at least one communication bus 43, and at least one memory 44. The communication interface 42 may include a display screen and a keyboard; optionally, the communication interface 42 may also include a standard wired interface or a wireless interface. The memory 44 may be a high-speed RAM memory or an unstable memory, such as at least one disk storage device. Optionally, the memory 44 may also be at least one storage device located remotely from the aforementioned processor 41. The processor 41 may be combined with... Figure 7The described apparatus has an application program stored in memory 44, and the processor 41 calls the program code stored in memory 44 to perform the steps of the skeleton human behavior recognition method of any of the above method embodiments.

[0069] The communication bus 43 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 43 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0070] The memory 44 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 44 may also include a combination of the above types of memory.

[0071] The processor 41 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

[0072] The processor 41 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0073] Optionally, memory 44 is also used to store program instructions. Processor 41 can invoke program instructions to implement the present invention. Figure 1The skeleton human behavior recognition method shown in the embodiment.

[0074] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the skeleton human behavior recognition method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0075] The above are merely preferred embodiments of the present invention, and are implementations based on the overall concept of the present invention. Furthermore, the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A skeleton human behavior recognition method based on derived data distillation, characterized in that, include: Obtain the skeleton data of the target object, and obtain the joint data through data preprocessing; Based on the joint data, the bone data is calculated; Calculate joint difference data based on the joint data, and calculate bone difference data based on the bone data; Based on the aforementioned skeletal data and skeletal difference data, a teacher network is trained. Based on the aforementioned skeletal data, skeletal difference data, and soft labels calculated by the teacher network, the student network is trained, including: Skeletal data is input into the spatial branch network of the teacher network or skeletal difference data is input into the temporal branch network of the teacher network. The weight parameters are frozen, and soft labels are calculated under the parameter distillation temperature T. Joint data is input into the spatial branch network of the student network or joint difference data is input into the temporal branch network of the student network. The weight parameters are updated with backpropagation to calculate hard predictions and soft predictions under the parameter distillation temperature T. Based on the soft label and soft prediction, the soft loss is calculated; Based on the hard labels and the hard predictions, the hard loss is calculated; Based on the soft loss and hard loss, the final loss is calculated; The data to be identified is input into the student network to obtain the prediction results.

2. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the joint data, bone data is calculated, including: Establish a joint coordinate system based on the joint data; The coordinates of the bone data are calculated based on the coordinates of the two ends of the bone.

3. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the joint data, joint difference data is calculated, including: Establish a joint coordinate system based on the joint data; Based on the joint coordinate system, joint change data within a preset time period is extracted, and the difference within the preset time period is calculated based on the joint change data and the joint data to obtain joint difference data.

4. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the aforementioned bone data, bone difference data is calculated, including: Establish a skeletal coordinate system based on the aforementioned skeletal data; Based on the skeletal coordinate system, extract skeletal change data within a preset time period, and calculate the difference within the preset time period based on the skeletal change data and the skeletal data, thereby obtaining skeletal difference data.

5. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the aforementioned skeletal data and skeletal difference data, a teacher network is trained, including: Skeletal data is input into the spatial branch network of the teacher network for training; Skeletal difference data is input into the temporal branch network of the teacher network for training; The teacher network weights are updated through backpropagation, and after a predetermined number of training rounds, the trained teacher network weights are obtained.

6. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the soft label and soft prediction, the algorithm for calculating the soft loss is as follows: Where K represents the total number of action categories; The representative teacher network's classification score for the i-th type of action using the softmax function under distillation temperature T is given by exp(·), where exp(·) is an exponential function with base e. The representative student network's classification score on the i-th action class using the softmax function at distillation temperature T; y i and z i represents the classification scores of the teacher network and the student network on the i-th label before passing through the softmax function, respectively.

7. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the hard labels and the hard predictions, the algorithm for calculating the hard loss is as follows: Among them, g i ∈{0, 1} represents the value of the true label corresponding to the i-th type of action. The student network's classification score for the i-th action after passing through the softmax function at a distillation temperature T=1 is represented.

8. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, Based on the soft loss and hard loss, the algorithm for calculating the final loss is as follows: L=aL hard +βL soft Among them, L hard For hard loss, L soft The loss is soft, and α and β are hyperparameters.

9. The skeleton human behavior recognition method based on derived data distillation according to claim 1, characterized in that, The data to be identified is input into the student network to obtain prediction results, including: The data to be identified is input into the student network to obtain the spatial branch network prediction score and the temporal branch network prediction score of the student network; The spatial branch network prediction scores and temporal branch network prediction scores of the student network are weighted and fused to obtain the final prediction score, wherein the class with the highest score in the final prediction score is the predicted action classification result.

10. A skeleton human behavior recognition device based on derived data distillation, characterized in that, include: The acquisition module is used to acquire joint data of the target object; A calculation module is used to calculate bone data based on the joint data; Calculate joint difference data based on the joint data, and calculate bone difference data based on the bone data; The network training module is used to train the teacher network based on the skeletal data and skeletal difference data. Based on the aforementioned skeletal data, skeletal difference data, and soft labels calculated by the teacher network, the student network is trained: Skeletal data is input into the spatial branch network of the teacher network or skeletal difference data is input into the temporal branch network of the teacher network. The weight parameters are frozen, and soft labels are calculated under the parameter distillation temperature T. Joint data is input into the spatial branch network of the student network or joint difference data is input into the temporal branch network of the student network. The weight parameters are updated with backpropagation to calculate hard predictions and soft predictions under the parameter distillation temperature T. Based on the soft label and soft prediction, the soft loss is calculated; Based on the hard labels and the hard predictions, the hard loss is calculated; Based on the soft loss and hard loss, the final loss is calculated; The prediction module is used to input the data to be identified into the student network and obtain the prediction result.

11. A skeleton human behavior recognition device based on derived data distillation, characterized in that, include: The method comprises a communication unit, a memory, and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the steps of the method according to any one of claims 1-9.

Images