Rope skipping action recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a rope skipping action recognition method and device, electronic equipment and a storage medium. The method comprises the following steps: detecting a target person located in a preset image area from each target video frame of a target video; extracting a human body key point of the target person from each target video frame as a target key point corresponding to each target video frame; inputting the target key point corresponding to each target video frame into a first graph neural network to obtain a rope skipping action state output by the first graph neural network, and taking the rope skipping action state as a rope skipping action state corresponding to the target video frame; and in response to a plurality of continuous target video frames which are detected from the plurality of target video frames and meet a preset rope skipping condition, inputting target key points corresponding to the plurality of continuous target video frames into the second graph neural network to obtain a rope skipping action type output by the second graph neural network. Normal rope skipping actions and abnormal rope skipping actions can be accurately distinguished, and the rope skipping counting accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a rope skipping action recognition method, device, electronic device and storage medium. Background Art

[0002] Usually, some physical health test items can be set for personnel to test their physical health, so that personnel can plan their own exercise methods according to the test results to enhance their physical health. For example, physical health test items can refer to rope skipping, pull-ups and other items.

[0003] In the scenario of the rope skipping test, in addition to normal rope skipping movements, people may also make abnormal rope skipping movements such as stumbling and stepping. In the related technology, it is possible to detect whether a person has performed rope skipping by analyzing the amplitude changes of the person's rope skipping movements or distinguishing the different motion states of the person, and count the rope skipping according to the detection results. However, the above rope skipping counting method cannot distinguish whether the person's rope skipping movements are normal rope skipping movements, resulting in a low accuracy rate of rope skipping counting. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for identifying rope skipping movements, so as to accurately distinguish normal rope skipping movements from abnormal rope skipping movements and improve the accuracy of rope skipping counting. The specific technical solutions are as follows:

[0005] The present application provides a method for identifying a rope skipping action, the method comprising:

[0006] From each target video frame of the target video, a person located in a preset image area is detected as a target person;

[0007] Extracting the human body key points of the target person from each of the target video frames respectively as the target key points corresponding to each of the target video frames;

[0008] Inputting the target key points corresponding to each of the target video frames into the first graph neural network, obtaining the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: an ascending state and a descending state;

[0009] In response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of the target video frames, the target key points corresponding to the plurality of consecutive target video frames are input into a second graph neural network to obtain a rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the plurality of consecutive target video frames is rope skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the plurality of consecutive target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state;

[0010] Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from a plurality of consecutive sample video frames obtained by shooting the second sample person, and the plurality of consecutive sample video frames are marked with the skipping action type of the second sample person.

[0011] In a possible embodiment, the first graph neural network is pre-trained in the following manner, including:

[0012] Inputting the first sample key point into the first original graph neural network to obtain a first key point feature of the first sample key point extracted by the first original graph neural network;

[0013] Determine a first difference between the first key point feature and the first positive sample feature, and determine a second difference between the first key point feature and the first negative sample feature; wherein the first positive sample feature is a feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is a feature of other first sample key points whose marked rope skipping action state is different from the first sample key point;

[0014] constructing a first loss function according to the first difference and the second difference, wherein the first loss function is positively correlated with the first difference and negatively correlated with the second difference;

[0015] Adjust the model parameters of the first original graph neural network in the direction of gradient descent of the first loss function to obtain a first graph neural network.

[0016] In a possible embodiment, a first loss function is constructed according to the first difference and the second difference by using the following formula, including:

[0017]

[0018] Among them, L ts1 is the first loss function; F check1 is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter.

[0019] In a possible embodiment, the second graph neural network is pre-trained in the following manner, including:

[0020] Inputting a plurality of second sample key points into a second original graph neural network to obtain second key point features of the plurality of second sample key points extracted by the second original graph neural network;

[0021] Determine a third difference between the second key point feature and the second positive sample feature, and determine a fourth difference between the second key point feature and the second negative sample feature; wherein the second positive sample feature is a feature of a human key point extracted from a plurality of positive sample video frames, the plurality of positive sample video frames are a plurality of consecutive video frames annotated with the same rope skipping action type as the plurality of consecutive sample video frames, and the second negative sample feature is a feature of a human key point extracted from a plurality of negative sample video frames, the plurality of negative sample video frames are a plurality of consecutive video frames annotated with a different rope skipping action type than the plurality of consecutive sample video frames;

[0022] constructing a second loss function according to the third difference and the fourth difference, wherein the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference;

[0023] In the direction of the gradient descent of the second loss function, the model parameters of the second original graph neural network are adjusted to obtain a second graph neural network.

[0024] In a possible embodiment, a second loss function is constructed according to the third difference and the fourth difference by using the following formula, including:

[0025]

[0026] Among them, L ts2 is the second loss function; F check2 is the second key point feature; positive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(F check2 , Fpositive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter.

[0027] In a possible embodiment, the rope skipping action types include: normal, stumbling, padding, background, and empty jump;

[0028] The rope skipping action type of the second sample person is marked in advance on the second sample key points by the following method, including:

[0029] If the second sample person in the plurality of consecutive sample video frames shows normal rope skipping behavior, marking the plurality of consecutive sample video frames as normal;

[0030] If the second sample person in the plurality of consecutive sample video frames stumbles, marking the plurality of consecutive sample video frames as stumbling;

[0031] If the second sample person in the plurality of consecutive sample video frames has a stepping behavior, marking the plurality of consecutive sample video frames as stepping;

[0032] If the second sample person in the plurality of consecutive sample video frames does not perform rope skipping behavior, marking the plurality of consecutive sample video frames as background;

[0033] If the second sample person in the plurality of consecutive sample video frames exhibits a skipping behavior, the plurality of consecutive sample video frames are marked as skipping.

[0034] The embodiment of the present application also provides a device for identifying a rope skipping action, the device comprising:

[0035] A target person detection module is used to detect a person located in a preset image area from each target video frame of the target video as a target person;

[0036] A target key point extraction module is used to extract the human body key points of the target person from each of the target video frames as the target key points corresponding to each of the target video frames;

[0037] A rope skipping action state obtaining module, used for inputting the target key points corresponding to each of the target video frames into the first graph neural network, obtaining the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: rising state and falling state;

[0038] A rope skipping action type obtaining module is used for, in response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of the target video frames, inputting the target key points corresponding to the consecutive plurality of target video frames into a second graph neural network, and obtaining the rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the consecutive plurality of target video frames is skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the consecutive plurality of target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state;

[0039] Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from a plurality of consecutive sample video frames obtained by shooting the second sample person, and the plurality of consecutive sample video frames are marked with the skipping action type of the second sample person.

[0040] In a possible embodiment, the first graph neural network is pre-trained in the following manner, including:

[0041] Inputting the first sample key point into the first original graph neural network to obtain a first key point feature of the first sample key point extracted by the first original graph neural network;

[0042] Determine a first difference between the first key point feature and the first positive sample feature, and determine a second difference between the first key point feature and the first negative sample feature; wherein the first positive sample feature is a feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is a feature of other first sample key points whose marked rope skipping action state is different from the first sample key point;

[0043] constructing a first loss function according to the first difference and the second difference, wherein the first loss function is positively correlated with the first difference and negatively correlated with the second difference;

[0044] Adjust the model parameters of the first original graph neural network in the direction of gradient descent of the first loss function to obtain a first graph neural network.

[0045] In a possible embodiment, a first loss function is constructed according to the first difference and the second difference by using the following formula, including:

[0046]

[0047] Among them, L ts1 is the first loss function; F check1 is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter.

[0048] In a possible embodiment, the second graph neural network is pre-trained in the following manner, including:

[0049] Inputting a plurality of second sample key points into a second original graph neural network to obtain second key point features of the plurality of second sample key points extracted by the second original graph neural network;

[0050] Determine a third difference between the second key point feature and the second positive sample feature, and determine a fourth difference between the second key point feature and the second negative sample feature; wherein the second positive sample feature is a feature of a human key point extracted from a plurality of positive sample video frames, the plurality of positive sample video frames are a plurality of consecutive video frames annotated with the same rope skipping action type as the plurality of consecutive sample video frames, and the second negative sample feature is a feature of a human key point extracted from a plurality of negative sample video frames, the plurality of negative sample video frames are a plurality of consecutive video frames annotated with a different rope skipping action type than the plurality of consecutive sample video frames;

[0051] constructing a second loss function according to the third difference and the fourth difference, wherein the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference;

[0052] In the direction of the gradient descent of the second loss function, the model parameters of the second original graph neural network are adjusted to obtain a second graph neural network.

[0053] In a possible embodiment, a second loss function is constructed according to the third difference and the fourth difference by using the following formula, including:

[0054]

[0055] Among them, L ts2 is the second loss function; F check2 is the second key point feature; positive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(Fcheck2 , F positive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter.

[0056] In a possible embodiment, the rope skipping action types include: normal, stumbling, padding, background, and empty jump;

[0057] The rope skipping action type of the second sample person is marked in advance on the second sample key points by the following method, including:

[0058] If the second sample person in the plurality of consecutive sample video frames shows normal rope skipping behavior, marking the plurality of consecutive sample video frames as normal;

[0059] If the second sample person in the plurality of consecutive sample video frames stumbles, marking the plurality of consecutive sample video frames as stumbling;

[0060] If the second sample person in the plurality of consecutive sample video frames has a stepping behavior, marking the plurality of consecutive sample video frames as stepping;

[0061] If the second sample person in the plurality of consecutive sample video frames does not perform rope skipping behavior, marking the plurality of consecutive sample video frames as background;

[0062] If the second sample person in the plurality of consecutive sample video frames exhibits a skipping behavior, the plurality of consecutive sample video frames are marked as skipping.

[0063] The present application also provides an electronic device, including:

[0064] Memory, used to store computer programs;

[0065] The processor is used to implement any of the above-mentioned rope skipping action recognition methods when executing the program stored in the memory.

[0066] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned rope skipping action recognition methods is implemented.

[0067] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned rope skipping action recognition methods.

[0068] Beneficial effects of the embodiments of the present application:

[0069] A rope skipping action recognition method, device, electronic device and storage medium provided in an embodiment of the present application can determine the target person who needs to perform rope skipping action recognition by detecting the person located in the preset image area from each target video frame of the target video, and extract the target person's human body key points from each target video frame respectively for the target person, and obtain the target key points corresponding to each target video frame, so that the rope skipping action can be recognized according to the characteristics of the target key points. It can be understood that in the process of completing a complete rope skipping action, the human body of the target person is usually in two states: an ascending state and a descending state. In the ascending state, the wrist of the target person moves downward, and the ankle of the target person moves upward; in the descending state, the wrist of the target person moves upward, and the ankle of the target person moves downward. The wrist and ankle are both the target key points of the target person. The characteristics of the target key points of the target person are different in different rope skipping action states. Therefore, the first graph neural network can be trained in advance based on the first sample key points of the first sample person, and the first sample key points are marked with the rope skipping action state of the first sample person, so that the trained first graph neural network can learn the correspondence between the characteristics of the first sample key points and the rope skipping action state of the first sample person, so that the first graph neural network can distinguish the rising state and the falling state of the target person, perform preliminary screening on each target video frame, and determine the rope skipping action state corresponding to each target video frame. Furthermore, the second graph neural network can be pre-trained based on multiple second sample key points of the second sample person. Since the multiple second sample key points are extracted from multiple consecutive sample video frames obtained by shooting the second sample person, and the multiple consecutive sample video frames are marked with the rope skipping action type of the second sample person, the second graph neural network can learn the correspondence between the features of the second sample key points of multiple consecutive sample video frames and the rope skipping action type of the second sample person, so that the second graph neural network can distinguish between normal rope skipping actions and abnormal rope skipping actions, and further screen each target video frame to determine whether the target person is rope skipping normally. It can be seen that the embodiment of the present application can achieve accurate distinction between normal rope skipping actions and abnormal rope skipping actions, thereby screening out abnormal rope skipping actions, so that in the process of rope skipping counting, the number of rope skipping can be increased by one only when a normal rope skipping action is detected, thereby improving the accuracy of rope skipping counting.

[0070] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0072] Figure 1 A schematic diagram of a process flow of a method for identifying a rope skipping action provided in an embodiment of the present application;

[0073] Figure 2a A schematic diagram of a preset image area in a target video frame provided in an embodiment of the present application;

[0074] Figure 2b A schematic diagram of target key points provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a target video frame input sequence provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of the structure of a third graph neural network obtained by cascading the first graph neural network and the second graph neural network provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of a flow chart of a method for training a first graph neural network provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a flow chart of a method for training a second graph neural network provided in an embodiment of the present application;

[0079] Figure 7a A characteristic schematic diagram of a target key point when the rope skipping action type provided in the embodiment of the present application is normal;

[0080] Figure 7b A characteristic schematic diagram of a target key point when the rope skipping action type provided in the embodiment of the present application is stumbling;

[0081] Figure 7c A characteristic schematic diagram of a target key point when the rope skipping action type provided in the embodiment of the present application is a stepping action;

[0082] Figure 7d A characteristic schematic diagram of a target key point when the rope skipping action type provided in an embodiment of the present application is background;

[0083] Figure 7e Another characteristic schematic diagram of target key points when the rope skipping action type provided in the embodiment of the present application is normal;

[0084] Figure 7f A characteristic schematic diagram of a target key point when the rope skipping action type provided in the embodiment of the present application is an empty jump;

[0085] Figure 8 Another schematic diagram of a process flow of a method for identifying a rope skipping action provided in an embodiment of the present application;

[0086] Fig. 9 A schematic diagram of a process of outputting a rope skipping action type by a second graph neural network provided in an embodiment of the present application;

[0087] Fig.10 A schematic diagram of the structure of a rope skipping action recognition device provided in an embodiment of the present application;

[0088] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0090] In order to more clearly explain the rope skipping action recognition method, device, electronic device and storage medium provided by the present application, some terms appearing in this article are explained below:

[0091] Human key point detection: Through deep learning methods, the human body in the image is detected and the human skeleton key points are extracted. The human skeleton key points are referred to as human key points below. The human key points include: the top of the head, upper neck, left shoulder, right shoulder, left hip, right hip, left knee, right knee and other parts of the key points.

[0092] GCN (Graph Convolutional Network): GCN is a deep learning model that is often used to effectively process irregular structured data.

[0093] In order to more clearly illustrate the rope skipping action recognition method provided by the present application, the possible application scenarios of the rope skipping action recognition method provided by the present application will be exemplified below. It can be understood that the following examples are only possible application scenarios of the rope skipping action recognition method provided by the present application. In other possible embodiments, the rope skipping action recognition method provided by the present application can be applied to other possible application scenarios, and the following examples do not impose any limitations on this.

[0094] At present, with the rapid development of the Internet, especially mobile Internet, deep learning and other artificial intelligence technologies, informatization has penetrated into all areas of society and all aspects of people's daily lives, and deep learning methods have also been applied to various fields. In the fields of sports, physical health tests, etc., some physical health test items can be set for personnel to detect the physical health of personnel, so that personnel can plan their own exercise methods according to the test results to enhance their own physical health. For example, physical health test items can refer to skipping, pull-ups and other items. In the skipping or pull-up items, deep learning methods can be used to automatically count the skipping or pull-up actions.

[0095] In the scenario of the rope skipping test project, the amplitude change of the rope skipping action of the person can be analyzed by the deep learning method or the different motion states of the person can be distinguished. The above method can detect whether the person has performed the rope skipping action. Each time a rope skipping action is detected, the number of rope skipping is increased by one, thereby realizing the rope skipping counting. However, it is understandable that in addition to normal rope skipping actions, people may also have abnormal rope skipping actions such as stumbling and stepping during the rope skipping process. When abnormal rope skipping actions occur, the number of rope skipping should not be increased by one. If the normal rope skipping action and the abnormal rope skipping action cannot be distinguished, the number of rope skipping generated by the abnormal rope skipping action will be incorrectly recorded in the number of rope skipping of the person, resulting in a low accuracy of the rope skipping count, that is, the abnormal rope skipping action will affect the accuracy of the rope skipping count. In the above rope skipping counting method, it is not possible to distinguish whether the rope skipping action of the person is a normal rope skipping action. Therefore, the accuracy of the above rope skipping technical method is low.

[0096] Based on this, in order to accurately distinguish normal rope skipping movements from abnormal rope skipping movements and improve the accuracy of rope skipping counting, the embodiment of the present application provides a rope skipping movement recognition method, see Figure 1 , methods include:

[0097] S101, detecting a person located in a preset image area from each target video frame of a target video as a target person.

[0098] S102, extracting the human body key points of the target person from each target video frame as the target key points corresponding to each target video frame.

[0099] S103, input the target key points corresponding to each target video frame into the first graph neural network, and obtain the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame.

[0100] The rope skipping action states include: rising state and falling state.

[0101] S104, in response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of target video frames, inputting target key points corresponding to the plurality of consecutive target video frames into a second graph neural network to obtain a rope skipping action type output by the second graph neural network.

[0102] Among them, the skipping rope action type is used to indicate whether the target person in multiple consecutive target video frames is skipping rope normally, and the preset skipping rope condition is: the skipping rope action state corresponding to the first multiple target video frames in the multiple consecutive target video frames is an ascending state, and the skipping rope action state corresponding to the remaining multiple target video frames is a descending state.

[0103] Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from multiple consecutive sample video frames obtained by shooting the second sample person, and the multiple consecutive sample video frames are marked with the skipping action type of the second sample person.

[0104] By using the embodiments of the present application, the target person who needs to perform rope skipping action recognition can be determined by detecting the person located in the preset image area from each target video frame of the target video, and the human body key points of the target person are extracted from each target video frame respectively for the target person, and the target key points corresponding to each target video frame are obtained, so that the rope skipping action can be recognized according to the characteristics of the target key points. It can be understood that in the process of completing a complete rope skipping action, the human body of the target person is usually in two states: an ascending state and a descending state. In the ascending state, the wrist of the target person moves downward, and the ankle of the target person moves upward; in the descending state, the wrist of the target person moves upward, and the ankle of the target person moves downward. The wrist and ankle are both the target key points of the target person. The characteristics of the target key points of the target person are different in different rope skipping action states. Therefore, the first graph neural network can be trained in advance based on the first sample key points of the first sample person, and the first sample key points are marked with the rope skipping action state of the first sample person, so that the trained first graph neural network can learn the correspondence between the characteristics of the first sample key points and the rope skipping action state of the first sample person, so that the first graph neural network can distinguish the rising state and the falling state of the target person, perform preliminary screening on each target video frame, and determine the rope skipping action state corresponding to each target video frame. Furthermore, the second graph neural network can be pre-trained based on multiple second sample key points of the second sample person. Since the multiple second sample key points are extracted from multiple consecutive sample video frames obtained by shooting the second sample person, and the multiple consecutive sample video frames are marked with the rope skipping action type of the second sample person, the second graph neural network can learn the correspondence between the features of the second sample key points of multiple consecutive sample video frames and the rope skipping action type of the second sample person, so that the second graph neural network can distinguish between normal rope skipping actions and abnormal rope skipping actions, and further screen each target video frame to determine whether the target person is rope skipping normally. It can be seen that the embodiment of the present application can achieve accurate distinction between normal rope skipping actions and abnormal rope skipping actions, thereby screening out abnormal rope skipping actions, so that in the process of rope skipping counting, the number of rope skipping can be increased by one only when a normal rope skipping action is detected, thereby improving the accuracy of rope skipping counting.

[0105] The above S101-S104 will be described exemplarily below:

[0106] In S101, the target video may be obtained by shooting the rope skipping process of a person through a video acquisition device, and the video acquisition device may refer to any device capable of shooting a video, such as a camera, a mobile phone, etc. The target video may include the rope skipping process of multiple persons, or the rope skipping process of one person.

[0107] It is understandable that, in addition to the process of skipping rope, the target video also includes background images such as the environment in which the person is located when the target video is shot, which may interfere with the recognition of the skipping rope action. Therefore, in order to reduce the interference of the background images in the target video on the recognition of the skipping rope action, a preset image area can be set, and the skipping rope action is only recognized for the person in the preset image area. Compared with the image area of ​​each target video frame in the target video, the preset image area is smaller, and the background images included in the preset image area are less, so that the interference of the background images on the recognition of the skipping rope action can be reduced.

[0108] Furthermore, it is understandable that if the preset image area is set to be relatively small or the position is relatively biased, for example, the preset image area is located at the far left of the target video frame and its size is one tenth of the target video frame, it may result in a situation where there is no person in the preset image area, and the subsequent rope skipping action recognition cannot be performed. Based on this, in order to allow the presence of people in the preset image area, the preset image area should be set to an area in the target video frame where the probability of the person appearing is relatively high. For example, when shooting the target video, the person being shot is generally in the center of the shooting, that is, the probability of a person jumping rope appearing in the center area of ​​the target video frame is relatively high, or the person is likely to be in the target video frame. The area is considered to be slightly left of the center or slightly right of the center. Exemplarily, the setting of the preset image area can be as follows: Figure 2a As shown, Figure 2a For a certain target video frame in the target video, the preset image area 21 is located in the area below the center of the target video frame.

[0109] Specifically, a detector can be used to detect a person in a target video frame to obtain the area where the person is located in the target video frame. If the area where a person is located overlaps with a preset image area, the person is considered to be in the preset image area and is used as the target person. After determining the target person in the target video frame, the target person can be followed by a follower to determine the positions of the target persons in all target video frames. Afterwards, the aforementioned S102-S104 can be performed on the target person to identify the rope skipping action of the target person. For example, see Figure 2a As shown, 21 is the preset image area, 22 is the area where person 1 is located in the target video frame, and 23 is the area where person 2 is located in the target video frame. The area overlapping with the preset image area is area 22 where person 1 is located in the target video frame. Then, the person located in the preset image area is person 1, that is, the target person is person 1.

[0110] There may be only one person or multiple people in the preset image area, that is, the number of target people may be one or more. If the number of target people is multiple, then for each target person, the aforementioned S102-S104 is executed to realize the rope skipping action recognition of each target person. The following will only take the number of target people as one as an example to exemplarily illustrate the rope skipping action recognition method provided by the present application. For the case where the number of target people is multiple, a follower can be used to follow a certain target person to determine the position of the target person in all target video frames. Therefore, the follower can determine the different target people in all video frames, that is, the position of the target person in all target video frames can be determined for each target person. Then, when the rope skipping action is recognized for multiple target people, the aforementioned S102-S104 can be executed for each target person respectively to realize the rope skipping action recognition for each target person. The way of rope skipping action recognition for different target people is the same, so the following will take a target person as an example to exemplarily illustrate how to recognize the rope skipping action for the target person.

[0111] The target videos, sample video frames and other video images in this application are acquired with the knowledge and permission of the personnel, and the purpose of acquiring the video images in this application is to improve the accuracy of rope skipping counting in the physical health test item of rope skipping, so as to accurately obtain the test results of the physical health level of the personnel, so that the personnel can plan their own exercise methods according to the test results and enhance their own physical health.

[0112] In S102, when the follower determines the target person in each target video frame, the target person in each target video frame is reflected in the form of a marked frame. Each target video frame is input into a pre-trained human posture estimation model, and the human posture estimation model extracts human key points for the target person in the marked frame in each target video frame, and outputs the human key points of the target person in each target video frame as the target key points corresponding to each target video frame. In addition, according to the time corresponding to each target video frame, the target key points corresponding to each target video frame are stored, that is, when storing the target key points corresponding to a certain target video frame, the time corresponding to the target video frame is stored accordingly. The process of S102 is the aforementioned human key point detection.

[0113] The target key points corresponding to each target video frame can be stored in the form of a one-dimensional array, where one one-dimensional array is the target key points corresponding to a target video frame, and the i-th bit in the array is the coordinate of the i-th target key point corresponding to the target video frame in the target video frame. The target key points corresponding to each target video frame can also be stored in the form of a two-dimensional array, where the x-th row and y-th column of the two-dimensional array are the coordinates of the y-th target key point corresponding to the x-th target video frame in the x-th target video frame. The target key points corresponding to each target video frame can also be stored in the form of a topological map. For example, the topological map formed by the target key points corresponding to a target video frame can be as follows. Figure 2b shown. Figure 2b The black dots in the figure are the target key points.

[0114] In S103, it can be understood that when the target person completes a complete rope skipping action, the human body is usually in two states: an ascending state and a descending state. In the ascending state, the wrist of the target person moves downward, and the ankle of the target person moves upward; in the descending state, the wrist of the target person moves upward, and the ankle of the target person moves downward. In different rope skipping action states, that is, the ascending state and the descending state, the characteristics of the target key points (i.e., the wrist and the ankle) are different, and the rope skipping action state of the target person in the target video frame can be judged according to the characteristics of the target key points corresponding to the target video frame.

[0115] The first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the rope skipping action state of the first sample person, so that the first graph neural network can learn the correspondence between the features of the first sample key points and the rope skipping action state of the first sample person. After the target key points corresponding to each target video frame are input into the first graph neural network, the first graph neural network can extract features of the target key points corresponding to each target video frame to obtain the target key point features corresponding to each target video frame, and based on the correspondence between the features of the first sample key points and the rope skipping action state of the first sample person, output the rope skipping action state corresponding to each target key point feature, that is, obtain the rope skipping action state corresponding to each target video frame. The following will provide an exemplary explanation of how to train the first graph neural network, which will not be repeated here.

[0116] If the target person in a target video frame is not skipping rope, for example, the target person is stationary, or is in a non-skipping rope action state such as turning around or running, then the characteristics of the target key points corresponding to the target person should not conform to the rising state or the falling state. For this target person, the skipping rope action state of the target video frame output by the first graph neural network is the background.

[0117] When inputting the target key points corresponding to each target video frame into the first graph neural network, the target key points corresponding to each target video frame can be input into the first graph neural network in the chronological order of the time corresponding to each target video frame. In this example, the first graph neural network outputs the rope skipping action state corresponding to each target video frame in the chronological order of the time corresponding to each target video frame.

[0118] When inputting the target key points corresponding to each target video frame into the first graph neural network, the target key points corresponding to each target video frame can also be randomly input into the first graph neural network. In this example, the first graph neural network outputs the rope skipping action state corresponding to each target video frame in the order in which each target video frame is input into the first graph neural network. Since the time corresponding to the target video frame is stored when the target key points corresponding to a target video frame are stored, the time sequence between the target key points corresponding to each target video frame can be determined. In this example, even if the target key points corresponding to each target video frame are not input into the first graph neural network according to the time sequence between each target video frame, the rope skipping action state corresponding to each target video frame can still be sorted according to the time sequence between the target key points corresponding to each target video frame, that is, according to the time sequence between each target video frame, to obtain the rope skipping action state corresponding to each target video frame arranged according to the time sequence between each target video frame.

[0119] For example, Figure 3 As shown, Figure 3 The boxes in represent the target key points corresponding to each target video frame. The numbers 1, 2, 3, 4, and 5 in the boxes are obtained by sorting the target key points corresponding to each target video frame according to the time sequence between each target video frame. The smaller the number, the earlier the time sequence of the target video frame. Assuming that the target key points corresponding to each target video frame are input into the first graph neural network in the order of 5, 2, 3, 4, and 1, the order of the rope skipping action states corresponding to each target video frame output by the first graph neural network is also 5, 2, 3, 4, and 1. According to the order between the target key points corresponding to each target video frame, the rope skipping action state corresponding to the target video frame with the order of 5 is swapped with the rope skipping action state corresponding to the target video frame with the order of 1, and the rope skipping action state corresponding to each target video frame arranged in the time sequence between each target video frame is obtained.

[0120] In S104, it is understandable that it takes a certain amount of time for the target person to complete a complete rope skipping action, that is, the human body is usually in a continuous rising state and a continuous falling state. In addition, when completing a normal rope skipping action, the target person must first jump and then land, that is, the human body of the target person must first be in a continuous rising state and then in a continuous falling state. Therefore, only when it is detected that the rope skipping action state corresponding to N consecutive target video frames is in an ascending state, and the rope skipping action state corresponding to the adjacent M consecutive target video frames is in a descending state, can the N+M consecutive target video frames be considered as a complete rope skipping action, and then the N+M consecutive target video frames, that is, the complete rope skipping action, need to be detected to determine whether the complete rope skipping action is a normal rope skipping action, and the judgment result is the rope skipping action type of the target person. Exemplarily, assuming that the target video includes 100 target video frames, if it is detected that the rope skipping action state corresponding to 6 consecutive target video frames 10-15 is a rising state, and the rope skipping action state corresponding to 5 consecutive target video frames 16-20 is a falling state, and target video frames 10-20 are 11 consecutive target video frames, then it can be considered that the 11 consecutive target video frames 10-20 are a complete rope skipping action.

[0121] That is to say, before detecting the type of rope skipping action, it is necessary to first detect a complete rope skipping action, that is, to detect a plurality of consecutive target video frames that meet the preset rope skipping condition from a plurality of target video frames. The preset rope skipping condition is a condition that makes the plurality of consecutive target video frames meet the complete rope skipping action. Specifically, the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the plurality of consecutive target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state.

[0122] It is understandable that different target persons have different skipping speeds, and the duration of the rising state and the descending state of different target persons is different, that is, for different target persons, the number of target video frames in multiple consecutive target video frames (hereinafter referred to as the number of video frames) is different. Therefore, when multiple consecutive target video frames are detected, different numbers of video frames can be set for different target persons. The number of video frames can be set by the user according to past experience for different skipping speeds, or it can be set by the user according to actual detection needs.

[0123] Furthermore, after determining the continuous multiple target video frames, it is necessary to continue to detect the target video frames other than the continuous multiple target video frames in each target video frame to obtain new continuous multiple target video frames. Exemplarily, assuming that the target video includes 100 target video frames, assuming that the target video frames 1-15 are detected to meet the preset rope skipping condition, then the target video frames 1-15 are regarded as continuous multiple target video frames, and the target video frames 16-100 are continuously detected. If the target video frames 16-30 meet the preset rope skipping condition, the target video frames 1-15 are regarded as new continuous multiple target video frames, and the target video frames 31-100 are continuously detected, and so on.

[0124] As described above, multiple groups of continuous target video frames that meet the preset rope skipping conditions can be detected. The target key points corresponding to all the continuous target video frames that meet the preset rope skipping conditions need to be input into the second graph neural network respectively to obtain the rope skipping action type output by the second neural network for each continuous target video frame. The rope skipping action type is used to indicate whether the target person in the continuous target video frames is jumping rope normally, that is, the rope skipping action type output by the second graph neural network can be normal or abnormal.

[0125] The second graph neural network can be obtained by training based on multiple second sample key points of the second sample person in advance. Since the multiple second sample key points are extracted from multiple continuous sample video frames obtained by shooting the second sample person, and the multiple continuous sample video frames are annotated with the rope skipping action type of the second sample person, the second graph neural network can learn the correspondence between the features of the second sample key points of the multiple continuous sample video frames and the rope skipping action type of the second sample person. After the target key points corresponding to the multiple continuous target video frames are input into the second graph neural network, the first graph neural network can extract features of the target key points corresponding to the multiple continuous target video frames, obtain the target key point features corresponding to the multiple continuous target video frames, and output the rope skipping action type of the target person in the multiple continuous target video frames based on the correspondence between the features of the second sample key points of the multiple continuous sample video frames and the rope skipping action type of the second sample person. The following will provide an exemplary description of how to train the second graph neural network, which will not be repeated here.

[0126] Rope skipping counting can be achieved by counting the normal number of rope skipping action types output by the second graph neural network.

[0127] In an embodiment of the present application, the structure of the first graph neural network and the second graph neural network is the same as the structure of the aforementioned graph convolutional network GCN. The first graph neural network and the second graph neural network can be two separate graph neural networks to implement the rope skipping action recognition method provided in the present application; the first graph neural network and the second graph neural network can also be cascaded to obtain a third graph neural network to implement the rope skipping action recognition method provided in the present application.

[0128] The structure of the third graph neural network obtained by cascading the first graph neural network and the second graph neural network can be as follows Figure 4 See Figure 4 The third graph neural network includes a first-level graph network (i.e., the aforementioned first graph neural network) and a second-level graph network (i.e., the aforementioned second graph neural network). According to the chronological order of each target video frame, that is, according to Temporal (timing), the target key points (i.e., the key point sequence) corresponding to each target video frame are input into the third graph neural network, and the category output of the third graph neural network can be obtained, that is, the type of rope skipping action output by the third graph neural network can be obtained.

[0129] The network structure of the first graph neural network can be as follows Figure 4 As shown, see Figure 4 , the network structure of the first graph neural network includes from input to output: the first Gconv (Graph Convolutional Networks), the first CNN (Convolutional Neural Network), the second Gconv, the second CNN, MLP (Multilayer Perceptron) and the output layer y. Among them, the first Gconv is Figure 4 The first graph convolutional neural subnetwork in the first CNN is Figure 4 The first convolutional neural subnetwork in the network, the second Gconv is Figure 4 The second graph convolutional neural subnetwork in the second CNN is Figure 4 The second convolutional neural sub-network in .

[0130] The aforementioned S101-S104 has been exemplarily described above. Based on the above description, the implementation of the rope skipping action recognition method provided by the present application depends on the first graph neural network and the second graph neural network. The higher the accuracy of the first graph neural network and the second graph neural network, the higher the accuracy of the rope skipping action recognition method provided by the present application. Based on this, in order to train a more accurate first graph neural network and a second graph neural network, the present application provides a method for training a first graph neural network and a method for training a second graph neural network. The following will first provide an exemplified description of the method for training a first graph neural network provided by the present application. See Figure 5 , the method of training the first graph neural network includes:

[0131] S501, input the first sample key point into the first original graph neural network to obtain the first key point feature of the first sample key point extracted by the first original graph neural network.

[0132] S502, determining a first difference between the first key point feature and the first positive sample feature, and determining a second difference between the first key point feature and the first negative sample feature.

[0133] Among them, the first positive sample feature is the feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is the feature of other first sample key points whose marked rope skipping action state is different from the first sample key point.

[0134] Exemplarily, assuming that among all the first sample key points, the rope skipping action state of the first sample person marked in a certain first sample key point is in an ascending state, then the first positive sample feature is the feature of the other first sample key points marked with the rope skipping action state in an ascending state, and the first negative sample feature is the feature of the other first sample key points marked with the rope skipping action state in a descending state. If the rope skipping action state of the first sample person marked in a certain first sample key point is in a descending state, then the first positive sample feature is the feature of the other first sample key points marked with the rope skipping action state in a descending state, and the first negative sample feature is the feature of the other first sample key points marked with the rope skipping action state in an ascending state.

[0135] The method for calculating the first difference and the second difference should be the same. Specifically, the first difference between the first key point feature and the first positive sample feature can be calculated by any distance calculation method such as Euclidean distance, Hamming distance, cosine distance, etc.; the first difference between the first key point feature and the first negative sample feature can also be calculated by any distance calculation method such as Euclidean distance, Hamming distance, cosine distance, etc.

[0136] S503: construct a first loss function according to the first difference and the second difference.

[0137] Among them, the first loss function is positively correlated with the first difference, and negatively correlated with the second difference.

[0138] It can be understood that since the first positive sample feature is the feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is the feature of other first sample key points whose marked rope skipping action state is different from the first sample key point, the first difference between the first positive sample feature and the first key point feature should be smaller, and the second difference between the first negative sample feature and the first key point feature should be larger. Therefore, when the first difference is smaller and the second difference is larger, it can be considered that the accuracy of the first key point feature extracted by the first original graph neural network is higher, and the value of the first loss function is smaller. When the first difference is larger and the second difference is smaller, it can be considered that the accuracy of the first key point feature extracted by the first original graph neural network is lower, and the value of the first loss function is larger, that is, the first loss function is positively correlated with the first difference, and negatively correlated with the second difference.

[0139] Specifically, the first loss function is constructed by the following formula (1):

[0140]

[0141] Among them, L ts1 is the first loss function; F check1 is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter; d(·,·) is any distance calculation method shown in S502.

[0142] When setting the first distance parameter α1, it is necessary to make the distance between the first key point feature and the first positive sample feature as small as possible, and the distance between the first key point feature and the first negative sample feature as large as possible, that is, to make the first difference as small as possible, the second difference as large as possible, and the value of the first loss function is at least α1. Therefore, the smaller α1 is, the first difference does not need to be too small and the second difference does not need to be too large, so that the first loss function can approach 0, which will lead to poor classification effect of the trained first graph neural network. For example, the first graph neural network will mistakenly classify the target person standing in place and swinging the rope as an ascending state or a descending state. The larger α1 is, the first loss function will maintain a large value and it will be difficult to approach 0, which will make it difficult for the model parameters of the first original graph neural network to converge. Based on this, the setting of the first distance parameter α1 should not be too small or too large. Specifically, the first distance parameter α1 can be set to 0.15. In other possible examples, the first distance parameter α1 can also be set to moderate values ​​such as 0.14 and 0.16.

[0143] By training the first original graph neural network according to the first loss function shown in formula (1), the obtained first graph neural network can learn the feature similarities of the first sample key points with the same rope skipping action state, and at the same time learn the differences between the features of the first sample key points with different rope skipping action states, so that the obtained first graph neural network can more accurately distinguish between the rising state and the falling state.

[0144] Compared with the existing cross entropy loss function, this embodiment uses the ternary loss function shown in formula (1) to train the first graph neural network, thereby improving the first graph neural network's ability to recognize two different rope skipping action states, namely, the rising state and the falling state, and improving the accuracy of the first graph neural network.

[0145] S504: Adjust the model parameters of the first original graph neural network in the direction of gradient descent of the first loss function to obtain a first graph neural network.

[0146] Along the direction of gradient descent of the first loss function, the model parameters of the first original graph neural network are adjusted, and the first original graph neural network after parameter adjustment is used as the new first original graph neural network, and the above-mentioned S501-S503 are returned to execute until the first preset termination condition is reached to obtain the first graph neural network.

[0147] The first preset termination condition may refer to the value of the first loss function being less than the first preset loss threshold, or the number of times the parameters of the first original graph neural network are adjusted reaching the first preset number threshold, or other first preset termination conditions, and this application does not impose any restrictions on this.

[0148] By selecting this embodiment, the trained first graph neural network can fully learn the similarities between the first key point features of the first sample key points corresponding to the same rope skipping action state, and the differences between the first key point features of the first sample key points corresponding to different rope skipping action states, so that the trained first graph neural network can better distinguish between the two different rope skipping action states, namely the rising state and the falling state, thereby improving the accuracy of the first graph neural network, and further improving the accuracy of distinguishing between normal rope skipping actions and abnormal rope skipping actions, thereby improving the accuracy of rope skipping counting.

[0149] The above has been an exemplary description of the method for training the first graph neural network. The following is an exemplary description of the method for training the second graph neural network provided by the present application. Figure 6 , the method of training the second graph neural network includes:

[0150] S601, input multiple second sample key points into the second original graph neural network to obtain second key point features of the multiple second sample key points extracted by the second original graph neural network.

[0151] S602, determining a third difference between the second key point feature and the second positive sample feature, and determining a fourth difference between the second key point feature and the second negative sample feature.

[0152] Among them, the second positive sample feature is the feature of the human body key points extracted from multiple positive sample video frames respectively, and the multiple positive sample video frames are multiple consecutive video frames annotated with the same skipping action type as the multiple consecutive sample video frames; the second negative sample feature is the feature of the human body key points extracted from multiple negative sample video frames respectively, and the multiple negative sample video frames are multiple consecutive video frames annotated with different skipping action types from the multiple consecutive sample video frames.

[0153] Exemplarily, assuming that the rope skipping action type of the second sample person marked in multiple second sample key points is normal, the multiple positive sample video frames refer to: multiple consecutive video frames in which the rope skipping action type of the second sample person is marked as normal, and the second positive sample features are the features of the human key points extracted from the multiple positive sample video frames respectively; the multiple negative sample video frames refer to: multiple consecutive video frames in which the rope skipping action type of the second sample person is marked as abnormal, and the second negative sample features are the features of the human key points extracted from the multiple negative sample video frames respectively.

[0154] Assuming that the rope skipping action type of the second sample person marked in the multiple second sample key points is abnormal, the multiple positive sample video frames refer to: multiple consecutive video frames in which the rope skipping action type of the second sample person is marked as abnormal, and the second positive sample features are the features of the human key points extracted from the multiple positive sample video frames respectively; the multiple negative sample video frames refer to: multiple consecutive video frames in which the rope skipping action type of the second sample person is marked as normal, and the second negative sample features are the features of the human key points extracted from the multiple negative sample video frames respectively.

[0155] The third difference and the fourth difference should be calculated in the same way. Specifically, the third difference between the second key point feature and the second positive sample feature can be calculated by any distance calculation method such as Euclidean distance, Hamming distance, cosine distance, etc.; the fourth difference between the second key point feature and the second negative sample feature can also be calculated by any distance calculation method such as Euclidean distance, Hamming distance, cosine distance, etc.

[0156] S603: construct a second loss function according to the third difference and the fourth difference.

[0157] Among them, the second loss function is positively correlated with the third difference, and negatively correlated with the fourth difference.

[0158] It can be understood that since the second positive sample feature is the feature of the human key point extracted from multiple positive sample video frames, multiple positive sample video frames are multiple continuous video frames annotated with the same skipping action type as the continuous multiple sample video frames, and the second negative sample feature is the feature of the human key point extracted from multiple negative sample video frames, multiple negative sample video frames are multiple continuous video frames annotated with different skipping action types as the continuous multiple sample video frames, the third difference between the second key point feature and the second positive sample feature should be small, and the fourth difference between the second negative sample feature and the second key point feature should be large. Therefore, when the third difference is small and the fourth difference is large, it can be considered that the accuracy of the second key point feature extracted by the second original graph neural network is high, and the value of the second loss function is small. When the third difference is large and the fourth difference is small, it can be considered that the accuracy of the second key point feature extracted by the second original graph neural network is low, and the value of the second loss function is large, that is, the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference.

[0159] Specifically, the second loss function is constructed by the following formula (2):

[0160]

[0161] Among them, L ts2 is the second loss function; F check2 is the second key point feature; Fpositive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(F check2 , F positive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter; d(·,·) is any distance calculation method shown in S602.

[0162] When setting the second distance parameter α2, it is necessary to make the distance between the second key point feature and the second positive sample feature as small as possible, and the distance between the second key point feature and the second negative sample feature as large as possible, that is, to make the third difference as small as possible, the fourth difference as large as possible, and the value of the second loss function is at least α2. Therefore, the smaller α2 is, the third difference does not need to be too small and the fourth difference does not need to be too large, so that the second loss function can approach 0, which will lead to poor classification effect of the trained second graph neural network. The larger α2 is, the second loss function will maintain a large value and it will be difficult to approach 0, which will make the model parameters of the second original graph neural network difficult to converge. Based on this, the setting of the second distance parameter α2 should not be too small or too large. Specifically, the second distance parameter α2 can be set to 0.15. In other possible examples, the second distance parameter α2 can also be set to moderate values ​​such as 0.14 and 0.16. The values ​​of the first distance parameter α1 and the second distance parameter α2 can be the same or different.

[0163] By training the second original graph neural network according to the second loss function shown in formula (2), the obtained second graph neural network can learn the feature similarities of multiple second sample key points of the same rope skipping action type, and at the same time learn the differences between the features of multiple second sample key points of different rope skipping action types, so that the obtained second graph neural network can more accurately distinguish different rope skipping action types.

[0164] Compared with the existing cross entropy loss function, this embodiment uses the ternary loss function shown in formula (2) to train the second graph neural network, thereby improving the second graph neural network's ability to recognize two different types of normal and abnormal rope skipping movements and improving the accuracy of the second graph neural network.

[0165] S604: Adjust the model parameters of the second original graph neural network in the direction of the gradient descent of the second loss function to obtain a second graph neural network.

[0166] Along the direction of the gradient descent of the second loss function, the model parameters of the second original graph neural network are adjusted, and the second original graph neural network with adjusted parameters is used as the new second original graph neural network, and the above-mentioned S601-S603 are returned to execute until the second preset termination condition is reached to obtain the second graph neural network.

[0167] The second preset termination condition may refer to the value of the second loss function being less than the second preset loss threshold, or the number of parameter adjustments to the second original graph neural network reaching the second preset number threshold, or other second preset termination conditions, and this application does not impose any restrictions on this.

[0168] By selecting this embodiment, the trained second graph neural network can fully learn the similarities between the second key point features of the second sample key points corresponding to the same rope skipping action type, and the differences between the second key point features of the second sample key points corresponding to different rope skipping action states, so that the trained second graph neural network can better distinguish different rope skipping action types, thereby improving the accuracy of the second graph neural network, and further improving the accuracy of distinguishing normal rope skipping actions from abnormal rope skipping actions, thereby improving the accuracy of rope skipping counting.

[0169] The above has been an exemplary description of the method for training the second graph neural network, and the following will be an exemplary description of the rope skipping action type in the aforementioned S104. It can be understood that when the rope skipping action type is abnormal, it may be divided into a variety of different abnormal rope skipping situations. Specifically, the abnormal rope skipping situation may include: stumbling, padding, background, and empty jump, that is, the aforementioned rope skipping action types include: normal, stumbling, padding, background, and empty jump.

[0170] Among them, when the target person's rope skipping action type is normal, the characteristics of the target key point of the left hip of the target person in the target video can be as follows: Figure 7a As shown in the figure, the target key points show regular movement changes in the time and space dimensions, and adjacent skipping actions maintain a certain interval.

[0171] When the target person's rope skipping action type is stumbling, the feature of the target key point of the target person's left hip in the target video can be as follows: Figure 7b As shown, the target key points are obviously truncated in the spatiotemporal dimensions.

[0172] When the target person's rope skipping action type is stepping, the feature of the target key point of the target person's left hip in the target video can be as follows: Figure 7c As shown in the figure, the target key points show a low and high operation pattern in the time and space dimensions.

[0173] When the target person's rope skipping action type is background, the feature of the target key point of the target person's left hip in the target video can be as follows: Figure 7d As shown, the target key points have non-periodic amplitude changes in the spatiotemporal dimension.

[0174] When the target person's rope skipping action type is empty jump, it is necessary to combine the features of the two target key points, wrist and ankle, to determine whether the target person's rope skipping action type is empty jump by judging whether the target person's rope skipping action is coordinated with his hands and feet. When the target person's rope skipping action type is normal, the features of the two target key points, wrist and ankle, can be as follows Figure 7e As shown in Figure 1, in normal rope skipping, the wrists and ankles show a staggered consistency, that is, the torso completes the jumping action while the hands are swinging down. When the target person's rope skipping action type is an empty jump, the characteristics of the two target key points of the wrist and ankle can be as follows: Figure 7f As shown, during the empty rope skipping action, the wrists and ankles do not show interlaced consistency.

[0175] exist Figure 7e as well as Figure 7f In FIG. 7 , curve 71 and curve 73 are characteristic curves of the wrist, and curve 72 and curve 74 are characteristic curves of the ankle. Figure 7a , Figure 7b , Figure 7c , Figure 7d , Figure 7e as well as Figure 7f The horizontal axis of the coordinate system is time, and the vertical axis is height.

[0176] In this embodiment, the second sample key points can be labeled with the skipping action type of the second sample person in the following manner: if the second sample person in multiple consecutive sample video frames shows normal skipping behavior, the multiple consecutive sample video frames are labeled as normal; if the second sample person in multiple consecutive sample video frames shows stumbling behavior, the multiple consecutive sample video frames are labeled as stumbling; if the second sample person in multiple consecutive sample video frames shows stepping behavior, the multiple consecutive sample video frames are labeled as stepping; if the second sample person in multiple consecutive sample video frames does not show skipping behavior, the multiple consecutive sample video frames are labeled as background; if the second sample person in multiple consecutive sample video frames shows empty jumping behavior, the multiple consecutive sample video frames are labeled as empty jumping.

[0177] And, in this embodiment, according to Figure 7a-7fThe motion patterns corresponding to the target key points in the different types of rope skipping actions shown can be used to select the wrist and ankle as the second sample key points for training when training the second graph neural network. The trained second graph neural network can learn the motion patterns corresponding to the second sample key points in different types of rope skipping actions, so that the second graph neural network can more finely distinguish between normal, stumbling, stepping, background, and empty jump types of rope skipping actions, so that the second graph neural network can effectively filter out abnormal rope skipping actions such as stepping, empty jump, and stumbling, thereby improving the accuracy of rope skipping counting.

[0178] In this example, if Figure 2a As shown, the rope skipping action recognition method provided by the present application is used to recognize the rope skipping action of the target person, and the obtained rope skipping action type result can be displayed in the left area of ​​the target video frame. Figure 2a "6 seconds in training" means: the target person in the target video is currently being identified for skipping rope action, and the target person's skipping rope action recognition has been completed for 6 seconds of the target video. For the sake of convenience in the following description, the 6 seconds of video that has completed the target person's skipping rope action recognition will be referred to as an identified video clip; "Valid number: 3" means: 3 of the identified video clips have normal skipping rope action types; "Number of tripping times: 0" means: 0 of the identified video clips have tripping rope action types; "Action violation: none" means: the target person in the identified video clip does not show any skipping rope action types such as padding, background, or empty jump, that is, there are 0 skipping rope action types such as padding, background, or empty jump in the identified video clip.

[0179] The above has provided an exemplary description of the rope skipping action types in the aforementioned S104. Figure 8 An overall flow chart of the rope skipping action recognition method provided in the embodiment of the present application is shown in FIG. Figure 8 The rope skipping action recognition method is implemented by a video data acquisition unit 801, a target tracking and detection unit 802, a human key point extraction unit 803, a rope skipping action counting unit 804, and an abnormal rope skipping action filtering unit 805.

[0180] The video data acquisition unit 801 is used to acquire the target video.

[0181] The target tracking and detecting unit 802 is used to execute the aforementioned S101. That is, the target tracking and detecting unit 802 is used to detect a person located in a preset image area from each target video frame of the target video as a target person.

[0182] The human body key point extraction unit 803 is used to execute the aforementioned S102. That is, the human body key point extraction unit 803 is used to extract the human body key points of the target person from each target video frame as the target key points corresponding to each target video frame.

[0183] The rope skipping action counting unit 804 is used to execute the aforementioned S103. That is, the rope skipping action counting unit 804 is used to input the target key points corresponding to each target video frame into the first graph neural network, and obtain the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame.

[0184] The abnormal rope skipping action filtering unit 805 is used to execute the aforementioned S104. That is, the abnormal rope skipping action filtering unit 805 is used to, in response to detecting a plurality of consecutive target video frames that meet the preset rope skipping condition from a plurality of target video frames, input the target key points corresponding to the consecutive plurality of target video frames into the second graph neural network to obtain the rope skipping action type output by the second graph neural network.

[0185] In the abnormal rope skipping action filtering unit 805, the process of the second graph neural network outputting the rope skipping action type can be as follows: Fig. 9 As shown, the second graph neural network (i.e. Fig. 9 The graph convolutional network (GCN) in the image processing method concatenates and fuses the features of the two target key points, namely, the wrist and the ankle, to obtain fused features, and classifies the fused features into one of the five rope skipping action types: normal, stumbling, stepping, background, and empty jump.

[0186] Corresponding to the aforementioned rope skipping action recognition method, the present application embodiment also provides a rope skipping action recognition device, see Fig.10 , the device comprises:

[0187] The target person detection module 1001 is used to detect a person located in a preset image area from each target video frame of the target video as a target person;

[0188] A target key point extraction module 1002 is used to extract the human body key points of the target person from each target video frame as the target key points corresponding to each target video frame;

[0189] The rope skipping action state obtaining module 1003 is used to input the target key points corresponding to each target video frame into the first graph neural network, and obtain the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: rising state and falling state;

[0190] The rope skipping action type obtaining module 1004 is used for, in response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of target video frames, inputting the target key points corresponding to the consecutive plurality of target video frames into the second graph neural network, and obtaining the rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the consecutive plurality of target video frames is rope skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the consecutive plurality of target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state;

[0191] Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from multiple consecutive sample video frames obtained by shooting the second sample person, and the multiple consecutive sample video frames are marked with the skipping action type of the second sample person.

[0192] In a possible embodiment, the first graph neural network is pre-trained in the following manner, including:

[0193] Inputting the first sample key point into the first original graph neural network to obtain a first key point feature of the first sample key point extracted by the first original graph neural network;

[0194] Determine a first difference between the first key point feature and the first positive sample feature, and determine a second difference between the first key point feature and the first negative sample feature; wherein the first positive sample feature is a feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is a feature of other first sample key points whose marked rope skipping action state is different from the first sample key point;

[0195] Constructing a first loss function according to the first difference and the second difference, wherein the first loss function is positively correlated with the first difference and negatively correlated with the second difference;

[0196] In the direction of gradient descent of the first loss function, the model parameters of the first original graph neural network are adjusted to obtain the first graph neural network.

[0197] In a possible embodiment, a first loss function is constructed according to the first difference and the second difference by using the following formula, including:

[0198]

[0199] Among them, L ts1 is the first loss function; F check1is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter.

[0200] In a possible embodiment, the second graph neural network is pre-trained in the following manner, including:

[0201] Inputting the plurality of second sample key points into the second original graph neural network to obtain second key point features of the plurality of second sample key points extracted by the second original graph neural network;

[0202] Determine a third difference between the second key point feature and the second positive sample feature, and determine a fourth difference between the second key point feature and the second negative sample feature; wherein the second positive sample feature is a feature of a human key point extracted from a plurality of positive sample video frames, the plurality of positive sample video frames are a plurality of consecutive video frames annotated with the same rope skipping action type as the plurality of consecutive sample video frames, and the second negative sample feature is a feature of a human key point extracted from a plurality of negative sample video frames, the plurality of negative sample video frames are a plurality of consecutive video frames annotated with different rope skipping action types than the plurality of consecutive sample video frames;

[0203] constructing a second loss function according to the third difference and the fourth difference, wherein the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference;

[0204] In the direction of the gradient descent of the second loss function, the model parameters of the second original graph neural network are adjusted to obtain the second graph neural network.

[0205] In a possible embodiment, a second loss function is constructed according to the third difference and the fourth difference by the following formula, including:

[0206]

[0207] Among them, L ts2 is the second loss function; F check2 is the second key point feature; F positive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(F check2 , F positive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter.

[0208] In a possible embodiment, the rope skipping action types include: normal, stumbling, padding, background, and empty jump;

[0209] The rope skipping action type of the second sample personnel is annotated in advance by the following methods:

[0210] If the second sample person in a plurality of consecutive sample video frames shows normal rope skipping behavior, the plurality of consecutive sample video frames are marked as normal;

[0211] If the second sample person in a plurality of consecutive sample video frames shows stumbling behavior, the plurality of consecutive sample video frames are marked as stumbling;

[0212] If the second sample person in a plurality of consecutive sample video frames shows a stepping behavior, the plurality of consecutive sample video frames are marked as stepping;

[0213] If the second sample person in a plurality of consecutive sample video frames does not perform rope skipping behavior, the plurality of consecutive sample video frames are marked as background;

[0214] If the second sample person in a plurality of consecutive sample video frames exhibits a skipping behavior, the plurality of consecutive sample video frames are marked as skipping.

[0215] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0216] Memory 1101, used for storing computer programs;

[0217] The processor 1102 is used to implement the following steps when executing the program stored in the memory 1101:

[0218] From each target video frame of the target video, a person located in a preset image area is detected as a target person;

[0219] Extracting the human body key points of the target person from each target video frame respectively as the target key points corresponding to each target video frame;

[0220] Input the target key points corresponding to each target video frame into the first graph neural network, and obtain the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: rising state and falling state;

[0221] In response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of target video frames, the target key points corresponding to the plurality of consecutive target video frames are input into a second graph neural network to obtain a rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the plurality of consecutive target video frames is rope skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the plurality of consecutive target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state;

[0222] Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from multiple consecutive sample video frames obtained by shooting the second sample person, and the multiple consecutive sample video frames are marked with the skipping action type of the second sample person.

[0223] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0224] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0225] The communication interface is used for communication between the above electronic device and other devices.

[0226] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0227] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0228] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned rope skipping action recognition methods are implemented.

[0229] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the rope skipping action recognition methods in the above embodiments.

[0230] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0231] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0232] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0233] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for identifying a rope skipping action, characterized in that: The method comprises: From each target video frame of the target video, a person located in a preset image area is detected as a target person; Extracting the human body key points of the target person from each of the target video frames respectively as the target key points corresponding to each of the target video frames; Inputting the target key points corresponding to each of the target video frames into the first graph neural network, obtaining the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: an ascending state and a descending state; In response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of the target video frames, the target key points corresponding to the plurality of consecutive target video frames are input into a second graph neural network to obtain a rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the plurality of consecutive target video frames is rope skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the plurality of consecutive target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state; Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from a plurality of consecutive sample video frames obtained by shooting the second sample person, and the plurality of consecutive sample video frames are marked with the skipping action type of the second sample person.

2. The method according to claim 1, characterized in that The first graph neural network is pre-trained in the following manner, including: Inputting the first sample key point into the first original graph neural network to obtain a first key point feature of the first sample key point extracted by the first original graph neural network; Determine a first difference between the first key point feature and the first positive sample feature, and determine a second difference between the first key point feature and the first negative sample feature; wherein the first positive sample feature is a feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is a feature of other first sample key points whose marked rope skipping action state is different from the first sample key point; constructing a first loss function according to the first difference and the second difference, wherein the first loss function is positively correlated with the first difference and negatively correlated with the second difference; Adjust the model parameters of the first original graph neural network in the direction of gradient descent of the first loss function to obtain a first graph neural network.

3. The method according to claim 2, characterized in that A first loss function is constructed according to the first difference and the second difference by the following formula, including: ; Among them, L ts1 is the first loss function; F check1 is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter.

4. The method according to claim 1, characterized in that: The second graph neural network is pre-trained in the following manner, including: Inputting a plurality of second sample key points into a second original graph neural network to obtain second key point features of the plurality of second sample key points extracted by the second original graph neural network; Determine a third difference between the second key point feature and the second positive sample feature, and determine a fourth difference between the second key point feature and the second negative sample feature; wherein the second positive sample feature is a feature of a human key point extracted from a plurality of positive sample video frames, the plurality of positive sample video frames are a plurality of consecutive video frames annotated with the same rope skipping action type as the plurality of consecutive sample video frames, and the second negative sample feature is a feature of a human key point extracted from a plurality of negative sample video frames, the plurality of negative sample video frames are a plurality of consecutive video frames annotated with a different rope skipping action type than the plurality of consecutive sample video frames; constructing a second loss function according to the third difference and the fourth difference, wherein the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference; In the direction of the gradient descent of the second loss function, the model parameters of the second original graph neural network are adjusted to obtain a second graph neural network.

5. The method according to claim 4, characterized in that A second loss function is constructed according to the third difference and the fourth difference by the following formula, including: ; Among them, L ts2 is the second loss function; F check2 is the second key point feature; positive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(F check2 , F positive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter.

6. The method according to claim 1, characterized in that The rope skipping action types include: normal, stumbling, padding, background, and empty jump; The rope skipping action type of the second sample person is marked in advance on the second sample key points by the following method, including: If the second sample person in the plurality of consecutive sample video frames shows normal rope skipping behavior, marking the plurality of consecutive sample video frames as normal; If the second sample person in the plurality of consecutive sample video frames stumbles, marking the plurality of consecutive sample video frames as stumbling; If the second sample person in the plurality of consecutive sample video frames has a stepping behavior, marking the plurality of consecutive sample video frames as stepping; If the second sample person in the plurality of consecutive sample video frames does not perform rope skipping behavior, marking the plurality of consecutive sample video frames as background; If the second sample person in the plurality of consecutive sample video frames exhibits a skipping behavior, the plurality of consecutive sample video frames are marked as skipping.

7. A rope skipping action recognition device, characterized in that: The device comprises: A target person detection module is used to detect a person located in a preset image area from each target video frame of the target video as a target person; A target key point extraction module is used to extract the human body key points of the target person from each of the target video frames as the target key points corresponding to each of the target video frames; A rope skipping action state obtaining module, used for inputting the target key points corresponding to each of the target video frames into the first graph neural network, obtaining the rope skipping action state output by the first graph neural network as the rope skipping action state corresponding to the target video frame; the rope skipping action state includes: rising state and falling state; A rope skipping action type obtaining module is used for, in response to detecting a plurality of consecutive target video frames that meet a preset rope skipping condition from a plurality of the target video frames, inputting the target key points corresponding to the consecutive plurality of target video frames into a second graph neural network, and obtaining the rope skipping action type output by the second graph neural network; wherein the rope skipping action type is used to indicate whether the target person in the consecutive plurality of target video frames is skipping normally, and the preset rope skipping condition is: the rope skipping action state corresponding to the first plurality of target video frames in the consecutive plurality of target video frames is an ascending state, and the rope skipping action state corresponding to the remaining plurality of target video frames is a descending state; Among them, the first graph neural network is pre-trained based on the first sample key points of the first sample person, and the first sample key points are marked with the skipping action status of the first sample person; the second graph neural network is pre-trained based on multiple second sample key points of the second sample person, and the multiple second sample key points are extracted from a plurality of consecutive sample video frames obtained by shooting the second sample person, and the plurality of consecutive sample video frames are marked with the skipping action type of the second sample person.

8. The device according to claim 7, characterized in that The first graph neural network is pre-trained in the following manner, including: Inputting the first sample key point into the first original graph neural network to obtain a first key point feature of the first sample key point extracted by the first original graph neural network; Determine a first difference between the first key point feature and the first positive sample feature, and determine a second difference between the first key point feature and the first negative sample feature; wherein the first positive sample feature is a feature of other first sample key points whose marked rope skipping action state is the same as the first sample key point, and the first negative sample feature is a feature of other first sample key points whose marked rope skipping action state is different from the first sample key point; constructing a first loss function according to the first difference and the second difference, wherein the first loss function is positively correlated with the first difference and negatively correlated with the second difference; Adjusting the model parameters of the first original graph neural network in the direction of gradient descent of the first loss function to obtain a first graph neural network; A first loss function is constructed according to the first difference and the second difference by the following formula, including: ; Among them, L ts1 is the first loss function; F check1 is the first key point feature; F positive1 is the first positive sample feature; F negative1 is the first negative sample feature; d(F check1 , F positive1 ) is the first difference; d(F check1 , F negative1 ) is the second difference; α1 is the first distance parameter; The second graph neural network is pre-trained in the following manner, including: Inputting a plurality of second sample key points into a second original graph neural network to obtain second key point features of the plurality of second sample key points extracted by the second original graph neural network; Determine a third difference between the second key point feature and the second positive sample feature, and determine a fourth difference between the second key point feature and the second negative sample feature; wherein the second positive sample feature is a feature of a human key point extracted from a plurality of positive sample video frames, the plurality of positive sample video frames are a plurality of consecutive video frames annotated with the same rope skipping action type as the plurality of consecutive sample video frames, and the second negative sample feature is a feature of a human key point extracted from a plurality of negative sample video frames, the plurality of negative sample video frames are a plurality of consecutive video frames annotated with a different rope skipping action type than the plurality of consecutive sample video frames; constructing a second loss function according to the third difference and the fourth difference, wherein the second loss function is positively correlated with the third difference and negatively correlated with the fourth difference; Adjusting the model parameters of the second original graph neural network in the direction of the gradient descent of the second loss function to obtain a second graph neural network; A second loss function is constructed according to the third difference and the fourth difference by the following formula, including: ; Among them, L ts2 is the second loss function; F check2 is the second key point feature; positive2 is the second positive sample feature; F negative2 is the second negative sample feature; d(F check2 , F positive2 ) is the third difference; d(F check2 , F negative2 ) is the fourth difference; α2 is the second distance parameter; The rope skipping action types include: normal, stumbling, padding, background, and empty jump; The rope skipping action type of the second sample person is marked in advance on the second sample key points by the following method, including: If the second sample person in the plurality of consecutive sample video frames shows normal rope skipping behavior, marking the plurality of consecutive sample video frames as normal; If the second sample person in the plurality of consecutive sample video frames stumbles, marking the plurality of consecutive sample video frames as stumbling; If the second sample person in the plurality of consecutive sample video frames has a stepping behavior, marking the plurality of consecutive sample video frames as stepping; If the second sample person in the plurality of consecutive sample video frames does not perform rope skipping behavior, marking the plurality of consecutive sample video frames as background; If the second sample person in the plurality of consecutive sample video frames exhibits a skipping behavior, the plurality of consecutive sample video frames are marked as skipping.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images