A fancy skipping rope recognition algorithm based on image recognition

Abstract

The application discloses a fancy skipping rope recognition algorithm based on image recognition, and the fancy skipping rope recognition algorithm comprises the following steps: S1, a camera collects video data and sends the video data into a processor, and the processor performs human body key point recognition frame by frame; S2, jumping feature key points are determined, and a target window is divided by using the jumping feature; S3, according to the visibility of the key points, whether the jumping target window is effective is judged; S4, data features of the target window are extracted; S5, according to the data features, whether the current window is a jump is judged; and S6, according to ankle features and wrist features, fancy movements of the ankle and the wrist are respectively recognized. According to the application, by dynamically selecting key points, the problem that fixed key points cannot complete skipping rope counting after being shielded is solved, by using a scale zooming technology, the problem that the skipping rope counting precision is influenced due to the fact that the human body occupies too small proportion in an image is solved, the fancy skipping rope movements can be recognized, and moreover, reasonable feature selection improves the accuracy of skipping rope counting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of skipping rope recognition processing, and in particular relates to a fancy skipping rope recognition algorithm based on image recognition. BACKGROUND

[0002] At present, the skipping rope counting algorithm based on image recognition mainly uses a camera to shoot a video, the video is input frame by frame into a pre-trained key point recognition model, the positions of human body key points such as nose, shoulder and hip in the frame image are found, the data of the selected key points in multiple frames of images are recorded in time sequence, and whether skipping is being performed is determined by judging whether the amplitude of the up and down of the selected key points is greater than a threshold value, and the number of skipping is calculated to realize skipping counting.

[0003] These implementation methods generally require the human body to face the camera, and once the body turns during the skipping process and some key points are blocked, the skipping counting task cannot be normally completed. And only simple jump count can be performed, and some fancy actions cannot be recognized.

[0004] These implementation methods often require the user to be in the image all over the body, and the proportion in the image cannot be too small, that is, the distance from the camera cannot be too far, otherwise the amplitude of the up and down of the key points may not reach the threshold value of skipping counting, resulting in a decrease in the accuracy of skipping counting. Generally, a display device is used to inform the user to move the body, and the body in the image is moved to a certain frame to limit the position of the user. For the case where the skipping speed is slow and only some simple actions are performed, it is relatively easy to achieve, but when the user performs some fancy actions, because the action is large, it is easy to deviate from the position, resulting in that the original counting algorithm cannot normally complete the skipping counting. Therefore, it is necessary to develop a more accurate fancy skipping rope recognition method based on image recognition. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a fancy skipping rope recognition algorithm based on image recognition.

[0006] An embodiment of the present application adopts the technical solution to solve its technical problem: a fancy skipping rope recognition algorithm based on image recognition, the fancy skipping rope recognition algorithm comprising the following steps:

[0007] S1: a camera collects video data and sends it to a processor, and the processor performs human key point recognition frame by frame;

[0008] S2: determine the jumping feature key points, and divide the target window by using the jumping feature;

[0009] S3: judge whether the jumping target window is valid according to the visibility of the key points;

[0010] S4: Extracting data features of the target window;

[0011] S5: According to the data features, judging whether the current window is a jump;

[0012] S6: According to the ankle features and wrist features, respectively identifying the ankle and wrist tricks.

[0013] As preferred, in step S1, the processor can perform real-time processing and recognition on the video data collected by the camera, or the processor can perform processing and recognition on the video data saved by the camera;

[0014] The information of the human body key points is put into a queue for caching, and a human body key point recognition model is selected, which has the following requirements:

[0015] A1: At least the following key points can be output: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left wrist, right wrist, left ankle and right ankle;

[0016] A2: The key point information contains x and y coordinates in the image and is normalized;

[0017] A3: The key point information contains visibility, and the output visibility index is normalized to 0 to 1, 0 indicating invisible and 1 indicating visible.

[0018] As preferred, step S2: determining the jump feature key points, the specific method of using the jump feature to divide the target window is as follows:

[0019] S21: Defining the top-left corner coordinate of the image as (0, 0) and the bottom-right corner coordinate as (1, 1), when the y coordinate of a point becomes smaller, the point is going up in the image; when the y coordinate becomes larger, the point is going down in the image;

[0020] S22: Selecting the data of the left shoulder and the right shoulder, or the left hip and the right hip as the jump feature key points Gleft and Gright, and intercepting the y coordinate data of the jump feature key points Gleft and Gright;

[0021] S23: The y coordinate data of the jump feature key points Gleft and Gright are put into a queue after being filtered by 4-point moving average, and the initial state is to default Gleft as the jump feature key point, when a certain data point is higher than the previous data and the next data, it is determined as a "peak", the visibility attribute of all data points between the two consecutive peaks is averaged, if the average visibility is greater than a threshold M1, it is considered that the target window T is found;

[0022] If the visibility average is less than threshold M1, the default jump feature key point is changed to G right, and the window search continues. When the visibility of G right does not meet the threshold requirement, the jump feature key point is changed back to G left. The sampling serial number corresponding to the first peak is recorded as the start point of the target window, and the sampling serial number corresponding to the second peak is recorded as the end point of the target window T.

[0023] As preferred, step S3: judging whether the jump target window is valid according to the visibility of the key points is specifically as follows:

[0024] The average of the visibility indexes of the shoulder, ankle and wrist key points of the left and right half bodies is calculated.

[0025] When the average of the visibilities of the left shoulder and right shoulder key points is less than threshold M2, it is judged that the target window T is invalid.

[0026] When the average of the visibilities of the left ankle and right ankle key points is less than threshold M2, it is judged that the target window T is invalid.

[0027] When the average of the visibilities of the left wrist and right wrist key points is less than threshold M2, it is judged that the target window T is invalid.

[0028] As preferred, step S4: extracting the data features of the target window is specifically as follows:

[0029] The scale factor L is calculated. If the jump feature key point is in the left half body, i.e. G left, the scale factor L is the average of the differences between all the y coordinates of the left hips and the y coordinate of the left shoulder in the window. If the jump feature key point is in the right half body, i.e. G right, the scale factor L is the average of the differences between all the y coordinates of the right hips and the y coordinate of the right shoulder in the window. The following features of the jump feature key point in the window are extracted:

[0030] Feature a1: the difference between the start point y coordinate and the minimum y coordinate in the window;

[0031] Feature a2: the difference between the end point y coordinate and the minimum y coordinate in the window;

[0032] Feature a3: the average of the absolute values of the y coordinate differences between the previous point and the next point in the window;

[0033] Feature a4: the average of the absolute values of the y coordinate differences between all the points and the average of the start point y coordinate and the end point y coordinate;

[0034] Feature a5: the sampling serial number of the end point minus the sampling serial number of the start point divided by the frame rate of the video;

[0035] Feature a6: the correlation coefficient between the jump feature point and the knee key point of the half body where the jump feature point is located is calculated.

[0036] The results of features a1 to a4 also need to be divided by a scale factor L for dynamic scaling of the y-axis data.

[0037] As preferred, step S5: judging whether the current window is a jump according to the data features is a feature threshold method or a label training method.

[0038] Feature threshold method: set a corresponding threshold for each feature in the target window, and when one or more features meet the threshold requirement, it is identified as a jump action.

[0039] Label training method: collect a large amount of skipping data, find the target window and extract the recorded features, label each window with whether it is a jump, train a machine learning model using the features and labels, obtain a prediction model, and use the prediction model to identify whether the target window provided later is a jump.

[0040] As preferred, step S6: according to the ankle features and wrist features, respectively, identify the fancy actions of the ankle and wrist, and according to the ankle features, identify the fancy actions of the ankle and wrist.

[0041] Jumping with both feet: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window, and when the correlation coefficient is greater than a threshold M3, it is a jump with both feet; if it cannot be identified as other types, it is defaulted as a jump with both feet.

[0042] Crossing left and right feet: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window, and when the correlation coefficient is less than a threshold M4, it is a left and right cross jump.

[0043] Jumping with both feet: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window, and when the correlation coefficient is greater than a threshold M3, it is a jump with both feet; if it cannot be identified as other types, it is defaulted as a jump with both feet.

[0044] As preferred, step S6: according to the ankle features and wrist features, respectively, identify the fancy actions of the ankle and wrist, and according to the wrist features, identify the fancy actions of the ankle and wrist.

[0045] As preferred, according to the wrist features, the specific method for identifying fancy actions is as follows:

[0046] S61: Select the effective hand; when the average visibility of the left wrist is greater than 0.8 and the average visibility of the right wrist is less than 0.8, select the left wrist; when the average visibility of the left wrist is less than 0.8 and the average visibility of the right wrist is greater than 0.8, select the right wrist; when the average visibility of both the left and right wrists is greater than 0.8, select the hand with more calculated shakes; when both the left and right hands are less than 0.8, do not perform shake recognition;

[0047] S62: In the target window, at any time, when the y coordinate of the effective hand is less than the average of the y coordinates of the shoulder and hip of the half body where the effective hand is located, do not perform shake recognition;

[0048] S63: Select the x coordinate of the wrist of the effective hand, and calculate the number of x coordinate peaks in the window;

[0049] S64: Perform difference filtering on the x coordinate of the effective hand, and then perform 4-point sliding average;

[0050] S65: Count the number of peaks and valleys in the window after difference and smoothing, and the larger number is the shake;

[0051] S66: Define shake 1 as single shake, shake 2 as double shake, and shake greater than or equal to 3 as a trick action of multiple shakes.

[0052] As a preferred embodiment, the specific method for recognizing trick actions according to wrist features further includes the step of recognizing cross actions, which is as follows:

[0053] S67: If the average visibility of the left wrist and the right wrist is greater than 0.8, perform cross action recognition, including the following steps:

[0054] S671: Calculate threshold M6: the average of the absolute value of the difference between the x coordinate of the left shoulder and the x coordinate of the right shoulder;

[0055] S672: Calculate threshold M7: the average of the absolute value of the difference between the x coordinate of the left shoulder and the x coordinate of the right shoulder, divided by 5;

[0056] S673: If the maximum value minus the minimum value of the x coordinates of the left wrist and the right wrist in the target window is greater than threshold M6, and the correlation coefficient of the x coordinates of the left wrist and the right wrist is less than threshold M8, then recognize it as a cross action;

[0057] S674: Count the cross action for each jump, taking the x coordinate of the left shoulder as the baseline for the left wrist and the x coordinate of the right shoulder as the baseline for the right wrist; according to the time sequence of the samples, the data in the target window is judged as follows:

[0058] When the result of the x coordinate of the wrist minus the baseline changes from less than threshold M7 to greater than threshold M7, the cross count is incremented by 1; when the result of the x coordinate of the wrist minus the baseline changes from greater than threshold M7 to less than threshold M7, the cross count is incremented by 1;

[0059] S675: The cross count is 1, which is identified as a single cross, the cross count is 2, which is identified as a double cross, and the cross count is greater than or equal to 3, which is identified as a multi-cross fancy action.

[0060] The application has the advantages that: by dynamically selecting key points, the problem that fixed key points cannot complete the skipping counting after being blocked is solved, by the scale scaling technology, the problem that the human body occupies too small in the image and affects the skipping counting accuracy is solved, the skipping fancy action can be identified, and reasonable feature selection improves the accuracy of skipping counting. BRIEF DESCRIPTION OF DRAWINGS

[0061] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the following drawings, in which:

[0062] Figure 1 The waveform diagram of the y coordinates of the shoulder, hip and knee key points of the left and right half bodies is one of the embodiments;

[0063] Figure 2 The waveform diagram of the y coordinates of the ankles of the left and right feet when the target window is the jumping jack is one of the embodiments;

[0064] Figure 3 The waveform diagram of the y coordinates of the ankles of the left and right feet when the target window is the cross jump is one of the embodiments;

[0065] Figure 4 The waveform diagram of the x coordinates of the ankles of the left and right feet when the target window is the jumping jack is one of the embodiments;

[0066] Figure 5 The waveform diagram of the visibility of the left and right wrists of the target window when the effective hand is the left hand is one of the embodiments, wherein the visibility of the left hand is close to 1, and almost coincides with the scale line of the y coordinate axis;

[0067] Figure 6 The waveform diagram of the x coordinates of the shoulder, hip, knee and left and right hands of the left half body when the effective hand is the left hand is one of the embodiments;

[0068] Figure 7 The waveform diagram of the x coordinates of the effective hand after difference filtering is one of the embodiments;

[0069] Figure 8 The waveform diagram of the x coordinates of the shoulder, hip, knee and left and right hands of the left and right half bodies when the fancy action is one jump and one cross is one of the embodiments;

[0070] Figure 9 The waveform diagram of the x coordinates of the shoulder, hip, knee and left and right hands of the left and right half bodies when the fancy action is one jump and two crosses is one of the embodiments. DETAILED DESCRIPTION

[0071] The specific embodiments of the present application will be described in detail below, the preferred embodiments of the present application are shown in the drawings, the role of the drawings is to supplement the description of the text part with graphics, so that one can intuitively and visually understand each technical feature and the overall technical solution of the present application, but it cannot be understood as a limitation on the scope of protection of the present application.

[0072] In the description of the present application, the meaning of multiple is more than two, greater than, less than, more than, etc. is not included in the number, above, below, within, etc. is included in the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0073] In the description of the present application, it is understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0074] In the present application, unless otherwise explicitly limited, the words "set", "install", "connect" and the like should be broadly understood, for example, they can be directly connected, or indirectly connected through an intermediate medium; can be fixedly connected, or can be detachably connected, or can be integrally formed; can be mechanically connected; can be the internal communication or interaction relationship of two elements. The skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0075] Embodiment

[0076] The present application provides a kind of based on image recognition's fancy skipping rope identification algorithm, fancy skipping rope identification algorithm includes the following steps:

[0077] S1: camera collects video data and sends into processor, processor is identified frame by frame human body key point;

[0078] As preferred, in step S1, the processor can identify the video data collected by the camera in real time, or the processor can process and identify the video data saved by the camera;Extract the frame rate, resolution and other information of the video.

[0079] The information of human body key point is put into queue for caching, and a human body key point identification model is selected, which has the following requirements:

[0080] A1: At least the following key points can be output: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left wrist, right wrist, left ankle, and right ankle;

[0081] A2: The key point information contains x and y coordinates in the image and is normalized; if the normalized coordinates are output, they can be used directly, and if pixel points are output, x is divided by the image width and y is divided by the image height, and then the normalized coordinates are used.

[0082] A3: The key point information contains visibility, and the output visibility index is normalized to 0 to 1, 0 indicating invisible and 1 indicating visible.

[0083] Existing key point recognition models such as OpenPose, AlphaPose, and Mediapipe Pose can meet the requirements.

[0084] S2: Determine the jumping feature key point, and divide the target window using the jumping feature.

[0085] As a preferred embodiment, the specific method of step S2: determining the jumping feature key point and dividing the target window using the jumping feature is as follows:

[0086] S21: Define the top-left corner of the image as (0, 0) and the bottom-right corner as (1, 1). When the y-coordinate of a point decreases, the point moves upward in the image; when the y-coordinate increases, the point moves downward in the image.

[0087] The process of jumping is generally that the knees are slightly bent, the entire body is displaced downward, then the thighs and calves exert force to jump upward, the entire body is displaced upward, and after reaching the highest point, it falls again. In the image, the y-coordinates of the shoulders and knees will first increase, then decrease, reach a minimum value, and then increase again. If jumping is continuous, the process of large to small and then from small to large will appear periodically.

[0088] S22: Select the data of the left shoulder and the right shoulder, or the left hip and the right hip, as the jumping feature key points Gleft and Gright, and intercept the y-coordinate data of the jumping feature key points Gleft and Gright.

[0089] S23: The y-coordinate data of the jumping feature key points Gleft and Gright are respectively put into the queue after 4-point moving average filtering. In the initial state, Gleft is the jumping feature key point by default. When a data point is higher than the previous and subsequent data points, it is determined to be a "peak". The visibility attribute of all data points between the two consecutive peaks is averaged, and if the average visibility is greater than a threshold M1 (default 0.9), it is considered that the target window T has been found.

[0090] If the visibility average is less than threshold M1, the default jump feature key point is changed to G right, and the window search continues. When the visibility of G right does not meet the threshold requirement, the jump feature key point is changed back to G left. The sampling sequence number corresponding to the first peak is recorded as the start point of the target window, and the sampling sequence number corresponding to the second peak is recorded as the end point of the target window T.

[0091] Please refer to Figure 1 An embodiment of the waveform diagram of the y coordinates of the shoulder, hip and knee key points of the left and right halves of the body is shown.

[0092] S3: judging whether the jump target window is valid according to the visibility of the key points;

[0093] As preferred, the specific method of step S3: judging whether the jump target window is valid according to the visibility of the key points is as follows:

[0094] The average value of the visibility indicators of the shoulder, ankle and wrist key points of the left and right halves of the body needs to be calculated. The judgment method of the validity of the target window is as follows:

[0095] When the average values of the visibilities of the two key points of the left shoulder and the right shoulder are both less than threshold M2, it is judged that the target window T is invalid.

[0096] When the average values of the visibilities of the two key points of the left ankle and the right ankle are both less than threshold M2, it is judged that the target window T is invalid.

[0097] When the average values of the visibilities of the two key points of the left wrist and the right wrist are both less than threshold M2, it is judged that the target window T is invalid.

[0098] Threshold M2 can be set to 0.8. The main purpose of increasing the validity judgment is to avoid the jump misjudgment when the whole body is not in the image. Since there may be self-occlusion, the key points of the left and right halves of the body are not visible in the image, and the window is judged to be invalid. Only when the window is valid, the next step of recognition is performed.

[0099] S4: extracting the data features of the target window;

[0100] As preferred, the specific method of step S4: extracting the data features of the target window is as follows:

[0101] The scale factor L is calculated. If the jump feature key point is in the left half of the body, i.e. G left, the scale factor L is the average value of the differences between all left hip y coordinates and left shoulder y coordinates in the window. If the jump feature key point is in the right half of the body, i.e. G right, the scale factor L is the average value of the differences between all right hip y coordinates and right shoulder y coordinates in the window. The following features of the jump feature key point in the window are extracted:

[0102] Feature a1: the difference between the start point y coordinate and the minimum y coordinate in the window;

[0103] Feature a2: the difference between the end point y coordinate and the minimum y coordinate in the window;

[0104] Feature a3: the average of the absolute value of the difference between the y coordinates of the previous point and the next point in the window;

[0105] Feature a4: the average of the absolute value of the difference between the y coordinates of all points and the average of the start point y coordinate and the end point y coordinate;

[0106] Feature a5: the sample number of the end point minus the sample number of the start point divided by the frame rate of the video;

[0107] Feature a6: calculate the correlation coefficient between the jump feature point and the knee key point of the half body in which the jump feature point is located;

[0108] The results of features a1 to a4 also need to be divided by a scale factor L for dynamic scaling of y-axis data. When the proportion of the human body in the image is small, the values of features a1 to a4 are also small, and the scale factor L is also small. When the proportion of the human body in the image is large, the values of features a1 to a4 are large, and the scale factor L is also large. Dividing the features by the scale factor L can reduce the difficulty of identifying the features caused by the large difference in the proportion of the human body in the image, and the threshold setting of each feature is also relatively simple, which improves the accuracy of the algorithm recognition. The value range of the correlation coefficient is [-1, 1], a positive number indicates a positive correlation, a negative number indicates a negative correlation, the closer to 1 indicates a higher (positive) correlation, the closer to -1 indicates a higher (negative) correlation, and 0 indicates no correlation.

[0109] S5: determine whether the current window is a jump according to the data features;

[0110] As a preferred, the method of step S5: determining whether the current window is a jump according to the data features is a feature threshold method or a label training method;

[0111] Feature threshold method: set a corresponding threshold for each feature in the target window, and when one or more features meet the threshold requirement, it is identified as a jump action;

[0112] Label training method: collect a large amount of jump rope data, find the target window and extract the recorded features, label each window as whether it is a jump, train a machine learning model using the features and labels, obtain a prediction model, and use the prediction model to identify whether the target window provided later is a jump. The label training method can use models such as support vector machines, decision trees, random forests, XGBoost, etc.

[0113] S6: identify the trick action of the ankle and the wrist according to the ankle features and the wrist features, respectively.

[0114] As preferred, according to ankle features, the following dance moves can be identified: jumping jacks, crisscrossing, or jumping jacks. The specific method is as follows:

[0115] Jumping jacks: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window. When the correlation coefficient is greater than threshold M3 (for example, 0.7), it is a jumping jack. If it cannot be identified as other types, it is a jumping jack by default. As shown in Figure 2 , it is a graph of the waveform of the y coordinates of the left and right ankles when the target window is a jumping jack.

[0116] Crisscrossing: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window. When the correlation coefficient is less than threshold M4 (for example, -0.8), it is a crisscrossing jump. As shown in Figure 3 , it is a graph of the waveform of the y coordinates of the left and right ankles when the target window is a crisscrossing jump.

[0117] Jumping jacks: the average visibility of the left and right ankles and the left and right shoulders is greater than 0.9. Calculate the correlation coefficient of the x coordinates of the left and right ankles in the window. When the correlation coefficient is less than threshold M5 (for example, -0.8), and the "x coordinate maximum minus minimum" of the left and right ankles in the window is greater than one-third of the "x coordinate difference absolute value" of the left and right shoulders, it is a jumping jack. As shown in Figure 4 , it is a graph of the waveform of the x coordinates of the left and right ankles when the target window is a jumping jack.

[0118] As preferred, step S6: according to wrist features, the following dance moves can be identified: single shake, double shake, multiple shake, single cross, double cross, or multiple cross.

[0119] The specific method for identifying dance moves is as follows:

[0120] S61: Select the effective hand; when the average visibility of the left wrist is greater than 0.8 and the average visibility of the right wrist is less than 0.8, select the left wrist; when the average visibility of the left wrist is less than 0.8 and the average visibility of the right wrist is greater than 0.8, select the right wrist; when the average visibility of the left and right wrists is greater than 0.8, select the hand with more shakes; if both are less than 0.8, do not identify the number of shakes;

[0121] As shown in Figure 5 , the average visibility of the left hand is greater than 0.8, and the average visibility of the right hand is less than 0.8, so the left hand is selected as the effective hand.

[0122] S62: In the target window, at any time, when the y coordinate of the effective hand is less than the average value of the y coordinates of the shoulder and hip of the half body where the effective hand is located, do not identify the number of shakes. That is, when the hand is higher than the midpoint of the shoulder and hip, the user is not doing a normal straight shake, and may be doing a cross.

[0123] S63: Select the x-coordinate of the wrist of the effective hand, and calculate the number of peak x-coordinates within the window; this will tell you how many rotations were performed.

[0124] by Figure 6 Taking the target window as an example, the bottom sub-image in the figure shows the waveforms of the left and right hand x-coordinates, where the left hand is the effective hand, and the waveform of the left hand has a higher signal-to-noise ratio than that of the right hand.

[0125] The windows framed in the image, from left to right, represent single rocking, double rocking, and triple rocking. It can be seen that single rocking has one peak, double rocking has two peaks, and triple rocking has three peaks.

[0126] Although it is clear that the number of rope swings and peaks are the same, sometimes the peaks are not so obvious and further data processing is required.

[0127] S64: Perform differential filtering on the effective x-coordinates, and then perform a moving average of the four points; for example... Figure 7 The image shows the waveform after differential filtering of the effective x-coordinate. The peaks and valleys after differential filtering correspond to the inflection points of the original signal. By calculating the number of inflection points, the number of peaks or valleys in the original signal can be more easily obtained.

[0128] S65: After statistical difference and smoothing, the number of peaks and valleys within the window, the larger number is the shake number. If the number of peaks is 3 and the number of valleys is 2, the shake number is 3.

[0129] S66: Define a shaking number of 1 as a single shaking, a shaking number of 2 as a double shaking, and a shaking number of 3 or more as a multi-shaking fancy move.

[0130] As a preferred embodiment, the specific method for recognizing fancy movements based on wrist characteristics also includes a step of recognizing cross movements, as detailed below:

[0131] S67: If the average visibility of both the left and right wrists is greater than 0.8, then perform cross-action recognition, including the following steps:

[0132] S671: Calculate threshold M6: the average of the absolute values ​​of the left shoulder x-coordinate minus the right shoulder x-coordinate;

[0133] S672: Calculate threshold M7: the average of the absolute values ​​of the left shoulder x-coordinate minus the right shoulder x-coordinate, then divide by 5;

[0134] S673: If the maximum value minus the minimum value of the x-coordinate of the left and right wrists within the target window is greater than the threshold M6, and the correlation coefficient of the x-coordinates of the left and right wrists is less than the threshold M8 (e.g., -0.5), then it is identified as a cross action.

[0135] S674: Count the cross action of each jump, take the left shoulder x coordinate as the baseline of the left wrist, and the right shoulder x coordinate as the baseline of the right wrist; the following judgments are made on the data in the target window in the order of the sampling time:

[0136] When the result of the x coordinate of the wrist minus the baseline changes from less than the threshold M7 to greater than the threshold M7, the cross count is incremented by 1; when the result of the x coordinate of the wrist minus the baseline changes from greater than the threshold M7 to less than the threshold M7, the cross count is incremented by 1; the cross counts calculated by the left and right hands are mostly the same, if there is a difference, take the larger one as the cross count of this jump;

[0137] S675: Identify single cross when the cross count is 1, identify double cross when the cross count is 2, and identify multi-cross when the cross count is greater than or equal to 3.

[0138] As shown in Figure 8 and Figure 9 , it is a waveform display of double-hand crossing. Among them, Figure 8 is the waveform diagram of the target window when jumping once and crossing once; Figure 9 is the waveform diagram of the target window when jumping once and crossing twice.

[0139] If the key point recognition model can provide finger key point monitoring, the finger key point and the wrist key point can be averaged and used as the wrist key point. Because not all models can provide finger key points, providing finger key points can improve the recognition accuracy of hand tricks.

[0140] In the present application, by dynamically selecting key points, the problem of fixed key points being blocked and unable to complete the skipping counting is solved, by using the scale scaling technology, the problem that the human body occupies too small in the image will affect the skipping counting accuracy is solved, the skipping trick action can be recognized, and the reasonable feature selection improves the accuracy of skipping counting.

[0141] Of course, the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications and replacements are all included in the scope defined by the claims of the present application.

Claims

1. An image recognition based fancy skipping identification algorithm, characterized in that, The fancy skipping recognition algorithm comprises the following steps: S1: the camera collects video data and sends it to the processor, and the processor identifies human key points frame by frame; S2: determine the jumping feature key point, and divide the target window by using the jumping feature; the specific method of determining the jumping feature key point and dividing the target window by using the jumping feature is as follows: S21: define the upper left corner coordinate of the image as (0, 0) and the lower right corner coordinate as (1, 1); when the y coordinate of a point becomes smaller, the point moves upward in the image; when the y coordinate becomes larger, the point moves downward in the image; S22: select the data of the left shoulder and the right shoulder, or the left hip and the right hip, as the jumping feature key points Gleft and Gright, and intercept the y coordinate data of the jumping feature key points Gleft and Gright; S23: the y coordinate data of the jumping feature key points Gleft and Gright are respectively put into the queue after 4-point moving average filtering; in the initial state, Gleft is the default jumping feature key point; when a certain data point is higher than the previous data and the next data, it is determined to be a "peak"; the visibility attribute of all data points between the two consecutive peaks is averaged, and if the visibility average is greater than a threshold M1, it is considered that the target window T is found; If the visibility average is less than the threshold M1, change the default jumping feature key point to Gright, and continue to search for the window; when the visibility of Gright does not meet the threshold requirement, change the jumping feature key point back to Gleft; record the sampling serial number corresponding to the first peak as the starting point of the target window, and the sampling serial number corresponding to the second peak as the end point of the target window T; S3: determine whether the jumping target window is valid according to the visibility of the key points; S4: extract the data features of the target window; the specific method of extracting the data features of the target window is as follows: Calculate the scale factor L; if the jumping feature key point is in the left half body, i.e. Gleft, the scale factor L is the average value of the difference between all left hip y coordinates and left shoulder y coordinates in the window; if the jumping feature key point is in the right half body, i.e. Gright, the scale factor L is the average value of the difference between all right hip y coordinates and right shoulder y coordinates in the window; extract the following features of the jumping feature key point in the window: Feature a1: the difference between the starting point y coordinate and the minimum y coordinate in the window; Feature a2: the difference between the end point y coordinate and the minimum y coordinate in the window; feature a3: the average value of the absolute value of the difference between the y coordinates of the previous point and the next point in the window; feature a4: the average value of the absolute value of the difference between the y coordinates of all points and the reference value, which is the average value of the starting point y coordinate and the end point y coordinate; feature a5: the sampling serial number of the end point minus the sampling serial number of the starting point and then divided by the frame rate of the video; feature a6: calculate the correlation coefficient of the jumping feature point and the knee key point of the half body where it is located; the results of features a1 to a4 also need to be divided by the scale factor L to realize dynamic scaling of the y axis data; S5: determine whether the current window is a jump according to the data features; the method for determining whether the current window is a jump according to the data features is feature threshold method or label training method; S6: recognize the fancy movements of the ankles and wrists according to the ankle features and wrist features, respectively.

2. The image recognition based trick jump rope identification algorithm of claim 1, wherein: In step S1, the processor can perform real-time processing and recognition on the video data collected by the camera, or the processor can perform processing and recognition on the video data saved by the camera; The information of the human body key points is put into a queue for caching, and a human body key point recognition model is selected, which has the following requirements: A1: At least the following key points can be output: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left wrist, right wrist, left ankle and right ankle; A2: The key point information includes x and y coordinates in the image and is normalized; A3: The key point information includes visibility, and the visibility index output is normalized to 0 to 1, 0 indicating invisible, and 1 indicating visible.

3. The image recognition based trick jump rope identification algorithm of claim 2, wherein: Step S3: According to the visibility of the key points, whether the jump target window is valid is judged by the following specific method: Calculate the average value of the visibility indexes of the shoulder, ankle and wrist key points of the left and right sides; When the average values of the visibility of the left shoulder and the right shoulder are both less than the threshold M2, it is judged that the target window T is invalid; When the average values of the visibility of the left ankle and the right ankle are both less than the threshold M2, it is judged that the target window T is invalid; When the average values of the visibility of the left wrist and the right wrist are both less than the threshold M2, it is judged that the target window T is invalid.

4. The image recognition based trick skipping recognition algorithm of claim 3, wherein: In step S5: Characteristic threshold method: set corresponding threshold values for each feature in the target window, and when one or more features meet the threshold requirements, it is identified as a jump action; Label training method: collect a large amount of jump rope data, find the target window and extract the recorded features, label each window with whether it is a jump or not, train a machine learning model using the features and labels, obtain a prediction model, and use the prediction model to identify whether the subsequent provided target window is a jump or not.

5. The image recognition based trick skipping recognition algorithm of claim 4, wherein: Step S6: According to the ankle features and wrist features, respectively, in the fancy action of the ankle and wrist, according to the ankle features, the following fancy actions can be identified: jump, left and right foot crossing or opening and closing jump, and the specific method is as follows: Jumping in parallel: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window, and when the correlation coefficient is greater than the threshold M3, it is a jump in parallel; If it cannot be identified as other types, it is defaulted as a jump in parallel; Left and right foot crossing: calculate the correlation coefficient of the y coordinates of the left and right ankles in the target window, and when the correlation coefficient is less than the threshold M4, it is a left and right crossing jump; Opening and closing jump: when the average visibility of the left and right ankles and the left and right shoulders is greater than 0.9, calculate the correlation coefficient of the x coordinates of the left and right ankles in the window, and when the correlation coefficient is less than the threshold M5, and the "x coordinate maximum value minus the minimum value" of the left ankle and the right ankle in the window is greater than one-third of the "x coordinate difference absolute value" of the left shoulder and the right shoulder, it is an opening and closing jump.

6. The image recognition based fancy skipping recognition algorithm of claim 4, wherein: Step S6: According to the ankle features and wrist features, In the fancy action of the ankle and wrist, according to the wrist features, the following fancy actions can be identified: single swing, double swing, multiple swing, single crossing, double crossing or multiple crossing.

7. The image recognition based trick skipping recognition algorithm of claim 6, wherein: According to the wrist features, the specific method for identifying fancy actions is as follows: S61: Select the effective hand; when the average visibility of the left wrist is greater than 0.8 and the average visibility of the right wrist is less than 0.8, select the left wrist; when the average visibility of the left wrist is less than 0.8 and the average visibility of the right wrist is greater than 0.8, select the right wrist; when the average visibility of the left and right wrists are both greater than 0.8, select the hand with more calculated shakes; when both the left and right hands are less than 0.8, do not perform shake recognition; S62: In the target window, at any time, if the y coordinate of the effective hand is less than the average of the y coordinates of the shoulder and hip of the half body in which the effective hand is located, do not perform shake recognition; S63: Select the x coordinate of the wrist of the effective hand and calculate the number of x coordinate peaks in the window; S64: Perform difference filtering on the x coordinate of the effective hand, and then perform 4-point sliding average; S65: Count the number of peaks and valleys in the window after difference and smoothing, and the larger number is the shake; S66: Define shake 1 as single shake, shake 2 as double shake, and shake greater than or equal to 3 as multi-shake trick action.

8. The image recognition based trick jump rope identification algorithm of claim 7, wherein: According to the wrist characteristics, the specific method of identifying trick actions further includes the step of identifying crossing actions, which is as follows: S67: If the average visibility of the left wrist and the right wrist is greater than 0.8, perform crossing action recognition, including the following steps: S671: Calculate threshold M6: the average of the absolute value of the difference between the x coordinates of the left shoulder and the right shoulder; S672: Calculate threshold M7: the average of the absolute value of the difference between the x coordinates of the left shoulder and the right shoulder, divided by 5; S673: If the maximum value minus the minimum value of the x coordinates of the left wrist and the right wrist in the target window is greater than threshold M6, and the correlation coefficient of the x coordinates of the left wrist and the right wrist is less than threshold M8, identify it as a crossing action; S674: Count the crossing action for each jump, taking the x coordinate of the left shoulder as the baseline for the left wrist and the x coordinate of the right shoulder as the baseline for the right wrist; in the order of the sampling time, the data in the target window is judged as follows: When the result of the x coordinate of the wrist minus the baseline changes from less than threshold M7 to greater than threshold M7, the crossing count is incremented by 1; when the result of the x coordinate of the wrist minus the baseline changes from greater than threshold M7 to less than threshold M7, the crossing count is incremented by 1; S675: Crossing count of 1 is identified as single crossing, crossing count of 2 is identified as double crossing, and crossing count greater than or equal to 3 is identified as multi-crossing trick action.

Images