Mixmlp model-based jump rope counting method and apparatus, electronic device, and storage medium

Abstract

The present application relates to a MixMLP model-based jump rope counting method and apparatus, an electronic device, and a storage medium. The method comprises: an image frame label value acquisition step: by processing N consecutive image frames at a time, processing image frames in a jump rope video stream using a MixMLP model, wherein the MixMLP model assigns, on the basis of different positions or states of the human body in space with respect to temporal information, an initial label value A to each image frame corresponding to one category and an initial label value B to each image frame corresponding to the other category; and a jump rope counting step: merging consecutive initial label values A in the jump rope video stream into a single final label value C, and counting the number of occurrences of the final label value C in the jump rope video stream, or counting the number of times the final label value C transitions to an initial label value B, or the number of times an initial label value B transitions to the final label value C, thereby obtaining a jump rope count.

Description

Skip counting method and device based on MixMLP model, electronic equipment and storage medium

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese patent application 202410619619.4, filed on May 17, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of motion detection technology, in particular to a skip counting method and device based on a MixMLP model, an electronic equipment and a storage medium. BACKGROUND

[0004] In the current common skip counting technology scheme based on video stream and artificial intelligence, the skip counting module is usually divided into two categories. One of the categories uses the spatial position information of the target in a single frame image, such as using the positional relationship between the human body and the skipping rope to determine whether a skip has occurred. This method only uses single frame image information to determine the state, and when the skipper's speed is fast, the light in the skipping area is complex, and the camera imaging is poor, etc., the skip counting is prone to obvious errors. Another type of skip counting module uses human body key point information in multiple frames of images, but this method mostly defines some rules to count the trend of human body key points, which relies heavily on human experience, and lacks the use of spatial position information of key points. The accuracy of skip state determination is not high enough, and skip counting is prone to errors. SUMMARY

[0005] Based on the above status, the present application provides a skip counting method and device based on a MixMLP model, which can significantly improve the accuracy of skipper skip state determination and complete high-precision skip counting.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a skip counting method based on a MixMLP model, comprising the following steps:

[0008] Frame image label value acquisition step: according to the way of processing continuous N frames of images each time, the MixMLP model is used to process each frame image in the skip video stream, to obtain the label value corresponding to each frame image which can be used for model counting determination, wherein the MixMLP model divides each frame in the image into two categories according to the different positions or states of the human body in the space in the time sequence information, and assigns an initial label value A to each frame image corresponding to one of the categories, and assigns an initial label value B to each frame image corresponding to the other category;

[0009] The skipping rope count statistics step: merging the continuous initial label value A in the skipping rope video stream into a final label value C, counting the number of times the final label value C appears in the skipping rope video stream, or the number of times the final label value C changes to the initial label value B, or the number of times the initial label value B changes to the final label value C, which is the skipping rope count corresponding to the video stream.

[0010] Optionally, in the skipping rope count statistics step, the continuous initial label value A in the skipping rope video stream is merged into a final label value C, and the continuous initial label value B is merged into a final label value D, and the number of times the final label value C appears in the skipping rope video stream, or the number of times the final label value C changes to the final label value D, or the number of times the final label value D changes to the final label value C, which is the skipping rope count corresponding to the video stream.

[0011] Optionally, in the frame image label value obtaining step, each frame image corresponding label value is obtained by the following sub-step:

[0012] The key point obtaining sub-step: taking the continuous N frame images as the input of the human body detection model to obtain N human body target boxes corresponding thereto, taking the N human body target regions corresponding to the N human body target boxes as the input of the human body pose estimation model in sequence to obtain the key points of M different parts of the human body corresponding to each human body target region;

[0013] The label value obtaining sub-step: taking the key points corresponding to the N human body target regions as the input of the MixMLP model to obtain the label values corresponding to the N frame images respectively; wherein, the MixMLP model divides each frame in the image into two categories according to the different positions or states of the human body in the space on the time sequence information, and assigns the initial label value A to each frame image corresponding to one of the two categories, and assigns the initial label value B to each frame image corresponding to the other of the two categories.

[0014] Optionally, in the label value obtaining sub-step, the MixMLP model assigns the initial label value A to the corresponding frame image of the human body in the highest position in the space on the time sequence information, and the two frame images adjacent to it before and after, and assigns the initial label value B to the remaining frame images.

[0015] Optionally, in the label value obtaining sub-step, the MixMLP model assigns the initial label value A to all corresponding frame images of the human body in the rising state in the space on the time sequence information; and assigns the initial label value B to all corresponding frame images of the human body in the falling state in the space on the time sequence information.

[0016] Optionally, in the label value obtaining sub-step, the MixMLP model assigns an initial label value A to the corresponding frame image of the human body at the spatial landing point position on the time sequence information and the two frame images adjacent before and after it, and assigns an initial label value B to the remaining frame images.

[0017] Optionally, the MixMLP model comprises a matrix transposition module, a feature enhancement module, a transformation module, a time sequence MLP module, a spatial MLP module, a preliminary fusion feature transposition module and an output module, the order of use of the time sequence MLP module and the spatial MLP module in the MixMLP model is determined in advance before use, and the label value obtaining sub-step comprises:

[0018] According to the key point matrix constructed according to the key points of the N human body target regions, the transposed matrix features are obtained by inputting the matrix transposition module;

[0019] The enhanced human body key points are obtained by inputting the transposed matrix features into the feature enhancement module;

[0020] The enhanced human body key points are input into the transformation module, and the transformation module transforms the enhanced human body key points into a feature sequence that can be processed by the first use module in the time sequence MLP module and the spatial MLP module according to the first use module;

[0021] The feature sequence is input into the first use module to obtain preliminary fusion features, wherein when the first use module is the time sequence MLP module, the preliminary fusion features are the preliminary fusion features of the time information between different frame images, and when the first use module is the spatial MLP module, the preliminary fusion features are the preliminary fusion features of the spatial information between different frame images;

[0022] The preliminary fusion features are input into the preliminary fusion feature transposition module to obtain transposed features of the preliminary fusion features;

[0023] The transposed features of the preliminary fusion features are input into the second use module in the time sequence MLP module and the spatial MLP module to obtain spatio-temporal fusion features of the time information and the spatial information between different frame images;

[0024] The spatio-temporal fusion features are input into the output module, the output module first determines the label probability corresponding to each of the N frame images according to the spatio-temporal fusion features, and then analyzes the label probability corresponding to each frame image, if the label probability is greater than a preset value, the initial label value of the corresponding frame image is assigned as A; otherwise, the initial label value of the frame image is assigned as B.

[0025] Optionally, when there is a probability that a single frame image corresponds to multiple labels, the output module averages all label probabilities corresponding to the frame image to obtain a real label probability of the frame, if the real label probability is greater than the preset value, the initial label value of the frame image is assigned as A; if the real label probability is less than the preset value, the initial label value of the frame image is assigned as B.

[0026] In a second aspect, the application provides a rope counting device based on a MixMLP model, the device comprising the following modules:

[0027] A frame image label value acquisition module is configured to process each frame image in a rope video stream using a MixMLP model in a manner of processing N consecutive frame images each time to obtain a label value corresponding to each frame image that can be used for model counting judgment, wherein the MixMLP model divides each frame in the image into two categories according to different positions or states of a human body in space on time sequence information, and assigns an initial label value A to each frame image corresponding to one of the two categories and an initial label value B to each frame image corresponding to the other of the two categories.

[0028] A rope count statistics module is configured to combine consecutive initial label values A in a rope video stream into a final label value C, and count the number of occurrences of the final label value C in the rope video stream, or the number of times of changing from the final label value C to the initial label value B, or the number of times of changing from the initial label value B to the final label value C, which is the rope count corresponding to the video stream.

[0029] Optionally, the rope count statistics module is configured to combine consecutive initial label values A in a rope video stream into a final label value C, combine consecutive initial label values B into a final label value D, and count the number of occurrences of the final label value C in the rope video stream, or the number of times of changing from the final label value C to the final label value D, or the number of times of changing from the final label value D to the final label value C, which is the rope count corresponding to the video stream.

[0030] Optionally, the frame image label acquisition module comprises the following sub-modules:

[0031] A key point acquisition sub-module is configured to take the N consecutive frame images as input of a human body detection model to obtain N human body target boxes corresponding thereto, take the N human body target boxes as input of a human body pose estimation model in sequence to obtain key points of M different parts of the human body corresponding to each human body target box.

[0032] The label value acquisition submodule is configured to take the key points corresponding to the N human body target areas as inputs of a MixMLP model to obtain label values corresponding to the N frames of images respectively; wherein the MixMLP model divides each frame of the images into two categories according to different positions or states of the human body in space in time sequence information, and assigns an initial label value A to each frame of image corresponding to one of the two categories and an initial label value B to each frame of image corresponding to the other of the two categories.

[0033] Optionally, the MixMLP model assigns the initial label value A to the frame of image corresponding to the highest position of the human body in space in time sequence information and two frames of images adjacent to the frame of image before and after the frame of image, and assigns the initial label value B to the remaining frames of images.

[0034] Optionally, the MixMLP model assigns the initial label value A to all frames of images corresponding to the rising state of the human body in space in time sequence information, and assigns the initial label value B to all frames of images corresponding to the falling state of the human body in space in time sequence information.

[0035] Optionally, the MixMLP model assigns the initial label value A to the frame of image corresponding to the landing position of the human body in space in time sequence information and two frames of images adjacent to the frame of image before and after the frame of image, and assigns the initial label value B to the remaining frames of images.

[0036] In a third aspect, the present application provides an electronic device, comprising:

[0037] a processor; and

[0038] a memory, wherein the memory has stored thereon a computer program, and the computer program is capable of implementing the MixMLP model-based skipping counting method when executed by the processor.

[0039] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program, and the computer program is used to run to implement the MixMLP model-based skipping counting method.

[0040] The present application proposes a brand-new skipping counting modeling method, and for the first time, proposes a MixMLP model structure in a skipping counting module, so that each frame of image is divided into two categories according to different positions or states of the human body in space in time sequence information, and is respectively assigned an initial label value A and an initial label value B, and then after merging consecutive initial label values A into a final value label value C, the skipping count corresponding to the video stream can be quickly obtained by counting the number of occurrences of the final value label value C of the skipping video stream. The skipping state judgment accuracy of the skipping player is obviously improved, the judgment accuracy of the skipping state of the player is significantly improved, and the accuracy and fairness of the skipping counting result are ensured.

[0041] Other beneficial effects of the present application will be illustrated in the specific embodiments by the introduction of specific technical features and technical solutions, which should be understood by those skilled in the art through the introduction of the technical features and technical solutions. BRIEF DESCRIPTION OF DRAWINGS

[0042] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. In the drawings:

[0043] FIG. 1 is a flowchart of an embodiment of a rope counting method based on a MixMLP model provided by the present application;

[0044] FIG. 2 is a schematic diagram of merging the continuous initial label value A and the initial label value B in an embodiment provided by the present application;

[0045] FIG. 3 is a schematic diagram of a matrix of M*2 provided by the present application;

[0046] FIG. 4 is a schematic diagram of the structure of an embodiment of a rope counting device based on a MixMLP model provided by the present application;

[0047] FIG. 5 is a schematic diagram of the model structure of an embodiment of a MixMLP model provided by the present application;

[0048] FIG. 6 is a schematic diagram of the model structure of another embodiment of a MixMLP model provided by the present application. DETAILED DESCRIPTION

[0049] The present application is described below based on examples, but the present application is not limited to only these examples. In the following detailed description of the present application, some specific details are described in detail in order to avoid obscuring the essence of the present application, and well-known methods, processes, procedures, elements are not described in detail.

[0050] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0051] Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to".

[0052] In the description of the present application, it should be understood that the terms "first", "second", etc. are only for the purpose of description and should not be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "multiple" is two or more.

[0053] Referring to FIG. 1, the application provides a skipping counting method based on a MixMLP model, comprising the following steps:

[0054] A frame image label value obtaining step: each frame image in the skipping video stream is processed by the MixMLP model in a manner of processing continuous N frame images each time, to obtain a label value corresponding to each frame image that can be used for model counting judgment, wherein the MixMLP model divides each frame in the image into two categories according to different positions or states of the human body in space on the time sequence information, and assigns an initial label value A to each frame image corresponding to one of the categories, and assigns an initial label value B to each frame image corresponding to the other category.

[0055] A skipping count step: the continuous initial label value A in the skipping video stream is combined into a final value label value C, and the number of occurrences of the final value label value C in the skipping video stream, or the number of times of changing from the final value label value C to the initial label value B, or the number of times of changing from the initial label value B to the final value label value C, is counted, which is the skipping count corresponding to the video stream.

[0056] Specifically, first, in the frame image label value obtaining step, the implementation of processing continuous N frame images each time can be to process the first frame to the Nth frame first, and then process the second frame to the N+1th frame, and then process the third frame to the N+2th frame, and so on, until all the images of the video stream are processed. This processing method often leads to the situation of processing the same frame multiple times, for example, in the case of N being 10, the first 1-10 frames are processed continuously the first time, the second 2-11 frames are processed continuously the second time, and the third 3-12 frames are processed continuously the third time. Obviously, only in the first three processing times, the second frame to the eleventh frame are processed at least once. This not only prevents the occurrence of frame loss, but also significantly improves the accuracy of skipping count statistics.

[0057] Because the MixMLP model can learn the spatial position information between different key points of the human body and the time sequence information between continuous frame images, based on the model, the different positions of the rope skipping player in space in the time sequence information can be accurately determined, and the fusion of the time sequence information and the spatial information is sufficient to accurately distinguish whether the rope skipping player is in the upward jumping state (rising state) or in the downward landing state (falling state) in the dynamic change process displayed by all frame images of the video stream, based on these state judgments, it is also easy to determine whether the rope skipping player is located at the highest point in space or at the landing point in space, and thus the entire video stream can be divided into two categories according to the different positions or states of the rope skipping player in space, and the initial label value A is assigned to the frame images corresponding to one category, and the initial label value B is assigned to the frame images corresponding to the other category, for example, the frame images corresponding to the rope skipping player being at the highest point in space (or the adjacent frame images thereof) are assigned the initial label value A, and all other frame images are assigned the initial label value B; or the frame images corresponding to the rope skipping player being in the falling state are all assigned the initial label value A, and all other frame images are assigned the initial label value B, and the like. It should be noted that the object of the two-category assignment of the present application is all frame images in the video stream, rather than being able to be divided into two categories for N frame images each time, which can be simply illustrated by an example: if the frame images corresponding to the rope skipping player being at the highest point in space are selected to be assigned the initial label value A, at this time, there may be no frame images in which the rope skipping player is at the highest point in space in the continuous N frame images processed at a certain time or even at certain times (for example, the players are all in the falling process in the N frame images), and therefore the label value B can be assigned to the N frame images.

[0058] The assignment method of the present application can make all the obtained initial label values A in each rope skipping process (referring to a process from jumping, to the highest point in space, and to the landing point) of the rope skipping player in the video stream continuous, and therefore, by merging the continuous initial label values A in the rope skipping video stream into one final label value C in the rope skipping times counting step, the number of times of the appearance of the label value C in the rope skipping video stream can be counted, and the total number of times of rope skipping of the rope skipping player in the video stream can be obtained. As can be understood by those skilled in the art, the values of A and C are pre-set, and the values of A and C can be equal or not equal, for example, in a specific implementation, the value of A can be pre-selected as 1, and the value of C can also be pre-selected as 1, at which time the initial label value 1 in the rope skipping video stream can be merged into one final label value 1 in the rope skipping times counting step; if the value of A is pre-selected as 1 and the value of C is pre-selected as 2, at which time the initial label value 1 in the rope skipping video stream can be merged into one final label value 2 in the rope skipping times counting step.

[0059] Of course, the number of transitions between the final label value C and the initial label value B can also be counted, i.e., without limiting whether it is from C to B or from B to C, but if it is selected from C to B, the counting is all started with C, and only the number of switching transitions from C to B is counted; if it is selected from B to C, it is all started with B, and only the number of switching transitions from B to C is counted as the skipping rope count corresponding to the video stream. In fact, this counting method is essentially the same as counting the number of times the C value appears as described in the previous paragraph, because only once the C value appears, a switching between the final label value C and the initial label value B will occur.

[0060] The present application proposes a brand-new skipping rope count modeling method, and for the first time proposes a MixMLP model structure in the skipping rope count module, thereby dividing each frame in the image into two categories according to different positions or states of the human body in space on the time sequence information, and assigning initial label values A and B respectively, and then merging consecutive initial label values A into a final label value C in a simple counting fusion manner, so that the skipping rope count corresponding to the video stream can be quickly obtained by counting the number of times the final label value C appears. The skipping rope state judgment accuracy of the present application for skipping rope athletes is obviously improved, and is suitable for various situations, whether the skipping rope athletes are fast or slow, whether the light of the skipping rope site is complex, whether the camera imaging effect is relatively poor or high, and can also avoid introducing false counts by non-skip rope actions such as rope swinging, and can obtain a high-precision skipping rope count result, ensuring the accuracy and fairness of the skipping rope count.

[0061] Optionally, in the skipping rope count step, consecutive initial label values A in the skipping rope video stream are merged into a final label value C, consecutive initial label values B are merged into a final label value D, and the number of times the final label value C appears in the skipping rope video stream, or the number of times the final label value C changes to the final label value D, or the number of times the final label value D changes to the final label value C is counted, i.e., the skipping rope count corresponding to the video stream

[0062] Specifically, in order to facilitate counting, consecutive initial label values B in the skipping rope video stream can also be merged into a final label value D, which further shortens the length of the consecutive label value sequence and makes the skipping rope count more convenient. Similarly, the values of B and D are pre-set, and the values of B and D can be equal or not equal.

[0063] In one specific embodiment, the initial label value A and the final label value C can be taken as 1, and the initial label value B and the final label value D can be taken as 0. An example of merging consecutive identical label values is shown in FIG. 2. In the case shown in the figure, it is clear that the number of jumps of the jumper is two, because in the binary sequence obtained after merging the label values, the 1 value appears twice, and the switching between 1 and 0 also occurs only twice (whether switching from 1 to 0 or from 0 to 1, each only twice).

[0064] Optionally, in the frame image label value obtaining step, each consecutive N frame image is obtained by the following sub-steps to obtain the label value corresponding to each frame image:

[0065] Key point obtaining sub-step: taking the consecutive N frame images as the input of the human body detection model to obtain N human body target boxes corresponding thereto, and taking the N human body target regions obtained according to the N human body target boxes as the input of the human body pose estimation model in sequence to obtain the key points of M different parts of the human body corresponding to each human body target region;

[0066] Label value obtaining sub-step: taking the key points corresponding to the N human body target regions as the input of the MixMLP model to obtain the label value corresponding to each of the N frame images; wherein the MixMLP model divides each frame in the image into two categories according to the different positions or states of the human body in the space on the time sequence information, and assigns an initial label value A to each frame image corresponding to one of the two categories, and assigns an initial label value B to each frame image corresponding to the other of the two categories.

[0067] In the key point obtaining sub-step, the video stream is first initialized, and the consecutive N frame images of the jump rope video stream are obtained. The consecutive N frame images are input into the human body detection model in sequence to obtain N human body target boxes corresponding to the consecutive N frame images in one-to-one correspondence. Then, the N human body target regions corresponding to the N human body target boxes in one-to-one correspondence are obtained (i.e., a human body target region is cut out on each frame image), and the N human body target regions are input into the human body pose estimation model in sequence to obtain N sets of human body key point coordinates corresponding to the N human body target regions, wherein each set of human body key points contains M key points, and the M key points correspond to different human body parts, respectively.

[0068] Considering the balance between the jump rope counting accuracy and the computing strength of the computing device, in one specific embodiment, N can be taken as 10, and M can be taken as 17, wherein the 17 different human body parts correspond to the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle of the human body of the jump rope player, respectively.

[0069] Optionally, in the sub-step of label value acquisition, the MixMLP model assigns an initial label value A to the corresponding frame image of the human body in the spatial highest point position on the time sequence information and the two adjacent frame images before and after it, and assigns an initial label value B to the remaining frame images.

[0070] The present application finds that in the application of the MixMLP model, the distribution of the initial label value A and the initial label value B does not need to be completely average, but it is better if it is relatively balanced. Therefore, by the above assignment method, the case of too many initial label values B and too few initial label values A can be avoided, and the case of very uneven distribution can be avoided, and because these A-value images are also continuous, the accuracy of the skipping rope count will not be affected.

[0071] Optionally, in the sub-step of label value acquisition, the MixMLP model assigns an initial label value A to all corresponding frame images of the human body in the rising state in the space on the time sequence information; and assigns an initial label value B to all corresponding frame images of the human body in the falling state in the space on the time sequence information.

[0072] It has been mentioned above that the MixMLP model can fuse time sequence information and spatial information, so it can be judged whether the human body state of the skipping rope player is in the rising state or the falling state in the space (the two states cannot be distinguished without the participation of time sequence information), so all corresponding frame images of the human body in the rising state can be assigned an initial label value A, and all corresponding frame images of the human body in the falling state can be assigned an initial label value B.

[0073] Because the rising and falling states are usually compared with the previous frame image, while the spatial highest point is also the starting point of the human body in the falling state, but the corresponding frame of the spatial highest point is also further rising compared with the previous frame image before this frame, so the corresponding frame image of the spatial highest point can also be assigned a label value A.

[0074] Optionally, in the sub-step of label value acquisition, the MixMLP model assigns an initial label value A to the corresponding frame image of the human body in the spatial highest point position on the time sequence information and the two adjacent frame images before and after it, and assigns an initial label value B to the remaining frame images.

[0075] By this assignment method, the case of too many initial label values B and too few initial label values A can also be avoided, and the case of very uneven distribution can be avoided, which is beneficial to the accurate statistics of the skipping rope count.

[0076] Optionally, referring to FIGS. 5-6, the MixMLP model comprises a matrix transposition module, a feature enhancement module, a transformation module, a time sequence MLP module, a spatial MLP module, a preliminary fusion feature transposition module, and an output module. The order of use of the time sequence MLP module and the spatial MLP module in the MixMLP model is determined in advance before use. The label value obtaining sub-step comprises:

[0077] According to the key point matrix corresponding to the key points of the N human target regions, the transposed matrix features are obtained by inputting the matrix transposition module;

[0078] The enhanced human key points are obtained by inputting the transposed matrix features into the feature enhancement module;

[0079] The enhanced human key points are input into the transformation module. The transformation module transforms the enhanced human key points into a feature sequence that can be processed by the first used module in the time sequence MLP module and the spatial MLP module according to the pre-determined first used module;

[0080] The feature sequence is input into the first used module to obtain the preliminary fusion features. When the first used module is the time sequence MLP module, the preliminary fusion features are the time information fusion features between different frame images. When the first used module is the spatial MLP module, the preliminary fusion features are the spatial information fusion features between different frame images.

[0081] The preliminary fusion features are input into the preliminary fusion feature transposition module to obtain the transposed features of the preliminary fusion features;

[0082] The transposed features of the preliminary fusion features are input into the second used module in the time sequence MLP module and the spatial MLP module to obtain the spatio-temporal fusion features that fuse the time information and the spatial information between different frame images.

[0083] The spatio-temporal fusion features are input into the output module. The output module first determines the label probability corresponding to each of the N frame images according to the spatio-temporal fusion features, and then analyzes the label probability corresponding to each frame image. If the label probability is greater than a preset value, the initial label value of the corresponding frame image is assigned as A. Otherwise, the initial label value of the frame image is assigned as B.

[0084] Specifically, the MixMLP model uses the pre-determined order of use of the temporal MLP module and the spatial MLP module therein, because the temporal MLP module and the spatial MLP module are not strongly constrained in the order of use (while other modules are usually not the case, for example, the feature enhancement module cannot work normally without the corresponding transposition processing of the data by the matrix transposition module), and for the temporal MLP module and the spatial MLP module, the temporal MLP module can be used first, and then the spatial MLP module can be used, or vice versa, as long as the pre-determination is achieved.

[0085] In the label value acquisition sub-step, first, a key point matrix is constructed according to the key points corresponding to the N human body target regions, and the matrix is an N*M*2 key point matrix, wherein N represents the number of groups of human body key point coordinates (also the number of images processed this time), M represents the number of key points in each group of human body key point coordinates, and 2 represents the corresponding coordinates (x, y) of each key point. Referring to FIG. 3, an M*2 matrix diagram is shown (the total number of rows is M), and N*M*2 indicates that the M*2 matrix has a total of N.

[0086] After the N*M*2 key point matrix is subjected to the normalization preprocessing operation, the MixMLP model is inputted to obtain the label value corresponding to each frame of image, and the specific operation mode can be:

[0087] The key point matrix after the normalization preprocessing operation is inputted into the matrix transposition module to obtain the matrix features after transposition;

[0088] The matrix features after transposition are inputted into the feature enhancement module to obtain the enhanced human body key points;

[0089] After the above step is completed, the enhanced human body key points are automatically transmitted to the transformation module in the model (i.e., the enhanced human body key points are inputted into the transformation module), and the transformation module transforms the enhanced human body key points into a feature sequence that can be processed by the first use module in the pre-determined temporal MLP module and spatial MLP module:

[0090] Specifically, referring to FIG. 5, if it is pre-determined to use the temporal MLP module first, the first use module is the temporal MLP module at this time, and the next step is to input the data features processed by the transformation module into the temporal MLP module to obtain the preliminary fusion features of the time information between different frames of images, at this time, the transformation module performs a Reshape deformation operation on the enhanced human body key points to transform the enhanced human body key points into a one-dimensional sequence, and the one-dimensional sequence is the feature sequence that can be processed by the temporal MLP module;

[0091] If it is determined in advance to use the spatial MLP module first, i.e., the first module to be used is the spatial MLP module, and the next step is to input the data features processed by the transformation module into the spatial MLP module to obtain the preliminary fusion features of fusing the spatial information between different frame images, at this time, the reshape transformation operation is performed on the enhanced human key points by the transformation module to change the enhanced human key points into a one-dimensional sequence, and the transpose operation is performed to transpose the one-dimensional sequence. The sequence after the transposition of the one-dimensional sequence is the feature sequence that can be processed by the spatial MLP module.

[0092] After the above step is completed, the feature sequence that can be processed by the temporal MLP module or the spatial MLP module is automatically transmitted to the temporal MLP module and the spatial MLP module in the model in the predetermined order (because different MLP modules have different requirements for data formats / arrangement modes, the preliminary fusion feature conversion module needs to be involved in the transposition processing between the two MLP modules to change the preliminary fusion feature output from the previous MLP module to be processed by the next MLP module), to obtain the spatio-temporal fusion features of fusing the temporal information and the spatial information between different frames. The spatio-temporal fusion features reflect the different positions or states of the human body in the space in the temporal information.

[0093] After the above step is completed, the spatio-temporal fusion features are automatically transmitted to the output module in the model. The output module first determines the label probability corresponding to each of the N frame images according to the spatio-temporal fusion features. The label probability reflects the probability of the human body being in the corresponding position or state that can be assigned the initial label value A (for example, the probability of being located at the highest point in the space). By setting a preset value, the relationship between the label probability and the preset value is compared. If the label probability is greater than the preset value, the initial label value of the corresponding frame image can be assigned as A. Otherwise, the initial label value of the frame image is assigned as B.

[0094] The preset value can be selected according to the actual situation. In a specific embodiment, it can be selected as 0.5.

[0095] In a specific embodiment, the feature enhancement model includes 2 layers of 2D convolution layers. The temporal MLP module and the spatial MLP module each include 3 layers of MLP layers. The main components of the two modules can be RNN, LSTM, or transformer structure. The function adopted by the output module is the Softmax function.

[0096] The temporal MLP module and the spatial MLP module in the MixMLP model adopted by the present application can effectively extract the temporal and spatial feature information of the human key points in the continuous frames. In combination with the counting modeling method described above, the characteristics of the MixMLP model structure and the counting modeling method are fully utilized to achieve the precise skipping rope counting function.

[0097] Optionally, when there are multiple label probabilities corresponding to a single frame image, the output module averages all the label probabilities corresponding to the frame image to obtain a real label probability of the frame, and if the real label probability is greater than a preset value, the initial label value of the frame image is assigned as A; if the real label probability is less than the preset value, the initial label value of the frame image is assigned as B.

[0098] As introduced above, the same frame image can be processed more than once, and thus there can be multiple label probabilities corresponding to a single frame image. At this time, the average of all the label probabilities corresponding to the frame image can be taken as the real label probability of the frame, which is compared with a preset value to determine how the label value of the frame image should be assigned.

[0099] Referring to FIG. 4, the application further provides a rope counting device based on a MixMLP model, which comprises the following modules:

[0100] A frame image label value acquisition module is configured to process each frame image in a rope video stream using a MixMLP model in a manner of processing consecutive N frame images each time to obtain a label value corresponding to each frame image and available for model counting determination, wherein the MixMLP model divides each frame in the image into two categories according to different positions or states of a human body in space on time sequence information, and assigns an initial label value A to each frame image corresponding to one of the two categories and an initial label value B to each frame image corresponding to the other of the two categories.

[0101] A rope count statistics module is configured to combine consecutive initial label values A in the rope video stream into a final label value C, and count the number of occurrences of the final label value C in the rope video stream, or the number of times of changing from the final label value C to the initial label value B, or the number of times of changing from the initial label value B to the final label value C, which is the rope count corresponding to the video stream.

[0102] Optionally, the rope count statistics module is configured to combine consecutive initial label values A in the rope video stream into a final label value C, combine consecutive initial label values B into a final label value D, and count the number of occurrences of the final label value C in the rope video stream, or the number of times of changing from the final label value C to the final label value D, or the number of times of changing from the final label value D to the final label value C, which is the rope count corresponding to the video stream.

[0103] Optionally, the frame image label acquisition module comprises the following sub-modules:

[0104] The key point acquisition submodule is configured to take the continuous N frames of images as inputs of a human body detection model to obtain N human body target boxes corresponding to the N frames of images, and take the N human body target boxes as inputs of a human body pose estimation model to obtain M key points of different parts of each human body in each human body target box;

[0105] The label value acquisition submodule is configured to take the key points corresponding to the N human body target boxes as inputs of a MixMLP model to obtain label values corresponding to the N frames of images respectively; wherein, the MixMLP model divides each frame of image into two categories according to different positions or states of the human body in space on time sequence information, and assigns an initial label value A to each frame of image corresponding to one category and an initial label value B to each frame of image corresponding to the other category.

[0106] Optionally, the MixMLP model assigns the initial label value A to the frame of image corresponding to the highest position of the human body in space on time sequence information and two adjacent frames of image before and after the frame of image, and assigns the initial label value B to the remaining frames of image.

[0107] Optionally, the MixMLP model assigns the initial label value A to all frames of image corresponding to the rising state of the human body in space on time sequence information, and assigns the initial label value B to all frames of image corresponding to the falling state of the human body in space on time sequence information.

[0108] Optionally, the MixMLP model assigns the initial label value A to the frame of image corresponding to the landing position of the human body in space on time sequence information and two adjacent frames of image before and after the frame of image, and assigns the initial label value B to the remaining frames of image.

[0109] Optionally, the MixMLP model comprises a matrix transposition module, a feature enhancement module, a transformation module, a time sequence MLP module, a space MLP module, a preliminary fusion feature transposition module and an output module, and the order of use of the time sequence MLP module and the space MLP module in the MixMLP model is determined in advance before use, wherein,

[0110] The matrix transposition module is configured to transpose a key point matrix constructed according to the key points corresponding to the N human body target boxes to obtain a transposed matrix feature;

[0111] The feature enhancement module is configured to enhance the transposed matrix feature to obtain an enhanced human body key point;

[0112] The transformation module is configured to transform the enhanced human body key point into a feature sequence processable by a first use module of the time sequence MLP module and the space MLP module according to the first use module;

[0113] The first use module is used for processing the feature sequence to obtain preliminary fusion features, wherein when the first use module is a time MLP module, the preliminary fusion features are fusion features of time information between different frame images, and when the first use module is a space MLP module, the preliminary fusion features are fusion features of space information between different frame images.

[0114] The preliminary fusion feature transposition module is used for transposing the preliminary fusion features to obtain features after transposition of the preliminary fusion features.

[0115] The post-use module in the time MLP module and the space MLP module is used for processing the features after transposition of the preliminary fusion features to obtain spatio-temporal fusion features of time information and space information between different frame images.

[0116] The output module is used for determining label probabilities corresponding to N frame images respectively according to the spatio-temporal fusion features, and then analyzing the label probabilities corresponding to each frame image, if the label probability is greater than a preset value, an initial label value of the corresponding frame image is assigned as A, otherwise, the initial label value of the frame image is assigned as B.

[0117] Optionally, when a single frame image corresponds to multiple label probabilities, the output module is further used for averaging all label probabilities corresponding to the frame image to obtain a real label probability of the frame, if the real label probability is greater than a preset value, an initial label value of the frame image is assigned as A, and if the real label probability is less than a preset value, the initial label value of the frame image is assigned as B.

[0118] The application provides an electronic device, comprising:

[0119] a processor; and

[0120] a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the skip counting method based on the MixMLP model.

[0121] The application provides a computer readable storage medium, which stores a computer program, and the computer program is used to run to implement the skip counting method based on the MixMLP model.

[0122] Those skilled in the art can understand that the above-mentioned preferred schemes can be freely combined and superimposed without conflict.

[0123] It should be understood that the above-mentioned embodiments are only exemplary and not limiting, and those skilled in the art can make various obvious or equivalent modifications or replacements to the above-mentioned details without departing from the basic principles of the application, which are all included in the scope of claims of the application.

Claims

1. A skipping counting method based on a MixMLP model, comprising the following steps: a frame image label value obtaining step: in a manner of processing continuous N frame images each time, each frame image in a skipping video stream is processed by a MixMLP model to obtain a label value corresponding to each frame image which can be used for model counting judgment, wherein the MixMLP model divides each frame in the image into two categories according to different positions or states of a human body in space on time sequence information, and assigns an initial label value A to each frame image corresponding to one of the categories and an initial label value B to each frame image corresponding to the other category; a skipping count statistics step: continuous initial label values A in the skipping video stream are combined into a final value label value C, and the number of occurrences of the final value label value C in the skipping video stream, or the number of times of changing from the final value label value C to the initial label value B, or the number of times of changing from the initial label value B to the final value label value C is counted, which is the skipping count corresponding to the video stream.

2. The method of claim 1, wherein, In the skipping count statistics step, continuous initial label values A in the skipping video stream are combined into a final value label value C, and continuous initial label values B are combined into a final value label value D, and the number of occurrences of the final value label value C in the skipping video stream, or the number of times of changing from the final value label value C to the final value label value D, or the number of times of changing from the final value label value D to the final value label value C is counted, which is the skipping count corresponding to the video stream.

3. The method of claim 1 or 2, wherein, In the frame image label value obtaining step, each continuous N frame image obtains a label value corresponding to each frame image through the following sub-steps: a key point obtaining sub-step: the continuous N frame images are taken as the input of a human body detection model to obtain N human body target boxes, and the N human body target boxes are taken as the input of a human body pose estimation model in sequence to obtain key points of M different parts of the human body corresponding to each human body target region; a label value obtaining sub-step: the key points corresponding to the N human body target regions are taken as the input of a MixMLP model to obtain a label value corresponding to each of the N frame images; wherein the MixMLP model divides each frame in the image into two categories according to different positions or states of a human body in space on time sequence information, and assigns an initial label value A to each frame image corresponding to one of the categories and an initial label value B to each frame image corresponding to the other category.

4. The method of claim 3, wherein, In the label value obtaining sub-step, the MixMLP model assigns an initial label value A to the frame image corresponding to the highest point position of the human body on time sequence information and the two frame images adjacent to it before and after, and assigns an initial label value B to the remaining frame images.

5. The method of claim 3, wherein, In the label value obtaining sub-step, the MixMLP model assigns an initial label value A to all frame images corresponding to the rising state of the human body in space on time sequence information; and assigns an initial label value B to all frame images corresponding to the falling state of the human body in space on time sequence information.

6. The method of claim 3, wherein, In the label value obtaining sub-step, the MixMLP model assigns an initial label value A to the corresponding frame image of the human body at the spatial landing position on the time sequence information and the two frame images adjacent before and after it, and assigns an initial label value B to the remaining frame images.

7. The method of claim 3, wherein, The MixMLP model includes a matrix transposition module, a feature enhancement module, a transformation module, a time sequence MLP module, a spatial MLP module, a preliminary fusion feature transposition module, and an output module. The order of use of the time sequence MLP module and the spatial MLP module in the MixMLP model is determined in advance before use. The label value obtaining sub-step includes: According to the key points corresponding to the target region of N human bodies, a key point matrix is constructed, which is input into the matrix transposition module to obtain transposed matrix features; The transposed matrix features are input into the feature enhancement module to obtain enhanced human key points; The enhanced human key points are input into the transformation module, which transforms the enhanced human key points into a feature sequence that can be processed by the first module of use in the time sequence MLP module and the spatial MLP module according to the pre-determined first module of use; The feature sequence is input into the first module of use to obtain preliminary fusion features. When the first module of use is the time sequence MLP module, the preliminary fusion features are the fusion of time information between different frame images. When the first module of use is the spatial MLP module, the preliminary fusion features are the fusion of spatial information between different frame images. The preliminary fusion features are input into the preliminary fusion feature transposition module to obtain transposed features of the preliminary fusion features; The transposed features of the preliminary fusion features are input into the second module of use in the time sequence MLP module and the spatial MLP module to obtain spatio-temporal fusion features that fuse the time information and the spatial information between different frame images. The spatio-temporal fusion features are input into the output module, which first determines the label probability corresponding to each of the N frame images according to the spatio-temporal fusion features, and then analyzes the label probability corresponding to each frame image. If the label probability is greater than a preset value, the initial label value of the corresponding frame image is A; otherwise, the initial label value of the frame image is B.

8. The method of claim 7, wherein, When there is a single frame image corresponding to multiple label probabilities, the output module averages all the label probabilities corresponding to the frame image to obtain the true label probability of the frame. If the true label probability is greater than the preset value, the initial label value of the frame image is A; if the true label probability is less than the preset value, the initial label value of the frame image is B.

9. A rope skipping counting device based on a MixMLP model, the device comprising the following modules: The frame image label value obtaining module is configured to process each frame image in the skipping rope video stream by using the MixMLP model in a manner of processing N continuous frame images each time, to obtain a label value corresponding to each frame image and available for model counting determination, wherein, The MixMLP model divides each frame in the image into two categories according to different positions or states of the human body in space on the time sequence information, assigns an initial label value A to each frame image corresponding to one of the two categories, and assigns an initial label value B to each frame image corresponding to the other of the two categories. The skipping rope frequency counting module is configured to combine continuous initial label values A in a skipping rope video stream into a final label value C, count the number of occurrences of the final label value C in the skipping rope video stream, or the number of times of transition from the final label value C to an initial label value B, or the number of times of transition from the initial label value B to the final label value C, and the skipping rope count corresponding to the video stream is obtained.

10. The apparatus of claim 9, wherein, The skipping rope frequency counting module is configured to combine continuous initial label values A in a skipping rope video stream into a final label value C, combine continuous initial label values B into a final label value D, count the number of occurrences of the final label value C in the skipping rope video stream, or the number of times of transition from the final label value C to the final label value D, or the number of times of transition from the final label value D to the final label value C, and the skipping rope count corresponding to the video stream is obtained.

11. The apparatus of claim 9 or 10, wherein, The frame image label obtaining module comprises the following sub-modules: The key point obtaining sub-module is configured to take the continuous N frame images as the input of a human body detection model to obtain N human body target boxes corresponding to the N frame images, take the N human body target boxes as the input of a human body pose estimation model to obtain key points of different parts of a human body corresponding to each human body target box, and take the N human body target boxes as the input of a MixMLP model to obtain label values corresponding to the N frame images. The label value obtaining sub-module is configured to take the key points corresponding to the N human body target boxes as the input of the MixMLP model to obtain the label values corresponding to the N frame images.

12. The apparatus of claim 11, wherein, The MixMLP model is configured to assign an initial label value A to a frame image corresponding to a highest point of a human body in a space and two frame images adjacent to the frame image in a time sequence, and assign an initial label value B to the remaining frame images.

13. The apparatus of claim 11, wherein, The MixMLP model is configured to assign an initial label value A to all frame images corresponding to a rising state of a human body in a space in a time sequence. The MixMLP model is configured to assign an initial label value B to all frame images corresponding to a falling state of a human body in a space in a time sequence.

14. The apparatus of claim 11, wherein, The MixMLP model is configured to assign an initial label value A to a frame image corresponding to a landing point of a human body in a space and two frame images adjacent to the frame image in a time sequence, and assign an initial label value B to the remaining frame images.

15. The apparatus of claim 11, wherein, The MixMLP model comprises a matrix transposition module, a feature enhancement module, a transformation module, a time sequence MLP module, a space MLP module, a preliminary fusion feature transposition module and an output module. The matrix transposition module is configured to transpose a key point matrix constructed according to the key points corresponding to the N human body target boxes to obtain a transposed matrix feature. The feature enhancement module is configured to enhance the transposed matrix feature to obtain an enhanced human body key point. The transformation module is configured to transform the enhanced human key points into a feature sequence processable by a first-in-use module of the predetermined temporal MLP module and the spatial MLP module; The first-in-use module is configured to process the feature sequence to obtain preliminary fusion features, wherein when the first-in-use module is the temporal MLP module, the preliminary fusion features are fusion features of time information between different frame images, and when the first-in-use module is the spatial MLP module, the preliminary fusion features are fusion features of spatial information between different frame images; The preliminary fusion feature transposition module is configured to transpose the preliminary fusion features to obtain features after transposition of the preliminary fusion features; The second-in-use module of the temporal MLP module and the spatial MLP module is configured to process the features after transposition of the preliminary fusion features to obtain spatio-temporal fusion features of time information and spatial information between different frame images; The output module is configured to determine label probabilities corresponding to N frame images according to the spatio-temporal fusion features, and then analyze the label probabilities corresponding to each frame image, if the label probability is greater than a preset value, an initial label value of the frame image is assigned as A; otherwise, the initial label value of the frame image is assigned as B.

16. The apparatus of claim 15, wherein, When there are multiple label probabilities corresponding to a single frame image, the output module averages all the label probabilities corresponding to the frame image to obtain a real label probability of the frame, if the real label probability is greater than the preset value, the initial label value of the frame image is assigned as A; if the real label probability is less than the preset value, the initial label value of the frame image is assigned as B.

17. An electronic device comprising: a processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the processor, enables the method of counting skipping rope based on the MixMLP model according to any one of claims 1-8.

18. A computer readable storage medium having a computer program stored thereon, wherein the computer program is configured to run to implement the method of counting skipping rope based on the MixMLP model according to any one of claims 1-8. ​

Images