Pull-up evaluation method, device, electronic device and storage medium

Abstract

The present invention provides a pull-up evaluation method, device, electronic device and storage medium, wherein the method comprises: obtaining a video stream to be tested; based on the position information of the skeleton points of a person in the current frame in the video stream to be tested, determining the state of the current frame, wherein the state is any one of a ready state, an upward state and a downward state; when the state of the current frame is an upward state or a downward state, based on the horizontal bar position information and the skeleton point position information of the current frame, determining the pull-up displacement of the current frame; and performing a pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested. According to the method, device, electronic device and storage medium provided by the present invention, the pull-up evaluation process does not need to rely on a tester composed of various sensors, thereby reducing the cost of the pull-up evaluation, and does not need to install a lot of additional supporting equipment, thereby improving the convenience of the pull-up evaluation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pull-up evaluation method, device, electronic equipment and storage medium. Background Art

[0002] Pull-ups are a common sport that involves hanging exercises in which the body relies on its own strength to overcome its own body weight and do upward work.

[0003] The current pull-up assessment mainly relies on manual counting or automated pull-up assessment methods. Manual counting is time-consuming and laborious, with large errors, and cannot accurately detect whether the person being tested has violated the rules. The automated pull-up assessment method mostly relies on testers composed of various sensors. The testers have many supporting equipment, high costs, complex installation, and inconvenient movement. Summary of the invention

[0004] The present invention provides a pull-up evaluation method, device, electronic equipment and storage medium, which are used to solve the defects of manual counting and automatic pull-up evaluation methods in the prior art.

[0005] The present invention provides a pull-up evaluation method, comprising:

[0006] Get the video stream to be tested;

[0007] Based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, determining the state of the current frame, wherein the state is any one of a ready state, an upward state and a downward state;

[0008] When the state of the current frame is an upward state or a downward state, based on the horizontal bar position information of the current frame and the position information of the skeleton points of the person, determining the pull-up displacement of the current frame;

[0009] A pull-up test is performed based on the pull-up displacement of each frame in the video stream to be tested.

[0010] According to a pull-up evaluation method provided by the present invention, the horizontal bar position information of the current frame is determined based on the following steps:

[0011] Inputting the current frame into a horizontal bar detection model to obtain horizontal bar position information of the current frame output by the horizontal bar detection model;

[0012] The horizontal bar detection model is trained based on a constraint condition that the person's position is within the horizontal bar position, and based on a first sample image, and a horizontal bar position label and a person's position label of the first sample image.

[0013] According to a pull-up evaluation method provided by the present invention, the horizontal bar detection model is trained based on the following steps:

[0014] Obtain an initial detection model;

[0015] Inputting the first sample image into the initial detection model to obtain a horizontal bar detection result and a person detection result of the first sample image output by the initial detection model;

[0016] Determine a first loss based on the horizontal bar detection result and the horizontal bar position tag, the person detection result and the person position tag;

[0017] determining a second loss based on whether the position of the person in the person detection result is at the horizontal bar position in the horizontal bar detection result;

[0018] Based on the first loss and the second loss, parameters of the initial detection model are iterated to obtain the horizontal bar detection model.

[0019] According to a pull-up evaluation method provided by the present invention, the horizontal bar detection model includes a feature extraction layer, and the feature extraction layer is used to extract image features for the horizontal bar detection model to perform horizontal bar position detection;

[0020] The feature extraction layer includes a channel attention module and a spatial attention module. The input of the channel attention module is the input of the feature extraction layer, and the output of the channel attention module is the input of the spatial attention module. The input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

[0021] According to a pull-up evaluation method provided by the present invention, the state of the current frame is determined based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, including:

[0022] When the forward and reverse grip detection result of the current frame is a forward grip, determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested;

[0023] The forward and reverse grip detection results are obtained by detecting the current frame and the hand position information of the current frame by a rule detection model, and the hand position information is determined based on the position information of the skeleton points of the person in the current frame.

[0024] According to a pull-up evaluation method provided by the present invention, the state of the current frame is determined based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, including:

[0025] When the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the personnel skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information, record the number of times the head passes over the bar, and determine the state of the current frame based on the number of times the head passes over the bar;

[0026] When the state of the previous frame in the video stream to be tested is a downward state, the arm angle of the person in the current frame is determined based on the position information of the person's skeletal points in the current frame, and the state of the current frame is determined based on the number of arm bends when the person's arm angle is less than a preset angle value.

[0027] According to a pull-up evaluation method provided by the present invention, the pull-up evaluation is performed based on the pull-up displacement of each frame in the video stream to be tested, including:

[0028] Based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the personnel bone point position information, and / or the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the personnel bone point position information of each frame, determine the area type of each frame;

[0029] Determine, based on the pull-up displacement of each frame in which the region type is the upper critical region, an inflection point frame corresponding to an extremely high value inflection point of the pull-up displacement;

[0030] Determine, based on the pull-up displacement of each frame in which the region type is the lower critical region, an inflection point frame corresponding to an extremely low value inflection point of the pull-up displacement;

[0031] Based on the inflection point frame, pull-up counting is performed.

[0032] The present invention also provides a pull-up evaluation device, comprising:

[0033] An acquisition unit, used for acquiring a video stream to be tested;

[0034] A state determination unit, used to determine the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, wherein the state is any one of a ready state, an upward state and a downward state;

[0035] A displacement determination unit, for determining the pull-up displacement of the current frame based on the horizontal bar position information and the personnel skeleton point position information of the current frame when the state of the current frame is an upward state or a downward state;

[0036] An evaluation unit is used to perform pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the pull-up evaluation method as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the pull-up evaluation methods described above.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the pull-up evaluation method as described above is implemented.

[0040] The pull-up evaluation method, device, electronic device and storage medium provided by the present invention can determine the pull-up displacement of each frame in the video stream to be tested based on the horizontal bar position information and the person's skeleton point position information of each frame in the video stream to be tested. Since the pull-up displacement is relative to the horizontal bar position information and the person's skeleton point position information, the calculation of the pull-up displacement does not require the calibration of the horizontal bar position information and the person's skeleton point position information, thereby simplifying the preliminary preparation process for the pull-up evaluation; based on the pull-up displacement of each frame in the video stream to be tested, the pull-up evaluation is performed, and this process does not need to rely on a tester composed of various sensors, thereby reducing the cost of the pull-up evaluation, and does not require the installation of a lot of additional supporting equipment, thereby improving the convenience of the pull-up evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of the process of the pull-up evaluation method provided by the present invention;

[0043] Figure 2 It is a flow chart of the training steps of the horizontal bar detection model provided by the present invention;

[0044] Figure 3 It is a structural schematic diagram of the feature extraction layer provided by the present invention;

[0045] Figure 4 is a structural diagram of the channel attention module provided by the present invention;

[0046] Figure 5It is a schematic diagram of the structure of the spatial attention module provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of the rule detection model provided by the present invention;

[0048] Figure 7 is a flow chart of step 140 in the pull-up evaluation method provided by the present invention;

[0049] Figure 8 It is a structural schematic diagram of the pull-up evaluation device provided by the present invention;

[0050] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0053] In the related art, the automatic evaluation method of pull-ups mostly relies on a tester composed of various sensors. The tester has many supporting equipment, complex structure, high cost, complex installation, and inconvenient movement. For example, when performing a pull-up evaluation, the pressure sensor needs to be set on the inside of the elbow of the person being tested, which is inconvenient. In view of this, the embodiment of the present invention improves the evaluation method of pull-ups.

[0054] The present invention provides a pull-up evaluation method. Figure 1 : is a schematic diagram of the process of the pull-up evaluation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0055] Step 110: Obtain the video stream to be tested.

[0056] Specifically, the video stream to be tested, i.e., the video stream for which the pull-up test is required, can be recorded in real time. The pull-up test can be to test the positive and negative grip of the pull-up of the person being tested, or to test whether the pull-up action of the person being tested is compliant, or to count the number of pull-ups of the person being tested, which is not specifically limited in the embodiment of the present invention.

[0057] Step 120, based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, determine the state of the current frame, where the state is any one of a ready state, an upward state and a downward state.

[0058] Specifically, after obtaining the video stream to be tested, the state of the current frame can be determined based on the position information of the skeleton points of the person in the current frame in the video stream to be tested. The current frame in the video stream to be tested here refers to the latest frame in the video stream to be tested, and the position information of the skeleton points of the person here can reflect the human body posture information of the person under test contained in the frame image of the current frame, and can include the position of the skeleton points of the hand of the person under test, and can also include the position of the skeleton points of the head of the person under test, and can also include the position of the skeleton points of the shoulder of the person under test, and the embodiment of the present invention does not make specific limitations on this.

[0059] The state of the current frame refers to the state of the person under test in the current frame during the pull-up process, which can be a preparation state, an upward state, or a downward state. The preparation state here can be that the person under test holds the horizontal bar with both hands, both hands are shoulder-width apart, arms are fully extended, the body is suspended and still, and the body is facing the camera. The preparation state can be understood as a static state; the upward state here can be that the overall state of the person under test is doing pull-ups from bottom to top; the downward state here can be that the overall state of the person under test is doing pull-ups from top to bottom. The upward state and the downward state can be understood as a dynamic state.

[0060] Step 130, when the state of the current frame is an upward state or a downward state, based on the horizontal bar position information of the current frame and the position information of the skeleton points of the person, determine the pull-up displacement of the current frame.

[0061] Specifically, when the state of the current frame is an upward state or a downward state, that is, when the current frame indicates that the person being tested is currently in a dynamic state of pull-ups, the pull-up displacement of the current frame can be determined based on the horizontal bar position information of the current frame and the position information of the person's skeletal points. Here, the horizontal bar position information of the current frame refers to the position information of the horizontal bar in the current frame, which may include the horizontal bar position information of the horizontal bar, and may also include the position information of the left and right pillars of the horizontal bar. The horizontal bar position information of the current frame can be obtained based on a horizontal bar detection model. For example, the current frame in the video stream to be tested can be input into the horizontal bar detection model to obtain the horizontal bar position information of the current frame output by the horizontal bar detection model. It can be understood that the horizontal bar detection model here is an image target detection model that can regard the horizontal bar as a detection target.

[0062] Here, the pull-up displacement of the current frame can be determined based on the relative position information of the horizontal bar position information and the position information of the person's skeleton point in the current frame, and the pull-up displacement refers to the vertical distance from the reference skeleton point of the person under test in the current frame to the horizontal bar in the horizontal bar. For example, the skeleton point on the top of the person under test's head can be used as the reference skeleton point.

[0063] The pull-up displacement of the current frame here refers to the displacement generated by the person being tested during the pull-up process when the current frame is in an upward state or a downward state. For example, when the reference bone point is higher than the horizontal bar, the pull-up displacement is positive; when the reference bone point is lower than the horizontal bar, the pull-up displacement is negative. The embodiment of the present invention does not make any specific limitation on this.

[0064] Step 140, performing pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0065] Specifically, after determining the pull-up displacement of the current frame and each frame before the current frame in the video stream to be tested, the pull-up evaluation can be performed based on the pull-up displacement of each frame in the video stream to be tested.

[0066] For example, the pull-up displacement of each frame in the video stream to be tested can be compared with a preset threshold value to determine the extremely high inflection point and the extremely low inflection point in the pull-up displacement, thereby performing pull-up counting. The inflection point frames in the video stream to be tested corresponding to the extremely high inflection point and the extremely low inflection point can also be determined, and the inflection point frames in the video stream to be tested corresponding to the extremely high inflection point and the extremely low inflection point can be input into the rule detection model to perform standardized detection of pull-up actions at extremely high and extremely low values.

[0067] The method provided by the embodiment of the present invention can determine the pull-up displacement of each frame in the video stream to be tested based on the horizontal bar position information and the person's skeleton point position information of each frame in the video stream to be tested. Since the pull-up displacement is relative to the horizontal bar position information and the person's skeleton point position information, the calculation of the pull-up displacement does not require the calibration of the horizontal bar position information and the person's skeleton point position information, which simplifies the preliminary preparation process for pull-up evaluation; based on the pull-up displacement of each frame in the video stream to be tested, pull-up evaluation is performed. This process does not need to rely on a tester composed of various sensors, thereby reducing the cost of pull-up evaluation, and does not require the installation of a lot of additional supporting equipment, thereby improving the convenience of pull-up evaluation.

[0068] Based on the above embodiment, the pull-up movement displacement here can be determined based on the following steps:

[0069] Based on the skeleton point network, the left shoulder skeleton point coordinates corresponding to the current frame, the previous frame of the current frame, and the previous frame of the current frame can be determined. i , right shoulder bone point coordinates right_shoulder i And the coordinates of the mandibular bone points chin i , where i is the frame number.

[0070] Each frame obtains a reference bone point coordinate based on the above three bone point coordinates. The formula is as follows:

[0071]

[0072] Among them, h avg_i is the reference bone point coordinate, left_shoulder i [0] is the x-axis coordinate of the left shoulder bone point output by the bone point network, right_shoulder i [0] The x-axis coordinate of the right shoulder bone point output by the bone point network, chin i [0] is the x-axis coordinate of the mandibular bone point output by the bone point network, left_shoulder i [1] is the y-axis coordinate of the left shoulder bone point output by the bone point network, right_shoulder i [1] is the y-axis coordinate of the right shoulder bone point output by the bone point network, i [1] is the y-axis coordinate of the mandibular bone point output by the bone point network.

[0073] Based on the leftmost point coordinate bar_p1 and the rightmost point coordinate bar_p2 of the horizontal bar in the horizontal bar position information, the pull-up movement displacement is calculated. The formula for the pull-up movement displacement is as follows:

[0074]

[0075] Among them, h avg_i is the coordinate of the reference bone point, h(i) is the pull-up movement displacement, bar_p1 is the coordinate of the leftmost point of the bar, bar_p2 is the coordinate of the rightmost point of the bar, and ||bar_p2-bar_p1||2 is the second norm of bar_p1 and bar_p2.

[0076] Based on the above embodiment, the horizontal bar position information of the current frame is determined based on the following steps:

[0077] Inputting the current frame into a horizontal bar detection model to obtain horizontal bar position information of the current frame output by the horizontal bar detection model;

[0078] The horizontal bar detection model is trained based on a constraint condition that the person's position is within the horizontal bar position, and based on a first sample image, and a horizontal bar position label and a person's position label of the first sample image.

[0079] Specifically, traditional horizontal bar detection models, such as the YOLOX model, only detect the position of the horizontal bar during training. In the pull-up scenario, since the person being tested is in the horizontal bar frame in each frame of the video stream to be tested, the horizontal bar detection model needs to detect not only the position of the horizontal bar and the position of the person, but also limit the position of the person to be within the horizontal bar position. Whether the person being tested is in the horizontal bar frame can be used as part of the loss function and the training of the horizontal bar detection model can be carried out together.

[0080] The horizontal bar detection model can be trained based on the first sample image, the horizontal bar position label and the person position label of the first sample image, with the constraint that the person position is within the horizontal bar position. The first sample image here is a sample image including the horizontal bar and the person being tested, which can be pre-captured by an image acquisition device or captured in real time.

[0081] The current frame in the video stream to be tested is input into the horizontal bar detection model, and the horizontal bar position information of the current frame output by the horizontal bar detection model can be obtained.

[0082] Based on the above embodiments, the horizontal bar detection model is applied in the special scenario of pull-ups, and the horizontal bar detection model takes the personnel position within the horizontal bar position as a constraint, and takes the correlation between the horizontal bar position and the personnel position into consideration. Therefore, the accuracy of the horizontal bar detection results can be further improved.

[0083] Based on the above embodiments, Figure 2 It is a flow chart of the training steps of the horizontal bar detection model provided by the present invention, such as Figure 2As shown, the horizontal bar detection model is trained based on the following steps:

[0084] Step 210, obtaining an initial detection model;

[0085] Step 220, inputting the first sample image into the initial detection model, and obtaining a horizontal bar detection result and a person detection result of the first sample image output by the initial detection model;

[0086] Step 230, determining a first loss based on the horizontal bar detection result and the horizontal bar position tag, the personnel detection result and the personnel position tag;

[0087] determining a second loss based on whether the position of the person in the person detection result is at the horizontal bar position in the horizontal bar detection result;

[0088] Step 240: Based on the first loss and the second loss, perform parameter iteration on the initial detection model to obtain the horizontal bar detection model.

[0089] Specifically, in order to better improve the detection effect of the horizontal bar detection model, an embodiment of the present invention sets an initial detection model, and the parameters of the initial detection model here can be randomly generated or pre-set.

[0090] After acquiring the initial detection model, the first sample image can be input into the initial detection model to obtain the horizontal bar detection result and the person detection result of the first sample image output by the initial detection model. The horizontal bar detection result here reflects whether the detection result includes information about the horizontal bar, and when the detection result includes the horizontal bar, the detection result also includes the position information of the horizontal bar.

[0091] Then, the first loss can be determined based on the horizontal bar detection result and the horizontal bar position label, the personnel detection result and the personnel position label, where the first loss is used to reflect the degree of difference between the horizontal bar detection result and the horizontal bar position label, the personnel detection result and the personnel position label. It can be understood that the greater the first loss, the greater the degree of difference between the horizontal bar detection result and the horizontal bar position label, the personnel detection result and the personnel position label, and the smaller the first loss, the smaller the degree of difference between the horizontal bar detection result and the horizontal bar position label, the personnel detection result and the personnel position label.

[0092] Here, the formula for the first loss is as follows:

[0093] L ori =λ1L iou +λ2L cls +λ3L obj

[0094] Among them, L oriis the first loss, L iou is the target box regression loss function, L cls is the classification loss function, L obj is the target loss function, and λ1, λ2, and λ3 are weight coefficients. The target box regression loss function is used to reflect the error between the detected horizontal bar position and the actual position label, the classification loss function is used to reflect the error of whether the detected target category is a horizontal bar, and the target loss function is used to reflect the error of whether the horizontal bar is detected.

[0095] It can be understood that the calculation method of the first loss is similar to the calculation method of the loss function of the horizontal bar detection model in the related art.

[0096] At the same time, the second loss can also be determined based on whether the person position in the person detection result is in the single bar position in the single bar detection result. The second loss here is used to reflect whether the person position in the person detection result is in the single bar position in the single bar detection result.

[0097] Here, the formula for the second loss is as follows:

[0098]

[0099] Among them, loc_person is the human frame coordinate frame, loc_bar is the bar frame, Indicates that the human body frame coordinate frame is within the bar frame. Indicates that the human frame coordinate frame is not within the bar frame, L person The second loss.

[0100] It is understandable that when the human body frame coordinate frame is not within the bar frame, a second loss will occur.

[0101] Finally, based on the first loss and the second loss, the parameters of the initial detection model can be iterated to obtain the horizontal bar detection model.

[0102] Among them, the total loss can be obtained based on the first loss and the second loss. The formula of the total loss is as follows:

[0103] L new =λ1L iou +λ2L cls +λ3L obj +λ4L person

[0104] Among them, L new is the total loss, L iou is the target box regression loss function, L cls is the classification loss function, L obj is the target loss function, L personis the second loss, and λ1, λ2, λ3, and λ4 are weight coefficients.

[0105] Based on the above embodiment, the horizontal bar detection model includes a feature extraction layer, and the feature extraction layer is used to extract image features for the horizontal bar detection model to perform horizontal bar position detection. Figure 3 is a schematic diagram of the structure of the feature extraction layer provided by the present invention, such as Figure 3 shown.

[0106] Specifically, the model structure of the horizontal bar detection model can be one of various target detection models, for example, a YOLOX model, a Faster R-CNN model, an SSD (Single Shot MultiBox Detector) model, etc., which is not specifically limited in the embodiment of the present invention.

[0107] It can be understood that in order to realize the horizontal bar position detection, the horizontal bar detection model needs to extract image features through the feature extraction layer. The feature extraction layer here can be one or more, and the multiple feature extraction layers can be cascaded or parallel. The embodiment of the present invention does not make specific limitations on this.

[0108] Based on the above embodiments, Figure 4 is a schematic diagram of the structure of the channel attention module provided by the present invention, Figure 5 is a schematic diagram of the structure of the spatial attention module provided by the present invention, such as Figure 4 , Figure 5 As shown, the feature extraction layer includes a channel attention module and a spatial attention module. The input of the channel attention module is the input of the feature extraction layer, and the output of the channel attention module is the input of the spatial attention module. The input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

[0109] Specifically, the traditional CBAM (Convolutional Block Attention Module) model includes a channel attention module (Channel Attention) and a spatial attention module (Spatial Attention), wherein the channel attention module in the traditional CBAM model is extended to the front of the spatial attention module through a residual structure.

[0110] The feature extraction layer here may include a channel attention module and a spatial attention module, wherein the input of the channel attention module is the input of the feature extraction layer, the output of the channel attention module is the input of the spatial attention module, and the input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

[0111] Compared with the traditional CBAM model, in the feature extraction layer here, the channel attention module is extended to the spatial attention module through the residual structure, so that the output of the spatial attention module can cover the information in the channel attention module, further enriching the information of the features extracted by the feature extraction layer, which helps to improve the reliability of model detection. The channel attention module here may include a pooling layer, a convolution layer, a fusion layer, and an activation layer. The pooling layer here can be a maximum pooling layer or an average pooling layer. The convolution layer here can use a deep full convolution network or a full convolution network. The activation layer here can use a Sigmoid activation function, a Softmax activation function, or a ReLU activation function, which is not specifically limited in the embodiment of the present invention. The spatial attention module here may include a pooling layer, a convolution layer, and an activation layer. The pooling layer here can be a maximum pooling layer or an average pooling layer. The convolution layer here can use a deep full convolutional network or a full convolutional network. The activation layer here can use a Sigmoid activation function, a Softmax activation function, or a ReLU activation function, which is not specifically limited in the embodiment of the present invention.

[0112] For example, assuming the input feature map is F = R C×H×W , the one-dimensional channel attention map M is derived sequentially through the feature extraction layer c ∈R C×1×1 and the 2D spatial attention map M s ∈R 1×H×W , the specific process is as follows:

[0113] F'=M c (F)

[0114]

[0115]

[0116] in

[0117]

[0118] Among them, F is the input feature map, F' is the feature map output by the channel attention module, and M s (F') is the feature map output by the spatial attention module, F" is the feature map obtained by weighted fusion of the feature map output by the spatial attention module and the feature map output by the channel attention module, F"' is the output result of the feature extraction layer, σ is the sigmoid function, AvgPool is the average pooling layer, MaxPool is the maximum pooling layer, W0∈R C / r×C , W1∈RC×C / r , f 1×1 is a convolution kernel of size 1×1, f 7×7 is a 7×7 convolution kernel.

[0119] In addition, the horizontal bar detection model can be in the form of an EfficientV2 network (EfficientNet v2) to improve the detection efficiency of the horizontal bar detection model.

[0120] Based on the above embodiments, the feature extraction layer of the horizontal bar detection model includes a channel attention module and a spatial attention module, which better improves the accuracy of feature extraction compared to the traditional connection method of the channel attention module and the spatial attention module.

[0121] Based on the above embodiments, Figure 6 It is a structural diagram of the rule detection model provided by the present invention, such as Figure 6 As shown, step 120 includes:

[0122] When the forward and reverse grip detection result of the current frame is a forward grip, determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested;

[0123] The forward and reverse grip detection results are obtained by detecting the current frame and the hand position information of the current frame by a rule detection model, and the hand position information is determined based on the position information of the skeleton points of the person in the current frame.

[0124] Specifically, the forward and reverse grip detection results are obtained by the rule detection model based on the current frame and the hand position information of the current frame. For example, the position information of the skeleton points of the person in the current frame can be extracted first, and the hand position information of the current frame can be determined based on the skeleton point position information of the current frame. Then, the current frame and the hand position information of the current frame can be input into the rule detection model for detection, or based on the hand position information of the current frame, the hand image in the current frame can be intercepted and input into the rule detection model for detection, thereby obtaining the forward and reverse grip detection results. The embodiment of the present invention does not make specific limitations on this. Here, the forward and reverse grip detection results can be either a forward grip or a reverse grip.

[0125] The forward and reverse grip detection results here can be obtained through the forward and reverse grip classification network in the rule detection model. The forward and reverse grip classification network here includes a convolution layer, a bottleneck layer (Bottleneck) and a pooling layer. The convolution layer here can use a deep fully convolutional network (Deep-ConvNet) or a fully convolutional network (Fully Convolutional Networks, FCN). The pooling layer here can be an average pooling layer or a maximum pooling layer, and the embodiment of the present invention does not specifically limit this.

[0126] The hand position information here is determined based on the position information of the skeleton points of the person in the current frame. The position information of the skeleton points of the person here can be obtained based on the skeleton point detection network. For example, the current frame in the video stream to be tested can be input into the skeleton point detection network to obtain the position information of the skeleton points of the person output by the skeleton point network. Here, the hand position information can be extracted using Roi align. The formula for extracting the hand position information is as follows:

[0127]

[0128] hand_center=(midfinger_pt[0],(wrist_pt[1]+palm_pt[1]) / 2-width / 5)

[0129] Wherein, width is the size of the hand rectangle, hand_center is the position of the hand center point in the hand position information, right_ear_pt[0] is the x-axis coordinate of the right ear point output by the skeleton point detection network, right_ear_pt[1] is the y-axis coordinate of the right ear point output by the skeleton point detection network, left_ear_pt[0] is the x-axis coordinate of the left ear point output by the skeleton point detection network, left_ear_pt[1] is the y-axis coordinate of the left ear point output by the skeleton point detection network, midfinger_pt[0] is the x-axis coordinate of the middle finger tip output by the skeleton point detection network, wrist_pt[1] is the y-axis coordinate of the wrist point output by the skeleton point detection network, and palm_pt[1] is the y-axis coordinate of the palm and middle finger output by the skeleton point detection network.

[0130] When the forward and reverse grip detection result of the current frame is a forward grip, the state of the current frame can be determined based on the position information of the skeleton points of the person in the current frame in the video stream to be tested. The state of the current frame here can be a ready state, an upward state, or a downward state. The embodiment of the present invention does not specifically limit this. It can be understood that the forward and reverse grip detection is an important part of the pull-up assessment. If the reverse grip of the hand of the person being tested is detected, it proves that the person being tested has violated the rules and there is no need to perform subsequent tests.

[0131] The method provided by the embodiment of the present invention determines the state of the current frame only when the forward and reverse grip detection result of the current frame is a forward grip, which saves a large amount of computing resources and improves the efficiency of pull-up evaluation.

[0132] Based on the above embodiment, the state of the current frame is determined based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, including:

[0133] When the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the person's skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information, the number of times the head passes the bar is recorded, and the state of the current frame is determined based on the number of times the head passes the bar.

[0134] Specifically, the state of the previous frame in the video stream to be tested can be determined based on the position information of the person's skeleton points in the previous frame in the video stream to be tested. The previous frame in the video stream to be tested here refers to the previous frame of the current frame in the video stream to be tested.

[0135] If the state of the previous frame in the video stream to be tested is the upward state, and the head vertex coordinates in the skeleton point position information of the current frame are out of the horizontal bar coordinates in the horizontal bar position information, record the number of times the head passes the bar. The head vertex coordinates here refer to the skeleton point coordinates of the vertex of the head of the person being tested, and the horizontal bar coordinates may include the coordinates of each pixel point on the horizontal bar of the horizontal bar.

[0136] After obtaining the number of times the head passes over the bar, the state of the current frame can be determined based on the number of times the head passes over the bar. The state of the current frame here can be an upward state or a downward state.

[0137] It can be understood that, when the number of times the head passes over the bar is 0, the top of the head of the person being tested in the current frame has not passed over the horizontal bar of the horizontal bar. When the number of times the head passes over the bar is 1, the top of the head of the person being tested in the current frame passes over the horizontal bar of the horizontal bar for the first time, and then the person being tested will continue to do pull-ups upward. When the number of times the head passes over the bar is 2, the top of the head of the person being tested in the current frame passes over the horizontal bar of the horizontal bar for the second time, and then the person being tested will continue to do pull-ups downward.

[0138] When the state of the previous frame in the video stream to be tested is a downward state, the arm angle of the person in the current frame is determined based on the position information of the person's skeletal points in the current frame, and the state of the current frame is determined based on the number of arm bends when the person's arm angle is less than a preset angle value.

[0139] Specifically, when the state of the previous frame in the video stream to be tested is a downward state, the arm angle of the person in the current frame is determined based on the skeleton point position information of the person in the current frame. The arm angle of the person here refers to the angle between the three skeleton point position information of the person's shoulder, elbow and wrist. The skeleton point position information of the person's shoulder, elbow and wrist here can be determined based on the skeleton point position information of the person, wherein the skeleton point position information of the person is obtained based on the skeleton point detection network.

[0140] At this time, the number of times the person's arm bends when the arm angle is less than a preset angle value can be calculated. The preset angle value here can be 90 degrees, 80 degrees, or 100 degrees, and the embodiment of the present invention does not make specific limitations on this.

[0141] After obtaining the number of arm bends when the person's arm angle is less than the preset angle value, the state of the current frame can be determined based on the number of arm bends when the person's arm angle is less than the preset angle value, and when the number of arm bends is 2, the state of the current frame is determined to be an upward state.

[0142] The method provided in the embodiment of the present invention accurately determines the state of the current frame by calculating the number of times the person passes the bar over the head and the number of times the person's arm is bent with an angle less than a preset angle value, thereby performing a pull-up assessment. The method does not need to rely on a tester composed of multiple types of sensors, has a low cost, reduces the cost of the pull-up assessment, and at the same time, improves the convenience of the pull-up assessment.

[0143] Based on the above embodiments, Figure 7 is a flow chart of step 140 in the pull-up evaluation method provided by the present invention, such as Figure 7 As shown, step 140 includes:

[0144] Step 141, based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the personnel bone point position information, and / or the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the personnel bone point position information of each frame, determine the area type of each frame;

[0145] Step 142, based on the pull-up displacement of each frame where the region type is the upper critical region, determining the inflection point frame corresponding to the extremely high value inflection point of the pull-up displacement;

[0146] Step 143, based on the pull-up displacement of each frame where the area type is the lower critical area, determine the inflection point frame corresponding to the extremely low value inflection point of the pull-up displacement;

[0147] Step 144, counting pull-ups based on the inflection point frame.

[0148] Specifically, the region type of each frame can be determined based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the person's bone point position information, and / or the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the person's bone point position information of each frame. The region type here can include an upper critical region and a lower critical region, and the region type of each frame refers to whether the region where each frame is located belongs to the upper critical region or the lower critical region.

[0149] That is, the type of region in which each frame is located can be determined based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the person's bone point position information, and the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the person's bone point position information of each frame; the type of region in which each frame is located can also be determined based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the person's bone point position information; the type of region in which each frame is located can also be determined based on the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the person's bone point position information of each frame, and the embodiment of the present invention does not make specific limitations on this.

[0150] For example, the region type of each frame where the top bone point position information in the person's bone point position information is above the horizontal bar position information can be used as the upper critical region. The region type of each frame where the angles between the shoulder bone points, elbow bone points, and wrist bone points in the person's bone point position information are greater than 90° can be used as the lower critical region, and the embodiments of the present invention do not specifically limit this.

[0151] After determining the region type of each frame, the inflection point frame corresponding to the extremely high inflection point of the pull-up displacement can be determined based on the pull-up displacement of each frame whose region type is the upper critical region. The extremely high inflection point of the pull-up displacement here refers to the point with the highest position of the top bone point of each frame whose region type is the upper critical region.

[0152] After determining the region type of each frame, the inflection point frame corresponding to the extremely low inflection point of the pull-up displacement can be determined based on the pull-up displacement of each frame whose region type is the lower critical region. The extremely low inflection point of the pull-up displacement here refers to the point with the largest angle among the shoulder bone point, elbow bone point and wrist bone point in the person's bone point position information of each frame whose region type is the lower critical region.

[0153] The inflection point frame here refers to the frame in the video stream to be tested corresponding to the extremely high inflection point and the extremely low inflection point of the upward displacement of the body. After obtaining the inflection point frame, the pull-up count can be performed. For example, when the position of the mandibular bone point of the person in the extremely high inflection point frame passes the horizontal bar position in the horizontal bar, and the person in the last extremely low inflection point frame meets the arm angle hanging (the first time to meet the standard for the preparation action), and the time between the last compliant extremely high inflection point frame (the first time to meet the standard for the preparation action) is within 10 seconds, the pull-up count is increased once.

[0154] In addition, the inflection point frames corresponding to the extremely high inflection point and the extremely low inflection point of the pull-up displacement can also be input into the rule detection model for compliance action detection, and the compliance action detection results are output. The compliance action detection results here include compliance actions and non-compliance actions. The compliance actions here include four types: high chin over the bar, high chin over the bar, low arm hanging, and low arm not hanging. The high chin over the bar here means that when the current frame is in an upward state, the pull-up displacement of the person is greater than the displacement from the extremely high inflection point to the horizontal bar of the horizontal bar. The high chin over the bar here means that when the current frame is in an upward state, the pull-up displacement of the person is less than the displacement from the extremely high inflection point to the horizontal bar of the horizontal bar. The low arm hanging here means that when the current frame is in a downward state, the arm is fully extended and the body is suspended and still. The low arm not hanging here means that when the current frame is in a downward state, the arm is not fully extended and the body is not suspended and still. The non-compliant action here refers to the detection results other than the compliance action.

[0155] In addition to compliance action detection, there are other violation detections. For example, when the current frame is in the ready state, the hands of the person being tested are not shoulder-width apart, that is,

[0156]

[0157] The detection results of other violations are non-compliant; when the current frame is in the ready state, the person being tested has his back to the camera, that is, the x-coordinate of the left shoulder bone point is less than or equal to the coordinate of the right shoulder bone point, and the detection results of other violations are non-compliant; when the current frame is in the ready state, the body of the person being tested is not hanging still, that is, the normalized similarity of the bone points in two adjacent frames is less than or equal to 0.99, and the detection results of other violations are non-compliant.

[0158] The method provided in the embodiment of the present invention counts pull-ups based on inflection point frames, which is more convenient and faster than manual counting and a tester composed of various sensors, and has no errors, further improving the accuracy of pull-up evaluation.

[0159] Based on any of the above embodiments, a pull-up evaluation method includes the following steps:

[0160] First, the video stream to be tested can be obtained, and then based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, the state of the current frame can be determined, where the state can be any one of the ready state, the upward state and the downward state.

[0161] When the current frame is in an upward state or a downward state, the pull-up displacement of the current frame can be determined based on the horizontal bar position information and the position information of the person's skeleton points in the current frame. Then, the pull-up evaluation is performed based on the pull-up displacement of each frame in the video stream to be tested.

[0162] The horizontal bar position information here can be determined based on the following steps: input the current frame into the horizontal bar detection model to obtain the horizontal bar position information of the current frame output by the horizontal bar detection model. The horizontal bar detection model here is trained based on the first sample image, the horizontal bar position label and the personnel position label of the first sample image, with the constraint that the personnel position is within the horizontal bar position.

[0163] The horizontal bar detection model here can be specifically trained based on the following steps: first, an initial detection model can be obtained, and then the first sample image can be input into the initial detection model to obtain the horizontal bar detection result and the person detection result of the first sample image output by the initial detection model. Then, the first loss can be determined based on the horizontal bar detection result and the horizontal bar position label, the person detection result and the person position label; the second loss can be determined based on whether the person position in the person detection result is in the horizontal bar position in the horizontal bar detection result; and then, based on the first loss and the second loss, the initial detection model can be iterated on parameters to obtain the horizontal bar detection model.

[0164] The horizontal bar detection model here includes a feature extraction layer, and the feature extraction layer here is used to extract image features for the horizontal bar detection model to detect the horizontal bar position. Among them, the feature extraction layer includes a channel attention module and a spatial attention module. The input of the channel attention module is the input of the feature extraction layer, and the output of the channel attention module is the input of the spatial attention module. The input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

[0165] In addition, the horizontal bar detection model can be in the form of an EfficientV2 network to improve the detection efficiency of the horizontal bar detection model.

[0166] Among them, when the forward and reverse grip detection result of the current frame is a forward grip, the state of the current frame can be determined based on the position information of the person's skeleton points in the current frame in the video stream to be tested, wherein the forward and reverse grip detection result is obtained by detecting the current frame and the hand position information of the current frame by a rule detection model, and the hand position information here is determined based on the position information of the person's skeleton points in the current frame.

[0167] The state of the current frame is determined based on the following steps: when the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the personnel skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information, the number of times the head passes over the bar is recorded, and the state of the current frame is determined based on the number of times the head passes over the bar;

[0168] When the state of the previous frame in the video stream to be tested is a downward state, the arm angle of the person in the current frame is determined based on the position information of the person's skeletal points in the current frame, the state of the current frame is determined based on the number of arm bends when the person's arm angle is less than a preset angle value, and the number of arm bends when the person's arm angle is less than the preset angle value.

[0169] Then, the region type of each frame can be determined based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the person's bone point position information, and / or the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the person's bone point position information of each frame;

[0170] Based on the pull-up displacement of each frame in which the region type is the upper critical region, an inflection point frame corresponding to an extremely high value inflection point of the pull-up displacement can be determined;

[0171] Based on the pull-up displacement of each frame in which the area type is the lower critical area, an inflection point frame corresponding to an extremely low value inflection point of the pull-up displacement can be determined;

[0172] Finally, pull-ups can be counted based on the inflection point frame.

[0173] The pull-up evaluation device provided by the present invention is described below. The pull-up evaluation device described below and the pull-up evaluation method described above can be referenced to each other.

[0174] Based on any of the above embodiments, Figure 8 : is a schematic diagram of the structure of the pull-up evaluation device provided by the present invention, such as Figure 8 As shown, the device comprises:

[0175] An acquisition unit 810 is used to acquire a video stream to be tested;

[0176] A state determination unit 820 is used to determine the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, wherein the state is any one of a ready state, an upward state and a downward state;

[0177] A displacement determination unit 830, configured to determine the pull-up displacement of the current frame based on the horizontal bar position information and the personnel skeleton point position information of the current frame when the state of the current frame is an upward state or a downward state;

[0178] The evaluation unit 840 is used to perform pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0179] The device provided by the embodiment of the present invention can determine the pull-up displacement of each frame in the video stream to be tested based on the horizontal bar position information and the person's skeleton point position information of each frame in the video stream to be tested. Since the pull-up displacement is relative to the horizontal bar position information and the person's skeleton point position information, the calculation of the pull-up displacement does not require the calibration of the horizontal bar position information and the person's skeleton point position information, which simplifies the preliminary preparation process for pull-up evaluation; based on the pull-up displacement of each frame in the video stream to be tested, pull-up evaluation is performed. This process does not need to rely on a tester composed of various sensors, thereby reducing the cost of pull-up evaluation, and does not require the installation of a lot of additional supporting equipment, thereby improving the convenience of pull-up evaluation.

[0180] Based on any of the above embodiments, the horizontal bar position information of the current frame is determined based on the following steps:

[0181] Inputting the current frame into a horizontal bar detection model to obtain horizontal bar position information of the current frame output by the horizontal bar detection model;

[0182] The horizontal bar detection model is trained based on a constraint condition that the person's position is within the horizontal bar position, and based on a first sample image, and a horizontal bar position label and a person's position label of the first sample image.

[0183] Based on any of the above embodiments, the horizontal bar detection model is trained based on the following steps:

[0184] Obtain an initial detection model;

[0185] Inputting the first sample image into the initial detection model to obtain a horizontal bar detection result and a person detection result of the first sample image output by the initial detection model;

[0186] Determine a first loss based on the horizontal bar detection result and the horizontal bar position tag, the person detection result and the person position tag;

[0187] determining a second loss based on whether the position of the person in the person detection result is at the horizontal bar position in the horizontal bar detection result;

[0188] Based on the first loss and the second loss, parameters of the initial detection model are iterated to obtain the horizontal bar detection model.

[0189] Based on any of the above embodiments, the horizontal bar detection model includes a feature extraction layer, and the feature extraction layer is used to extract image features for the horizontal bar detection model to perform horizontal bar position detection;

[0190] The feature extraction layer includes a channel attention module and a spatial attention module. The input of the channel attention module is the input of the feature extraction layer, and the output of the channel attention module is the input of the spatial attention module. The input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

[0191] Based on any of the above embodiments, the state determination unit includes:

[0192] When the forward and reverse grip detection result of the current frame is a forward grip, determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested;

[0193] The forward and reverse grip detection results are obtained by detecting the current frame and the hand position information of the current frame by a rule detection model, and the hand position information is determined based on the position information of the skeleton points of the person in the current frame.

[0194] Based on any of the above embodiments, the state determination unit is specifically used for:

[0195] When the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the personnel skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information, record the number of times the head passes over the bar, and determine the state of the current frame based on the number of times the head passes over the bar;

[0196] When the state of the previous frame in the video stream to be tested is a downward state, the arm angle of the person in the current frame is determined based on the position information of the person's skeletal points in the current frame, and the state of the current frame is determined based on the number of arm bends when the person's arm angle is less than a preset angle value.

[0197] Based on any of the above embodiments, the evaluation unit specifically includes:

[0198] Based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the personnel bone point position information, and / or the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the personnel bone point position information of each frame, determine the area type of each frame;

[0199] Determine, based on the pull-up displacement of each frame in which the region type is the upper critical region, an inflection point frame corresponding to an extremely high value inflection point of the pull-up displacement;

[0200] Determine, based on the pull-up displacement of each frame in which the region type is the lower critical region, an inflection point frame corresponding to an extremely low value inflection point of the pull-up displacement;

[0201] Based on the inflection point frame, pull-up counting is performed.

[0202] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the pull-up evaluation method, which includes: obtaining a video stream to be tested; based on the position information of the skeleton points of the personnel in the current frame in the video stream to be tested, determining the state of the current frame, the state being any one of the ready state, the upward state and the downward state; when the state of the current frame is the upward state or the downward state, determining the pull-up displacement of the current frame based on the horizontal bar position information and the skeleton point position information of the current frame; and performing pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0203] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0204] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pull-up evaluation method provided by the above-mentioned methods, which includes: obtaining a video stream to be tested; based on the position information of the skeleton points of a person in the current frame in the video stream to be tested, determining the state of the current frame, wherein the state is any one of a ready state, an upward state and a downward state; when the state of the current frame is an upward state or a downward state, based on the horizontal bar position information and the skeleton point position information of the current frame, determining the pull-up displacement of the current frame; and performing a pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0205] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the pull-up evaluation method provided by the above-mentioned methods, the method comprising: obtaining a video stream to be tested; based on the position information of the skeleton points of a person in the current frame in the video stream to be tested, determining the state of the current frame, wherein the state is any one of a ready state, an upward state and a downward state; when the state of the current frame is an upward state or a downward state, based on the horizontal bar position information and the skeleton point position information of the current frame, determining the pull-up displacement of the current frame; and performing a pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested.

[0206] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0207] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pull-up evaluation method, characterized in that: include: Get the video stream to be tested; Based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, determining the state of the current frame, wherein the state is any one of a ready state, an upward state and a downward state; The step of determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested includes: When the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the personnel skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information of the current frame, record the number of times the head passes over the bar, and determine the state of the current frame based on the number of times the head passes over the bar; In the case where the state of the previous frame in the video stream to be tested is a downward state, determining the arm angle of the person in the current frame based on the position information of the person's skeletal points in the current frame, and determining the state of the current frame based on the number of arm bends when the person's arm angle is less than a preset angle value; When the state of the current frame is an upward state or a downward state, based on the horizontal bar position information of the current frame and the position information of the skeleton points of the person, the pull-up displacement of the current frame is determined, and the pull-up displacement of the current frame refers to the displacement generated by the person under test during the pull-up process when the state of the current frame is an upward state or a downward state; Performing a pull-up test based on the pull-up displacement of each frame in the video stream to be tested; The pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested includes: Based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the personnel bone point position information, and the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the personnel bone point position information of each frame, determine the area type of each frame; Based on the pull-up displacement of each frame whose region type is the upper critical region, determine the inflection point frame corresponding to the extremely high value inflection point of the pull-up displacement, wherein the extremely high value inflection point of the pull-up displacement refers to the point with the highest position of the top bone point of each frame whose region type is the upper critical region; Based on the pull-up displacement of each frame whose region type is the lower critical region, determine the inflection point frame corresponding to the extremely low value inflection point of the pull-up displacement, wherein the extremely low value inflection point of the pull-up displacement refers to the point with the largest angle among the shoulder bone point, the elbow bone point and the wrist bone point in the personnel bone point position information of each frame whose region type is the lower critical region; Based on the inflection point frame, pull-up counting and compliance action detection are performed.

2. The pull-up evaluation method according to claim 1, characterized in that: The horizontal bar position information of the current frame is determined based on the following steps: Inputting the current frame into a horizontal bar detection model to obtain horizontal bar position information of the current frame output by the horizontal bar detection model; The horizontal bar detection model is trained based on a constraint condition that the person's position is within the horizontal bar position, and based on a first sample image, and a horizontal bar position label and a person's position label of the first sample image.

3. The pull-up evaluation method according to claim 2, characterized in that: The horizontal bar detection model is trained based on the following steps: Obtain an initial detection model; Inputting the first sample image into the initial detection model to obtain a horizontal bar detection result and a person detection result of the first sample image output by the initial detection model; Determine a first loss based on the horizontal bar detection result and the horizontal bar position tag, the person detection result and the person position tag; determining a second loss based on whether the position of the person in the person detection result is at the horizontal bar position in the horizontal bar detection result; Based on the first loss and the second loss, parameters of the initial detection model are iterated to obtain the horizontal bar detection model.

4. The pull-up evaluation method according to claim 2, characterized in that: The horizontal bar detection model includes a feature extraction layer, and the feature extraction layer is used to extract image features for the horizontal bar detection model to perform horizontal bar position detection; The feature extraction layer includes a channel attention module and a spatial attention module. The input of the channel attention module is the input of the feature extraction layer, and the output of the channel attention module is the input of the spatial attention module. The input of the feature extraction layer, the output of the channel attention module and the output of the spatial attention module are fused as the output of the feature extraction layer.

5. The pull-up evaluation method according to claim 1, characterized in that: The step of determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested includes: When the forward and reverse grip detection result of the current frame is a forward grip, determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested; The forward and reverse grip detection results are obtained by detecting the current frame and the hand position information of the current frame by a rule detection model, and the hand position information is determined based on the position information of the skeleton points of the person in the current frame.

6. A pull-up evaluation device, characterized in that: include: An acquisition unit, used for acquiring a video stream to be tested; A state determination unit, used to determine the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested, wherein the state is any one of a ready state, an upward state and a downward state; The step of determining the state of the current frame based on the position information of the skeleton points of the person in the current frame in the video stream to be tested includes: When the state of the previous frame in the video stream to be tested is an upward state and the head vertex coordinates in the personnel skeleton point position information of the current frame are at the horizontal bar coordinates in the horizontal bar position information of the current frame, record the number of times the head passes over the bar, and determine the state of the current frame based on the number of times the head passes over the bar; In the case where the state of the previous frame in the video stream to be tested is a downward state, determining the arm angle of the person in the current frame based on the position information of the person's skeletal points in the current frame, and determining the state of the current frame based on the number of arm bends when the person's arm angle is less than a preset angle value; A displacement determination unit, for determining the pull-up displacement of the current frame based on the horizontal bar position information of the current frame and the position information of the skeleton points of the person, when the state of the current frame is an upward state or a downward state, wherein the pull-up displacement of the current frame refers to the displacement generated by the person under test during the pull-up process when the state of the current frame is an upward state or a downward state; An evaluation unit, configured to perform a pull-up evaluation based on the pull-up displacement of each frame in the video stream to be tested; The evaluation unit is specifically used for: Based on the relative position information between the horizontal bar position information of each frame and the top bone point position information in the personnel bone point position information, and the relative position information between the shoulder bone points, elbow bone points and wrist bone points in the personnel bone point position information of each frame, determine the area type of each frame; Based on the pull-up displacement of each frame whose region type is the upper critical region, determine the inflection point frame corresponding to the extremely high value inflection point of the pull-up displacement, wherein the extremely high value inflection point of the pull-up displacement refers to the point with the highest position of the top bone point of each frame whose region type is the upper critical region; Based on the pull-up displacement of each frame whose region type is the lower critical region, determine the inflection point frame corresponding to the extremely low value inflection point of the pull-up displacement, wherein the extremely low value inflection point of the pull-up displacement refers to the point with the largest angle among the shoulder bone point, the elbow bone point and the wrist bone point in the personnel bone point position information of each frame whose region type is the lower critical region; Based on the inflection point frame, pull-up counting and compliance action detection are performed.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the pull-up evaluation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the pull-up evaluation method as described in any one of claims 1 to 5.

Images