Automatic detection method for rule violation actions in race walking video based on 2D human pose estimation

Abstract

The application discloses a race walking video rule violation action automatic detection method based on 2D human posture estimation and belongs to the technical field of computer vision and sports science, in particular to the following: a video 2D human posture estimation network based on space-time self-attention is constructed and trained, including a picture human posture estimation network, a joint feature embedding module and a space-time self-attention Transformer network with embedded time sequence block moving operation; expert classification of race walking video data sets is performed, input into the trained video 2D human posture estimation network, and the output joint node estimation two-dimensional coordinates are input into a logistic regression model together with the calculated left and right knee joint angles for training to obtain a logistic regression model for different rule violation actions, which is used for rule violation action automatic detection. The application uses the space-time self-attention mechanism to fuse the rich time information between video frames, obtains more accurate human two-dimensional coordinates, and then combines the logistic regression model to learn the race walking rule violation action, so that the automatic detection and classification of the race walking rule violation action are realized, and the detection accuracy is high.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and sports science, specifically relating to an automatic detection method for foul actions in race walking videos based on 2D human posture estimation. Background Technology

[0002] Race walking is a traditional track and field event. During race walking, athletes must adhere to two rules. First, athletes must always keep one foot on the ground; a violation of this rule is called loss of contact (LC). Second, from the moment the leading foot lands until the leg reaches a vertical position, the leg must remain straight, and the knee must not bend; otherwise, it is considered a knee-bent (KB). Currently, violations are primarily judged by coaches' visual observation. However, during the long duration of a race walk, some technical violations are difficult to observe. Therefore, ensuring objectivity and fairness in the judgment process is crucial. To achieve fairness and objectivity, professionals have conducted extensive research on the automatic detection of race walking violations. However, these methods require sensors to be installed on athletes, but these sensors can interfere with the athlete's normal posture, making them unsuitable for formal competitions.

[0003] With the rapid development of internet technology, artificial intelligence is ushering in a new wave. The continuous improvement of deep learning technology based on neural networks and related hardware technologies is profoundly impacting the world, playing an indispensable role in the development of many fields such as security, transportation, healthcare, and sports. Currently, the research and application of information and intelligent technologies in the field of sports and health are receiving increasing attention. Among these applications, motion analysis technology based on human posture estimation has attracted much attention. This technology aims to track and record human movement in real time without delaying the digitization process and has been widely used for quantitative purposes in several sports disciplines and activities, such as technical and competition evaluation, serving as a major data source for researchers to obtain knowledge of human biomechanics. In recent years, with the widespread adoption of deep learning technology, many revolutionary technologies have emerged in the field of label-free motion capture, leading to even more extensive research into deep learning-based human posture estimation technology.

[0004] Human pose estimation (HPE) is a crucial task in computer vision, aiming to identify the coordinates of key points (joints) of all people in an input image or video. This technology builds a computer image model by recording human movements, collecting and analyzing human motion information, and obtaining the contained human pose data and motion information. It tracks and records human motion in real time and has been widely applied in real-world scenarios such as human-computer interaction, healthcare, security surveillance, and sports.

[0005] 2D human pose estimation can be broadly categorized into two types based on the input data: image-based human pose estimation and video-based human pose estimation. Image-based human pose estimation methods (such as the Image Human Pose Estimation Network HRNet) already possess high prediction accuracy when processing static human figures in single images. However, in real-world scenarios, such as security surveillance and human-computer interaction, video data is crucial. When image-based human pose estimation methods are directly applied to video tasks, their performance significantly degrades. This is because video introduces challenges such as camera shift, motion blur, defocus, and occlusion, leading to a decrease in video frame quality. Furthermore, unlike image-based methods, video frames contain rich temporal information, such as temporal dependencies and geometric consistency, which can provide valuable information for human pose estimation. Therefore, a video-based human pose estimation method is urgently needed to achieve high-performance automatic detection of race walking violations. Summary of the Invention

[0006] To address the problems existing in traditional image-based human pose estimation, this invention provides an automatic detection method for race walking video fouls based on 2D human pose estimation. It utilizes a video 2D human pose estimation network based on spatiotemporal self-attention to learn rich temporal information between video frames and obtain accurate two-dimensional coordinates of key points, thereby achieving high-accuracy automatic detection of race walking fouls.

[0007] To achieve the above objectives, the technical method employed in this invention is as follows:

[0008] An automatic detection method for race walking video violations based on 2D human pose estimation includes the following steps:

[0009] Step 1: Construct a video 2D human pose estimation network based on spatiotemporal self-attention, including an image human pose estimation network, a joint feature embedding module, and a spatiotemporal self-attention Transformer network with embedded temporal block movement operations.

[0010] Using RGB videos from a video human pose estimation dataset as input, and the real 2D coordinates of the joints in each frame of the RGB video as the estimation target, a video 2D human pose estimation network based on spatiotemporal self-attention is trained to obtain the trained video 2D human pose estimation network based on spatiotemporal self-attention.

[0011] The image human pose estimation network is used to extract global features from each frame of the RGB video.

[0012] The joint feature embedding module is used to convert the global features of each frame into an N×P-dimensional token sequence composed of feature embeddings corresponding to N joint points; the N joint points include the left hip, right hip, left knee, right knee, left ankle, right ankle, and nose; P represents the feature dimension;

[0013] The spatiotemporal self-attention Transformer network with embedded temporal block shifting operation includes multiple stacked encoding and decoding blocks. Each encoding block includes, in sequence, a first layer normalization (LN), a temporal block shifting operation, a multi-head self-attention layer (MHSA), a temporal block reset operation, a second layer normalization, and a feedforward neural network layer (FFN). The input of the first encoding block is a token sequence with position embedding. The features output by the previous encoding block serve as the input of the next encoding block. The output of the last encoding block is processed by multiple decoding blocks to output the estimated two-dimensional coordinates of the key points for each frame. The temporal block shifting operation is used to fuse the token sequences of the M consecutive frames before and after each frame into an N×P dimensional fused feature for the corresponding frame. The temporal block reset operation is used to restore the fused feature to its original position.

[0014] Step 2: Obtain the race walking video dataset. Experts categorize all race walking videos into three types: standard race walking movements, KB foul movements, and LC foul movements, which will be used as the training set.

[0015] Step 3: Input the training set into the trained spatiotemporal self-attention-based video 2D human pose estimation network, output the estimated two-dimensional coordinates of the joints in each frame of each race walking video, and after the two-dimensional coordinates are standardized, calculate the left knee joint angle θ1 and the right knee joint angle θ2 based on the standardized estimated two-dimensional coordinates of the left hip, right hip, left knee, right knee, left ankle and right ankle.

[0016] Step 4: Construct two logistic regression models. The models are trained by taking the standardized estimated y-coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle of each frame in the race walking video corresponding to the KB foul and LC foul, as well as the left knee joint angle θ1 and the right knee joint angle θ2, as inputs. The result is a logistic regression model for the KB foul and a logistic regression model for the LC foul.

[0017] Step 5: The race walking video to be detected is input into the trained spatiotemporal self-attention-based video 2D human pose estimation network to obtain the estimated two-dimensional coordinates of the key points of each frame. After the two-dimensional coordinates are standardized, they are input into the logistic regression model for KB fouls or LC fouls. The output 0 indicates that the race walking video to be detected is the corresponding KB foul or LC foul, and the output 1 indicates that the race walking video to be detected is a standard race walking action.

[0018] Furthermore, M=9, N=15, and the 15 joints are the head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0019] Furthermore, the joint feature embedding module includes a convolutional layer, a flattening layer, and a feedforward neural network layer in sequence; wherein, the convolutional layer contains N convolutional kernels, the global features are processed by the convolutional layer into features of N joint points, flattened into a one-dimensional form by the flattening layer, and then input into the feedforward neural network layer to obtain the feature embedding of each joint point, that is, the token of each joint point, and then concatenated to obtain the token sequence.

[0020] Furthermore, for each frame, a 3×3 motion pattern window matrix is ​​used to fuse the token sequences of the corresponding preceding and following M=9 frames. The specific process is as follows:

[0021] Assume the current frame is frame t, and let Z be the token sequence of frame t. t =[z0,z1,z2,…,z] N ], where z n n = 1, 2, ..., N represents the feature embedding of the nth joint.

[0022] The token sequences of the first 4 frames and the last 4 frames of frame t are fused into the fused feature of frame t. The fusion formula is as follows:

[0023]

[0024] In the formula, Z represents the fusion feature of frame t; ⊙ represents the Hadamard product; i, i = 1, 2, ..., 9 represents the frame number of the token sequence to be fused in 9 consecutive frames; t+i-5 Represents the token sequence of frame t+i-5; A i This represents the movement matrix corresponding to the i-th frame out of 9 consecutive frames; A represents i The element value of the nth joint is either 0 or 1; I represents a 1×N dimensional matrix with all elements being 1.

[0025] The movement mode window matrix is:

[0026]

[0027] The value of each element represents the frame number i of the token sequence to be merged in 9 consecutive frames;

[0028] In a continuous N×P-dimensional token sequence of 9 frames, complete N×P-dimensional fusion is achieved by sliding a 3×3-dimensional motion pattern window matrix N×P / 9 times. Specifically, in each sliding motion pattern window matrix, based on the element position of the i-th frame in the motion pattern window matrix, the fusion feature of that element position is set as the feature embedding of the corresponding key point in the i-th frame, i.e., A i The element value corresponding to the key point is set to 1, and the other element values ​​are set to 0, therefore A i There are N / 3 elements with a value of 1 and 2N / 3 elements with a value of 0.

[0029] Furthermore, in step 1, when training the video 2D human pose estimation network based on spatiotemporal self-attention, the loss function Loss used is:

[0030]

[0031] In the formula, (x n ,y n (x) represents the standardized estimated two-dimensional coordinates of the nth joint; n * ,y n * ) represents the true two-dimensional coordinates of the nth joint.

[0032] Furthermore, the specific process of the two-dimensional coordinate standardization process described in step 3 is as follows:

[0033] The midpoint of the estimated y-coordinates between the left and right hips is used as the estimated y-coordinate of the buttocks; the scaling factor K is calculated based on the reciprocal of the vertical distance between the estimated y-coordinates of the buttocks and the nose; the standardized estimated 2D coordinates of the joints in each frame are multiplied by the scaling factor K to obtain the standardized estimated 2D coordinates of the joints.

[0034] Furthermore, the image human pose estimation network mentioned in step 1 is specifically HRNet.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] To address the challenges of motion blur and occlusion that often occur in race walking videos, this invention provides an automatic detection method for race walking videos based on 2D human pose estimation, building upon an image-based human pose estimation network. It utilizes a spatiotemporal self-attention mechanism to fuse rich temporal information between video frames, obtaining more accurate two-dimensional human coordinates. This is then combined with a logistic regression model to learn race walking fouls, achieving automatic detection and classification of race walking fouls with high accuracy. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a framework diagram of the video 2D human pose estimation network based on spatiotemporal self-attention proposed in Example 1;

[0039] Figure 2 This is a schematic diagram of the timing block movement operation process in Example 1, taking three consecutive frames as an example;

[0040] Figure 3 This is a schematic diagram of the structure of the coding block in the standard self-attention Transformer network in Example 1;

[0041] Figure 4 This is a schematic diagram of the structure of the coded block in the spatiotemporal self-attention Transformer network with embedded temporal block movement operation in Example 1. Detailed Implementation

[0042] To further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, and not for limiting the scope of the claims.

[0043] Example 1

[0044] This embodiment proposes an automatic detection method for foul actions in race walking videos based on 2D human pose estimation, including the following steps:

[0045] Step 1: Construct a video 2D human pose estimation network based on spatiotemporal self-attention, with the framework structure as follows: Figure 1 As shown, it includes, in sequence, an image human pose estimation network HRNet, a joint feature embedding module, and a spatiotemporal self-attention Transformer network with embedded temporal block movement operations;

[0046] Using RGB videos from a video human pose estimation dataset as input, and the real 2D coordinates of the joints in each frame of the RGB video as the estimation target, a video 2D human pose estimation network based on spatiotemporal self-attention is trained to obtain the trained video 2D human pose estimation network based on spatiotemporal self-attention.

[0047] The image human pose estimation network HRNet is used to extract global features from each frame of RGB video.

[0048] The joint feature embedding module includes a 1×1 convolutional layer, a flattening layer, and a feedforward neural network layer in sequence. The convolutional layer contains N convolutional kernels. The global features are processed by the convolutional layer into features of N joint points. After being flattened into a one-dimensional form by the flattening layer, the features are input to the feedforward neural network layer to obtain the 1×P-dimensional feature embedding of each joint point, i.e., the token of each joint point. Then, the tokens are concatenated to obtain an N×P-dimensional token sequence. P represents the feature dimension and takes a value of 1024. The N joint points are the head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0049] The spatiotemporal self-attention Transformer network with embedded temporal block movement operation includes multiple repeatedly stacked encoding blocks and decoding blocks; Figure 3 This is a schematic diagram of the structure of a coding block in a standard self-attention Transformer network. This embodiment embeds time block shifting and time block resetting operations within it. Specifically, as shown... Figure 4 As shown, each coding block includes, in sequence, a first-layer normalization, a time block shifting operation, a multi-head self-attention layer, a time block reset operation, a second-layer normalization, and a feedforward neural network layer;

[0050] The first coding block takes a token sequence with position embedding as input, i.e., the input dimension is N×(P+1). The features output by the previous coding block are used as the input of the next coding block. The output of the last coding block is processed by multiple decoding blocks to output the estimated two-dimensional coordinates of the key points in each frame.

[0051] The temporal block shifting operation is used to fuse the token sequence of each frame corresponding to the preceding and following M=9 frames into the corresponding frame's N×P dimensional fusion feature. Figure 2 This is a schematic diagram of the temporal block movement operation process with three consecutive frames as an example. By exchanging the feature blocks (token sequences) of multiple consecutive frames, the temporal information between consecutive frames is effectively fused.

[0052] For each frame, a 3×3 motion pattern window matrix is ​​used to fuse the token sequences of the corresponding nine consecutive frames. The specific process is as follows:

[0053] Assume the current frame is frame t, and let Z be the token sequence of frame t. t =[z0,z1,z2,…,z] N ], where z n n = 1, 2, ..., N represents the feature embedding of the nth joint.

[0054] The token sequences of the first 4 frames and the last 4 frames of frame t are fused into the fused feature of frame t. The fusion formula is as follows:

[0055]

[0056] In the formula, Z represents the fusion feature of frame t; ⊙ represents the Hadamard product; i, i = 1, 2, ..., 9 represents the frame number of the token sequence to be fused in 9 consecutive frames; t+i-5 Represents the token sequence of frame t+i-5; A i This represents the movement matrix corresponding to the i-th frame out of 9 consecutive frames; A represents i The element value of the nth joint is either 0 or 1; I represents a 1×N dimensional matrix with all elements being 1.

[0057] The 3×3 dimensional movement pattern window matrix is ​​as follows:

[0058]

[0059] The value of each element represents the frame number i of the token sequence to be merged in 9 consecutive frames;

[0060] In a continuous N×P-dimensional token sequence of 9 frames, complete N×P-dimensional fusion is achieved by sliding a 3×3-dimensional motion pattern window matrix N×P / 9 times. Specifically, in each sliding motion pattern window matrix, based on the element position of the i-th frame in the motion pattern window matrix, the fusion feature of that element position is set as the feature embedding of the corresponding key point in the i-th frame, i.e., A i The element value corresponding to the key point is set to 1, and the other element values ​​are set to 0, therefore A i There are N / 3 = 5 elements with a value of 1, and 2N / 3 = 10 elements with a value of 0.

[0061] The time block reset operation is used to restore the fused features to their original positions;

[0062] When training a video 2D human pose estimation network based on spatiotemporal self-attention, end-to-end optimization is employed to minimize the error between the normalized estimated 2D coordinates and the true 2D coordinates of the joints. The loss function used is:

[0063]

[0064] In the formula, (x n ,y n (x) represents the standardized estimated two-dimensional coordinates of the nth joint; n * ,y n * ) represents the true two-dimensional coordinates of the nth joint.

[0065] Step 2: Obtain the race walking video dataset, which was filmed by four race walkers. Each independent race walking video is 80 frames. The four race walkers filmed a total of 741 videos. All race walking videos were divided into three categories by experts (professional race walking coaches): standard race walking movements, KB (Best Kick) foul movements, and LC (Lowest Criterion) foul movements, and were labeled accordingly. Among them, there are 248 videos of standard race walking movements, 245 videos of KB foul movements, and 248 videos of LC foul movements.

[0066] In the categorized race walking video dataset, 70% was used as the training set and 30% as the test set.

[0067] Step 3: Input the training set into the trained spatiotemporal self-attention-based video 2D human pose estimation network, and output the estimated two-dimensional coordinates of the joints for each frame in each race walking video. Since the size of the human body in different frames of the video will change during the movement due to factors such as the shooting angle, the two-dimensional coordinates of the human joints in different frames need to be standardized. The specific process is as follows:

[0068] The midpoint of the estimated y-coordinates between the left and right hips is used as the estimated y-coordinate of the buttocks; the scaling factor K is calculated based on the reciprocal of the vertical distance between the estimated y-coordinates of the buttocks and the nose; the estimated two-dimensional coordinates of the joints in each frame are multiplied by the scaling factor K to obtain the standardized estimated two-dimensional coordinates of the joints.

[0069] Since race walking fouls generally only occur in the lower body of athletes, the two-dimensional coordinates of the lower body joints are very important, especially the y-coordinate. The two-dimensional coordinates of the upper body joints have a very limited impact on the model, so the upper body is not considered.

[0070] Meanwhile, the angle between the left and right knee joints is very helpful in judging fouls. Therefore, it is necessary to calculate the angle θ1 of the left knee joint and the angle θ2 of the right knee joint based on the standardized estimated two-dimensional coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0071] Step 4: Since the BK foul and LC foul have different sensitivities to each input feature, it is necessary to build and train logistic regression models for the two types of fouls respectively. Specifically, two logistic regression models are constructed and trained with the standardized estimated y-coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle of each frame in the race walking video corresponding to the KB foul and the LC foul, as well as the left knee joint angle θ1 and the right knee joint angle θ2, as inputs to obtain the logistic regression model for the KB foul and the logistic regression model for the LC foul.

[0072] Step 5: Use the test set to test and evaluate the automatic detection method for race walking video violations based on 2D human pose estimation proposed in this embodiment. Specifically:

[0073] The test set of race walking videos to be detected is input into the trained spatiotemporal self-attention-based video 2D human pose estimation network to obtain the estimated two-dimensional coordinates of the key points in each frame. After the two-dimensional coordinates are standardized, they are input into the logistic regression model for KB fouls or LC fouls. The output 0 indicates that the race walking video to be detected is the corresponding KB foul or LC foul, and the output 1 indicates that the race walking video to be detected is a standard race walking action.

[0074] The automatic detection results were compared and evaluated with the classification labels given by experts, and the evaluation results are shown in Table 1. It can be seen that the automatic detection accuracy of KB foul and LC foul is excellent.

[0075] Table 1

[0076]

[0077] The above embodiments are provided to better understand the present invention and are not limited to the preferred embodiments described. They do not constitute a limitation on the content and scope of protection of the present invention. Any product that is the same as or similar to the present invention, derived by any person under the guidance of the present invention or by combining the features of the present invention with other prior art, is within the scope of protection of the present invention.

Claims

1. A method for automatic detection of rule violation actions in race walking videos based on 2D human pose estimation, characterized in that, The method comprises the following steps: Step 1, constructing a video 2D human pose estimation network based on spatio-temporal self-attention, comprising a picture human pose estimation network, a joint feature embedding module and a spatio-temporal self-attention Transformer network with embedded time block moving operation in sequence; Taking the RGB video of a video human pose estimation dataset as input, the real two-dimensional coordinates of the joint nodes of each frame in the RGB video as the estimation target, training the video 2D human pose estimation network based on spatio-temporal self-attention to obtain the trained video 2D human pose estimation network based on spatio-temporal self-attention; The picture human pose estimation network is used to extract the global features of each frame in the RGB video; The joint feature embedding module is used to convert the global features of each frame into an N×P-dimensional token sequence composed of feature embeddings corresponding to N joint nodes; N = 15, and the 15 joint nodes are head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle; and P represents the feature dimension; The spatio-temporal self-attention Transformer network with embedded time block moving operation comprises a plurality of repeatedly stacked encoding blocks and decoding blocks, each encoding block comprises, in sequence, a first layer normalization, a time block moving operation, a multi-head self-attention layer, a time block reset operation, a second layer normalization and a feedforward neural network layer; wherein the input of the first encoding block is a token sequence with position embedding; the features output by the previous encoding block are used as the input of the subsequent encoding block, and the output of the last encoding block is processed by a plurality of decoding blocks to output the estimated two-dimensional coordinates of the joint nodes of each frame; the time block moving operation is used to fuse the token sequences of the corresponding previous and subsequent M = 9 frames into an N×P-dimensional fusion feature corresponding to the frame; and the time block reset operation is used to restore the fusion feature to the original position; For each frame, a 3×3-dimensional moving pattern window matrix is used to fuse the token sequences of the corresponding previous and subsequent M = 9 frames, and the specific process is as follows: Assume the current frame is frame t, and denote the token sequence of frame t as follows: ,in For the first Feature embedding of each key point; The token sequences of the previous 4 frames and the subsequent 4 frames of the t-th frame are fused into the fusion feature of the t-th frame, and the fusion formula is as follows: ； ； ； In the formula, represents the fusion feature of the tth frame; represents Hadamard product; represents the frame number of the token sequence to be fused in the consecutive 9 frames; represents the token sequence of the tth frame; represents the token sequence of the tth frame; represents the motion matrix corresponding to the tth frame in the consecutive 9 frames; represents the motion matrix corresponding to the tth frame in the consecutive 9 frames; represents the element value of the tth node in the motion matrix, which is specifically 0 or 1; represents the element value of the tth node in the motion matrix, which is specifically 0 or 1; represents the element value of the tth node in the motion matrix, which is specifically 0 or 1; represents the 1xN matrix with all elements being 1; The moving pattern window matrix is as follows: ； Wherein, the value of each element represents the frame sequence number of the token sequence to be fused in the continuous 9 frames ; In a continuous N×P-dimensional token sequence of 9 frames, complete N×P-dimensional fusion is achieved by sliding a 3×3-dimensional motion pattern window matrix N×P / 9 times. Specifically, in each sliding motion pattern window matrix, according to the... The element position of the frame in the motion mode window matrix, and the fusion feature of that element position is set as the first element. Feature embedding of frame corresponding to key points, that is The element value corresponding to the joint is set to 1, and the other element values ​​are set to 0. There are N / 3 elements with a value of 1 and 2N / 3 elements with a value of 0; Step 2, obtaining a race walking video dataset, dividing all race walking videos into three categories of standard race walking actions, KB rule violation actions and LC rule violation actions by experts as a training set; Step 3, input the training set into the trained spatio-temporal self-attention-based video 2D human pose estimation network, output the estimated 2D coordinates of each frame in each race video, after 2D coordinate standardization, according to the standardized estimated 2D coordinates of the left crotch, the right crotch, the left knee, the right knee, the left ankle and the right ankle, calculate the left knee joint angle and the right knee joint angle ; Step 4, construct two logistic regression models, respectively with KB violation actions and LC violation actions corresponding to the standardized estimated y coordinates of the left hip, right hip, left knee, right knee, left ankle, right ankle, and the left knee joint angle and right knee joint angle of each frame in the race video as input to train, get the logistic regression model for KB violation actions and the logistic regression model for LC violation actions; and right knee joint angle as input to train, get the logistic regression model for KB violation actions and the logistic regression model for LC violation actions; Step 5, inputting the race walking video to be detected into the trained video 2D human pose estimation network based on spatio-temporal self-attention to obtain the estimated two-dimensional coordinates of the joint nodes of each frame, inputting the two-dimensional coordinates after standardization into the logic regression model for KB rule violation actions or the logic regression model for LC rule violation actions, and outputting 0 to represent that the race walking video to be detected is the corresponding KB rule violation action or LC rule violation action, and outputting 1 to represent that the race walking video to be detected is a standard race walking action.

2. The method according to claim 1, wherein the method further comprises: determining the body posture of the runner in the frame of the video based on the 2D human pose estimation; and determining the body posture of the runner in the frame of the video based on the 3D human pose estimation. The joint feature embedding module comprises a convolution layer, a flattening layer and a feedforward neural network layer in sequence; wherein the convolution layer contains N convolution kernels, the global feature is processed by the convolution layer into the features of N joint nodes, after being flattened into one-dimensional form by the flattening layer, the features are input into the feedforward neural network layer to obtain the feature embedding of each joint node, i.e. the token of each joint node, and then the token sequence is obtained by splicing. 3.The method of claim 1, wherein, Step 1. The loss function used when training the spatio-temporal self-attention based video 2D human pose estimation network is: is: ； wherein denotes the normalized estimated two-dimensional coordinate of the denotes the real two-dimensional coordinate of the denotes the real two-dimensional coordinate of the 4. The method according to any one of claims 1-3, wherein, The specific process of the two-dimensional coordinate standardization processing in step 3 is as follows: Taking the estimated y-coordinate midpoint position between the left crotch and the right crotch as the estimated y-coordinate of the hip; then according to the reciprocal of the estimated vertical distance of the y-coordinate between the hip and the nose, a scaling factor K is calculated; Multiply the estimated two-dimensional coordinates of each frame of joint node by the scaling factor K to obtain the standardized estimated two-dimensional coordinates of the joint node.

5. The method according to any one of claims 1-3, wherein, The picture human body posture estimation network in step 1 is HRNet.

Images