Walking race video foul action automatic detection method based on 2D human body posture estimation

Abstract

The invention discloses a walking race video foul action automatic detection method based on 2D human body posture estimation, and belongs to the technical field of crossing of computer vision and sports science, and the method specifically comprises the steps: constructing and training a video 2D human body posture estimation network based on space-time self-attention; comprising a picture human body posture estimation network, a joint feature embedding module and a time-space self-attention Transform network embedded with time sequence block moving operation in sequence. Classifying a walking race video data set through an expert, inputting the classified walking race video data set into the trained video 2D human body posture estimation network, outputting joint point estimation two-dimensional coordinates, and inputting the joint point estimation two-dimensional coordinates and the calculated included angle between the joint point estimation two-dimensional coordinates and the left knee joint and the right knee joint into a logistic regression model for training to obtain the logistic regression model for different foul actions for automatic foul action detection. According to the method, rich time information between video frames is fused by utilizing a space-time self-attention mechanism, more accurate human body two-dimensional coordinates are acquired, the walking race foul actions are learned in combination with the logistic regression model, automatic detection and classification of the walking race foul actions are realized, and the detection accuracy is relatively high.

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of computer vision and sports science, and in particular relates to an automatic detection method for foul actions in race walking videos based on 2D human posture estimation. Background Art

[0002] Race walking is one of the traditional track and field events. During race walking, athletes must abide by two rules. First, athletes must always keep one foot on the ground, and the corresponding foul behavior is called loss of contact (LC). Second, from the moment the forward foot lands until the leg reaches a vertical position, the athlete's legs must remain straight and the knees cannot be bent, otherwise it will be considered knee-bent (KB). At present, foul actions are mainly judged by coaches' naked eye observation. During the long-term race walking process of athletes, some technical foul actions are difficult to observe. Therefore, it is crucial to ensure objectivity and fairness in the judgment process. In order to achieve fairness and objectivity, professionals have conducted a lot of research on the automatic detection of race walking foul actions. However, these methods require sensors to be installed on athletes, but the sensors will affect the normal posture of athletes, so they are not applicable in official competitions.

[0003] With the rapid development of Internet technology, artificial intelligence is setting off a new wave. The continuous improvement of deep learning technology based on neural networks and related hardware technology has profoundly affected the world and plays an indispensable role in the development of many fields such as security, transportation, medical care, and sports. At present, the research and application of information and intelligent technology in the field of sports health are increasingly valued. Among these applications, motion analysis technology based on human posture estimation has attracted much attention. This technology aims to track and record human movements in real time without delaying the digitization process. It has been widely used for quantitative purposes in several sports disciplines and activities, such as technical and competition evaluation. It is the main source of data to help researchers obtain knowledge of human motion mechanics. In recent years, with the widespread popularization of deep learning technology, many revolutionary technologies have been born in the field of markerless motion capture, and human posture estimation technology based on deep learning has been more widely studied.

[0004] 2D Human Pose Estimation (HPE) technology is an important task in the field of computer vision. Its purpose is to identify the key point (joint) position coordinates of all people in the input image or video. This technology records the human body's motion process, collects and analyzes human motion information to build its computer image model, and obtains the human body posture data and motion information it contains. It tracks and records the human body's motion status in real time and has been widely used in real-life scenarios such as human-computer interaction, medical health, monitoring security, and sports.

[0005] 2D human pose estimation can be divided into two categories according to the input data: image-based human pose estimation and video-based human pose estimation. Image-based human pose estimation methods (such as the image human pose estimation network HRNet) have achieved high prediction accuracy when processing static human bodies in a single image. In actual scenarios, such as surveillance and security, human-computer interaction, etc., video data must be processed. However, when image-based human pose estimation methods are directly used to process video tasks, the performance will be significantly reduced. This is because videos bring challenges such as camera shift, motion blur, defocus, and occlusion, which will lead to a decrease in video frame quality. In addition, unlike image-based human pose estimation methods, there is rich temporal information between video frames, such as temporal dependence and geometric consistency, which can provide valuable information for human pose estimation. Therefore, there is an urgent need for a video-based human pose estimation method to achieve automatic detection of race walking fouls with excellent performance. Summary of the invention

[0006] In view of the problems existing in traditional image-based human posture estimation, the present invention provides a method for automatic detection of race walking foul actions in videos based on 2D human posture estimation. A video 2D human posture estimation network based on spatiotemporal self-attention is used to learn the rich temporal information between video frames and obtain precise two-dimensional coordinates of joint points, so as to achieve high-accuracy automatic detection of race walking foul actions.

[0007] In order to achieve the above purpose, the technical method adopted by the present invention is as follows:

[0008] The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation includes the following steps:

[0009] Step 1: Build a video 2D human pose estimation network based on spatiotemporal self-attention, including a sequential image human pose estimation network, a joint feature embedding module, and a spatiotemporal self-attention Transformer network with embedded sequential block movement operations;

[0010] Taking the RGB video of the video human pose estimation dataset as input and the real two-dimensional coordinates of the joint points in each frame of the RGB video as the estimation target, the video 2D human pose estimation network based on spatiotemporal self-attention is trained to obtain the trained video 2D human pose estimation network based on spatiotemporal self-attention;

[0011] The image human posture estimation network is used to extract global features of each frame in the RGB video;

[0012] The joint feature embedding module is used to convert the global features of each frame into an N×P-dimensional token sequence consisting of feature embeddings corresponding to N joint points; the N joint points include the left hip, the right hip, the left knee, the right knee, the left ankle, the right ankle and the nose; P represents the feature dimension;

[0013] The spatiotemporal self-attention Transformer network with embedded temporal block shifting operation includes a plurality of repeatedly stacked encoding blocks and decoding blocks, each encoding block includes a first layer normalization (LN), a temporal block shifting operation, a multi-head self-attention layer (MHSA), a temporal block reset operation, a second layer normalization and a feedforward neural network layer (FFN) in sequence; wherein the input of the first encoding block is a token sequence with position embedding; the features output by the previous encoding block are used as the input of the next encoding block, and the output of the last encoding block is processed by a plurality of decoding blocks to output the estimated two-dimensional coordinates of the joint points of each frame; the temporal block shifting operation is used to fuse the token sequences of the consecutive M frames before and after each frame into the N×P-dimensional fused features of the corresponding frame; the temporal block reset operation is used to restore the fused features to the original position;

[0014] Step 2: Obtain a race walking video dataset, and have experts classify all race walking videos into three categories: standard race walking movements, KB foul movements, and LC foul movements, as a training set;

[0015] Step 3: Input the training set into the trained spatiotemporal self-attention-based video 2D human posture estimation network, output the estimated 2D coordinates of the joint points in each frame of each race walking video, and calculate the left knee joint angle θ according to the standardized estimated 2D coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle after the 2D coordinates are standardized. 1 and the right knee joint angle θ 2 ;

[0016] Step 4: Construct two logistic regression models, using the KB foul action and the LC foul action to correspond to the standardized estimated y coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle in each frame of the race walking video, as well as the left knee joint angle θ 1 and the right knee joint angle θ 2 The input is used for training, and a logistic regression model for KB's foul action and a logistic regression model for LC's foul action are obtained;

[0017] Step 5. The race walking video to be detected is input into the trained spatiotemporal self-attention-based video 2D human posture estimation network to obtain the estimated two-dimensional coordinates of the joint points of each frame. After the two-dimensional coordinates are standardized, they are input into the logistic regression model for the KB foul action or the logistic regression model for the LC foul action. The output 0 indicates that the race walking video to be detected is the corresponding KB foul action or LC foul action, and the output 1 indicates that the race walking video to be detected is the standard race walking action.

[0018] Furthermore, M=9, N=15, and the 15 joints are head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0019] Furthermore, the joint feature embedding module includes a convolution layer, a flattening layer and a feedforward neural network layer in sequence; wherein the convolution layer contains N convolution kernels, the global features are processed into features of N joint points by the convolution layer, flattened into a one-dimensional form by the flattening layer, and then input into the feedforward neural network layer to obtain the feature embedding of each joint point, that is, the token of each joint point, which is then spliced ​​to obtain a token sequence.

[0020] Furthermore, for each frame, a 3×3 dimensional moving pattern window matrix is ​​used to realize the fusion of the token sequence corresponding to the previous and subsequent M=9 consecutive frames. The specific process is as follows:

[0021] Assume that the current frame is the tth frame, and the token sequence of the tth frame is Z t =[z 0 ,z 1 ,z 2 ,…,z N ], where z n ,n=1,2,…,N is the feature embedding of the nth joint point;

[0022] The token sequences of the first 4 frames and the last 4 frames of the t-th frame are fused into the fusion features of the t-th frame. The fusion formula is:

[0023]

[0024] In the formula, represents the fusion feature of the tth frame; ⊙ represents the Hadamard product; i, i = 1, 2, ..., 9 represents the frame number of the token sequence to be fused in 9 consecutive frames; Z t+i-5 A represents the token sequence of the t+i-5th frame; i Represents the motion matrix corresponding to the i-th frame in 9 consecutive frames; Indicates A i The element value of the nth joint point in is 0 or 1; I represents a 1×N-dimensional matrix with all elements being 1;

[0025] The mobile mode window matrix is:

[0026]

[0027] The value of each element represents the frame number i of the token sequence to be fused in 9 consecutive frames;

[0028] In the N×P-dimensional token sequence of 9 consecutive frames, the complete fusion of N×P dimensions is completed by sliding the 3×3-dimensional moving pattern window matrix N×P / 9 times. Specifically, in each sliding moving pattern window matrix, according to the element position of the i-th frame in the moving pattern window matrix, the fusion feature of the element position is set as the feature embedding of the corresponding joint point of the i-th frame, that is, A i The element value corresponding to the joint point in is set to 1, and the other element values ​​are set to 0, so A i There are N / 3 elements with the value 1 and 2N / 3 elements with the value 0.

[0029] Furthermore, in step 1, when training the video 2D human pose estimation network based on spatiotemporal self-attention, the loss function Loss used is:

[0030]

[0031] In the formula, (x n ,y n ) represents the normalized estimated two-dimensional coordinates of the nth joint point; (x n * ,y n * ) represents the real two-dimensional coordinate of the nth joint point.

[0032] Furthermore, the specific process of the two-dimensional coordinate standardization processing in step 3 is:

[0033] The estimated y coordinate midpoint between the left hip and the right hip is used as the estimated y coordinate of the buttocks. The scaling factor K is calculated based on the inverse of the vertical distance between the estimated y coordinates of the buttocks and the nose. The estimated two-dimensional coordinates of the joint points of each frame are multiplied by the scaling factor K to obtain the standardized estimated two-dimensional coordinates of the joint points.

[0034] Furthermore, the image human posture estimation network described in step 1 is specifically HRNet.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] In view of the challenges such as motion blur and occlusion that often occur in race walking videos, the present invention provides an automatic detection method for race walking video foul actions based on 2D human posture estimation on the basis of the image human posture estimation network. The spatiotemporal self-attention mechanism is used to fuse the rich time information between video frames to obtain more accurate two-dimensional coordinates of the human body. Then, the logistic regression model is combined to learn the race walking foul actions, thereby realizing automatic detection and classification of race walking foul actions with high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is a framework diagram of the video 2D human posture estimation network based on spatiotemporal self-attention proposed in Example 1;

[0039] Figure 2 This is a schematic diagram of the sequential block movement operation process taking three consecutive frames as an example in Example 1;

[0040] Figure 3 Schematic diagram of the structure of the encoding block in the standard self-attention Transformer network in Example 1;

[0041] Figure 4 Schematic diagram of the structure of the encoding block in the spatiotemporal self-attention Transformer network with embedded temporal block shift operation in Example 1. DETAILED DESCRIPTION

[0042] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples, but it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the invention.

[0043] Example 1

[0044] This embodiment proposes a method for automatically detecting foul actions in a race walking video based on 2D human posture estimation, comprising the following steps:

[0045] Step 1: Build a video 2D human posture estimation network based on spatiotemporal self-attention. The framework structure is as follows: Figure 1 As shown, it includes the image human pose estimation network HRNet, the joint feature embedding module and the spatiotemporal self-attention Transformer network with embedded temporal block movement operation;

[0046] Taking the RGB video of the video human pose estimation dataset as input and the real two-dimensional coordinates of the joint points in each frame of the RGB video as the estimation target, the video 2D human pose estimation network based on spatiotemporal self-attention is trained to obtain the trained video 2D human pose estimation network based on spatiotemporal self-attention.

[0047] The image human posture estimation network HRNet is used to extract global features of each frame in the RGB video.

[0048] The joint feature embedding module includes a 1×1 convolution layer, a flattening layer and a feedforward neural network layer in sequence; wherein the convolution layer contains N convolution kernels, the global features are processed into features of N joint points by the convolution layer, flattened into a one-dimensional form by the flattening layer, and then input into the feedforward neural network layer to obtain a 1×P-dimensional feature embedding of each joint point, that is, a token of each joint point, and then spliced ​​to obtain an N×P-dimensional token sequence; P represents the feature dimension, and its value is 1024; the N joint points are respectively the head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0049] The spatiotemporal self-attention Transformer network with embedded temporal block shifting operation includes a plurality of repeatedly stacked encoding blocks and decoding blocks; Figure 3 is a schematic diagram of the structure of the encoding block in the standard self-attention Transformer network, in which the present embodiment embeds the time block shift operation and the time block reset operation. Specifically, Figure 4 As shown, each encoding block includes the first layer normalization, time block shift operation, multi-head self-attention layer, time block reset operation, second layer normalization and feedforward neural network layer in sequence;

[0050] The input of the first encoding block is a token sequence with position embedding, that is, the input dimension is N×(P+1); the features output by the previous encoding block are used as the input of the next encoding block, and the output of the last encoding block is processed by multiple decoding blocks to output the estimated two-dimensional coordinates of the joint points of each frame;

[0051] The sequential block shift operation is used to fuse the token sequences of M=9 consecutive frames before and after each frame into N×P-dimensional fusion features of the corresponding frame; Figure 2 The schematic diagram of the sequential block movement operation process is shown in Figure 1, taking three consecutive frames as an example. By exchanging the feature blocks (token sequences) of multiple consecutive frames, the temporal information between consecutive frames can be effectively integrated.

[0052] For each frame, a 3×3 dimensional moving pattern window matrix is ​​used to achieve the fusion of the token sequences corresponding to the previous and next 9 consecutive frames. The specific process is as follows:

[0053] Assume that the current frame is the tth frame, and the token sequence of the tth frame is Z t =[z 0 ,z 1 ,z 2 ,…,z N ], where z n ,n=1,2,…,N is the feature embedding of the nth joint point;

[0054] The token sequences of the first 4 frames and the last 4 frames of the t-th frame are fused into the fusion features of the t-th frame. The fusion formula is:

[0055]

[0056] In the formula, represents the fusion feature of the tth frame; ⊙ represents the Hadamard product; i, i = 1, 2, ..., 9 represents the frame number of the token sequence to be fused in 9 consecutive frames; Z t+i-5 A represents the token sequence of the t+i-5th frame; i Represents the motion matrix corresponding to the i-th frame in 9 consecutive frames; Indicates A i The element value of the nth joint point in is 0 or 1; I represents a 1×N-dimensional matrix with all elements being 1;

[0057] The 3×3 dimensional moving mode window matrix is:

[0058]

[0059] The value of each element represents the frame number i of the token sequence to be fused in 9 consecutive frames;

[0060] In the N×P-dimensional token sequence of 9 consecutive frames, the complete fusion of N×P dimensions is completed by sliding the 3×3-dimensional moving pattern window matrix N×P / 9 times. Specifically, in each sliding moving pattern window matrix, according to the element position of the i-th frame in the moving pattern window matrix, the fusion feature of the element position is set as the feature embedding of the corresponding joint point of the i-th frame, that is, A i The element value corresponding to the joint point in is set to 1, and the other element values ​​are set to 0, so A i There are N / 3=5 elements with the value of 1 and 2N / 3=10 elements with the value of 0.

[0061] The time block resetting operation is used to restore the fused features to their original positions;

[0062] When training the video 2D human pose estimation network based on spatiotemporal self-attention, end-to-end optimization is used to minimize the error between the standardized estimated 2D coordinates of the joint points and the true 2D coordinates. The loss function Loss is:

[0063]

[0064] In the formula, (x n ,y n ) represents the normalized estimated two-dimensional coordinates of the nth joint point; (x n * ,y n * ) represents the real two-dimensional coordinate of the nth joint point.

[0065] Step 2: Obtain a race walking video dataset, which was shot by four race walking athletes. Each independent race walking video has 80 frames. The four race walking athletes shot a total of 741 videos. Experts (race walking professional coaches) divided all the race walking videos into three categories: standard race walking movements, KB foul movements, and LC foul movements, and affixed classification labels. There are 248 standard race walking movement videos, 245 BK foul movement videos, and 248 LC foul movement videos.

[0066] In the classified race walking video dataset, 70% is taken as the training set and 30% is taken as the test set.

[0067] Step 3: Input the training set into the trained spatiotemporal self-attention-based 2D human posture estimation network, and output the estimated 2D coordinates of the joints of each frame in each race walking video. Since the size of the human body in different frames in the video will change due to shooting angles and other reasons during the movement, the 2D coordinates of the human joints in different frames need to be standardized. The specific process is as follows:

[0068] The estimated y coordinate midpoint between the left hip and the right hip is used as the estimated y coordinate of the buttocks; the scaling factor K is calculated based on the inverse of the vertical distance between the estimated y coordinates of the buttocks and the nose; the estimated two-dimensional coordinates of the joint points of each frame are multiplied by the scaling factor K to obtain the standardized estimated two-dimensional coordinates of the joint points;

[0069] Since race walking fouls usually only occur in the lower body of the athlete, the two-dimensional coordinates of the lower body joints are very important, especially the y coordinates. The two-dimensional coordinates of the joints of the upper body have a very limited impact on the model, so the upper body is not considered;

[0070] At the same time, the left and right knee joint angles are very helpful for judging foul actions. Therefore, it is necessary to calculate the left knee joint angle θ based on the standardized estimated two-dimensional coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle.1 and the right knee joint angle θ 2 .

[0071] Step 4: Since the BK foul action and the LC foul action have different sensitivities to each input feature, it is necessary to establish a logistic regression model for each of the two foul actions for training. Specifically, two logistic regression models are constructed, and the standardized estimated y coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle in each frame of the race walking video are used for the KB foul action and the LC foul action, respectively, as well as the left knee joint angle θ. 1 and the right knee joint angle θ 2 The input is used for training, and a logistic regression model for KB's foul action and a logistic regression model for LC's foul action are obtained;

[0072] Step 5: Use the test set to test and evaluate the automatic detection method for foul actions in race walking videos based on 2D human posture estimation proposed in this embodiment. Specifically:

[0073] The race walking video to be detected in the test set is input into the trained video 2D human posture estimation network based on spatiotemporal self-attention to obtain the estimated two-dimensional coordinates of the joint points of each frame. After the two-dimensional coordinates are standardized, they are input into the logistic regression model for KB foul action or the logistic regression model for LC foul action. The output 0 indicates that the race walking video to be detected is the corresponding KB foul action or LC foul action, and the output 1 indicates that the race walking video to be detected is the standard race walking action.

[0074] The automatic detection results are compared and evaluated with the classification labels given by experts, and the evaluation results shown in Table 1 are obtained. It can be seen that the automatic detection accuracy of KB foul actions and LC foul actions is very good.

[0075] Table 1

[0076]

[0077] The above embodiments are intended to provide a better understanding of the present invention, but are not limited to the optimal implementation mode and do not constitute a limitation on the content and protection scope of the present invention. Any product identical or similar to the present invention obtained by anyone under the inspiration of the present invention or by combining the present invention with the features of other prior arts shall be within the protection scope of the present invention.

Claims

1. An automatic detection method for foul actions in race walking videos based on 2D human posture estimation, characterized in that: The following steps are involved: Step 1: Build a video 2D human pose estimation network based on spatiotemporal self-attention, including a sequential image human pose estimation network, a joint feature embedding module, and a spatiotemporal self-attention Transformer network with embedded sequential block movement operations; Taking the RGB video of the video human pose estimation dataset as input and the real two-dimensional coordinates of the joint points in each frame of the RGB video as the estimation target, the video 2D human pose estimation network based on spatiotemporal self-attention is trained to obtain the trained video 2D human pose estimation network based on spatiotemporal self-attention; The image human posture estimation network is used to extract global features of each frame in the RGB video; The joint feature embedding module is used to convert the global features of each frame into an N×P-dimensional token sequence consisting of feature embeddings corresponding to N joint points; the N joint points include the left hip, the right hip, the left knee, the right knee, the left ankle, the right ankle and the nose; P represents the feature dimension; The spatiotemporal self-attention Transformer network with embedded temporal block shifting operation includes a plurality of repeatedly stacked encoding blocks and decoding blocks, each encoding block includes a first layer normalization, a temporal block shifting operation, a multi-head self-attention layer, a temporal block reset operation, a second layer normalization and a feedforward neural network layer in sequence; wherein the input of the first encoding block is a token sequence with position embedding; the features output by the previous encoding block are used as the input of the next encoding block, and the output of the last encoding block is processed by a plurality of decoding blocks to output the estimated two-dimensional coordinates of the joint points of each frame; the temporal block shifting operation is used to fuse the token sequences of the consecutive M frames before and after each frame into the N×P-dimensional fused features of the corresponding frame; the temporal block reset operation is used to restore the fused features to the original position; Step 2: Obtain a race walking video dataset, and have experts classify all race walking videos into three categories: standard race walking movements, KB foul movements, and LC foul movements, as a training set; Step 3, input the training set into the trained spatiotemporal self-attention-based video 2D human posture estimation network, output the estimated two-dimensional coordinates of the joint points in each frame of each race walking video, and after the two-dimensional coordinates are standardized, calculate the left knee joint angle θ1 and the right knee joint angle θ2 according to the standardized estimated two-dimensional coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle; Step 4, construct two logistic regression models, respectively using the standardized estimated y coordinates of the left hip, right hip, left knee, right knee, left ankle, and right ankle in each frame of the race walking video corresponding to the KB foul action and the LC foul action, as well as the left knee joint angle θ1 and the right knee joint angle θ2 as input for training, to obtain a logistic regression model for the KB foul action and a logistic regression model for the LC foul action; Step 5. The race walking video to be detected is input into the trained spatiotemporal self-attention-based video 2D human posture estimation network to obtain the estimated two-dimensional coordinates of the joint points of each frame. After the two-dimensional coordinates are standardized, they are input into the logistic regression model for the KB foul action or the logistic regression model for the LC foul action. The output 0 indicates that the race walking video to be detected is the corresponding KB foul action or LC foul action, and the output 1 indicates that the race walking video to be detected is the standard race walking action.

2. According to claim 1, the method for automatically detecting foul actions in race walking videos based on 2D human posture estimation is characterized in that: M=9, N=15, the 15 joints are head, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

3. The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation according to claim 2 is characterized in that: The joint feature embedding module includes a convolution layer, a flattening layer and a feedforward neural network layer in sequence; wherein the convolution layer contains N convolution kernels, the global features are processed into features of N joint points by the convolution layer, flattened into a one-dimensional form by the flattening layer, and then input into the feedforward neural network layer to obtain the feature embedding of each joint point, that is, the token of each joint point, and then spliced ​​to obtain a token sequence.

4. The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation according to claim 2, characterized in that: For each frame, a 3×3 dimensional moving pattern window matrix is ​​used to realize the fusion of the token sequence corresponding to the previous and subsequent M=9 consecutive frames. The specific process is as follows: Assume that the current frame is the tth frame, and the token sequence of the tth frame is Z t =[z0,z1,z2,…,z N ], where z n ,n=1,2,…,N is the feature embedding of the nth joint point; The token sequences of the first 4 frames and the last 4 frames of the t-th frame are fused into the fusion features of the t-th frame. The fusion formula is: In the formula, represents the fusion feature of the tth frame; ⊙ represents the Hadamard product; i, i = 1, 2, ..., 9 represents the frame number of the token sequence to be fused in 9 consecutive frames; Z t+i-5 A represents the token sequence of the t+i-5th frame; i Represents the motion matrix corresponding to the i-th frame in 9 consecutive frames; Indicates A i The element value of the nth joint point in is 0 or 1; I represents a 1×N-dimensional matrix with all elements being 1; The mobile mode window matrix is: The value of each element represents the frame number i of the token sequence to be fused in 9 consecutive frames; In the N×P-dimensional token sequence of 9 consecutive frames, the complete fusion of N×P dimensions is completed by sliding the 3×3-dimensional moving pattern window matrix N×P / 9 times. Specifically, in each sliding moving pattern window matrix, according to the element position of the i-th frame in the moving pattern window matrix, the fusion feature of the element position is set as the feature embedding of the corresponding joint point of the i-th frame, that is, A i The element value corresponding to the joint point in is set to 1, and the other element values ​​are set to 0, so A i There are N / 3 elements with the value 1 and 2N / 3 elements with the value 0.

5. The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation according to claim 2, characterized in that: Step 1: When training the video 2D human pose estimation network based on spatiotemporal self-attention, the loss function Loss used is: In the formula, (x n ,y n ) represents the normalized estimated two-dimensional coordinates of the nth joint point; (x n * ,y n * ) represents the real two-dimensional coordinate of the nth joint point.

6. The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation according to any one of claims 1 to 5, characterized in that: The specific process of the two-dimensional coordinate standardization process in step 3 is as follows: The estimated y coordinate midpoint between the left hip and the right hip is used as the estimated y coordinate of the buttocks; and the scaling factor K is calculated according to the inverse of the vertical distance between the estimated y coordinate of the buttocks and the nose; Multiply the estimated two-dimensional coordinates of the joint points of each frame by the scaling factor K to obtain the standardized estimated two-dimensional coordinates of the joint points.

7. The method for automatically detecting foul actions in race walking videos based on 2D human posture estimation according to any one of claims 1 to 5, characterized in that: The image human posture estimation network described in step 1 is specifically HRNet.

Images