A multi-keypoint detection-based multi-person sports event counting method, system and medium

Abstract

The application discloses a kind of multi-person sports project counting method, system and medium based on multi-key point detection, and the multi-person sports project counting method based on multi-key point detection includes the video stream data of detection site is acquired in real time by camera, the video frame in video stream is preprocessed to obtain the final video frame for key point detection, the bounding box position of personnel to be detected in video frame is detected and the image area for key point detection is intercepted, personnel in video frame is tracked simultaneously, the coordinate value of 33 key points in human body is detected, and the counting of corresponding project is completed for different sports projects.The application can accurately identify the key point information of personnel to be detected under different resolutions, different illumination intensities and different site backgrounds, and accurately complete the counting work of corresponding project, with the advantages of high accuracy, good generalization and strong expansibility, and can be widely applied in counting, timing and other work related to sports.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multi-person sports event counting method, device and medium based on multi-key point detection. Background Art

[0002] Sports counting (timing) is widely used in various scenarios, such as competitions and sports exams. Current counting methods can be roughly divided into three categories: the first is traditional manual counting, which is time-consuming and labor-intensive, and is subject to subjective judgment by the counter; the second is the use of multiple sensor-based physical devices to assist in counting, which can significantly save labor costs but still requires the purchase of a large amount of physical equipment, which is relatively expensive; and the last is a pure vision solution based on deep neural networks. This solution only requires physical equipment that can record video of the detection site and computing equipment to achieve a highly accurate and robust counting solution.

[0003] Current computer vision-based counting solutions have certain issues. For example, most only support single-person detection or detect a small number of key points on the person, failing to meet the requirements of item counting. There are two mainstream approaches for detecting key points on people. One is a top-down approach, which first detects a bounding box around the person and then performs key point detection on the person within the detected box. This approach can detect key points on multiple people, but existing solutions detect a small number of key points. For example, YOLO-POSE only detects 17 key points, which cannot meet the requirements of sophisticated algorithms. Furthermore, its model structure is weak in detecting small objects, making it difficult to detect multiple key points. The other is a bottom-up approach, which directly identifies the key points of people in the image and selects the most likely key points to form a human structure. This approach can detect a large number of key points, but it cannot detect multiple people. Furthermore, multi-person detection requires determining the identity information of each person to be detected. Each person should have a unique and unchanging ID. Therefore, relying solely on key point detection algorithms is difficult to achieve accurate multi-person detection and counting. Summary of the Invention

[0004] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems in the prior art, a method, system and medium for counting multi-person sports events based on multi-key point detection are provided. The present invention can accurately complete the counting (timing) of corresponding multi-person sports events under different resolutions, different venue backgrounds and different light intensities. It has the advantages of high accuracy and good generalization, and can be widely used in sports events, sports examinations and other scenarios.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A multi-player sports event counting method based on multi-keypoint detection includes the following steps:

[0007] 1) Get the video stream data of the venue.

[0008] 2) Preprocessing the video frames in the video stream to obtain the final video frames for key point detection, wherein the preprocessing of the video frames includes sampling the video frames, pre-storing the video frames and possible scaling operations on the video frames.

[0009] 3) Detect the bounding box position of the person to be detected in the video frame and intercept the image area for key point detection. The bounding box position is detected using a target detection model based on YOLO V8. This model has a fast inference speed and can meet the requirements of real-time detection. For the identified area of ​​the person to be detected, on the one hand, it is necessary to intercept the area for subsequent key point detection, and on the other hand, it is necessary to use it for subsequent target tracking tasks. After completing target detection, the person to be detected needs to be tracked. This part uses the DeepSort algorithm to assign a unique and unchanging ID to each person to be detected to facilitate subsequent counting operations.

[0010] 4) Detect the coordinates of 33 key points on the human body, including 11 on the head, 10 on the arms (on both sides), 4 on the torso, 2 on the legs (on both sides), and 6 on the feet (on both sides), using the posture detection module of mediapipe;

[0011] 5) Count the number of completed items for different sports. Taking the sit-up count as an example, the four states of the person to be tested are defined: preparation state, supine state, sit-up state, and end state. The corresponding rules are used to determine the transition between states. When the person to be tested completes the state transition from supine state to sit-up state to supine state, it is considered that he has completed one sit-up action.

[0012] A multi-person sports event counting system based on multi-key point detection includes a computer device that is programmed or configured to execute the steps of any one of the multi-person sports event counting methods based on multi-key point detection described above, or a storage medium of the computer device stores a computer program that is programmed or configured to execute any one of the multi-person sports event counting methods based on multi-key point detection described above.

[0013] A computer-readable storage medium stores a computer program programmed or configured to execute any one of the above-mentioned methods for counting multi-person sports events based on multi-key point detection.

[0014] Compared with the existing technology, the present invention has the following advantages: it pre-processes the video stream to prevent the problem of video stream buffer overflow when the processing speed is insufficient; it realizes multi-key point detection of multiple people, while taking into account the efficiency of key point detection and improving the precision and accuracy of the counting algorithm; at the same time, due to the use of DeepSort for target tracking, it can effectively avoid tracking failures caused by instantaneous occlusion, and can accurately complete the counting of multiple people in sports events under different venue backgrounds, light intensities, and number of people. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0016] Figure 2 Schematic diagram of the basic flow of the tracking algorithm according to an embodiment of the present invention.

[0017] Figure 3 This is the network architecture diagram of the DeepSort algorithm feature extraction network.

[0018] Figure 4 This is a key point annotation diagram for key point detection of the human body according to an embodiment of the present invention.

[0019] Figure 5 This is a state transition diagram of the sit-up counting algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] like Figure 1 As shown, the implementation steps of the multi-person sports event counting method based on multi-key point detection in this embodiment include:

[0021] 1. Obtain the site's video stream data. This data is typically obtained from a camera that supports wired or wireless video streaming, typically supporting RTSP, RTMP, or encoded video streams. This method also supports local cameras, and then counts (times) the stored video to be detected.

[0022] 2. Preprocess the video frames in the video stream to obtain the final video frames for key point detection. The detailed steps of the preprocessing include:

[0023] 2.1) The video frames in the original video stream are uniformly sampled. For video streams with a speed higher than 30 FPS, the frame rate after sampling is 20 frames per second.

[0024] 2.2) Create a data structure called FramePool. Its size is determined by the specific computing device's hardware requirements, such as memory. The video frames sampled in 2.1) are preferentially stored in this FramePool. When the FramePool runs out of capacity and a new video frame arrives, the first video frame in the FramePool is deleted and the newly arrived video frame is stored in the FramePool. Subsequent processes will then retrieve the first video frame from the FramePool for processing. This process is asynchronous, with a corresponding locking mechanism to ensure correct access to the FramePool.

[0025] 2.3) For video frames sampled from the FramePool, if their resolution is higher than 1280×720, they are scaled to 1280×720 for ease of subsequent processing. This preprocessing effectively avoids video stream buffer overflows caused by insufficient computing power on the device.

[0026] 3. Detect the bounding box position of the person to be detected in the video frame and intercept the image area for key point detection. At the same time, track the person to be detected. The specific implementation steps are as follows:

[0027] 3.1) A YOLO V8-based object detection model is used to detect people in video frames. The model's Backbone uses the C2f module as its basic building block. Specifically, it consists of five convolutional modules and four convolutional-to-fractal modules. Compared to the previous generation C3 module, it has fewer parameters and better feature extraction capabilities, making it more suitable for real-time detection of video stream data. Its Neck utilizes multi-scale feature fusion technology to fuse feature maps from different stages of the Backbone to enhance feature representation capabilities. This component plays a key role in feature extraction and fusion. The Head is responsible for the final object detection and classification tasks and includes a detection head and a classification head. The detection head comprises a series of convolutional and deconvolutional layers to generate detection results, while the classification head uses global average pooling to classify each feature map. Using the resulting bounding box information, the key point detection area in the image, i.e., the area containing the person to be detected, is cropped.

[0028] 3.2) Based on the bounding box information obtained in 3.1) and the captured key point detection area image, use the DeepSort algorithm to track the target person in the video frame. The specific implementation steps are as follows: Figure 2 As shown: Feature extraction is performed on the image captured in 3.1), and the feature will be used for cascade matching later. The structure of the feature extraction network is as follows Figure 3As shown: The pre-convolutional layer consists of 2D convolution and a batch normalization layer. After ReLU and max pooling operations, it is processed by the subsequent convolutional layer composed of basic blocks. The basic block consists of two 2D convolutional layers connected to a batch normalization layer, using ReLU as the activation function. In particular, the basic block provides optional downsampling. The backbone network consists of 8 basic blocks and an average pooling layer, where each two basic blocks form a group. If a group of basic blocks is downsampled, only the first basic block in the group is downsampled, that is, the 3rd, 5th, and 7th basic blocks in the current structure are downsampled. The classifier consists of a linear layer and a batch normalization layer. After ReLU activation and dropout operations, it is connected to a final classification linear layer to obtain the result. The extracted features are used for subsequent cascade matching.

[0029] The bounding box trajectory, or trace, has two states: confirmed and unconfirmed. An unconfirmed trajectory can be converted to a confirmed trajectory under certain conditions. The subsequent matching process uses different processing rules for unconfirmed and confirmed trajectories. Each existing trajectory is predicted using a Kalman filter (KF).

[0030] For the first frame where the bounding box of a person is detected, the bounding box is initialized as the original trajectory. All trajectories are in an unconfirmed state when they are first generated. For the unconfirmed trajectory, its KF predicted trajectory will be IOU matched with the detection box of the subsequent video frame. The match will produce three matching results through the Hungarian algorithm. If the trajectory is not successfully matched, the trajectory will be deleted (since the trajectory is in an unconfirmed state, it can be deleted directly). If the detection box is not matched successfully, the detection box will be initialized to a new trajectory. If the two match successfully, the trajectory will be updated according to the position of the detection box.

[0031] If a track matches a detected bounding box multiple times in a row, the track is transformed into a confirmed state. The number of consecutive matches is a variable threshold, usually set to 3. For confirmed tracks, its KF predicted track will be cascade matched with the detection box in the subsequent video frame. This cascade match will use the features extracted by the deep neural network mentioned above. Similarly, this match will have two results. If the detection box does not match or the track does not match, a subsequent IOU match will be performed. If the match is successful, the track information is updated based on the detection box position. For confirmed tracks, if the IOU match fails, the track will not be deleted directly. Instead, it will be deleted after multiple consecutive matching failures. The number of consecutive failures is a variable threshold, usually set to 30.

[0032] 4. Detect the coordinate values ​​of 33 key points on the human body, see the attached Figure 4, including 11 for the head, 10 for the arms (on both sides), 4 for the torso, 2 for the legs (on both sides), and 6 for the feet (on both sides); the specific key point parameters are 0-nose、1-left eye(inner)、2-left eye、3-left eye(outer)、4-right eye(inner)、5-right eye、6-righteye(outer)、7-left ear、8-right ear、9-mouth(left)、10-mouth(right)、11-leftshoulder、12-right shoulder、13-left elbow、14-right elbow、15-left wrist、16-right wrist、17-leftpinky、18-rightpinky、19-left index、20-right index、21-leftthumb、22-right thumb、23-lefthip、24-righthip、25-left knee、26-right knee、27-left ankle、28-right ankle、29-leftheel、30-right heel、31-left foot index, 32-right foot index. This key point detection model uses the pose module of mediapipe. Compared with mainstream models such as YOLO-POSE, this module recognizes more key points and has a faster recognition speed. The subdivision of the key points of the human foot gives it a unique advantage when processing some projects that require fine foot judgment. Its disadvantage is that it uses a bottom-up model architecture, that is, it identifies all possible human key points in the image and selects several key points with the highest confidence to form the human skeletal structure. Therefore, the model only supports single-person human key point detection. When there are multiple people to be detected in the image, the model only outputs the key point of the "most credible" person to be detected. Therefore, to address the above problems, this method uses this model in combination with target detection and target tracking to achieve multi-key point and fast multi-person skeletal key point recognition.

[0033] 5. For different sports, the corresponding counting algorithms are used to complete the counting of the corresponding items. The following takes the counting of sit-ups as an example. See the attached Figure 5 , the specific steps of this embodiment are as follows:

[0034] 5.1) Define the status of the person to be tested. There are four states in total, namely, the ready state, the supine state, the sitting state, and the end state. The specific definitions of each state are as follows. Ready state: The person appears in the picture and is detected by the target detection model, but has not started the sit-up exercise; Supine state: The person lies in the test position, with the torso horizontal, the shoulder blades touching the ground, the legs naturally bent, the hands placed on both sides of the body close to the ground, and the soles of the feet touching the ground; Sit-up state: The tester relies on the strength of the abdomen to lift the upper body to a certain angle, and at the same time moves the fingers forward to the standard line position. During the process, the feet and buttocks are prohibited from leaving the ground; End state: When the person to be tested stands up from the supine state or the sitting state, it enters the end state. After entering the end state, no counting judgment will be performed.

[0035] 5.2) Ready state determination: After the person to be detected appears in the video screen and is detected by the aforementioned target detection model, the tracking algorithm assigns a unique ID to the person and enters the ready state. At this time, the key points of the human skeleton of the person to be detected are identified.

[0036] 5.3) Detected keypoints will be screened for confidence before subsequent use. Keypoints with confidence levels below a set threshold will not be used. For data with bilateral keypoints, if both keypoints are greater than the set threshold, their average value will be taken to reduce misjudgments due to errors in the keypoint detection model.

[0037] 5.4) Supine state determination: The person to be tested should be in a horizontal lying state rather than an upright state. At this time, the x-direction coordinate difference between the key point of the head and the key point of the foot should be greater than the y-direction coordinate difference, and the x-direction coordinate difference between the key point of the hip and the key point of the foot should be greater than the y-direction coordinate difference. The key point of the head uses the coordinates of the key point of the nose position, and the key point of the foot uses the average value of the coordinates of the three key points of the foot, namely the ankle, heel and toe. At the same time, it is necessary to ensure that the person is lying horizontally on the ground, and the shoulder blades are close to the ground, that is, the angle formed by the three key points of the head, hip and foot is close to 180 degrees; the legs are naturally bent, and the angle formed by the three key points of the hip, knee and foot is close to 90 degrees; the feet touch the ground, that is, while meeting the requirements of lying horizontally, the angles of the three key points of the toes, heels and hips are close to 180 degrees. Among them, the method for calculating the angle based on the coordinates of the three points is

[0038]

[0039] Note: x2 and y2 are the coordinates of the key points of the vertex of the angle.

[0040] 5.5) Sitting-up position determination: After the person being tested enters the supine position, the determination is made as to whether the person has entered the sitting-up position. First, check whether the frames in the sitting-up process show any phenomenon such as hip lifting or foot lifting. If so, the count will be invalidated. When the character enters the supine state, the coordinates of the character's hip and foot key points will be recorded. The changes in the y-direction coordinates of the character's hip and foot during the standing-up process will be used to determine whether the above-mentioned errors have occurred. Foot lifting includes instep lifting and foot lifting. For instep lifting and foot lifting, it is only necessary to compare the average value of the three key points of the foot, namely the ankle, heel, and toe, with the average value of the initially recorded foot key points to simultaneously determine whether the above-mentioned two errors have occurred; if no such errors occur, detect the character's standing-up angle, and determine the character's standing-up angle based on the angle formed by the three key points of the shoulder, hip, and foot. When the angle meets a certain threshold, it is considered to have met the requirements, which is usually set to 150 degrees, that is, the angle between the upper body of the person to be tested and the ground is greater than 30 degrees; finally, detect whether the hand has moved to the standard line, and determine whether the hand has crossed the line based on the finger coordinates and the standard line coordinate position.

[0041] 5.6) End state determination: If the current state of the person to be tested is supine or sitting up, and the current person no longer meets the horizontal lying state described in 5.4), that is, the person to be tested has stood up, then the current person enters the end state, and no counting determination will be performed after entering the end state.

[0042] 5.7) When the person to be tested goes from the supine position to the sit-up position and then back to the supine position, it is considered that he has completed one sit-up movement, and the person's sit-up count is increased accordingly. It should be noted that the supine position can be obtained by changing from the ready position and the sit-up position, the sit-up position can only be obtained by changing from the supine position, and the end state can be obtained by changing from the supine position and the sit-up position.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-player sports event counting method based on multi-keypoint detection, characterized in that: include: 1) Obtain the video stream data of the venue; 2) Preprocess the video frames in the video stream to obtain the final video frames for key point detection; 3) Detect the bounding box position of the person to be detected in the video frame and intercept the image area for key point detection, while tracking the person to be detected; 4) Detect the coordinates of 33 key points on the human body, including 11 on the head, 10 on both sides of the arms, 4 on the torso, 2 on both sides of the legs, and 6 on both sides of the feet; 5) Complete the corresponding counts for different sports; Step 2) includes: 2.1) Uniformly sample the video frames. The sampling step size is set based on the actual computing device and the original video stream frame rate. Typically, for video streams with a frame rate greater than or equal to 30, the sampled video frames are 20 frames per second. 2.2) Establish a FramePool to store sampled video frames. Set the size of the FramePool based on the actual computing device. When the FramePool is full and a new frame enters, discard the first frame that enters the FramePool. 2.3) The processing thread continuously retrieves the earliest frames from the FramePool, resizes them, and then performs subsequent processing. Specifically, video frames with a resolution greater than 1280×720 will be scaled to 1280×720, and frames with a resolution smaller than 1280×720 will not be processed. This pre-processing effectively avoids video stream buffer overflows caused by insufficient computing power on the computing device. Step 3) involves using a pre-trained object detection model to detect the video frames to locate the person to be detected, and then using a tracking algorithm to track the detected person. This involves determining the bounding box and unique ID of each person to be detected in the video. The unique ID is assigned by the system and is used only for target tracking. The pre-trained target detection model uses a YOLO V8-based deep convolutional neural network model to perform target detection on the video frames obtained in step 2.3). The BackBone of this model uses the C2f module as its basic building block. Compared with the previous generation C3 module, it has fewer parameters and better feature extraction capabilities, which is more in line with the needs of real-time detection of video stream data. Its Neck uses multi-scale feature fusion technology to fuse feature maps from different stages of the Backbone to enhance feature representation capabilities. The Head part is responsible for the final target detection and classification tasks, including a detection head and a classification head. The detection head contains a series of convolutional layers and deconvolution layers for generating detection results. The classification head uses global average pooling to classify each feature map. The bounding box information obtained is used to crop the key point detection area in the image, that is, the area containing the person to be detected. The target tracking of the detected personnel is performed using the DeepSort algorithm, and the steps include: s1) Extract features from the captured image, which will be used for cascade matching later; s2) The person detection frame in the first processed frame containing several person detection frames is set as the initial trajectory; s3) Use Kalman filtering to predict the person's trajectory. The trajectory is divided into confirmed and unconfirmed states. For unconfirmed trajectory boxes, the predicted person's bounding box is matched with the subsequently detected ground truth for IOU matching. The Hungarian algorithm is used to obtain a linear matching result. If the predicted trajectory does not match, the trajectory is deleted. If the detected box does not match, a new trajectory is generated, indicating that a new person has been detected. If there is a match, the trajectory information is updated. For confirmed trajectories, cascade matching is performed with the detected bounding box. In this step, the appearance features and trajectory information extracted in s1) are simultaneously used. If the match is successful, the trajectory is updated. Trajectories that fail to match continue to undergo subsequent IOU matching. s4) For a trajectory in the unconfirmed state, if it is successfully matched several times in a row, it can be converted to the confirmed state; s5) For a confirmed trajectory, if the number of IOU matching failures is greater than the set threshold, the trajectory is deleted; otherwise, the trajectory is put back into the trajectory set and waits for the next match; For sit-ups, the steps are as follows: c1) Define the status of the person to be tested: Ready state: The person appears on the screen but has not yet started the sit-up exercise; Supine state: The person lies in the test position, with the torso horizontal, shoulder blades touching the ground, legs naturally bent, hands placed on both sides of the body close to the ground, and feet touching the ground; Sitting state: The tester uses abdominal strength to lift the upper body to a certain angle, while moving the fingers forward to the standard line position. During this process, the feet and buttocks must not leave the ground; End state: When the person to be tested stands up from the supine or sitting state, the end state is entered, and no counting judgment is performed after entering the end state; c2) Ready state determination: When a person to be detected appears in the video and is detected by the target detection model, the tracking algorithm assigns the person a unique ID and the person enters the ready state; c3) Supine position determination: The person to be tested should be lying horizontally rather than standing upright. The difference in the x-coordinates between the head keypoint and the foot keypoint should be greater than the difference in the y-coordinates, and the difference in the x-coordinates between the hip keypoint and the foot keypoint should be greater than the difference in the y-coordinates. Furthermore, the person must be lying horizontally on the ground with the shoulder blades close to the ground. This means the angles formed by the head, hip, and foot keypoints must be close to 180 degrees. The legs must be naturally bent, requiring the angles formed by the hip, knee, and foot keypoints to be close to 90 degrees. The feet must be in contact with the ground, meaning that the angles between the toes, heels, and hip keypoints must be close to 180 degrees while meeting the horizontal lying requirement. c4) Sitting-up state determination: After the person enters the supine state, the determination is made whether they have entered the sitting-up state. First, the frames during the sitting-up process are checked to see if there is any hip or foot lifting. If so, the count is invalidated. When the person enters the supine state, the coordinates of the key points of the person's hips and feet are recorded. The changes in the y-direction coordinates of the person's hips and feet during the standing-up process are used to determine whether the above-mentioned invalidation has occurred. If no such invalidation has occurred, the person's standing-up angle is detected. The angle formed by the three key points of the shoulder, hip, and foot is used to determine the standing-up angle. When the angle meets a certain threshold, it is considered to have met the requirements. Finally, the hand is checked to see if it has moved to the standard line. The hand is determined to have crossed the line based on the coordinates of the finger and the coordinates of the standard line. c5) End state determination: If the current state of the person to be detected is supine or sitting up, and the current person no longer meets the horizontal lying state described in c3), the current person enters the end state, and no counting determination is performed after entering the end state; c6) When the person being tested moves from the supine position to the sit-up position and then back to the supine position, it is considered that he has completed one sit-up action, and the person's sit-up count will be increased accordingly.

2. The multi-player sports event counting method based on multi-key point detection according to claim 1 is characterized in that: The coordinate values ​​of 33 key points of the human body are detected using the posture detection module of mediapipe. The specific key points detected are 11 on the head, including the eyes, nose, mouth, and ears; 10 on both sides of the arms, including the elbows, wrists, and fingers; 4 on the torso, including the shoulders and hips; 2 on both sides of the legs, including the knees; and 6 on both sides of the feet, including the ankles, heels, and toes. The obtained coordinates are two-dimensional coordinates and the corresponding visibility information, that is, the coordinate format of a single key point is [x, y, visibility].

3. A multi-player sports event counting system based on multi-key point detection, comprising a computer device, characterized in that: The computer device is programmed or configured to execute the steps of the multi-person sports event counting method based on multi-key point detection as described in any one of claims 1-2, or the storage medium of the computer device stores a computer program that is programmed or configured to execute the multi-person sports event counting method based on multi-key point detection as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program that is programmed or configured to execute the multi-person sports event counting method based on multi-key point detection as described in any one of claims 1-2.

Images