Method and apparatus for detecting posture of player in net-separated ball game
By collecting videos from multiple perspectives and screening players' poses using three-dimensional spatial geometric relationship constraints, and directly estimating positions with backprojection technology, the problems of high error detection rate and large resource consumption in the existing technology are solved, and efficient and accurate posture detection.
Patent Information
- Application Number
- CN202411118083.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-14
AI Technical Summary
In the prior art, players in sports have high pose detection error rate, high resource consumption, slow calculation and reasoning speed, long matching time for multi-view triangulation and high matching error rate, which limits the application effect of pose detection.
By collecting videos from multiple perspectives, determining the two-dimensional posture of the human body target, and screening the three-dimensional posture of the player based on the three-dimensional spatial geometric relationship constraints associated with the camera parameters and the court, using back projection technology to estimate the position range at the next moment, skip Object detection directly estimates two-dimensional poses and performs triangulation to reduce computing resource consumption.
It improves the efficiency and accuracy of posture detection, reduces processing time and computing resource consumption, and alleviates the problems of matching time and high matching error rate.
Smart Images

Figure CN118967793A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to gesture detection in motion. Background Art
[0002] Detecting the posture of players in motion has a wide range of applications. For example, video frames of sports games or training can be collected, and the posture of players can be detected on the collected video frames. The determined three-dimensional posture data of the players can be used for rendering to replay the game process, analyze the technical and tactical performance of the players in the game or training, etc.
[0003] As a popular sport, tennis sports have fast ball speed, many rounds and fierce competition during the competition and training. Effective technical means are needed to provide viewers with a review of the fierce competition rounds that have already occurred, or to focus on the details of the competition that may be controversial during the game. Players also need to understand their technical and tactical performance in the game and training in a timely manner. Therefore, posture detection of players in tennis sports is of great significance.
[0004] Figure 1 The process 100 of traditional top-down posture detection is shown, which may include target detection 110, target tracking 130 and posture recognition 150. As an example, Chinese patent application publication CN 116958872 A uses this traditional top-down "target detection-target tracking-gesture recognition" process. Specifically, firstly, the bounding box of all identifiable human targets is obtained by using the target detection algorithm, and the court sideline is used for filtering, and the corresponding bounding box of the player is retained. Then, the bounding box sequence corresponding to each player is obtained by using a two-dimensional tracking algorithm. The image in the bounding box is cropped and enlarged, and the two-dimensional posture of the player is obtained by using a two-dimensional posture estimation algorithm. Thereafter, triangulation is performed based on the two-dimensional information of the badminton and the player in the monocular or binocular image coordinate system and the camera parameters to obtain the three-dimensional position of the badminton and the player, and finally the motion trajectory of the badminton and the player is composed using continuous images or videos.
[0005] The posture detection method based on this traditional top-down target detection has a high false detection rate, high resource consumption, slow computational reasoning speed, long matching time and high matching error rate during multi-view triangulation, all of which limit the application and effect of this posture detection method.
[0006] An efficient and accurate posture detection technology is needed in this field. Summary of the invention
[0007] In order to improve an efficient and accurate posture detection technology, the present invention is provided.
[0008] One aspect of the present invention provides a method for detecting the posture of a player in a tennis sport, comprising: determining the two-dimensional posture of all detectable human targets from multiple videos of the sport captured from multiple perspectives; determining the three-dimensional posture of the human target based on the determined two-dimensional posture of the human target and camera parameters; and determining the three-dimensional posture of the player through the three-dimensional posture of the human target based on a three-dimensional spatial geometric relationship constraint associated with a court.
[0009] The method as described above, wherein determining the two-dimensional posture of all detectable human targets includes: identifying all detectable human targets through a target detection algorithm; generating a bounding box sequence for each detected human target using a two-dimensional tracking algorithm; cropping and enlarging the image in each bounding box in the bounding box sequence; and determining the two-dimensional posture of the human target using a two-dimensional posture estimation algorithm.
[0010] A method as described in any of the above items, wherein determining the three-dimensional posture of a player through the three-dimensional posture of a human target includes: based on the three-dimensional spatial geometric relationship constraints associated with the court, screening a predetermined number of players' three-dimensional postures from all detectable three-dimensional postures of human targets, wherein the predetermined number is determined at least in part based on the number of players specified in the sports event.
[0011] A method as described in any of the above items, wherein, based on the three-dimensional spatial geometric relationship constraints associated with the court, screening out the three-dimensional postures of a predetermined number of players from the three-dimensional postures of all detectable human targets includes: screening out the three-dimensional postures of the predetermined number of players with the highest degree of match with the three-dimensional position of the court from the three-dimensional postures of all detectable human targets.
[0012] A method as described in any one of the above, wherein the three-dimensional spatial geometric relationship constraint associated with the court includes a court area defined by a boundary of the court.
[0013] A method as described in any of the above items, wherein determining the three-dimensional posture of a player through the three-dimensional posture of a human target includes: estimating the position range of the player's appearance in the multiple video frames at the next moment through back projection technology based on the player's three-dimensional posture and the camera parameters; skipping target detection within the position range and directly estimating the two-dimensional posture of the player at the next moment.
[0014] A method as described in any of the above items, wherein determining the three-dimensional posture of the player through the three-dimensional posture of the human target further comprises: for the estimated two-dimensional posture of the player at the next moment, skipping the matching task and directly performing single-person triangulation on the player.
[0015] The method as described in any of the above items also includes: using a neural network model to calculate the camera parameters based on position information of elements associated with the stadium in the multiple video frames.
[0016] Another aspect of the present invention provides a computer device, comprising a memory, a processor, and computer instructions stored in the memory, wherein the processor executes the computer instructions to implement the steps of any one of the above methods.
[0017] Another aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the steps of any of the above methods.
[0018] Another aspect of the present invention provides a computer program product, which comprises computer instructions, and when the computer instructions are executed by a processor, the steps of any one of the above methods are implemented.
[0019] The method and computer system for gesture detection according to the embodiments of the present invention are highly efficient and accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Embodiments of the present invention are described with reference to the accompanying drawings.
[0021] Figure 1 The conventional top-down posture detection process is shown.
[0022] Figure 2 is a flow chart illustrating a method for detecting a player's gesture in a tennis game according to some embodiments of the present invention.
[0023] Figure 3 is a flow chart illustrating a process for determining the two-dimensional poses of all detectable human targets according to some embodiments of the present invention.
[0024] Figure 4 is a flow chart illustrating a process for determining a three-dimensional pose of a player according to some embodiments of the present invention.
[0025] Figure 5 is a schematic diagram showing the camera arrangement of video acquisition cameras for acquiring video frames.
[0026] Figure 6 is a block diagram illustrating a computer device according to some embodiments of the present invention.
[0027] Figure 7 is a block diagram illustrating a computer-readable storage medium according to some embodiments of the present invention.
[0028] Figure 8is a block diagram illustrating a computer program product according to some embodiments of the present invention. DETAILED DESCRIPTION
[0029] In the present application, the term "netball sports" refers to a type of sports that generally includes two teams competing against each other, the playing field is divided into two parts by a net, each team competes with a ball in its respective part, and the players of each team do not have direct physical contact and confrontation with the opposing players. As examples, netball sports may include table tennis, badminton, tennis, volleyball, etc.
[0030] According to one aspect of the present invention, a method for detecting a player's posture in a tennis game is provided.
[0031] Figure 2 is a flow chart illustrating a method 200 for detecting a player's gesture in a tennis game according to some embodiments of the present invention.
[0032] Method 200 may include, at block 210 , determining two-dimensional poses of all detectable human targets in a plurality of video frames of motion captured from a plurality of viewpoints.
[0033] In some embodiments, multiple video frames of motion may be combined using the following Figure 5 The video capture camera described above is arranged to capture the video, but the scope of the present invention is not limited to this. Figure 3 The specific implementation of the step at block 210 is described.
[0034] Method 200 may include, at block 230 , determining a three-dimensional pose of the human target based on the determined two-dimensional pose of the human target and camera parameters.
[0035] As an example, the camera parameters may include the camera's extrinsic parameters, which are parameters for describing the position and posture of the camera in a world coordinate system.
[0036] The method 200 may include, at block 250 , determining a three-dimensional pose of the player through a three-dimensional pose of the human target based on three-dimensional spatial geometric relationship constraints associated with the court.
[0037] As an example, the three-dimensional geometric relationship constraints associated with the court may include a court area defined by the court boundary, the number of players defined by the sports rules, an area where players move defined by the sports rules, and any combination of the above. Figure 4 Specific implementations of the steps at block 250 are further described.
[0038] According to some embodiments of the present invention, the method 200 avoids the complete execution of the traditional top-down "target detection-target tracking-gesture recognition" process by utilizing the three-dimensional spatial geometric relationship constraints associated with the tennis court, thereby improving the efficiency and accuracy of gesture detection.
[0039] Optionally, in some embodiments, the method 200 may include, at block 270 , calculating camera parameters based on position information of elements associated with the court in the plurality of video frames using a neural network model.
[0040] For example, a plurality of key points of a court in a plurality of video frames may be detected. As an example, the key points may include points on a line on the court, intersections of lines, etc. The plurality of key points may be matched with a predefined standard court template. As an example, the plurality of key points may be densely matched with corresponding points on the standard court template. Based on the matching results, an algorithm may be used to estimate camera parameters. As an example, the algorithm may include a PnP algorithm, which is a method for solving a correspondence between three-dimensional and two-dimensional points.
[0041] In some embodiments, the steps at block 270 may be performed before, after, or at least partially in parallel with the steps at block 210. In some embodiments, the steps at block 270 may be performed before, after, or at least partially in parallel with the steps at block 230.
[0042] Figure 3 is a flow chart illustrating a process 300 for determining the two-dimensional poses of all detectable human targets according to some embodiments of the present invention. The process 300 may be Figure 2 The specific implementation of the step at box 210 in method 200 in FIG. 2 is described below, but the scope of the present invention is not limited thereto.
[0043] Process 300 may include, at block 310 , identifying all detectable human targets via a target detection algorithm.
[0044] Process 300 may include, at block 330 , generating a sequence of bounding boxes for each detected human target using a two-dimensional tracking algorithm.
[0045] Process 300 may include, at block 350 , cropping and enlarging the image within each bounding box in the sequence of bounding boxes.
[0046] Process 300 may include, at block 370 , determining a two-dimensional pose of the human target using a two-dimensional pose estimation algorithm.
[0047] Figure 4 is a flow chart illustrating a process for determining a three-dimensional pose of a player according to some embodiments of the present invention. Figure 4The process shown in can be Figure 2 The specific implementation of the steps at box 250 of method 200 in FIG. 2 is described below, but the scope of the present invention is not limited thereto.
[0048] In some embodiments, the process may include: at box 410, based on the three-dimensional spatial geometric relationship constraints associated with the court, screening a predetermined number of players' three-dimensional postures from all detectable three-dimensional postures of human targets, wherein the predetermined number is determined at least in part based on the number of players specified in the sports event.
[0049] As an example, for table tennis, badminton, or tennis, the predetermined number can be that each side has one player (singles match) or two players (doubles match). As an example, for volleyball, the predetermined number is that each side has six players.
[0050] In some embodiments, the step at box 410 may include: at box 415, filtering out the three-dimensional poses of the predetermined number of players that have the highest matching degree with the three-dimensional position of the field from the three-dimensional poses of all detectable human targets.
[0051] As an example, the three-dimensional geometric relationship constraints associated with the court may include the activity area of the players defined by the rules of the sport, based on which other detected persons may be excluded. For example, a person detected to stay near the net for a long time is likely to be a referee and therefore is not identified as a player. For example, a person detected to move only outside the court boundary is likely to be a caddie.
[0052] In some embodiments, the process may also include step sequence 450 .
[0053] The step sequence 450 may include: at box 451, estimating the position range of the player in the multiple video frames at the next moment based on the three-dimensional posture of the player and the camera parameters by back-projection technology.
[0054] The sequence of steps 450 may include: at block 453 , skipping target detection within the position range and directly estimating the two-dimensional pose of the player at the next moment.
[0055] The above steps effectively utilize the continuity of the player's movement and the three-dimensional spatial geometric relationship constraints associated with the court. The back-projection technology is used to estimate the position range of the player in multiple video frames at the next moment and directly estimate his two-dimensional posture, skipping the traditional target detection process, reducing processing time and computing resource consumption without affecting the accuracy of the results.
[0056] In some embodiments, the sequence of steps 450 may further include: at block 455 , for the estimated two-dimensional pose of the player at the next moment, skipping the matching task and directly performing single-person triangulation for the player.
[0057] Furthermore, the above steps effectively utilize the continuity of the player's movement and the three-dimensional spatial geometric relationship constraints associated with the court, skipping the matching task in the traditional process and directly triangulating the players, effectively alleviating the problems of long matching time and high matching error rate in the triangulation process.
[0058] Figure 5 is a schematic diagram showing a camera arrangement 500 of video capture cameras for capturing video frames.
[0059] Figure 5 The camera arrangement 500 of the video acquisition cameras for the badminton court 510 is specifically shown. Video acquisition cameras 521, 522, 523, 524, 525, 526 can be arranged around the badminton court 510 to capture motion data in the badminton court 510. The video acquisition cameras 521, 522, 523, 524, 525, 526 can be set as time-synchronized cameras, and their shooting frame rates are variable, and the default frame rate can be 50 frames per second. The video acquisition cameras 521, 522, 523, 524, 525, 526 can be coupled to the server 530 via optical fibers, and the collected signals are transmitted to the server 530 via optical fibers. The collected signals reflecting the motion data can be provided to perform the method for detecting the posture of a player in a tennis sport according to an embodiment of the present invention (for example, the above combined method). Figure 1-Figure 4 methods and processes described).
[0060] Figure 5 The number and position of the video acquisition cameras shown in are exemplary. In some embodiments, more or fewer video acquisition cameras can be used. In some embodiments, the position of each video acquisition camera can be adjusted.
[0061] According to another aspect of the present invention, a computer device is provided.
[0062] Figure 6 is a block diagram illustrating a computer device 600 according to some embodiments of the present invention.
[0063] The computer device 600 may include a processor 610 and a memory 620. The memory 620 stores computer instructions 625. The processor 610 may execute the computer instructions 625 to implement the above-mentioned Figure 1-Figure 4 Each method or step described.
[0064] According to another aspect of the present invention, a computer-readable storage medium is provided.
[0065] Figure 7 is a block diagram illustrating a computer-readable storage medium 700 according to some embodiments of the present invention.
[0066] The computer readable storage medium 700 may include computer instructions 750 stored thereon. The computer instructions 750, when executed by a processor, may implement the above-mentioned Figure 1-Figure 4 Each method or step described.
[0067] According to another aspect of the present invention, a computer program product is provided.
[0068] Figure 8 is a block diagram illustrating a computer program product 800 according to some embodiments of the invention.
[0069] The computer program product 800 may include computer instructions 850. When the computer instructions 850 are executed by a processor, the above-mentioned Figure 1-Figure 4 Each method or step described.
[0070] Embodiments of the present invention have been described with reference to the accompanying drawings. The embodiments are illustrative rather than restrictive.
Claims
1. A method for detecting the posture of a player in a tennis game, comprising: Determine the two-dimensional pose of all detectable human objects from multiple videos of the motion captured from multiple viewpoints; Determine a three-dimensional pose of the human target based on the determined two-dimensional pose of the human target and camera parameters; as well as The three-dimensional posture of the player is determined through the three-dimensional posture of the human target based on the three-dimensional spatial geometric relationship constraints associated with the court.
2. The method of claim 1, wherein: Determine the 2D pose of all detectable human objects including: Identify all detectable human targets through target detection algorithms; Generate a sequence of bounding boxes for each detected human target using a two-dimensional tracking algorithm; Cropping and enlarging the image in each bounding box in the bounding box sequence; and The two-dimensional pose estimation algorithm is used to determine the two-dimensional pose of the human target.
3. The method of claim 1, wherein: Determining the three-dimensional posture of the player through the three-dimensional posture of the human target includes: Based on the three-dimensional spatial geometric relationship constraints associated with the court, a predetermined number of players' three-dimensional postures are screened from all detectable three-dimensional postures of human targets, wherein the predetermined number is determined at least in part based on the number of players specified in the sport.
4. The method of claim 3, wherein: Based on the three-dimensional spatial geometric relationship constraints associated with the court, the three-dimensional postures of a predetermined number of players are screened from the three-dimensional postures of all detectable human targets, including: The three-dimensional postures of the predetermined number of players having the highest matching degree with the three-dimensional position of the field are screened out from the three-dimensional postures of all detectable human targets.
5. The method of claim 4, wherein: The three-dimensional spatial geometric relationship constraints associated with the court include a court area defined by the boundaries of the court.
6. The method of claim 1, wherein: Determining the three-dimensional posture of the player through the three-dimensional posture of the human target includes: Based on the three-dimensional posture of the player and the camera parameters, the position range of the player appearing in the multiple video frames at the next moment is estimated by back-projection technology; The target detection is skipped within the position range and the two-dimensional posture of the player at the next moment is directly estimated.
7. The method of claim 6, wherein: Determining the three-dimensional posture of the player through the three-dimensional posture of the human target also includes: For the estimated two-dimensional pose of the player at the next moment, the matching task is skipped and single-person triangulation of the player is directly performed.
8. The method of claim 1, further comprising: The camera parameters are calculated based on position information of elements associated with the court in the plurality of video frames using a neural network model.
9. A computer device comprising a memory, a processor, and computer instructions stored in the memory, wherein: The processor executes the computer instructions to implement the steps of the method according to any one of claims 1-8.
10. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.
11. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Intelligent auxiliary training method and system for badminton sports
CN116958872A
Badminton game human body posture estimation and ball path tracking method
CN115100744A
Method and device for constructing three-dimensional attitude estimation data set based on multiple view angles
CN115841602A
Three-dimensional human pose estimation method and apparatus, device, and storage medium
WO2022178951A1
Cited By
Exercise load analysis system and method
CN119625020A
Human motion action detection method and device, electronic equipment and storage medium
CN121963315A
Human motion action detection method and device, electronic equipment and storage medium
CN121963315B
