Sports scoring method, system, terminal and storage medium

Abstract

The present invention discloses a motion scoring method, system, terminal and storage medium. The method includes: determining an actual motion trajectory and a standard motion trajectory corresponding to a sports action of an athlete in a video, wherein the standard motion trajectory is determined according to the standard action corresponding to the sports action; determining a scaling factor corresponding to each segment of the standard motion trajectory; scaling each segment of the standard motion trajectory according to the scaling factor; determining a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; and determining a scoring result corresponding to the athlete according to the segmented comparison result. The technical solution of the present application is used to improve the accuracy of the motion scoring.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a sports scoring method, system, terminal and storage medium. Background Art

[0002] In sports events, the scoring standards for scoring competitive sports are relatively strict. Currently, the main sports action scoring schemes used are based on computer vision, including weighted scoring of technical indicators and graded scoring of overall action recognition. Among them, the weighted scoring scheme of technical indicators is to shoot the athletes' sports actions from multiple angles by setting up multiple cameras, and then perform 3D reconstruction based on the multi-channel video to generate 3D video. The deep learning method is used to perform action recognition on the technical indicators in the 3D video, such as the number of torso rotations and take-off height, and finally the technical scoring is performed according to the same referee rules and scoring standards. However, each athlete has different physical qualities such as weight and height. If the same scoring standards are used to score the trajectory of each athlete's sports action, there will be deviations, resulting in inaccurate sports scoring. Summary of the Invention

[0003] The embodiments of the present application aim to solve the problem of inaccurate motion scoring by providing a motion scoring method, system, terminal, and storage medium.

[0004] The present invention provides a motion scoring method, which includes:

[0005] Determine an actual motion trajectory corresponding to a sports action of an athlete in a video, and a standard motion trajectory, wherein the standard motion trajectory is determined according to a standard action corresponding to the sports action;

[0006] Determining the scaling factor corresponding to each segment of the standard motion trajectory;

[0007] scaling each segment of the standard motion trajectory according to the scaling factor;

[0008] Determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory;

[0009] The scoring result corresponding to the athlete is determined according to the segment comparison result.

[0010] In one embodiment, the step of determining the actual motion trajectory corresponding to the athlete's sports action in the video includes:

[0011] Determine key point information of each segment of the actual motion trajectory described in the video;

[0012] The key point information of each segment is fitted to obtain the actual motion trajectory of the athlete.

[0013] In one embodiment, the step of scaling each segment of the standard motion trajectory according to the scaling factor includes:

[0014] Obtaining key point information of the segments of the actual motion trajectory corresponding to the scaling factor; wherein the scaling factor corresponding to each segment is different;

[0015] The scaling factor is multiplied by the key point information to obtain the scaled segments of the standard motion trajectory.

[0016] In one embodiment, before the step of determining the actual motion trajectory corresponding to the athlete's sports action in the video, the following steps are included:

[0017] Determine the reference line segment corresponding to the athlete at the starting point of the movement;

[0018] Obtaining a standard reference line segment corresponding to the reference line segment;

[0019] The standard reference line segment and the reference line segment are aligned in a reference coordinate system.

[0020] In one embodiment, the step of aligning the standard reference line segment with the reference line segment in a reference coordinate system includes:

[0021] translating the end position of the standard reference line segment to the end position of the reference line segment;

[0022] Determining the angle between the translated standard reference line segment and the reference line segment in the directions of each coordinate axis of the reference coordinate system;

[0023] The coordinate transformation is performed on the standard reference line segment based on the included angle, so that the standard reference line segment and the reference line segment are aligned in a reference coordinate system.

[0024] In one embodiment, the step of determining a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory includes:

[0025] Mapping the actual motion trajectory and the scaled standard motion trajectory into a camera coordinate system of a live camera;

[0026] The segments of the mapped actual motion trajectory and the mapped and scaled standard motion trajectory are compared to obtain a segment comparison result between the actual motion trajectory and the scaled standard motion trajectory.

[0027] In one embodiment, the step of mapping the actual motion trajectory and the scaled standard motion trajectory into a camera coordinate system of a live camera includes:

[0028] Extracting and matching feature points from the image frames of the live camera and the image frames of the fixed camera at the same moment to obtain matching point pairs;

[0029] Obtaining an intrinsic matrix based on the matching point pairs and camera intrinsic parameters;

[0030] Decomposing the intrinsic matrix to obtain the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera;

[0031] The actual motion trajectory is transformed according to the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera to obtain the actual motion trajectory in the camera coordinate system of the live camera, and the scaled standard motion trajectory is transformed according to the rotation and translation matrix of the camera and the rotation and translation matrix of the fixed camera to obtain the scaled standard motion trajectory in the camera coordinate system of the live camera.

[0032] In addition, to achieve the above-mentioned purpose, the present invention further provides a sports scoring system, which includes:

[0033] a trajectory determination module, configured to determine an actual motion trajectory corresponding to a sports action of an athlete in a video, and a standard motion trajectory, wherein the standard motion trajectory is determined based on a standard action corresponding to the sports action;

[0034] A scaling factor determination module, configured to determine a scaling factor corresponding to each segment of the standard motion trajectory;

[0035] A scaling module, configured to scale each segment of the standard motion trajectory according to the scaling factor;

[0036] a comparison module, configured to determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory;

[0037] The scoring module is used to determine the scoring result corresponding to the athlete according to the segment comparison result.

[0038] In addition, to achieve the above-mentioned objectives, the present invention also provides an intelligent terminal, which includes: a memory, a processor, and a motion scoring program stored in the memory and executable on the processor, wherein the motion scoring program, when executed by the processor, implements the steps of the above-mentioned motion scoring method.

[0039] In addition, to achieve the above-mentioned object, the present invention further provides a computer-readable storage medium having a motion scoring program stored thereon, which implements the steps of the above-mentioned motion scoring method when executed by a processor.

[0040] A technical solution of a sports scoring method, system, terminal and storage medium provided in the embodiments of the present application adopts the following methods: determining the actual motion trajectory and standard motion trajectory corresponding to the sports movements of athletes in a video, wherein the standard motion trajectory is determined according to the standard movement corresponding to the sports movement; determining the scaling coefficient corresponding to each segment of the standard motion trajectory; scaling each segment of the standard motion trajectory according to the scaling coefficient; determining the segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; and determining the scoring result corresponding to the athlete according to the segmented comparison result. Since the actual motion trajectory corresponding to the sports movement of the athlete is segmentedly compared with the scaled standard motion trajectory, and the scoring result of the athlete is determined according to the segmented comparison result of each segment, the scoring result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the structure of an intelligent terminal involved in an embodiment of the present invention;

[0042] Figure 2 This is a flow chart of a first embodiment of the motion scoring method of the present invention;

[0043] Figure 3 This is a flow chart of an embodiment of a motion scoring method of the present invention;

[0044] Figure 4 This is a schematic diagram of the 3D motion trajectory of the present invention;

[0045] Figure 5 Schematic diagram of the angle change curve of key points of the present invention;

[0046] Figure 6 This is a schematic diagram of the imaging principle of the camera of the present invention;

[0047] Figure 7 This is a functional module diagram of the sports scoring system of the present invention.

[0048] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire invention. DETAILED DESCRIPTION

[0049] In order to solve the problem of inaccurate sports scoring during exercise, the present application adopts a technical solution of determining the actual motion trajectory corresponding to the sports action of the athlete in the video and the standard motion trajectory, wherein the standard motion trajectory is determined according to the standard action corresponding to the sports action; determining the scaling factor corresponding to each segment of the standard motion trajectory; scaling each segment of the standard motion trajectory according to the scaling factor; determining the segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; and determining the scoring result corresponding to the athlete according to the segmented comparison result. Since the actual motion trajectory corresponding to the sports action of the athlete is compared with the scaled standard motion trajectory in segments, the scoring result of the athlete is determined according to the segmented comparison result of each segment, so that the scoring result is more accurate.

[0050] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0052] It should be noted that Figure 1 This is a structural diagram of the hardware operating environment of the smart terminal.

[0053] like Figure 1 As shown, the intelligent terminal may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0054] Those skilled in the art will understand that Figure 1The structure of the smart terminal shown in the figure does not constitute a limitation on the smart terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0055] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a motion scoring program. The operating system is a program that manages and controls the hardware and software resources of the intelligent terminal, and the motion scoring program and other software or programs are executed.

[0056] exist Figure 1 In the smart terminal shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to communicate data with the background server; the processor 1001 can be used to call the sports scoring program stored in the memory 1005.

[0057] In this embodiment, the intelligent terminal includes: a memory 1005, a processor 1001, and a sports scoring program stored in the memory and executable on the processor, wherein:

[0058] When the processor 1001 calls the motion scoring program stored in the memory 1005, it performs the following operations:

[0059] Determine an actual motion trajectory corresponding to a sports action of an athlete in a video, and a standard motion trajectory, wherein the standard motion trajectory is determined according to a standard action corresponding to the sports action;

[0060] Determining the scaling factor corresponding to each segment of the standard motion trajectory;

[0061] scaling each segment of the standard motion trajectory according to the scaling factor;

[0062] Determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory;

[0063] The scoring result corresponding to the athlete is determined according to the segment comparison result.

[0064] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0065] Determine key point information of each segment of the actual motion trajectory described in the video;

[0066] The key point information of each segment is fitted to obtain the actual motion trajectory of the athlete.

[0067] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0068] Obtaining key point information of the segments of the actual motion trajectory corresponding to the scaling factor; wherein the scaling factor corresponding to each segment is different;

[0069] The scaling factor is multiplied by the key point information to obtain the scaled segments of the standard motion trajectory.

[0070] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0071] Determine the reference line segment corresponding to the athlete at the starting point of the movement;

[0072] Obtaining a standard reference line segment corresponding to the reference line segment;

[0073] The standard reference line segment and the reference line segment are aligned in a reference coordinate system.

[0074] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0075] translating the end position of the standard reference line segment to the end position of the reference line segment;

[0076] Determining the angle between the translated standard reference line segment and the reference line segment in the directions of each coordinate axis of the reference coordinate system;

[0077] The coordinate transformation is performed on the standard reference line segment based on the included angle, so that the standard reference line segment and the reference line segment are aligned in a reference coordinate system.

[0078] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0079] Mapping the actual motion trajectory and the scaled standard motion trajectory into a camera coordinate system of a live camera;

[0080] The segments of the mapped actual motion trajectory and the mapped and scaled standard motion trajectory are compared to obtain a segment comparison result between the actual motion trajectory and the scaled standard motion trajectory.

[0081] When the processor 1001 calls the motion scoring program stored in the memory 1005, it also performs the following operations:

[0082] Extracting and matching feature points from the image frames of the live camera and the image frames of the fixed camera at the same moment to obtain matching point pairs;

[0083] Obtaining an intrinsic matrix based on the matching point pairs and camera intrinsic parameters;

[0084] Decomposing the intrinsic matrix to obtain the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera;

[0085] The actual motion trajectory is transformed according to the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera to obtain the actual motion trajectory in the camera coordinate system of the live camera, and the scaled standard motion trajectory is transformed according to the rotation and translation matrix of the camera and the rotation and translation matrix of the fixed camera to obtain the scaled standard motion trajectory in the camera coordinate system of the live camera.

[0086] The technical solutions of this application will be introduced below in the form of embodiments.

[0087] This application mainly targets a specific sports event scene, and constructs a 3D standard motion trajectory database for the actions and postures that need to be scored according to the scoring criteria; by selecting suitable reference objects for the sports scene in the lens, such as springboards, diving platforms, etc., they are detected and fitted with the center line as the starting curve segment for matching the standard database, and then the 3D-2D coordinate mapping is used to project the 3D standard motion trajectory into the lens, and the scale is adjusted according to information such as the athlete's take-off height; finally, the athlete's actual motion trajectory is compared with the standard motion trajectory in the 3D standard motion trajectory database, as well as the athlete's actual motion trajectory segmentation and the standard motion trajectory segmentation in the 3D standard motion trajectory database, and finally a reference score is given.

[0088] Specifically, the present application performs 3D trajectory modeling on each action of scoring sports and forms a standard motion trajectory database of corresponding sports categories-action categories-trajectories. During the actual competition or training process, first, deep learning technology is used to identify the environmental features of the starting point of the movement, such as the diving platform or springboard in diving, and the center line of the diving platform or springboard is extracted; based on the center line reference, the starting line segment of the 3D standard motion trajectory is aligned with the center line in the current video frame under the reference coordinate system; then the athlete's take-off height is tracked and identified, and the 3D standard motion trajectory and the athlete's actual motion trajectory are corrected at the starting point position; the athlete's sports movements are tracked, and the athlete's actual motion trajectory is compared with the 3D standard motion trajectory in real time. Finally, the overall actual motion trajectory of the athlete is drawn and compared with the 3D standard motion trajectory to achieve the objectivity of the referee's subjective scoring; at the same time, according to the specific live broadcast or rebroadcast requirements, the athlete's actual motion trajectory and the standard motion trajectory are projected into the live image in real time through the posture relationship between the coordinate system of the live camera and the reference coordinate system. The overall flow chart is as follows: Figure 3 shown.

[0089] like Figure 2 As shown, in the first embodiment of the present application, the motion scoring method of the present application includes the following steps:

[0090] Step S110 , determining an actual motion trajectory corresponding to the sports action of the athlete in the video, and a standard motion trajectory, wherein the standard motion trajectory is determined according to the standard action corresponding to the sports action.

[0091] In this embodiment, the hardware system of this application consists of a fixed camera and several mobile live cameras, where the fixed camera is the reference camera and is used to construct a global reference coordinate system. Several mobile live cameras are optional, and the coordinate system mapping is selected according to different scenarios. The standard motion trajectory and the athlete's actual motion trajectory are generated and aligned in the reference coordinate system. The mobile coordinate system is obtained by calculating the relative position relationship between the live camera and the reference camera, and the standard motion trajectory and the actual motion trajectory are projected onto the image of the live camera through coordinate system transformation.

[0092] In this embodiment, the video is captured by a mobile live camera. The actual motion trajectory is the trajectory corresponding to the athlete's performance of the sports maneuver. The standard motion trajectory is stored in a standard motion trajectory database, and the standard motion trajectory is determined based on the standard motion corresponding to the sports maneuver. Each sports maneuver in a scoring sport can be modeled using 3D trajectories, and each trajectory can be stored in the standard motion trajectory database in the format of sport category-action category-trajectory. The trajectory can be an overall motion trajectory, i.e., the motion trajectory of the entire process from ascent to descent. For example, the standard motion trajectory database stores a diving-forward somersault 3.5 twists-tucked overall motion trajectory. The trajectory can also be a segmented motion trajectory. For example, the standard motion trajectory database may store: a diving-forward somersault 3.5 twists-tucked ascent trajectory, a diving-forward somersault 3.5 twists-tucked three rolls trajectory, a diving-forward somersault 3.5 twists-tucked semicircular curve trajectory, a diving-forward somersault 3.5 twists-tucked descent trajectory, etc.

[0093] Specifically, the standard motion trajectory is determined based on the standard movement corresponding to the sports action. Different sports have different motion trajectories. Taking diving's 107B movement as an example, based on the trajectory characteristics, the standard motion trajectory can be segmented and modeled into curves based on different time stages, such as the take-off segment (parabola), rotation segment (spiral curve), and descent segment (parabola). The corresponding spatial curve equation is constructed using the take-off point of the movement as the coordinate origin, which can be set based on actual conditions.

[0094] Reference Figure 4 The take-off point of the springboard, that is, the intersection of the reference line and the take-off segment, is taken as the coordinate origin. The standard motion trajectory consists of 6 segmented curves: the take-off segment U, 3 circular curve segments C 1-3 , a semicircular curve segment SC, and a falling segment D. The takeoff segment is represented by a three-dimensional equation of a straight line, the circular curve segment is represented by a three-dimensional equation of a circle, the semicircular curve segment is represented by a three-dimensional equation of a semicircle, and the falling segment is represented by a three-dimensional equation of a straight line. Considering the acceleration of gravity, a small straight line spacing is designed between the circular and semicircular curve segments, as shown below:

[0095] .

[0096] Since specific actions may require multiple parts of the human body to form a whole, in the specific modeling process, the human body can be modeled separately according to the joints. For example, 7 key points of the human body are selected: head, left hand, right hand, left knee joint, right knee joint, left foot, and right foot. Then, the trajectory of these 7 key points in the process of movement is designed by segmented curves, with a total of 7 standard curve segments (C Ref_head , C Ref_lefth , C Ref_righth , CRef_leftn , C Ref_rightn , C Ref_leftf , C Ref_rightf ) and are combined according to the specific movement components. For example, in 107B, the knee-hugging movement requires combining the left and right hands and the left and right knee joints. Therefore, when determining the standard motion trajectory, the left and right hand and left and right knee joint curves need to be combined. These four trajectories should completely overlap in the knee-hugging movement. That is, the standard motion trajectory can be expressed as:

[0097] ;

[0098] The origin of the standard motion trajectory is the origin O (0, 0, 0) in the reference coordinate system. If the length of the preset reference line segment is 1 unit and the preset reference line segment is along the z-axis, then the end vertex of the preset reference line segment is The coordinates are (0, 0, 1).

[0099] After the standard motion trajectory is segmented and modeled in the above manner, the standard motion trajectory is stored in a standard motion trajectory database.

[0100] In this embodiment, the actual motion trajectory corresponding to the sports action in the current video is identified, and a standard motion trajectory matching the actual motion trajectory is obtained from a standard motion trajectory database.

[0101] Step S120 : determining the scaling coefficient corresponding to each segment of the standard motion trajectory.

[0102] In the present embodiment, after obtaining the standard motion trajectory segments that match the actual motion trajectory segments, the scaling factor corresponding to each segment is determined. Among them, each segment has a corresponding scaling factor. Since the standard motion trajectory is constructed based on a unit scale, when doing the actual motion trajectory comparison, it is necessary to first scale the various segments of the standard motion trajectory according to the scaling factor. Before scaling the various segments of the standard motion trajectory according to the scaling factor, it is necessary to determine the scaling factor corresponding to the various segments of the standard motion trajectory. For example, taking the 107B sports action of diving as an example, the sports action is divided into an ascending segment, three tumbling segments, a semicircular curve segment, and a falling segment. Among them, real is used to represent the actual motion trajectory, and Ref is used to represent the standard motion trajectory. The scaling factors of different segments are calculated as follows:

[0103] Rising segment U: The scaling factor of the rising segment can be directly obtained by Calculated.

[0104] Three rolling segments C 1-3 :The scaling factors of the three rolling segments are obtained by the radius ratio of each circular motion, that is, by Calculated.

[0105] Semicircular curve segment SC: The scaling factor of the semicircular curve segment is obtained by the radius ratio of the semicircular curve, that is, by Calculated.

[0106] Drop segment D: The scaling factor calculation method of this drop segment can be directly calculated by Step S130: scaling each segment of the standard motion trajectory according to the scaling factor;

[0107] In this embodiment, each segment of each standard motion trajectory has a corresponding scaling factor, and the scaling factor can be used to scale each segment of the standard motion trajectory. For example, the scaling factor of the ascending segment can be used to scale the ascending segment of the standard motion trajectory. The scaling factor of the descending segment can be used to scale the descending segment of the standard motion trajectory.

[0108] Step S140 : determining a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory.

[0109] In this embodiment, after scaling each segment of the standard motion trajectory, each segment of the actual motion trajectory is compared with the corresponding segment of the standard motion trajectory, thereby determining the segment comparison results for each segment. The segment comparison results are represented by comparison scores. For example, the rising segment of the actual motion trajectory can be compared with the rising segment of the standard motion trajectory, and the comparison score is determined based on the degree of overlap between the rising segment of the actual motion trajectory and the rising segment of the standard motion trajectory. For example, when the degree of overlap between the rising segment of the actual motion trajectory and the rising segment of the standard motion trajectory reaches 98%, the comparison score is determined to be 98 points.

[0110] Step S150: determining the scoring result corresponding to the athlete according to the segment comparison result.

[0111] In this embodiment, after determining the segment comparison results for each segment, the segment comparison results for each segment can be weighted, and the athlete's corresponding scoring result can be determined based on the weighted value. The segment comparison results for each segment can also be combined with the proportion of each segment to determine the athlete's corresponding scoring result. For example, taking the 107B sports movement of diving as an example, assuming that the ascending segment accounts for 15%, the three tumbling segments account for 45%, the semicircular curve segment accounts for 25%, and the falling segment accounts for 15%. If the ascending segment has a score of 100, the three tumbling segments account for 95, the semicircular curve segment accounts for 90, and the falling segment accounts for 100, then the final score is: 15%*100+45%*95+25%*90+15%*100=95.25, i.e., the final score is 95.25.

[0112] In one embodiment, the athletes' sports movements can also be scored by using weighted scoring of technical indicators and graded scoring of overall movement recognition. Among them, the weighted scoring scheme of technical indicators is to shoot the athletes' sports movements from multiple angles by setting up multiple cameras, perform 3D reconstruction based on the multi-channel video to generate 3D video, use deep learning methods to perform movement recognition on the technical indicators in the 3D video, such as the number of torso rotation circles, take-off height, etc., and finally perform technical scoring according to the scoring standards of referee rules. The graded scoring of overall movement recognition is to use the past athletes' competition or training data, and through deep learning algorithms, train the whole set of movements according to the scoring level of sports referees. Finally, the trained model is used to infer and score the whole set of athletes' sports movements.

[0113] According to the above technical solution, this embodiment adopts the following technical solutions: determining the actual motion trajectory corresponding to the sports action of the athlete in the video, and the standard motion trajectory, the standard motion trajectory is determined according to the standard action corresponding to the sports action; determining the scaling coefficient corresponding to each segment of the standard motion trajectory; scaling each segment of the standard motion trajectory according to the scaling coefficient; determining the segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; and determining the scoring result corresponding to the athlete according to the segmented comparison result. Since the actual motion trajectory corresponding to the sports action of the athlete is compared with the scaled standard motion trajectory in segments, and the scoring result of the athlete is determined according to the segmented comparison result of each segment, the scoring result is more accurate.

[0114] In one embodiment, determining the actual motion trajectory corresponding to the sports action of the athlete in the video specifically includes the following steps:

[0115] Step S111, determining key point information of each segment of the actual motion trajectory in the video;

[0116] Step S112: Fitting the key point information of each segment to obtain the actual motion trajectory of the athlete.

[0117] In this embodiment, when determining the actual motion trajectory corresponding to an athlete's athletic action, it is necessary to simultaneously consider the position information of each joint of the athlete's body during the athletic action, i.e., key point information. For example, the key moment or key period corresponding to the athlete's athletic action in the video can be determined; at least two key point information of the athlete at the key moment can be extracted, or at least two key point information of the athlete at each moment within the key period can be extracted; and each key point information is fitted to obtain the actual motion trajectory corresponding to the athlete's athletic action.

[0118] Specifically, open-source target detection algorithms, such as RetinaNet and the Yolo series, are first used to train target detection and tracking models for athletes in the target scene. Real-time inference is then performed on the video stream, outputting the detected athlete target frames. Deep learning methods, such as OpenPose, are then used to detect key points within the athlete region within the target frame, obtaining the coordinates of key points such as the head, hands, knees, and feet in the camera coordinate system. Due to algorithm estimation errors, key points are smoothed in the previous and next image frames using filtering methods such as Gaussian smoothing to generate a real-time point sequence. After obtaining the key point sequence, the key moments of different sports movements are broken down and curve fitting is performed on the athlete's actual motion trajectory.

[0119] For example, taking the 107B action of diving as an example, the specific steps of curve fitting are as follows:

[0120] Rising segment U: The athlete's take-off time is t0, and the first coincidence time of the athlete's hand key point and the knee joint key point is t1. This time period is the take-off segment. At this time, the least squares fitting of the point sequence of the 7 key points mentioned above is performed on the straight line. Due to the influence of movements such as bending over and flexing the elbows, the rising segment straight line only takes the key point positions at the take-off time t0 and the key point positions at the coincidence time t1.

[0121] Three rolling segments C 1-3 : The key point vector of the head key point and the hand (knee joint) at the first overlap moment is recorded as V0, and the key point vector of the head key point and the hand (knee joint) in each frame image is calculated as V1-V k The angle between the head and the hand and V0 can be obtained as an angle curve. According to the trajectory characteristics of the rolling action, the angle curve is a semi-sine. As the rolling begins, the line connecting the head and the hand rotates clockwise with the head as the symmetrical point. V k It will deviate more and more from V0 until it is in the opposite direction of V0, at which point the maximum angle is 180 degrees. k Approaching V0 and finally coinciding, this is a complete circular motion. Figure 5 The starting and ending moments of each complete circular motion can be obtained from (t1, t2), (t2, t3), and (t3, t4). Take the 7 key points in this time period and perform the least square fitting of the circular curve to obtain the three rolling segment curves of the 7 key points, where C Ref_lefth , C Ref_righth , C Ref_leftn , C Ref_rightn Share the same roll curve segment.

[0122] Semicircular curve segment SC: from Figure 5The start and end time (t4, t5) of the half-cycle motion can be obtained, and the least square fitting of the semicircular curve is performed on the point sequence of the 7 key points in this time period.

[0123] Falling segment D: record the time t when the key point enters the water and disappears end is the end time of the entire action, take the time period (t5, t end ) is used to perform the least squares fitting of the straight line on the point sequence of the 7 key points within .

[0124] According to the above technical solution, this embodiment adopts a technical means of obtaining the actual motion trajectory by fitting the key point information of each segment, thereby improving the accuracy of the actual motion trajectory.

[0125] In one embodiment, scaling each segment of the standard motion trajectory according to the scaling factor specifically includes the following steps:

[0126] Step S131, obtaining key point information of the segments of the actual motion trajectory corresponding to the scaling coefficient; wherein the scaling coefficient corresponding to each segment is different;

[0127] Step S132: multiplying the scaling factor by the key point information to obtain the scaled segments of the standard motion trajectory.

[0128] In this embodiment, the scaling factor corresponding to each segment is different, that is, the scaling factor is associated with the corresponding segment of the actual motion trajectory and the segment of the standard motion trajectory. The key point information of the segment of the actual motion trajectory corresponding to the scaling factor is obtained. By multiplying the scaling factor with the key point information of the corresponding segment, the scaled segments of the standard motion trajectory are obtained. For example, the key point information of the rising segment in the actual motion trajectory is multiplied by the scaling factor of the rising segment to obtain the rising segment of the scaled standard motion trajectory. The segments of the standard motion trajectory are subjected to the same scaling processing method as the rising segment, thereby obtaining the scaled segments of the standard motion trajectory.

[0129] According to the above technical solution, this embodiment adopts a technical means of scaling each segment of the standard motion trajectory using a scaling factor, thereby converting the standard motion trajectory and the actual motion trajectory into the same scale, preparing conditions for subsequent comparison of the standard motion trajectory and the actual motion trajectory.

[0130] In one embodiment, determining the segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory specifically includes the following steps:

[0131] Step S141, mapping the actual motion trajectory and the scaled standard motion trajectory into the camera coordinate system of the live camera;

[0132] Step S142 : comparing each segment of the mapped actual motion trajectory with the mapped and scaled standard motion trajectory to obtain a segment comparison result between the actual motion trajectory and the scaled standard motion trajectory.

[0133] In this embodiment, the above-mentioned first embodiment records that the actual motion trajectory of the athlete is compared with the standard motion trajectory in the reference coordinate system of the fixed camera. In actual application scenarios, due to the needs of live broadcast or recorded broadcast, the actual motion trajectory and the standard motion trajectory in the reference coordinate system need to be projected into the corresponding live camera. At this time, it is necessary to calculate the projection transformation matrix based on the relative posture relationship between the live camera and the fixed camera, first transform the actual motion trajectory and the standard motion trajectory into the camera coordinate system of the live camera, and then project them back into the picture according to the camera imaging principle. In the final visualization effect, the motion trajectories are compared in all directions through two methods: full trajectory comparison and segmented trajectory comparison, providing an objective visual basis for the referee's scoring.

[0134] According to the above technical solution, this embodiment adopts a technical means of mapping the actual motion trajectory and the standard motion trajectory into the camera coordinate system of the live camera, and then performing segmented comparison on the mapped motion trajectories, thereby realizing the visualization of the motion trajectory and providing an objective visual basis for the referee's scoring.

[0135] In one embodiment, mapping the actual motion trajectory and the scaled standard motion trajectory to the camera coordinate system of the live camera specifically includes the following steps:

[0136] Step S1411, extracting and matching feature points of the image frames of the live camera and the image frames of the fixed camera at the same time to obtain matching point pairs;

[0137] Step S1412: Obtain an intrinsic matrix based on the matching point pairs and camera intrinsic parameters;

[0138] Step S1413, decomposing the intrinsic matrix to obtain the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera;

[0139] Step S1414: transform the actual motion trajectory according to the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera to obtain the actual motion trajectory in the camera coordinate system of the live camera, and transform the scaled standard motion trajectory according to the rotation and translation matrix of the camera and the rotation and translation matrix of the fixed camera to obtain the scaled standard motion trajectory in the camera coordinate system of the live camera.

[0140] In this embodiment, the relative pose transformation matrix between the live camera and the fixed camera is calculated as follows: feature points are extracted and matched for the live camera's image frames and the fixed camera's image frames at the same time to obtain a series of matching point pairs; the fundamental matrix F is solved for every 8 matching point pairs; the intrinsic matrix E is calculated using the fundamental matrix F and the camera's intrinsic parameters (fx, fy, u0, v0); the intrinsic matrix E is decomposed by SVD to obtain the rotation and translation matrices R,t of the two camera poses. After obtaining the rotation and translation matrices R,t, the actual motion trajectory and standard motion trajectory of the live camera in the camera coordinate system are calculated as follows:

[0141]

[0142] .

[0143] According to the above technical solution, this embodiment adopts the technical means of mapping the actual motion trajectory and the standard motion trajectory in the fixed camera coordinate system to the live camera coordinate system, thereby realizing the visualization of the motion trajectory and providing an objective visual basis for the referee's scoring.

[0144] In the second embodiment of the present application, before step S110 of the first embodiment, the motion scoring method of the present application includes the following steps:

[0145] Step S210, determining the reference line segment corresponding to the athlete at the starting point of the movement;

[0146] Step S220, obtaining a standard reference line segment corresponding to the reference line segment;

[0147] Step S230 , aligning the standard reference line segment and the reference line segment in a reference coordinate system.

[0148] In this embodiment, to effectively match the standard motion trajectory with the actual motion trajectory, it is necessary to unify the standard motion trajectory and the actual motion trajectory in the reference coordinate system of the fixed camera. Because the 3D standard motion model is curved information, it is necessary to extract the corresponding reference line segments from the real scene and calculate the coordinate system transformation matrix based on the spatial positional relationship between the reference line segments and the standard reference line segments of the standard motion model.

[0149] The 3D standard action model sets reference segments according to specific sports events, i.e. Figure 4The reference line segments in the actual motion trajectory can be extracted from the environment on which the athlete's effective movements depend. Taking diving as an example, the center line of the springboard can be selected as the reference line segment of the athlete's actual motion trajectory. Specifically, using a deep learning algorithm, according to different sports, the motion environment where the movement starting part of the action scene is located is detected, and then the center line of the detected area is extracted to characterize the reference line of the 3D trajectory model in the reference coordinate system. Taking diving as an example, video data of different competition or training environments are collected, the springboard or diving platform is finely marked, and then the target springboard or diving platform is segmented using an instance segmentation algorithm. For example, DeepMask can be used to train the collected data set, and the trained network model can be used to perform segmentation inference on the target springboard or diving platform for each frame of the video stream. Finally, the corresponding center line of the rectangular area of the springboard or diving platform is extracted based on the corner points of the rectangle. According to the relationship between the camera coordinate system and the image coordinate system, the coordinates of the center line in the reference coordinate system can be obtained, and its two end vertices are (O S , ).

[0150] As mentioned above, the reference coordinate system is the camera coordinate system of the fixed camera, in which the imaging principle (such as Figure 6 As shown in ), the relationship between the image coordinate system and the camera coordinate system, that is, the reference coordinate system, can be obtained by calculating the camera intrinsic parameters. Figure 6 In, O c is the optical center of the camera, that is, the coordinate origin of the reference coordinate system, and O is the coordinate origin of the image coordinate system. According to the imaging theorem, we can get:

[0151] .

[0152] Among them, (x, y) is the pixel coordinate in the image coordinate system, (fx, fy, u0, v0) is the camera internal parameter, which can be obtained by Zhang Zhengyou camera calibration method, ( ) are the coordinates in the camera coordinate system.

[0153] According to the above technical solution, this embodiment objectively presents the subjective evaluation method of competitive sports in the video by using standard motion trajectories, coordinate system alignment and scale correction methods, so that the live broadcast, rebroadcast and recorded broadcast of scoring sports events break the traditional visual experience mode and are more enjoyable.

[0154] In one embodiment, aligning the standard reference line segment with the reference line segment in a reference coordinate system comprises the following steps:

[0155] Step S231, translating the end position of the standard reference line segment to the end position of the reference line segment;

[0156] Step S232, determining the angle between the translated standard reference line segment and the reference line segment in the directions of each coordinate axis of the reference coordinate system;

[0157] Step S233 : performing coordinate transformation on the standard reference line segment based on the included angle, so that the standard reference line segment and the reference line segment are aligned in a reference coordinate system.

[0158] In this embodiment, the standard motion trajectory is calculated according to the end vertex of the standard reference line segment. Align the end vertex of the reference line segment detected in the real scene by translation ; Calculate the angle between the three axes of the standard reference line segment of the standard motion trajectory after translation and the detected reference line segment ( ); According to the coordinate system conversion formula, the standard reference line segment is rotated to the reference line segment of the actual scene:

[0159] , ,

[0160] .

[0161] According to the above technical solution, this embodiment achieves the effect of superimposing presentation in the video by aligning the coordinate system and scale.

[0162] The embodiments of the present invention provide embodiments of a motion scoring method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0163] like Figure 7 As shown, the present application provides a sports scoring system, which includes:

[0164] The trajectory determination module 10 is configured to determine an actual motion trajectory corresponding to an athlete's sports action in a video, as well as a standard motion trajectory, wherein the standard motion trajectory is determined based on the standard motion corresponding to the sports action. In one embodiment, the trajectory determination module 10 is configured to determine key point information for each segment of the actual motion trajectory in the video; and to fit the key point information for each segment to obtain the athlete's actual motion trajectory.

[0165] A scaling factor determination module 20 is configured to determine a scaling factor corresponding to each segment of the standard motion trajectory;

[0166] The scaling module 30 is configured to scale each segment of the standard motion trajectory according to the scaling factor. In one embodiment, the scaling module 30 is configured to obtain key point information of the segment of the actual motion trajectory corresponding to the scaling factor; wherein each segment corresponds to a different scaling factor; and multiply the scaling factor by the key point information to obtain the scaled segments of the standard motion trajectory.

[0167] The comparison module 40 is configured to determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory. In one embodiment, the comparison module 40 is configured to map the actual motion trajectory and the scaled standard motion trajectory into a camera coordinate system of a live camera; and compare each segment of the mapped actual motion trajectory with the mapped and scaled standard motion trajectory to obtain a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory. In one embodiment, the comparison module 40 is used to extract and match feature points of the image frames of the live camera and the image frames of the fixed camera at the same time to obtain matching point pairs; obtain an intrinsic matrix based on the matching point pairs and camera internal parameters; decompose the intrinsic matrix to obtain the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera; transform the actual motion trajectory according to the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera to obtain the actual motion trajectory in the camera coordinate system of the live camera; and transform the scaled standard motion trajectory according to the rotation and translation matrix of the camera and the rotation and translation matrix of the fixed camera to obtain the scaled standard motion trajectory in the camera coordinate system of the live camera.

[0168] The scoring module 50 is configured to determine a scoring result corresponding to the athlete based on the segment comparison result.

[0169] In one embodiment, an alignment module is connected before the trajectory determination module 10. This alignment module is configured to determine the reference line segment corresponding to the athlete at the starting point of the movement; obtain the standard reference line segment corresponding to the reference line segment; and align the standard reference line segment with the reference line segment in a reference coordinate system. In one embodiment, the alignment module is further configured to translate the end position of the standard reference line segment to the end position of the reference line segment; determine the angle between the translated standard reference line segment and the reference line segment in the directions of various coordinate axes of the reference coordinate system; and perform coordinate transformation on the standard reference line segment based on the angle to align the standard reference line segment with the reference line segment in the reference coordinate system.

[0170] The specific implementation of the motion scoring system of the present invention is basically the same as the above-mentioned embodiments of the motion scoring method, and will not be repeated here.

[0171] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a motion scoring program. When the motion scoring program is executed by a processor, the motion scoring program implements the various steps of the motion scoring method described above and can achieve the same technical effects. To avoid repetition, it will not be described here.

[0172] Since the storage medium provided in the embodiments of this application is the storage medium used to implement the method of the embodiments of this application, those skilled in the art will be able to understand the specific structure and variations of the storage medium based on the method described in the embodiments of this application, and therefore will not be described in detail here. All storage media used in the method of the embodiments of this application fall within the scope of protection to be provided by this application.

[0173] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0177] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0178] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0179] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A motion scoring method, characterized in that: The motion scoring method comprises: Determine the reference line segment corresponding to the athlete at the starting point of the movement; Obtaining a standard reference line segment corresponding to the reference line segment; translating the end position of the standard reference line segment to the end position of the reference line segment; Determining the angle between the translated standard reference line segment and the reference line segment in the directions of each coordinate axis of the reference coordinate system; Performing coordinate transformation on the standard reference line segment based on the included angle so that the standard reference line segment and the reference line segment are aligned in a reference coordinate system; Determine an actual motion trajectory corresponding to a sports action of an athlete in a video, and a standard motion trajectory, wherein the standard motion trajectory is determined according to a standard action corresponding to the sports action; Determining the scaling factor corresponding to each segment of the standard motion trajectory; scaling each segment of the standard motion trajectory according to the scaling factor; Determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; The scoring result corresponding to the athlete is determined according to the segment comparison result.

2. The motion scoring method according to claim 1, wherein: The step of determining the actual motion trajectory corresponding to the athlete's sports action in the video includes: Determine key point information of each segment of the actual motion trajectory described in the video; The key point information of each segment is fitted to obtain the actual motion trajectory of the athlete.

3. The motion scoring method according to claim 1, wherein: The step of scaling each segment of the standard motion trajectory according to the scaling factor comprises: Obtaining segmented key point information of the actual motion trajectory corresponding to the scaling factor; The scaling factor is multiplied by the key point information to obtain the scaled segments of the standard motion trajectory.

4. The motion scoring method according to claim 1, wherein: The step of determining a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory comprises: Mapping the actual motion trajectory and the scaled standard motion trajectory into a camera coordinate system of a live camera; The segments of the mapped actual motion trajectory and the mapped and scaled standard motion trajectory are compared to obtain a segment comparison result between the actual motion trajectory and the scaled standard motion trajectory.

5. The motion scoring method according to claim 4, wherein: The step of mapping the actual motion trajectory and the scaled standard motion trajectory into the camera coordinate system of the live camera includes: Extracting and matching feature points from the image frames of the live camera and the image frames of the fixed camera at the same moment to obtain matching point pairs; Obtaining an intrinsic matrix based on the matching point pairs and camera intrinsic parameters; Decomposing the intrinsic matrix to obtain the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera; The actual motion trajectory is transformed according to the rotation and translation matrix of the live camera and the rotation and translation matrix of the fixed camera to obtain the actual motion trajectory in the camera coordinate system of the live camera, and the scaled standard motion trajectory is transformed according to the rotation and translation matrix of the camera and the rotation and translation matrix of the fixed camera to obtain the scaled standard motion trajectory in the camera coordinate system of the live camera.

6. A sports scoring system, characterized in that: The sports scoring system includes: a trajectory determination module, configured to determine an actual motion trajectory corresponding to a sports action of an athlete in a video, and a standard motion trajectory, wherein the standard motion trajectory is determined based on a standard action corresponding to the sports action; A scaling factor determination module, configured to determine a scaling factor corresponding to each segment of the standard motion trajectory; A scaling module, configured to scale each segment of the standard motion trajectory according to the scaling factor; a comparison module, configured to determine a segmented comparison result between the actual motion trajectory and the scaled standard motion trajectory; A scoring module, configured to determine a scoring result corresponding to the athlete based on the segment comparison result; Before determining the actual motion trajectory corresponding to the athlete's sports action in the video, the method further includes: Determine the reference line segment corresponding to the athlete at the starting point of the movement; Obtaining a standard reference line segment corresponding to the reference line segment; translating the end position of the standard reference line segment to the end position of the reference line segment; Determining the angle between the translated standard reference line segment and the reference line segment in the directions of each coordinate axis of the reference coordinate system; The coordinate transformation is performed on the standard reference line segment based on the included angle, so that the standard reference line segment and the reference line segment are aligned in a reference coordinate system.

7. An intelligent terminal, characterized in that: The intelligent terminal includes: a memory, a processor, and a motion scoring program stored in the memory and executable on the processor. When the motion scoring program is executed by the processor, the steps of the motion scoring method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a motion scoring program, which, when executed by a processor, implements the steps of the motion scoring method according to any one of claims 1 to 5.

Images