Method for evaluating hitting process of golf ball and golf club

Abstract

The embodiment of the invention provides a method for evaluating the ball hitting process of a golf ball and a golf club, the method for evaluating the ball hitting process of the golf ball is operated on the golf club provided with a processor and multiple types of sensors, and the method comprises the steps that multiple types of information collected in the ball hitting process is obtained, the multiple types of information at least comprise a video frame sequence, sound information and attitude information; according to the multi-class information, a ball hitting process evaluation result is obtained, and the ball hitting process evaluation result comprises the contact position of the ball and the club face, the contact attribute of the club face, the ball and the ground, the movement track of the club and the rotation speed of the ball. According to the embodiment of the invention, the contact condition between the head of the golf club and the golf ball can be accurately measured, the swing path and speed can be autonomously tracked and accurately measured, the rotation and flight path of the golf ball can be calculated, and launching analysis, ball contact measurement and swing analysis are integrated.

Description

Technical Field

[0001] The present application relates to the field of information processing, and in particular, embodiments of the present application relate to a method for evaluating a golf hitting process and a golf club. Background Art

[0002] The contact between the club head (specifically the club face) and the ball is crucial in the process of hitting a golf ball, as it determines the trajectory and landing point of the ball. Golf learners often spend years trying to achieve stable and good contact with the ball, but because the ball flies too fast, players cannot directly observe the moment of impact, so they need to use equipment to evaluate the contact position.

[0003] In order to obtain the contact position, the related technology uses common cameras, emission analyzers or radar speed measurement devices that are independent of the club as independent devices to obtain the evaluation results of the hitting process. However, when using cameras and emission analyzers to obtain the evaluation results, the system is large and needs to be placed in front of the hitter, which brings a lot of inconvenience to the hitting; the radar speed measurement system is also large and needs to be placed behind the hitter, which affects the hitting process. In addition, the related technical solutions have at least the following technical defects: the corresponding equipment is expensive and has limited functions, and the accuracy of the evaluation results is poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and a golf club for evaluating the golf hitting process. The embodiments of the present application use various sensors (including image acquisition sensors, inertial measurement units, microphones, etc.) as part of the golf club, which can accurately measure the contact between the club head and the golf ball. At the same time, the embodiments of the present application can also autonomously track and extremely accurately measure the swing path and speed and calculate the rotation and flight trajectory of the ball, integrating launch analysis, ball contact measurement and swing analysis.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating a golf hitting process, which runs on a golf club provided with a processor and multiple types of sensors, the method comprising: acquiring multiple types of information collected during the current hitting process, wherein the multiple types of information include at least a video frame sequence, sound information, and posture information; acquiring a hitting process evaluation result based on the multiple types of information, wherein the hitting process evaluation result includes: the contact position between the ball and the club face, the contact properties between the club face and the ball and the ground, the movement trajectory of the club, and the rotation speed of the ball.

[0006] The embodiments of the present application collect multiple types of information in real time through sensors integrated in the golf club and then evaluate the hitting process based on the information, thereby reducing costs while improving the accuracy of the evaluation results and reducing the impact of obtaining the evaluation results on the hitting process of the hitter.

[0007] In some embodiments, the number of image acquisition sensors used to acquire the video frame sequence is one, wherein obtaining an evaluation result of the hitting process based on the multiple types of information includes: acquiring a first image of the ball contacting the club face from the video frame sequence; acquiring a homography matrix; converting the first image into a front-view contact map through the homography matrix; determining the pixel coordinate value of the contact point between the ball and the club face based on the front-view contact map to obtain the contact position between the ball and the club.

[0008] Some embodiments of the present application determine the contact position between the ball and the club head through an image obtained by a camera, thereby improving the speed of data processing.

[0009] In some embodiments, determining the pixel coordinate value of the contact point between the ball and the club face according to the front-view contact map includes: selecting a target area on the front-view contact map, wherein the target area is the contact area between the club face and the ball on the front-view contact map; constructing a circular projection surface area in an area adjacent to the target area of ​​the front-view contact map according to the actual size and shape of the ball; and using the center position of the circular projection surface area as the contact position between the ball and the club face.

[0010] The embodiments of the present application determine the final contact position between the golf ball and the club head in combination with the actual size of the golf ball, which can improve the accuracy of the obtained contact position.

[0011] In some embodiments, the image acquisition sensor used to acquire the video frame sequence includes a first image acquisition sensor and a second image acquisition sensor, and the first image acquisition sensor and the second image acquisition sensor are set on the club at a predetermined rotation angle and spatial displacement, wherein obtaining the evaluation result of the hitting process based on the multiple types of information includes: obtaining a first-perspective video frame sequence acquired by the first image acquisition sensor; obtaining a second-perspective video frame sequence acquired by the second image acquisition sensor; and obtaining the three-dimensional coordinates of the contact position between the ball and the club face based on the first-perspective video frame sequence and the second-perspective video frame sequence.

[0012] The embodiment of the present application can accurately obtain the three-dimensional coordinates of the contact position between the ball and the club head through images of different perspectives taken by two cameras.

[0013] In some embodiments, obtaining the evaluation result of the hitting process based on the multiple types of information includes: obtaining the deformation amount of the club during the hitting process; determining the initial motion trajectory of the club based on at least the video frame sequence; and correcting the initial motion trajectory based on the deformation amount to obtain the target motion trajectory of the club.

[0014] Some embodiments of the present application determine the motion trajectory of the club through images captured by an image acquisition sensor and a pre-estimated deformation amount, thereby improving the accuracy of the obtained evaluation results.

[0015] In some embodiments, determining the initial motion trajectory of the club face at least based on the video frame sequence includes: acquiring posture information at a first moment through a posture information acquisition sensor to obtain a first position and a first direction; acquiring a club posture image taken at a moment closest to the first moment from the video frame sequence, and acquiring a second position and a second direction through the club posture image; fusing the first position and the second position to obtain a target position at the first moment, and fusing the first direction and the second direction to obtain a target direction at the first moment, wherein the initial motion trajectory is a motion path determined by target positions and target directions at multiple moments.

[0016] Some embodiments of the present application obtain a more accurate initial motion trajectory by fusing the posture information obtained by the image and the posture information obtained by using the IMU, thereby improving the accuracy of the club's motion trajectory.

[0017] In some embodiments, obtaining the evaluation result of the hitting process according to the multiple types of information includes: identifying a rotation mark on the corresponding ball in the video frame sequence; and estimating the rotation speed of the ball according to the rotation mark.

[0018] Some embodiments of the present application evaluate the rotation speed of the ball by analyzing marks set on the surface of the golf ball captured in the image, thereby improving the accuracy of the value.

[0019] In some embodiments, the contact attributes include contact order and contact severity, wherein obtaining the evaluation result of the hitting process based on the multiple types of information includes: determining the contact order between the club and the ball and the club and the ground based on the attribute information of the sound information, or determining the contact severity between the club and the ground or lawn based on the attribute information of the sound information.

[0020] Some embodiments of the present application determine the contact properties between the club and the ball and the ground by analyzing the sound information collected by the microphone, thereby reducing costs and increasing the speed of obtaining contact property results.

[0021] In some embodiments, obtaining the evaluation result of the hitting process based on the multiple types of information includes: determining the moment T1 when the ball contacts the club through the sound information collected by the sound information collection sensor; filtering out the target image taken at the moment closest to the moment T1 from the video frame sequence; filtering out the second image when the club contacts the ground for the first time from the video frame sequence; and confirming the relative shooting order of the target image and the second image in the video frame sequence to obtain the contact order of the club with the ball and the ground.

[0022] Some embodiments of the present application obtain the contact sequence between the club, the ball and the ground through images and sound information collected by a microphone, thereby improving the accuracy of the evaluation result.

[0023] In some embodiments, the video frame sequence is acquired by an event sensor.

[0024] Some embodiments of the present application acquire sufficient image frames through a high-speed camera (eg, an event sensor), thereby improving the accuracy of the evaluation results acquired through the images and reducing the time for synchronizing different sensors to increase the data processing speed.

[0025] In a second aspect, some embodiments of the present application provide a data acquisition and processing device, which can be mounted on a golf club, and comprises: an image acquisition sensor, configured to acquire multiple frames of images of a golf club hitting a ball to obtain a video frame sequence; a posture information acquisition sensor, configured to acquire posture information of the golf club; a sound information acquisition sensor, configured to acquire sound information of the hitting process; and a computing processing unit, configured to execute the method described in any one of the embodiments of the first aspect.

[0026] In some embodiments, the data acquisition and processing device further includes an ambient light sensor configured to obtain ambient light information of a scene where the club is located; wherein the exposure parameter value of the image acquisition sensor is determined based on the ambient light information.

[0027] In a third aspect, an embodiment of the present application provides a golf club, which includes the data acquisition and processing device as described in the second aspect and a golf club.

[0028] The embodiments of the present application collect multiple types of information through multiple types of sensors disposed on the golf club, thereby obtaining a variety of evaluation results, thereby improving the versatility of the technical solution.

[0029] In some embodiments, the data collection and processing device includes a posture information collection sensor, a sound information collection sensor, and a computing and processing unit which are arranged inside the club.

[0030] The embodiment of the present application sets the sensor inside the golf club, which can at least minimize the impact of the sensor on the balance and weight of the club compared to the method of setting it outside the golf club.

[0031] In some embodiments, at least a portion of the image acquisition sensor is located inside the club.

[0032] In some embodiments, an optical portion of the image acquisition sensor is located outside the club and a remaining portion is located inside the club.

[0033] In some embodiments of the present application, only the optical part of the image acquisition sensor is disposed outside the club, which can solve the problem of affecting the sight line of the ball due to the placement of the image acquisition sensor outside the club to the greatest extent.

[0034] In a second aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any one of the embodiments of the first aspect.

[0035] In a third aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method described in any one of the embodiments of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 One of the schematic diagrams of a golf club provided in an embodiment of the present application;

[0038] Figure 2 A second schematic diagram of a golf club provided in an embodiment of the present application;

[0039] Figure 3 A flowchart of a method for evaluating a golf shot process provided by an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the club face composition provided in an embodiment of the present application;

[0041] Figure 5a A comparison diagram before and after the homography matrix transformation provided in an embodiment of the present application;

[0042] Figure 5b A schematic diagram of a process for obtaining a contact position according to the size of a golf ball provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a golf club provided with two cameras according to an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the process of fusing two types of sensor information to obtain the position and direction of any point on the motion trajectory of a golf club provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0047] At least in order to solve the problems existing in the background technology part, the embodiments of the present application use a high-speed camera integrated with the club to directly observe the contact between the ball and the club face, and through the visual algorithm homography (i.e., homography matrix), the coordinate values ​​of the contact points between the ball and the club face can be very accurately measured. The embodiments of the present application also use posture information acquisition sensors (e.g., inertial measurement units IMU) and sound information sensors (e.g., microphones) to accurately measure the contact sequence and degree of contact between the club head and the ball and grass, or obtain the movement trajectory of the club through the information of the corresponding sensors (because the sensor is fixed on the club, the collected information can more accurately reflect the movement trajectory of the club itself, that is, obtain the direction angle and position of the club at each moment of movement, etc.), which can provide a more complete evaluation result of the hitting process and improve the accuracy of the evaluation result.

[0048] That is, the embodiments of the present application use cameras and sensors (e.g., an inertial measurement unit for collecting posture information and a microphone for collecting sound information) as part of the golf club, which can accurately measure the contact between the club head and the golf ball. At the same time, the golf club of the embodiments of the present application can also autonomously track and very accurately measure the swing path and speed. In addition, the golf club of the embodiments of the present application can also calculate the rotation and flight trajectory of the ball, integrating launch analysis, ball contact measurement and swing analysis (i.e., all are performed by the golf club).

[0049] It is not difficult to understand that the data acquisition and processing device provided in the embodiment of the present application can be used as an accessory to an existing golf club, or it can be fixed on a club to use the club as a replacement for the club shaft, or it can be fixed on a completed golf club with a club and a club head to use it as a special practice club or a golf club equipped with new functions.

[0050] Please see Figure 1 , Figure 1 A golf club is provided for an embodiment of the present application, and the golf club includes a grip 110, a club 120, a club head 130, and a data acquisition and processing device 140. It should be noted that the real-time stream data acquisition and processing device 140 of the present application exemplarily includes multiple types of sensors and a computing processing unit. The embodiment of the present application collects multiple types of information through multiple types of sensors arranged on the club, and thus the types of evaluation results that can be obtained improve the versatility of the technical solution.

[0051] For example, Figure 2 As shown, in some embodiments of the present application, the data acquisition and processing device 140 exemplarily includes: an image acquisition sensor 141, a posture information acquisition sensor 142, a sound information acquisition sensor 143 and a computing processing unit 145 arranged inside the club 120.

[0052] The embodiment of the present application sets the sensor inside the golf club, which can at least minimize the impact of the sensor on the balance and weight of the club compared to the method of setting it outside the golf club.

[0053] In some embodiments of the present application, at least a portion of the image acquisition sensor 141 is located inside the club 120 .

[0054] In some embodiments of the present application, the optical part (not shown) of the image acquisition sensor 141 is located outside the club 120, and the remaining part is located inside the club. Figure 1 It is only used to illustrate the approximate positional relationship between the golf club and the data acquisition and processing device, and does not limit the data acquisition and processing device to be located entirely inside the golf club.

[0055] It is not difficult to understand that some embodiments of the present application only set the optical part of the image acquisition sensor outside the club, which can solve the problem of affecting the sight line of the ball due to setting the image acquisition sensor outside the club to the greatest extent.

[0056] like Figure 2 As shown, in some embodiments of the present application, the data acquisition and processing device also includes an ambient light sensor configured to obtain ambient light information of the scene where the club is located; wherein the exposure parameter value of the image acquisition sensor is determined based on the ambient light information.

[0057] For example, in some embodiments of the present application, the ambient light information is collected by an ambient light sensor, and then the image acquisition sensor or the computing processing unit adjusts the exposure parameter value of the image acquisition sensor according to the ambient light information to obtain a captured image with better quality.

[0058] like Figure 2 As shown, the evaluation results of the hitting process obtained by using the data acquisition and processing device or golf club provided by the embodiment of the present application include: the contact position between the ball and the club face, the contact properties of the club, the ball and the ground (contact sequence or contact severity), the movement trajectory of the club (determined by the direction and position of each movement trajectory point) and the rotation speed of the ball.

[0059] The following is an example of Figure 1 A method for evaluating a golf hitting process is performed by a data acquisition and processing device.

[0060] like Figure 3 As shown, an embodiment of the present application provides a method for evaluating a golf ball hitting process, which is run on a golf club (as shown in 1) provided with a processor and multiple types of sensors. The method includes:

[0061] S101, obtaining multiple types of information collected during the current hitting process.

[0062] It should be noted that the multiple types of information described in S101 at least include video frame sequences, sound information, and posture information. For example, in some embodiments of the present application, the video frame sequence is obtained by shooting the hitting process with an image acquisition sensor (e.g., a camera) disposed on the golf club, the sound information is obtained by collecting the sound information of the hitting process with a sound information acquisition sensor (e.g., a microphone) disposed on the golf club, and the posture information is obtained by collecting the posture signal of the hitting process with a posture information acquisition sensor (e.g., an IMU) disposed on the golf club.

[0063] For example, in some embodiments of the present application, a high-speed camera with a shooting rate greater than or equal to 1000 frames / second can be selected to capture a video frame sequence, in some embodiments of the present application, a sensor that can detect six-degree-of-freedom motion IMU can be selected as a sound information collection sensor, and in some embodiments of the present application, sound information can also be collected through a microphone, and an ambient light sensor can be used to use ambient light information to adjust the exposure parameters of the image acquisition sensor according to the ambient light information. The mechanical housing is firmly mounted on the club through a designed rigid connection.

[0064] S102, obtaining a ball hitting process evaluation result according to the multiple types of information, wherein the ball hitting process evaluation result includes: a contact position between the ball and the club face, contact properties between the club face, the ball and the ground, a movement trajectory of the club, and a rotation speed of the ball.

[0065] For example, some embodiments of the present application are Figure 2 The computing processing unit 145 executes the method for obtaining the evaluation result of the hitting process according to the multiple types of information. For example, the computing processing module includes a computing module (SoC based on ARM or QCOM) and a communication module, and the computing processing module is used to connect various types of sensors.

[0066] It is not difficult to understand that the embodiments of the present application collect multiple types of information in real time through sensors integrated on the golf club and then evaluate the hitting process based on this information, thereby reducing costs while improving the accuracy of the evaluation results.

[0067] The following is an example of obtaining the contact position between the ball and the club face (i.e., the club face included in the club head) through an image acquisition sensor, wherein the club face The Club Face refers to the plane on the golf club that contacts the ball, that is, the part that directly collides with the ball when hitting the ball. Its design and material are crucial to the accuracy, power and rotation of the shot.

[0068] like Figure 4 As shown in the figure, the club face is a schematic diagram of the composition of the club face, and the club face shown in the figure includes: top edge 131, sweet spot 132, toe 133, leading edge 134, root 135 and hosel 136, wherein the top edge TopEdge is the uppermost edge of the golf ball face, which affects the aerodynamic characteristics and flight stability of the ball; the sweet spot of the club face Sweet Spot is the best hitting position of a golf club. When the club hits the ball at the correct angle and path, the sweet spot can produce the best hitting effect and make the ball fly straighter and farther; the hosel is the part connecting the shaft and the head of the club. It ensures a firm connection between the shaft and the head and helps to transmit the power when hitting the ball; the toe is the front part of the club head, relative to the heel. When hitting the ball, avoiding hitting the ball with the toe can reduce the possibility of deviating from the target line; the heel is the rear end of the club head, which is opposite to the toe. During normal hitting, the ball should contact the center or sweet spot of the club face, not the heel; the leading edge is the front edge of the club face and the first part of the club that contacts the ball when hitting the ball. Its design and shape are crucial to the accuracy and control of the shot.

[0069] It should be noted that, through the embodiments of the present application, the contact position between the ball and the club face can be accurately determined, and then the effect of the current shot can be judged, which can ultimately help the batsman improve his batting skills.

[0070] For example, in some embodiments of the present application, the number of image acquisition sensors used to acquire the video frame sequence is one, and the process of acquiring the evaluation result of the hitting process according to the multiple types of information in S102 exemplarily includes:

[0071] In the first step, a first image of the ball contacting the club face is obtained from the video frame sequence.

[0072] For example, during a ball hitting process, the first image is as follows: Figure 5a The image shown on the left shows the process of golf ball 150 contacting club face 137.

[0073] The second step is to obtain the homography matrix.

[0074] The following is an example of how to obtain the homography matrix and the homography transformation process. The first sub-step is feature point matching: detect and match the corresponding feature points in the oblique view image and the desired front view (or reference image). This can be done through feature detection algorithms such as SIFT, SURF or ORB. The second sub-step is to calculate the homography matrix. With enough matching points, you can use the RANSAC algorithm or the least squares method to estimate a homography matrix. This matrix describes the transformation from the oblique view of the club face to the front view.

[0075] It should be noted that the homography matrix is ​​used to describe the mapping relationship between two planes. Figure 5a As shown in the figure on the right, the homography matrix is ​​used to describe the plane 15 parallel to the paper surface and Figure 5a The mapping relationship between the middle club face 137.

[0076] The third step is to convert the first image into a front view contact map by using the homography matrix.

[0077] Some embodiments of the present application use the calculated homography matrix to map all pixels in the oblique perspective image, i.e., the first image, to a new normal perspective, which usually involves image reprojection and interpolation to ensure that the image at the new perspective is continuous and complete. It should be noted that in some embodiments of the present application, the image can be post-processed, for example, cropped, scaled, or the color adjusted to obtain a better visual effect.

[0078] like Figure 5b The right figure shows the contact diagram from the front view, which shows the Figure 5aThe first image on the left is the corresponding image after being converted into the normal view contact map.

[0079] The fourth step is to determine the pixel coordinate value of the contact point between the ball and the club face according to the front view contact map, and obtain the contact position between the ball and the club.

[0080] That is to say, before the homography matrix conversion is used in some embodiments of the present application, the contact position between the ball and the club face is the position of the ball on the image taken from an oblique perspective, that is, the projection position of the ball on the club face when the club face forms any angle with the horizontal plane during the actual hitting process; after the homography matrix conversion is used, the position of the ball on the image taken from a normal perspective is obtained, and the image corresponding to the normal perspective, that is, the normal perspective contact map, is the position where the projection of the ball falls on the surface when the club face is parallel to the paper surface.

[0081] For example, in some embodiments of the present application, the fourth step of contacting the image from the front view (such as Figure 5b The process of determining the pixel coordinate value of the contact point between the ball and the club face on the right image) includes: Figure 5b The target area is selected on the image corresponding to the right figure, wherein the target area (not shown in the figure, i.e. Figure 5b The contact area between the ball and the club face in the right figure) is the contact area between the club face and the ball in the front view contact diagram; a circular projection surface area (such as Figure 5b The circular projection surface area 153 shown in the left figure is a circular area obtained by the perspective matrix on the imaging plane of the front view contact diagram according to the actual shape and size of the golf ball); the center position of the circular projection surface area (such as Figure 5b The center of the circular projection area in the left figure (not shown in the figure) is used as the contact position between the ball and the club face. It is not difficult to understand that the embodiment of the present application combines the actual size of the golf ball to determine the final contact position between the ball and the club head to improve the accuracy of the obtained contact position.

[0082] Combine the following Figure 5a and Figure 5b The method and principle of obtaining the contact position between the ball and the club face through homography are exemplified.

[0083] like Figure 5a The left figure to the right figure of the present invention shows the process of straightening the club face by homography in some embodiments of the present application. Figure 5b The right figure to the left figure shows some embodiments of the present application using the size of the golf ball (the size is mapped on the imaging plane of the front view through the perspective matrix to obtain Figure 5bThe circular shadow area shown is the circular projection area 153) and the information of the club surface to accurately estimate the contact point on the club surface. Figure 5a The right picture and Figure 5b In the right picture, the surface of the club is parallel to the page. Figure 5a The right figure of shows a plane parallel to the page (i.e., a plane 152 parallel to the paper surface). Some embodiments of the present application straighten by homography Figure 5a The club face on the left. In computer vision and image processing, a homography is a 2D transformation that describes the mapping relationship between two planes. It is often used to map points in one image to corresponding points in another image, which may be from different perspectives or taken by different cameras. When converting from a slanted perspective to an on-axis perspective, a homography can be used to correct the perspective distortion in the image so that it looks like it was taken from a frontal perspective. This is very common in applications such as document scanning and image foot prints inside buildings.

[0084] Figure 5a The left picture is an image of the moment when the club and the ball touch each other, which is selected from the video sequence frames of the hitting process shot by a high-speed camera (it can also be an image synthesized by algorithms such as interpolation). Figure 5a The right figure uses the pre-obtained homography matrix and Figure 5a The contact diagram of the front view obtained by multiplying the matrix corresponding to the left figure of the drawing by the left figure, combined with the drawing, it can be seen that the ball is deformed when the club face is transformed into a perspective parallel to the paper surface. Therefore, it is unreasonable to estimate the contact point between the club and the ball by the deformed ball. In order to solve this technical problem, the embodiment of the present application is from Figure 5a A target area is selected from the right image of the graph. The target area is the area where the club contacts the ball (this area is less deformed during homography transformation). Then, a circular virtual golf ball is generated based on the area and the size of the golf ball (the size of the pixel image of the golf ball on the plane is determined according to the internal or external parameters of the high-speed camera). Figure 5b The left picture, Figure 5b The right picture and Figure 5a The right image is the same image. Figure 5b The club face in the left picture has the groove area marked on it, so after determining the contact position between the club face and the ball, the effect of the shot can be determined by judging whether it is in this area.

[0085] That is, some embodiments of the present application can determine the contact point on the club surface through a calibrated perspective / distortion single camera (a single camera continuously captures the ball contacting the club to obtain a video frame sequence) using the following information and techniques: 1) accurate ball size (all golf balls are the same size); 2) perspective transformation, converting the oblique perspective to the axial perspective (refer to Figure 5a and Figure 5b ).1111

[0086] It is not difficult to understand that some embodiments of the present application determine the contact position between the ball and the club head (for example, the ball and the club face located at the club head) through an image obtained by a camera, thereby improving the speed of data processing.

[0087] Combine the following Figure 6 The method of acquiring the three-dimensional position points of the contact between the ball and the club face by using two image acquisition sensors is exemplified.

[0088] like Figure 6 As shown, for example, in some embodiments of the present application, the image acquisition sensor for acquiring the video frame sequence includes a first image acquisition sensor 146 and a second image acquisition sensor 147, and the first image acquisition sensor 146 and the second image acquisition sensor 147 are set on the club 10 with a predetermined rotation angle and spatial displacement. The corresponding S102 process of obtaining the evaluation result of the hitting process according to the multiple types of information exemplarily includes: obtaining a first-perspective video frame sequence acquired by the first image acquisition sensor; obtaining a second-perspective video frame sequence acquired by the second image acquisition sensor; and obtaining the three-dimensional coordinates of the contact position between the ball and the club face according to the first-perspective video frame sequence and the second-perspective video frame sequence.

[0089] That is to say, in some other embodiments of the present application, a dual-camera architecture may be used and the two cameras may be installed at a certain rotation angle and spatial displacement, so as to stereo-synthesize the 3D position of the ball relative to the contact point of the club surface. The position, orientation and design (focal length, field of view, resolution) of the camera are optimized through end-to-end performance prediction. For example, two cameras can measure the relative position of the ball and the club head through triangulation. For example, a pair of cameras can achieve 2D to 3D mapping or a 2D to 3D conversion can be achieved through the continuous change of the position of a camera. This is very helpful for estimating the position of the ball relative to the surface of the club head. The specific details are as follows: by using images taken from different perspectives, feature matching between images can be used to calculate the points in the 3D scene and the position of the camera. This process is called 3D reconstruction. The 3D reconstruction process includes: the first step, feature detection and matching, that is, first, key points (such as corner points, edges, etc.) are detected in the input image, and these key points should be able to be stably identified between different images. Then, these key points are described by feature descriptors (such as SIFT, SURF, ORB, etc.), and matching feature point pairs are found between different images); the second step is to use the matching feature point pairs to estimate the fundamental matrix (FundamentalMatrix), which is a 3x3 matrix that describes the geometric relationship between corresponding points in different perspectives, especially the epipolar geometry (Epipolar Geometry); the third step is to calculate the intrinsic moment from the fundamental matrix if the intrinsic parameters of the camera (focal length, optical center, etc.) are known; the fourth step is to "Triangulate to get the 3D points" in computer vision and 3D reconstruction. It refers to the calculation of the 3D coordinate points of an object from images under multiple perspectives by triangulation. This process exemplarily involves camera calibration, feature extraction, feature matching, and triangulation. The specific 3D reconstruction process can refer to the relevant prior art and will not be described in detail here to avoid repetition.

[0090] Typically, an initial factory calibration is done during product manufacturing to obtain "intrinsic parameters" of camera properties (e.g., focal length, center pixel), and "extrinsic parameters" of camera orientation and position relative to a global coordinate system (e.g., club face). During the life of the club, the camera may drift and move relative to the initial calibration. Extrinsic parameters may be calibrated periodically relative to the club face, and intrinsic parameters may require the club to be sent to a service center or calibrated in the field using calibration accessories.

[0091] It is not difficult to understand that the embodiments of the present application can accurately obtain the three-dimensional coordinates of the contact position between the ball and the club head through images taken from different perspectives by two cameras.

[0092] The following is an illustrative description of the process of measuring the swing trajectory provided by some embodiments of the present application.

[0093] For example, in some embodiments of the present application, the process of obtaining the batting process evaluation result according to the multiple types of information in S102 exemplarily includes:

[0094] The first step is to obtain the deformation of the club during the hitting process.

[0095] The second step is to determine the initial motion trajectory of the club at least based on the video frame sequence.

[0096] In some embodiments of the present application, if a high-speed camera is used to directly determine the initial motion trajectory based on the image; if a low-speed camera is used, the IMU and image information need to be fused to obtain the initial motion trajectory. For example, in some embodiments of the present application, the process of determining the initial motion trajectory of the club face at least according to the video frame sequence in the second step exemplarily includes: obtaining the posture information of the first moment (any moment of the current hitting process) through the posture information acquisition sensor to obtain the first position, the first direction and the first speed; obtaining the club posture image taken at the moment closest to the first moment from the video frame sequence, and obtaining the second position and the second direction through the club posture image; fusing (for example, selecting one of the two or a fusion algorithm such as weighted summation) the first position and the second position to obtain the target position at the first moment, and fusing the first direction and the second direction to obtain the target direction at the first moment, wherein the initial motion trajectory is a motion path determined by the target positions and target directions at multiple moments. It is not difficult to understand that some embodiments of the present application obtain a more accurate initial motion trajectory by fusing the posture information obtained by the image and the posture information obtained by the IMU, thereby improving the accuracy of the obtained club motion trajectory.

[0097] That is to say, Figure 7 The process of fusing visual information and posture information in some embodiments of the present application shown in the figure includes: using a camera to capture images of the hitting process, extracting and matching features of the captured images, and performing posture estimation based on image features to obtain second position and second direction information; acquiring information of each axis collected through the IMU, and performing IMU data integration on these information to obtain the first position, first direction, and first speed; fusing the first position and the second position through a fusion module, and fusing the first direction and the second direction to obtain the target position and target direction (corresponding to a certain acquisition time). Connecting these trajectory points can obtain real-time trajectory information of the image acquisition sensor and the IMU unit. Since the image acquisition sensor and the IMU are set on the golf club, the data collected by these sensors can initially reflect the movement trajectory of the club in real time.

[0098] The third step is to correct the initial motion trajectory according to the deformation amount to obtain the target motion trajectory of the club.

[0099] Some embodiments of the present application determine the motion trajectory of the club through images captured by an image acquisition sensor and a pre-estimated deformation amount, thereby improving the accuracy of the obtained evaluation results.

[0100] That is to say, some embodiments of the present application determine the swing angle by the club deformation, and the swing angle is positively correlated with the club deformation. Since the camera can not only measure the sphere, but also capture the external environment, some embodiments of the present application can use the Visual Inertial Odometry (VIO) method to estimate the golf club's own posture (position and direction, 6 degrees of freedom) in real time, so it can measure the golf swing path and speed very accurately. In some embodiments of the present application, the club shaft will bend during the swing process and at the moment of hitting the ball. The embodiments of the present application model this bending through the known shaft model and swing speed, or capture and measure it through the camera, and compensate in the algorithm when reporting the swing path. For example, if the image acquisition sensor has a camera with 1000 frames per second, it may be possible to reduce or weaken the influence of the inertial measurement unit (IMU) signal when calculating its own posture. Usually, the IMU reads data at a frequency of 200 to 1200 Hz, but consumes less power than the camera.

[0101] Visual-Inertial Odometry (VIO) is a common algorithm in computer vision (CV). Through this algorithm, the spatial position, angle, speed and angular velocity of the sensor itself can be inferred. Visual Inertial Odometry (VIO) is a position tracking technology that combines visual cameras and inertial sensors (IMU). Visual Inertial Odometry (VIO) is a technology that uses one or more cameras and one or more inertial sensors (IMU) to measure the state of the sensor. The "state" here refers to the physical quantities such as the posture and speed of the intelligent body (such as a drone) under specific degrees of freedom. Visual Inertial Odometry (VIO) is an important position tracking technology that accurately estimates the state of the swing trajectory of a golf club by fusing vision (i.e., the video sequence frames of the hitting process collected by a high-speed camera) and inertial information (i.e., the information collected and output by the inertial sensor IMU). For example, in some embodiments of the present application, the method for estimating the real-time position, speed and direction of the club exemplarily includes: acquiring a video frame sequence corresponding to the hitting process from an image acquisition sensor; acquiring measurement values ​​of an accelerometer and a gyroscope from an inertial sensor IMU, for example, these measurement values ​​include linear acceleration and angular velocity, etc.; using the raw data acquired by at least one of the visual and inertial sensors IMU to jointly estimate the values ​​of a set of variables related to the club, for example, estimating the real-time position, speed and direction of the club, etc. For example, in some embodiments of the present application, if it is confirmed that the frequency of the video sequence acquired by the visual information acquisition unit is greater than a set threshold, the swing trajectory is estimated based on the information of the video acquisition unit; if it is confirmed that it is less than the set threshold, it is necessary to combine the data acquired by the visual information acquisition unit and the IMU unit to jointly determine the swing attribute information.

[0102] The technical solution for obtaining the rotation speed of the ball is exemplified below.

[0103] In some embodiments of the present application, the process of obtaining the batting process evaluation result according to the multiple types of information in S102 exemplarily includes:

[0104] The first step is to identify the rotation marks on the corresponding ball in the video frame sequence.

[0105] For example, in some embodiments of the present application, images containing balls are screened out from video sequence frames by computer vision algorithms, and these images are arranged in the order of shooting, and rotation attribute marks are further identified from each ball according to the computer vision algorithm. For example, these attribute marks can be points or lines or other marks or features that are easier to identify.

[0106] The second step is to estimate the rotation speed of the ball according to the rotation mark.

[0107] For example, in some embodiments of the present application, the rotation attribute mark is a plurality of marks distributed on the sphere at different and equal intervals. For example, the rotation speed of the ball is determined by identifying the frequency of occurrence of the same rotation attribute mark or the time interval between adjacent or spaced marks.

[0108] It is not difficult to understand that some embodiments of the present application evaluate the rotation speed of the ball by analyzing the marks set on the surface of the golf ball captured in the image, thereby improving the accuracy of the value.

[0109] That is to say, in some embodiments of the present application, obtaining the evaluation result of the hitting process also includes obtaining the rotation and flight trajectory of the ball. In some embodiments of the present application, due to the high frame rate of the camera corresponding to the image acquisition sensor, the rotation speed of the ball (2000 to 9000 rpm) can be captured by capturing the features or marks on the ball. Some embodiments of the present application capture the flight path of the ball after being hit by the club. Based on these images, its flight trajectory (direction and speed) can be calculated.

[0110] The following is an exemplary description of the process of obtaining the contact properties of the club, the ball and the ground. This contact is also a parameter for obtaining the evaluation results of the hitting process in some embodiments of the present application.

[0111] For example, in some embodiments of the present application, the contact attributes include contact order and contact severity, wherein obtaining the evaluation result of the hitting process based on the multiple types of information includes: determining the contact order between the club and the ball and the club and the ground based on the attribute information of the sound information (for example, the attribute information here exemplarily includes the change characteristics of the timbre frequency over time), or, determining the contact severity between the club and the ground or lawn based on the attribute information of the sound information (for example, the attribute information includes the strength of the sound and / or the sound spectrum information, for example, the sound of grass is high frequency and the sound of the ground is mostly low frequency).

[0112] Some embodiments of the present application determine the contact properties between the club and the ball and the ground by analyzing the sound information collected by the microphone, thereby reducing costs and increasing the speed of obtaining contact property results.

[0113] For example, in some embodiments of the present application, obtaining the evaluation result of the hitting process based on the multiple types of information includes: determining the moment T1 when the ball contacts the club through the sound information collected by a sound information collection sensor (for example, a microphone); filtering out a target image taken at a moment closest to the moment T1 from the video frame sequence; filtering out a second image when the club contacts the ground for the first time from the video frame sequence; and confirming the relative shooting order of the target image and the second image in the video frame sequence to obtain the contact order of the club with the ball and the ground.

[0114] Some embodiments of the present application obtain the contact sequence between the club, the ball and the ground through images and sound information collected by a microphone, thereby improving the accuracy of the evaluation result.

[0115] It should be noted that in some embodiments of the present application, the video frame sequence is acquired by an event sensor. Some embodiments of the present application acquire sufficient image frames through a high-speed camera (e.g., an event sensor), thereby improving the accuracy of the evaluation results acquired through the image and reducing the time for synchronizing different sensors to improve the data processing speed.

[0116] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method described in the above embodiments can be implemented.

[0117] Some embodiments of the present application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the method described in the above embodiments is implemented.

[0118] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0120] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0123] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for evaluating a golf shot, which is run on a golf club provided with a processor and multiple types of sensors, the method comprising: Acquire multiple types of information collected during the current hitting process, wherein the multiple types of information at least include video frame sequences, sound information, and posture information; The evaluation result of the ball hitting process is obtained according to the multiple types of information, wherein the evaluation result of the ball hitting process includes: the contact position between the ball and the club face, the contact properties between the club face and the ball and the ground, the movement trajectory of the club and the rotation speed of the ball.

2. The method according to claim 1, characterized in that The number of image acquisition sensors used to acquire the video frame sequence is one, wherein: The step of obtaining the batting process evaluation result according to the multiple types of information includes: Acquire a first image of the ball in contact with the club face from the sequence of video frames; Get the homography matrix; Converting the first image into an orthographic contact map by using the homography matrix; The pixel coordinate values ​​of the contact points between the ball and the club face are determined according to the front-view contact map to obtain the contact positions between the ball and the club.

3. The method according to claim 2, characterized in that Determining the pixel coordinate value of the contact point between the ball and the club face according to the front view contact map includes: selecting a target area from the front-view contact map, wherein the target area is a contact area between the club face and the ball on the front-view contact map; Constructing a circular projection surface area in the vicinity of the target area of ​​the orthographic contact diagram according to the actual size and shape of the ball; The center position of the circular projection surface area is used as the contact position between the ball and the club face.

4. The method according to claim 1, characterized in that The image acquisition sensor for acquiring the video frame sequence includes a first image acquisition sensor and a second image acquisition sensor, wherein the first image acquisition sensor and the second image acquisition sensor are arranged on the club at a predetermined rotation angle and spatial displacement, wherein: The step of obtaining the batting process evaluation result according to the multiple types of information includes: Acquire a first-viewing angle video frame sequence acquired by the first image acquisition sensor; Acquire a second-viewing angle video frame sequence acquired by the second image acquisition sensor; The three-dimensional coordinates of the contact position between the ball and the club face are acquired according to the first-view video frame sequence and the second-view video frame sequence.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining the batting process evaluation result according to the multiple types of information includes: Obtaining the deformation amount of the club during the ball hitting process; determining an initial motion trajectory of the club at least based on the video frame sequence; The initial motion trajectory is corrected according to the deformation amount to obtain the target motion trajectory of the club.

6. The method according to claim 5, characterized in that The determining of the initial motion trajectory of the club face at least according to the video frame sequence comprises: Acquire posture information at a first moment through a posture information acquisition sensor to obtain a first position and a first direction; Acquire a club posture image taken at a moment closest to the first moment from the video frame sequence, and acquire a second position and a second direction through the club posture image; The first position and the second position are fused to obtain the target position at the first moment, and the first direction and the second direction are fused to obtain the target direction at the first moment, wherein the initial motion trajectory is a motion path determined by the target positions and target directions at multiple moments.

7. The method according to claim 6, characterized in that The step of obtaining the batting process evaluation result according to the multiple types of information includes: identifying a rotation mark on a corresponding ball in the sequence of video frames; The rotation speed of the ball is estimated based on the rotation mark.

8. The method according to claim 7, characterized in that The contact attributes include contact order and contact severity, where: The step of obtaining the batting process evaluation result according to the multiple types of information includes: The contact order between the club and the ball and between the club and the ground is determined according to the attribute information of the sound information, or the contact severity between the club and the ground or lawn is determined according to the attribute information of the sound information.

9. The method according to claim 7, characterized in that The step of obtaining the batting process evaluation result according to the multiple types of information includes: Determine the moment T1 when the ball contacts the club by using the sound information collected by the sound information collection sensor; Filtering from the video frame sequence to obtain a target image captured at a time closest to the time T1; Filtering out a second image in which the club first contacts the ground from the video frame sequence; The relative shooting order of the target image and the second image in the video frame sequence is confirmed to obtain the contact order of the club, the ball and the ground.

10. The method according to claim 1, characterized in that The video frame sequence is acquired through an event sensor.

11. A data acquisition and processing device, which can be mounted on a golf club, comprising: An image acquisition sensor is configured to acquire multiple frames of images of a club hitting a ball to obtain a video frame sequence; A posture information collection sensor configured to collect the posture information of the club; A sound information collection sensor is configured to collect sound information of a ball hitting process; as well as, A computing processing unit, configured to execute the method according to any one of claims 1 to 10.

12. The data acquisition and processing device according to claim 11, characterized in that: The data acquisition and processing device also includes an ambient light sensor configured to obtain ambient light information of the scene where the club is located; Wherein, the exposure parameter value of the image acquisition sensor is determined according to the ambient light information.

13. A golf club, characterized in that: The golf club comprises the data collection and processing device as claimed in claim 11 and a golf club.

14. The golf club according to claim 13, wherein: The data collection and processing device includes a posture information collection sensor, a sound information collection sensor and a calculation processing unit which are arranged inside the club.

15. The golf club according to claim 14, wherein: At least a portion of the image acquisition sensor is located inside the club.

16. The golf club according to claim 15, wherein: An optical portion of the image acquisition sensor is located outside the club, and a remaining portion is located inside the club.

17. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 10 can be implemented.

18. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images