Method and device for estimating information about golf swing, and non-transitory computer-readable recording medium

Abstract

To provide a method for estimating information about a golf swing.SOLUTION: According to one aspect of the present invention, a method for estimating information about a golf swing is provided, the method comprising the steps of: when a captured image of a user's golf swing is acquired, detecting at least one of a plurality of keypoints on a shaft of a club and a plurality of keypoints on a face of the club, and at least one joint of the user from the captured image by using an artificial neural network model; and estimating a stance of the club on the basis of at least one of positions of the plurality of keypoints on the shaft and positions of the plurality of keypoints on the face, and estimating information about the user's golf swing by referring to the posture of the club and a position of the at least one joint of the user.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to a method, device and non-transitory computer-readable medium for estimating information about a golf swing. [Background technology]

[0002] Recently, a technology has been introduced that analyzes a video of a golfer's swing to provide the golfer with useful information.

[0003] An example of related art in this regard is the technology relating to a golf clinic system and its operation method using an image processor method disclosed in Korean Patent Publication No. 2009-105031. According to this technology, the system includes a plurality of markers attached to the body and golf club of a golf trainee, a plurality of cameras that collect images of the golf trainee's swing, an image analyzer that reconstructs the 2D images collected by the plurality of cameras into 3D images, extracts the spatial coordinates of the markers according to the movement, analyzes the angle values ​​of the body segments and data for each phase in real time, and outputs the clinic results in the form of a report, and a database in which kinetic clinic information for the swing is matched with member information and stored as digital data.

[0004] However, with the above-mentioned conventional technologies and other technologies introduced so far, in order to analyze a golfer's swing, it has been necessary to use separate expensive equipment to recognize the golfer's posture and movement, or to attach separate sensors (markers) to the golfer's body and golf club. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Publication No. 2009-105031 Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to solve all of the problems of the prior art mentioned above. [Means for solving the problem]

[0007] Another object of the present invention is to, when a photographed image of a user's golf swing is acquired, detect at least one of a plurality of key points related to a club shaft and a plurality of key points related to a club face, and at least one joint of the user, from the photographed image using an artificial neural network model; estimate a posture of the club based on at least one of the positions of the plurality of key points related to the shaft and the positions of the plurality of key points related to the face; and estimate information about the user's golf swing by referring to the posture of the club and the position of at least one joint of the user.

[0008] Another object of the present invention is to reduce the weight of an artificial neural network model by using depthwise convolution and pointwise convolution, and to estimate information about a club posture and a user's golf swing from a captured image of the user's golf swing by using the reduced weighted artificial neural network model.

[0009] Another object of the present invention is to estimate information about a user's golf swing by referring to not only the position of at least one joint of the user but also the posture of the club estimated from a photographed image of the user's golf swing.

[0010] A typical configuration of the present invention to achieve the above object is as follows.

[0011] According to one aspect of the present invention, there is provided a method including the steps of: when a photographed image of a user's golf swing is acquired, detecting at least one of a plurality of key points related to a club shaft and a plurality of key points related to a face of the club, and at least one joint of the user, from the photographed image using an artificial neural network model; estimating a posture of the club based on at least one of positions of the plurality of key points related to the shaft and positions of the plurality of key points related to the face, and estimating information about the user's golf swing by referring to the posture of the club and the position of at least one joint of the user.

[0012] According to another aspect of the present invention, there is provided a device including: a club and joint detection unit that, when a photographed image of a user's golf swing is acquired, detects at least one of a plurality of key points related to a club shaft and a plurality of key points related to a face of the club and at least one joint of the user from the photographed image using an artificial neural network model; and a golf swing information estimation unit that estimates an attitude of the club based on at least one of positions of the plurality of key points related to the shaft and positions of the plurality of key points related to the face, and estimates information about the user's golf swing by referring to the attitude of the club and a position of at least one joint of the user.

[0013] In addition, other methods and devices for embodying the present invention, and a non-transitory computer-readable recording medium having a computer program for carrying out the method are also provided. [Effects of the Invention]

[0014] According to the present invention, when a photographed image of a user's golf swing is acquired, at least one of a plurality of key points related to the shaft of the club and a plurality of key points related to the face of the club and at least one joint of the user is detected from the photographed image using an artificial neural network model, and the posture of the club is estimated based on at least one of the positions of the plurality of key points related to the shaft and the positions of the plurality of key points related to the face, and information about the user's golf swing is estimated by referring to the posture of the club and the position of at least one joint of the user, thereby making it possible to estimate information about the user's golf swing using only the photographed image without using a separate sensor (marker) or equipment.

[0015] In addition, according to the present invention, a lightweight artificial neural network model is made using depthwise convolution and pointwise convolution, and the lightweight artificial neural network model is used to estimate the club posture from a photographed image of a user's golf swing, thereby enabling accurate and efficient estimation of information about the user's golf swing on a mobile device without using a separate sensor (marker) or equipment.

[0016] Furthermore, according to the present invention, when estimating information about a user's golf swing, by referring not only to the position of at least one joint of the user but also to the posture of the club estimated from a photographed image of the user's golf swing, it becomes possible to estimate information that is difficult to grasp from the position of at least one joint of the user alone (for example, information about wrist control and / or club control). [Brief explanation of the drawings]

[0017] FIG. 1 is a diagram illustrating in detail the internal configuration of a device according to an embodiment of the present invention.

[0018] FIG. 2(a) is a diagram illustrating an example of a process of performing a general convolution according to an embodiment of the present invention.

[0019] FIG. 2(b) is a diagram illustrating an example of a process of performing depthwise convolution and pointwise convolution according to an embodiment of the present invention.

[0020] 3 to 7 are diagrams illustrating a process of estimating information about a club posture and a user's golf swing according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] The following detailed description of the present invention refers to the accompanying drawings, which show, by way of example, specific embodiments in which the present invention may be practiced. These embodiments are described in detail to enable those skilled in the art to fully practice the present invention. It should be understood that the various embodiments of the present invention, although different from one another, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be embodied in different embodiments without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention should be understood to encompass the scope of the appended claims and all equivalents thereof. Like reference numerals in the drawings indicate the same or similar components throughout the various aspects.

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, various preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

[0023] Although the golf swing described herein focuses on a full swing, the golf swing in the present invention should be understood as the broadest concept including all actions for moving a golf club. For example, a golf swing according to an embodiment of the present invention may include a full swing, a half swing, a chip shot, a lob shot, putting, etc.

[0024] Configuring Devices

[0025] The internal configuration of the device 100 and the function of each component that performs important functions for implementing the present invention will now be discussed in detail.

[0026] FIG. 1 is a diagram illustrating in detail the internal configuration of a device 100 according to an embodiment of the present invention.

[0027] As shown in FIG. 1 , a device 100 according to an embodiment of the present invention may include a club and joint detection unit 110, a golf swing information estimation unit 120, a communication unit 130, and a control unit 140. According to an embodiment of the present invention, the club and joint detection unit 110, the golf swing information estimation unit 120, the communication unit 130, and the control unit 140 may be program modules, at least some of which communicate with an external system (not shown). Such program modules may be included in the device 100 in the form of an operating system, application program module, or other program module, and may be physically stored in various known storage devices. Furthermore, such program modules may be stored in a remote storage device that can communicate with the device 100. Meanwhile, such program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types, as described below, according to the present invention.

[0028] Meanwhile, although the device 100 has been described above, such description is illustrative, and it will be apparent to those skilled in the art that at least some of the components or functions of the device 100 may be embodied in a server (not shown) or included in an external system (not shown) as needed.

[0029] Meanwhile, the device 100 according to an embodiment of the present invention is a digital device equipped with a memory means and a microprocessor to provide computing capabilities, and may include a smartphone, tablet, smart watch, smart band, smart glasses, desktop computer, laptop, workstation, PDA, web pad, mobile phone, etc. However, the device 100 may be modified in any manner within the scope of the above-described exemplary embodiments as long as the object of the present invention can be achieved.

[0030] In particular, the device 100 may include an application (not shown) that assists the user in receiving services such as golf swing information estimation from the device 100. Such an application may be downloaded from an external application distribution server (not shown). Meanwhile, the nature of such an application may be generally similar to the club and joint detection unit 110, golf swing information estimation unit 120, communication unit 130, and control unit 140 of the device 100, which will be described later. Here, at least a portion of the application may be replaced with a hardware or firmware device that can perform substantially the same or equivalent functions, if necessary.

[0031] First, the club and joint detection unit 110 according to one embodiment of the present invention can perform a function of detecting at least one of a plurality of key points related to the shaft of the club and a plurality of key points related to the face of the club, and at least one joint of the user, from the captured image of a golf swing of the user using an artificial neural network model when the captured image of the golf swing of the user is acquired.

[0032] Specifically, according to one embodiment of the present invention, the captured image of the user's golf swing may be captured by device 100 or may be an image captured by another device (not shown) and acquired by device 100. According to one embodiment of the present invention, the captured image of the user's golf swing may refer to an image captured by an RGB camera. That is, the club and joint detection unit 110 according to one embodiment of the present invention can detect a plurality of key points related to the club shaft, at least one of a plurality of key points related to the club face, and at least one joint of the user, using only an RGB image of the user's golf swing, without using depth information acquired from equipment such as a depth camera or depth sensor or a sensor (marker) attached to the user's body or golf club. Meanwhile, in the present invention, the captured image of the user's golf swing primarily refers to a video, but should be understood as a broadest concept including all data capable of visually expressing the user's golf swing, regardless of its format.

[0033] The club and joint detection unit 110 according to one embodiment of the present invention can perform a function of deriving probability information regarding the position of at least one of a plurality of key points related to the shaft of the club and a plurality of key points related to the face of the club, and probability information regarding the position of at least one joint of the user, from a photographed image of the user's golf swing using an artificial neural network model.

[0034] Specifically, the probability information regarding the positions of multiple key points on the club shaft, the probability information regarding the positions of multiple key points on the club face, and the probability information regarding the position of at least one joint of the user, which can be derived by the club and joint detection unit 110 according to one embodiment of the present invention, can be included in a probability map (i.e., output data of the artificial neural network model) generated using photographed images of the user's golf swing as input data for the artificial neural network model.

[0035] For example, according to one embodiment of the present invention, the probability map may refer to a two-dimensional heat map. The club and joint detection unit 110 according to one embodiment of the present invention may generate at least one two-dimensional heat map image for each of the user's at least one joint using an artificial neural network model. The probability information regarding the two-dimensional position of the user's at least one joint may be derived based on the following characteristics: the larger the pixel value constituting the generated at least one heat map image, the higher the probability that the two-dimensional position of the joint corresponds to the corresponding pixel; the wider the distribution of small pixel values ​​in the heat map, the lower the probability that the joint's position will be accurately identified; and the narrower the distribution of large pixel values, the higher the probability that the joint's position will be accurately identified. The club and joint detection unit 110 according to one embodiment of the present invention may determine the two-dimensional position of the user's at least one joint by referring to the derived probability information and detect the position as the user's at least one joint.

[0036] Meanwhile, the above-mentioned content regarding deriving probability information regarding the position of the user's joints can be similarly applied to the case where multiple key points regarding the club shaft and / or multiple key points regarding the club face are detected by the club and joint detection unit 110 according to one embodiment of the present invention (i.e., at least one 2D heat map image is generated for each of the multiple key points), so detailed explanation will be omitted.

[0037] Meanwhile, the artificial neural network model according to an embodiment of the present invention may include a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a deep belief network (DBN) model, or an artificial neural network model in which the above models are combined, etc. However, the artificial neural network model according to an embodiment of the present invention is not limited to the above-listed models and may be variously modified within the scope that can achieve the object of the present invention.

[0038] The artificial neural network model according to an embodiment of the present invention may be a lightweight model using depthwise convolution and pointwise convolution.

[0039] Furthermore, the artificial neural network model according to an embodiment of the present invention may be a lightweight model that uses a lightweighting algorithm such as pruning, weight quantization, residual learning, etc. Furthermore, the artificial neural network model according to an embodiment of the present invention may be a model with an encoder-decoder structure, and for lightweighting, the decoder may be configured with a much smaller number of channels and layers than the encoder.

[0040] Specifically, an artificial neural network model commonly used in object recognition technology requires high levels of computing resource consumption to achieve high levels of recognition performance, and is therefore often difficult to use in environments with limited computing resources, such as mobile devices. Therefore, according to one embodiment of the present invention, a lightweight artificial neural network model is made using depthwise convolution and pointwise convolution, and the lightweight artificial neural network model is used on a mobile device to detect multiple key points related to the club shaft, at least one of multiple key points related to the club face, and at least one joint of the user from a captured image of a user's golf swing.

[0041] Here, the depthwise convolution according to an embodiment of the present invention may refer to a convolution process in which a kernel is applied to each depth of an input layer (i.e., channel of an input layer) when performing convolution in an artificial neural network model according to an embodiment of the present invention. Meanwhile, since the method of applying a kernel and performing an operation is the same as that of general convolution, a detailed description thereof will be omitted.

[0042] Pointwise convolution according to one embodiment of the present invention may refer to a convolution process in which a kernel of 1x1xM size (i.e., a kernel with a width of 1, a height of 1, and a depth of M) is applied to each point of an input layer when performing convolution in an artificial neural network model according to one embodiment of the present invention.

[0043] FIG. 2(a) is a diagram illustrating a process of performing a general convolution according to an embodiment of the present invention.

[0044] FIG. 2(b) is a diagram illustrating an example of a process of performing depthwise convolution and pointwise convolution according to an embodiment of the present invention.

[0045] 2(a), according to one embodiment of the present invention, it can be assumed that the width, height, and depth of an input layer 211 are F, F, and N, respectively, the width, height, and depth of each kernel 212 are K, K, and N, respectively, and the width, height, and depth of an output layer 213 are F, F, and M, respectively. It is also assumed here that appropriate padding and stride are set to prevent a change in width and height between the input layer 211 and the output layer 213. In this case, in a typical convolution, a kernel 212 is applied to the input layer 211 to form one depth of the output layer 213 (i.e., F×F×K×K×N operations), and since this operation is performed for M kernels 212, a total of F×F×K×K×N×M operations are performed.

[0046] 2(b), according to one embodiment of the present invention, it can be assumed that the width, height, and depth of an input layer 221 are F, F, and N, respectively, the width, height, and depth of each kernel 222 in the depth-specific convolution are K, K, and 1, respectively, the width, height, and depth of each kernel 224 in the point-specific convolution are 1, 1, and N, respectively, and the width, height, and depth of an output layer 225 are F, F, and M, respectively. In this case, a kernel 222 is applied to the input layer 221 for each depth to form each depth of the hidden layer 223 (i.e., F×F×K×K×1×N operations). Next, a kernel 224 is applied to the hidden layer 223 for each point to form one depth of the output layer 225 (i.e., F×F×1×1×N operations). Since these operations are performed for M kernels 224, a total of F×F×1×1×N×M operations are performed in the point-specific convolution. Therefore, when the depth-based convolution and point-based convolution operations according to one embodiment of the present invention are added together, a total of (F×F×K×K×1×N)+(F×F×1×1×N×M) operations are performed, which has the effect of reducing the amount of calculation compared to general convolution.

[0047] Meanwhile, the algorithm for weight reduction according to one embodiment of the present invention is not necessarily limited to the above algorithm (depth-based or point-based convolution), and the order or number of times each algorithm is applied may also be changed in various ways.

[0048] Meanwhile, according to one embodiment of the present invention, the multiple key points on the club shaft may include at least two of an arbitrary point located at the top of the club shaft, an arbitrary point located at the bottom of the club shaft, and any one position between the arbitrary point located at the top and the arbitrary point located at the bottom.

[0049] FIG. 3 is a diagram illustrating a process of estimating information about a club posture and a user's golf swing according to an embodiment of the present invention.

[0050] For example, according to one embodiment of the present invention, referring to FIG. 3(a), the multiple key points on the club shaft may include any point located at the top of the club shaft (313; i.e., the handle portion of the club), any point located at the bottom of the shaft (311; i.e., the head portion of the club), and a point intermediate between any point located at the top 313 and any point located at the bottom 311 (312; i.e., the middle of the club shaft).

[0051] As another example, according to one embodiment of the present invention, referring to Figure 3(b), it may be assumed that a part of the club (e.g., the club head) is not included in the captured image of the user's golf swing. In this case, according to one embodiment of the present invention, the multiple key points on the club shaft may include any point located at the top of the club shaft (322; i.e., the handle of the club) and the middle of the club shaft 321.

[0052] However, the key points regarding the shaft of the club according to one embodiment of the present invention are not limited to those described above, and may be variously modified within the scope of achieving the object of the present invention.

[0053] According to one embodiment of the present invention, the plurality of key points on the club face may include at least three positions that allow for estimation of the club face posture, which may include, but is not limited to, the direction the face is facing, the angle the face makes with a specific reference line (e.g., a reference line related to the user's body), and the degree to which the club is open or closed during a golf swing.

[0054] FIG. 4 is a diagram illustrating a process of estimating information about a club posture and a user's golf swing according to an embodiment of the present invention.

[0055] For example, according to one embodiment of the present invention, and referring to FIG. 4, key points on the club face 410 may include the middle of the line forming the top of the club face 410 (411; or the top of the club face), the middle of the line forming the bottom of the club face 410 (413; or the bottom of the club face), and the middle of the line forming the outside of the club face (412; or the outermost part of the club face).

[0056] However, the multiple key points on the face of the club according to one embodiment of the present invention are not limited to those described above, and may be modified in various ways within the scope of achieving the objectives of the present invention, unless it makes it impossible to estimate the posture of the face, such as when the multiple key points on the face of the club form a straight line.

[0057] Meanwhile, the club and joint detection unit 110 according to one embodiment of the present invention can deform a captured image of a user's golf swing, and determine the positions of the multiple key points related to the shaft, the positions of the multiple key points related to the face, and the position of at least one joint of the user based on the multiple key points related to the club shaft detected from the deformed image, the multiple key points related to the club face detected from the deformed image, and the position of at least one joint of the user detected from the deformed image, respectively.

[0058] Specifically, the club and joint detection unit 110 according to an embodiment of the present invention may deform at least some of the frames included in the captured image of the user's golf swing. According to an embodiment of the present invention, such deformation may include flipping the frame horizontally or moving the position of the frame (or at least some of the objects included in the frame). The club and joint detection unit 110 according to an embodiment of the present invention may detect at least one of a plurality of key points related to the club shaft and a plurality of key points related to the club face, and at least one joint of the user, from the deformed captured image of the user's golf swing. The club and joint detection unit 110 according to an embodiment of the present invention may determine the position of at least one of a plurality of key points related to the club shaft and a plurality of key points related to the club face, and the position of at least one joint of the user, by referring to both the detection results from the deformed captured image of the user's golf swing and the detection results from the original captured image of the user's golf swing (e.g., by referring to the average value of heat maps generated from each image), thereby improving the detection accuracy of the artificial neural network model.

[0059] However, the frame deformation method and number of deformations according to one embodiment of the present invention are not limited to those described above, and may be variously modified and / or combined within the scope that can achieve the object of the present invention.

[0060] Meanwhile, according to one embodiment of the present invention, the frame to be transformed as described above may be a frame that is highly important in analyzing a user's golf swing (e.g., a frame related to the top of a backswing).The club and joint detection unit 110 according to one embodiment of the present invention may determine the frame to be transformed as described above by referring to at least one of a plurality of key points related to the club shaft and a plurality of key points related to the club face, which are primarily detected from a photographed image related to the user's golf swing, and at least one joint of the user.

[0061] In addition, the club and joint detection unit 110 according to an embodiment of the present invention may perform multiple detections on a captured image of a user's golf swing and refer to all of the detection results to improve the detection accuracy of the artificial neural network model. For this purpose, an ensemble technique such as Random Forest or AdaBoost may be used.

[0062] Next, the golf swing information estimation unit 120 according to an embodiment of the present invention may perform a function of estimating the posture of the club based on at least one of the positions of a plurality of key points on the shaft of the club and the positions of a plurality of key points on the face of the club.

[0063] Specifically, the golf swing information estimating unit 120 according to an embodiment of the present invention can estimate the attitude of the club based on at least one of the positional relationships between a plurality of key points on the club shaft and the positional relationships between a plurality of key points on the club face. Here, according to an embodiment of the present invention, the attitude of the club may include the attitude of the club shaft, the attitude of the club face, etc., and the attitude of the shaft may include, but is not limited to, the position of the shaft, the direction in which the shaft faces, and the angle between the shaft and a specific reference line (e.g., a reference line related to the user's body).

[0064] In addition, the golf swing information estimation unit 120 according to an embodiment of the present invention can estimate the posture of the club based on probability information about at least two positions among an arbitrary point located at the top of the club shaft, an arbitrary point located at the bottom of the shaft, and any position between the arbitrary point located at the top and the arbitrary point located at the bottom.

[0065] 3(a), for example, it can be assumed that the club and joint detection unit 110 according to an embodiment of the present invention detects an arbitrary point 313 located at the top of the club shaft, an arbitrary point 311 located at the bottom of the shaft, and a middle point 312 of the shaft as multiple key points related to the shaft from a captured image of a user's golf swing, and the probability of the position of the arbitrary point 311 located at the bottom exceeds a predetermined critical value. In this case, the golf swing information estimation unit 120 according to an embodiment of the present invention can estimate the arbitrary point 313 located at the top of the club shaft to the arbitrary point 311 located at the bottom as the posture of the club (or the direction in which the club is heading).

[0066] 3(a), the club and joint detection unit 110 according to an embodiment of the present invention may detect an arbitrary point 313 located at the top of the club shaft, an arbitrary point 311 located at the bottom of the shaft, and a middle point 312 of the shaft as multiple key points related to the shaft from a captured image of a user's golf swing, and may assume that the probability of the position of the arbitrary point 311 located at the bottom is equal to or less than a predetermined critical value, and the probability of the position of the middle point 312 exceeds a predetermined critical value. In this case, the golf swing information estimation unit 120 according to an embodiment of the present invention may estimate the arbitrary point 313 located at the top of the club shaft to the middle point 312 as the posture of the club (or the direction in which the club is heading).

[0067] 3(b), it may be assumed that the club and joint detection unit 110 according to an embodiment of the present invention detects an arbitrary point 322 located on the top of the club shaft and a middle 321 of the shaft from a photographed image of a user's golf swing as multiple key points related to the shaft, and the probability of the position of the middle 321 exceeds a predetermined threshold value. In this case, the golf swing information estimation unit 120 according to an embodiment of the present invention may estimate the middle 321 from the arbitrary point 322 located on the top of the club shaft as the posture of the club (or the direction in which the club is heading).

[0068] FIG. 5 is a diagram illustrating a process of estimating information about a club posture and a user's golf swing according to an embodiment of the present invention.

[0069] 5, for example, the club and joint detection unit 110 according to an embodiment of the present invention can detect a plurality of key points 511 related to a club face 510 from a captured image of a user's golf swing. Then, the golf swing information estimation unit 120 according to an embodiment of the present invention can estimate the posture of the club face 510 (e.g., the direction the face is facing, the angle between the face and a specific reference line (e.g., a reference line related to the user's body), the degree to which the club is opened or closed during the golf swing, etc.) based on the positional relationship of the plurality of key points 511 (e.g., the shape and area of ​​the surface formed by the plurality of key points).

[0070] In addition, the golf swing information estimation unit 120 according to an embodiment of the present invention may perform a function of estimating information about the user's golf swing by referring to the estimated club posture and the position of at least one joint of the user, as described above.

[0071] Specifically, the golf swing information estimating unit 120 according to an embodiment of the present invention can estimate at least one of the type of at least one joint of the user, the position of at least one joint of the user, the distance between at least one joint of the user and at least one other joint of the user, and the angle formed between at least one joint of the user and at least one other joint of the user, based on the position of at least one joint of the user, and can estimate the posture of the user based on the estimated type of at least one joint of the user, the position of at least one joint of the user, the distance between at least one joint of the user and at least one other joint of the user, and the angle formed between at least one joint of the user and at least one other joint of the user, and can estimate the posture of the user based on the estimated posture of the user and the posture of the club.

[0072] According to an embodiment of the present invention, the information about the user's golf swing may include information that is difficult to understand from the position of at least one joint of the user alone, such as information about wrist control and club control. For example, according to an embodiment of the present invention, the information about the user's golf swing may include information about the swing plane (e.g., whether it is a one-plane swing, a two-plane swing, a shallowing swing, etc.), information about cocking (e.g., whether early cocking, cocking, overswing, casting, scooping, etc. occurred), information about the club path during the backswing (e.g., whether the club is released to the inside or outside), information about the club face (e.g., whether the club is closed or open during the takeback or at the top of the backswing, etc.). However, the information about the user's golf swing according to an embodiment of the present invention is not limited to the above and may be variously modified within the scope of achieving the objectives of the present invention.

[0073] Meanwhile, according to one embodiment of the present invention, information about a user's golf swing can be estimated by classifying it into parts of the golf swing.

[0074] Specifically, a golf swing according to an embodiment of the present invention may be composed of eight partial movements, such as address, takeaway, backswing, top of swing, downswing, impact, follow-through, and finish. The golf swing information estimation unit 120 according to an embodiment of the present invention may refer to the posture of the club and the position of at least one joint of the user to determine which of the eight movements the captured image of the user's golf swing belongs to, and classify the golf swing into the partial movements constituting the golf swing to perform a function of estimating information about the user's golf swing.

[0075] Meanwhile, the golf swing according to an embodiment of the present invention is not necessarily divided into the above eight stages, that is, the swing may be divided into substages that constitute each of the eight stages, or at least some of the eight stages may be divided into a single stage.

[0076] 6 and 7 are diagrams illustrating a process of estimating information about a club posture and a user's golf swing according to an embodiment of the present invention.

[0077] For example, referring to FIG. 6, the golf swing information estimation unit 120 according to one embodiment of the present invention can estimate information about a user's golf swing by referring to the angle (or direction) of the user's arm (i.e., the user's posture) (611, 621, and 631), which can be estimated based on the position of at least one joint of the user, and the posture of the club face (e.g., the direction the face is facing) (610, 620, and 630), which can be estimated based on a plurality of key points on the club face.

[0078] As another example, referring to FIG. 7, the golf swing information estimation unit 120 according to one embodiment of the present invention can estimate information related to a user's golf swing by referring to the direction (or angle, i.e., the user's posture) 720 of the user's wrist, which can be estimated based on the position of at least one joint of the user, and the posture of the club shaft (e.g., the direction in which the shaft is facing) 710, which can be estimated based on a plurality of key points related to the club shaft.

[0079] Next, the communication unit 130 according to an embodiment of the present invention may perform a function that enables transmission and reception of data to / from the club and joint detection unit 110 and the golf swing information estimation unit 120.

[0080] Finally, the control unit 140 according to an embodiment of the present invention may perform a function of controlling data flow between the club and joint detection unit 110, the golf swing information estimation unit 120, and the communication unit 130. That is, the control unit 140 according to an embodiment of the present invention may control the club and joint detection unit 110, the golf swing information estimation unit 120, and the communication unit 130 to perform their respective unique functions by controlling the data flow from / to the outside of the device 100 or the data flow between each component of the device 100.

[0081] The above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions stored on the computer-readable recording medium may be specially designed and constructed for the present invention or may be readily available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. A hardware device may be replaced by one or more software modules to perform the processes of the present invention, and vice versa.

[0082] Although the present invention has been described above using specific details such as specific components and limited examples and drawings, this is merely provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art to which the present invention pertains can make various modifications and changes from such descriptions.

[0083] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and all scopes equivalent to or modified equivalently from the scope of the claims below as well as the scope of the invention can be said to fall within the scope of the concept of the present invention. [Explanation of symbols]

[0084] 100:Device 110: Club and joint detector 120: Golf swing information estimation unit 130: Communications Department 140: Control unit

Claims

1. 1. A method for estimating information about a golf swing, comprising: When a photographed image of a user's golf swing is acquired, detecting a plurality of key points related to a shaft of a club, a plurality of key points related to a face of the club, and at least one joint of the user from the photographed image using an artificial neural network model; deforming at least some frames included in the captured image, and detecting a plurality of key points related to the shaft of the club, a plurality of key points related to the face of the club, and at least one joint of the user from the deformed captured image using the artificial neural network model; determining the positions of a plurality of key points on the shaft of the club, the positions of a plurality of key points on the face of the club, and the position of at least one joint of the user, by referring to both the detection results from the photographed image before deformation and the detection results from the photographed image after deformation; and estimating an attitude of the club based on positions of a plurality of key points on the shaft and positions of a plurality of key points on the face, and estimating information about the golf swing of the user by referring to the attitude of the club and a position of at least one joint of the user, In the detecting step, probability information regarding the position of at least one of a plurality of key points related to the shaft of the club and a plurality of key points related to the face of the club, and probability information regarding the position of the at least one joint of the user are derived from the captured image using the artificial neural network model; wherein at least one of probability information regarding the positions of a plurality of key points on the shaft of the club, probability information regarding the positions of a plurality of key points on the face of the club, and probability information regarding the position of the at least one joint of the user is included in a probability map generated using the captured images as input data for the artificial neural network model.

2. 2. The method of claim 1, wherein the estimating step estimates the attitude of the club based on probability information regarding at least two positions among an arbitrary point located at the top of the shaft, an arbitrary point located at the bottom of the shaft, and any one position between the arbitrary point located at the top and the arbitrary point located at the bottom.

3. In the detecting step, the plurality of key points and at least one joint of the user are detected from the captured image a plurality of times; The method according to claim 1 , wherein the step of estimating estimates information about the user's golf swing by referring to all of the plurality of detection results.

4. 2. The method of claim 1, wherein the estimating step estimates at least one of a type of the at least one joint of the user, a distance between the at least one joint of the user and at least one other joint of the user, and an angle formed between the at least one joint of the user and at least one other joint of the user, based on a position of the at least one joint of the user.

5. A non-transitory computer-readable recording medium having a computer program recorded thereon for executing the method of claim 1.

6. 1. A device for estimating information about a golf swing, comprising: Once a photographed image of a user's golf swing is acquired, detecting from the photographed image a plurality of key points related to a shaft of a club, a plurality of key points related to a face of the club, and at least one joint of the user using an artificial neural network model; deforming at least some frames included in the captured image, and detecting a plurality of key points related to the shaft of the club, a plurality of key points related to the face of the club, and at least one joint of the user from the deformed captured image using an artificial neural network model; a club and joint detection unit that determines the positions of a plurality of key points on the shaft of the club, the positions of a plurality of key points on the face of the club, and the position of at least one joint of the user by referring to both the detection results from the photographed image before deformation and the detection results from the photographed image after deformation; and a golf swing information estimation unit that estimates an attitude of the club based on positions of a plurality of key points on the shaft and positions of a plurality of key points on the face, and estimates information about the golf swing of the user by referring to the attitude of the club and the position of at least one joint of the user; the club and joint detection unit is configured to derive, from the captured image, probability information regarding the position of at least one of a plurality of key points related to the shaft of the club and a plurality of key points related to the face of the club, and probability information regarding the position of the at least one joint of the user, using the artificial neural network model; at least one of probability information regarding the positions of a plurality of key points on the shaft of the club, probability information regarding the positions of a plurality of key points on the face of the club, and probability information regarding the position of the at least one joint of the user is included in a probability map generated using the captured image as input data for the artificial neural network model.

7. 7. The device according to claim 6, wherein the golf swing information estimator is configured to estimate the attitude of the club based on probability information regarding at least two positions of an arbitrary point located at an upper part of the shaft, an arbitrary point located at a lower part of the shaft, and any one position between the arbitrary point located at the upper part and the arbitrary point located at the lower part.

8. the club and joint detection unit is configured to detect the plurality of key points and at least one joint of the user from the captured image a plurality of times; The device according to claim 6 , wherein the golf swing information estimator is configured to estimate information about the user's golf swing by referring to all of the plurality of detection results.

9. 7. The device of claim 6, wherein the golf swing information estimator is configured to estimate at least one of a type of the at least one joint of the user, a distance between the at least one joint of the user and at least one other joint of the user, and an angle formed between the at least one joint of the user and at least one other joint of the user, based on a position of the at least one joint of the user.

Images