Information processing device, information processing system, information processing method and program

Abstract

To provide a technique capable of three-dimensionally grasping a swing motion of golf with a relatively simple configuration.SOLUTION: An information processing device 300 according to one embodiment of the present disclosure recognizes a position of a head of a golf club over time of moving images on the basis of the moving images that are captured by a single camera 100 and show a swing motion of golf of a person being the subject, acquires feature quantities (angle difference Δθ) related to the head trajectory on the basis of the recognized position of the head over time, and estimates the head trajectory (angle θh) in the swing motion of golf on the basis of the acquired feature quantities.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and the like. [Background technology]

[0002] Conventionally, a technique for three-dimensionally measuring a golf swing motion is known (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6908312 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in order to perform three-dimensional measurement of a golf swing, multiple measuring devices such as cameras are required, and markers or the like must be attached to the subject person or golf club.

[0005] In view of the above problems, an object of the present invention is to provide a technique that makes it possible to grasp a golf swing motion in three dimensions using a relatively simple configuration. [Means for solving the problem]

[0006] In order to achieve the above object, in one embodiment of the present disclosure, Based on a video captured by a single camera and showing a golf swing motion of a subject person, the positions of feature points of at least one of the person and the golf club in a time series of the video are recognized, and based on the positions of the feature points in the recognized time series, feature amounts relating to the three-dimensional movement states of the feature points are acquired, and based on the acquired feature amounts, the three-dimensional movement states of the feature points in the golf swing motion are estimated. An information processing device is provided.

[0007] In another embodiment of the present disclosure, the camera including an imaging unit that captures the moving image of the person as the subject, and a display unit that displays an image of an imaging range of the imaging unit in real time; The information processing device, the display unit displays an auxiliary image for adjusting a relative positional relationship between the camera and a target for hitting a ball by a golf swing motion to a predetermined state, superimposed on an image in a real-time imaging range of the imaging unit. An information processing system is provided.

[0008] In still another embodiment of the present disclosure, an information processing device, based on a video captured by a single camera and showing a golf swing motion of a subject person, recognizes positions of feature points of at least one of the person and the golf club in a time series of the video, acquires feature amounts relating to a three-dimensional movement state of the feature points based on the recognized positions of the feature points in the time series, and estimates a three-dimensional movement state of the feature points in the golf swing motion based on the acquired feature amounts; A method for processing information is provided.

[0009] In still another embodiment of the present disclosure, In the information processing device, Based on a video captured by a single camera and showing a golf swing motion of a subject person, the positions of feature points of at least one of the person and the golf club in a time series of the video are recognized, a feature amount relating to a three-dimensional movement state of the feature point is acquired based on the recognized positions of the feature point in the time series, and a three-dimensional movement state of the feature point in the golf swing motion is estimated based on the acquired feature amount. The program is provided. Effect of the Invention

[0010] According to the above-described embodiment, a golf swing motion can be grasped three-dimensionally with a relatively simple configuration. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of a swing diagnosis system. [Diagram 2] FIG. 2 is a diagram showing an example of an image captured by a camera. [Diagram 3] FIG. 4 is a diagram showing an example of a screen of a display unit of the camera. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal. [Diagram 5] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a functional block diagram showing an example of a functional configuration of the swing diagnosis system. [Figure 7] FIG. 1 is a diagram illustrating features of a golf swing motion. [Figure 8] 4 is a functional block diagram showing an example of a functional configuration of an image extraction unit. FIG. [Figure 9] 11A and 11B are diagrams illustrating an example of a feature amount relating to a head trajectory. [Figure 10] 11 is a diagram showing an example of information indicating a correlation between a feature amount and a head trajectory. FIG. [Figure 11] FIG. 11 is a sequence diagram showing an example of an operation of the swing diagnosis system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment will be described with reference to the drawings.

[0013] [Swing diagnosis system overview] An overview of a swing diagnosis system 1 according to the present embodiment will be described with reference to FIGS.

[0014] Fig. 1 is a diagram showing an example of a swing diagnosis system 1. Fig. 2 is a diagram showing an example of an image captured by the camera 100 (captured image 20).

[0015] As shown in FIG. 1, the swing diagnosis system 1 includes a camera 100, a user terminal 200, and an information processing device 300.

[0016] The swing diagnosis system 1 performs diagnosis of a user's golf swing motion (hereinafter, "swing diagnosis") in the information processing device 300 based on video image data capturing the state of the user's golf swing motion captured by the camera 100. Then, the swing diagnosis system 1 provides the user with the diagnosis result of the swing diagnosis through the user terminal 200.

[0017] Camera 100 captures an image of a golf swing by a user and acquires a video showing the swing motion. Camera 100 may acquire the video of the user's golf swing in response to the operation of a photographer different from the user who performs the golf swing, or may acquire the video of the user's golf swing in response to the operation of the user using a self-timer function or the like.

[0018] In this specification, "imaging" refers to obtaining an image showing the surroundings of the camera 100, and "photographing" refers to recording the image obtained by imaging.

[0019] 1, camera 100 and user terminal 200 are depicted separately, but camera 100 may be built into user terminal 200, or may be provided separately from user terminal 200. In the latter case, the output of camera 100 (video image data) may be imported into user terminal 200 by communication, or may be imported into user terminal 200 via recording medium 201A described below.

[0020] For example, as shown in Fig. 2, camera 100 captures an image of a golf swing from behind the user. "Behind the user" means behind the user when the direction of the ball flight line imagined by the user (hereinafter, "virtual ball flight line") is defined as "forward." A captured image 20 shows a user 21 as a subject, a golf club 22 moving together with the user 21, and a teed-up ball 23. The captured image 20 captures a state at a point during the take-back or downswing in a golf swing.

[0021] The camera 100 may capture the golf swing motion from the front of the user, facing the user directly. The camera 100 may also capture both a video image representing the golf swing motion from the front of the user and a video image representing the golf swing motion from behind the user. In this case, for example, a plurality of cameras 100 capture both a video image representing the golf swing motion from the front of the user and a video image representing the golf swing motion from behind the user at the same timing. Also, by changing the position of one camera 100, both a video image representing the golf swing motion from the front of the user and a video image representing the golf swing motion from behind the user may be captured sequentially.

[0022] The user terminal 200 is a terminal device used by a user who receives a swing diagnosis. The user terminal 200 may be, for example, a terminal device installed in a golf lesson facility or shop, or may be a terminal device owned by a user.

[0023] The user terminal 200 is, for example, a portable terminal device, that is, a mobile terminal. The mobile terminal is, for example, a smartphone, a tablet terminal, a laptop type PC (Personal Computer), etc. The user terminal 200 may also be a stationary terminal device. The stationary terminal device is, for example, a desktop PC.

[0024] The user terminal 200 is communicatively connected to the information processing device 300 through a predetermined communication line. The predetermined communication line includes, for example, a local area network (LAN). The predetermined communication line may also include a wide area network (WAN). The wide area network includes, for example, the Internet. The wide area network may also include a mobile communication network terminated in a base station or a satellite communication network using a communication satellite. The predetermined communication line may also include, for example, WiFi, Bluetooth (registered trademark), local 5G (5G), etc. thAlternatively, the communication line may be a short-distance communication line using a predetermined wireless communication standard such as a standard for wireless communication using a wireless communication protocol ...

[0025] The user terminal 200 captures a video showing the state of a golf swing motion by the user from the camera 100, and transmits it to the information processing device 300. Then, the user terminal 200 presents the result of the swing diagnosis returned from the information processing device 300 to the user via the display device 208 described later.

[0026] The information processing device 300 performs swing diagnosis based on the video image showing the state of the golf swing motion of the user received from the user terminal 200, and returns the result of the swing diagnosis to the user terminal 200.

[0027] The information processing device 300 is, for example, a server device with a relatively high processing capacity. The server device may be a cloud server, an on-premise server, or an edge server. The information processing device may be a terminal device with a lower processing capacity than the server device, depending on the processing capacity required for swing diagnosis. The terminal device may be a stationary terminal device or a portable terminal device (mobile terminal).

[0028] The functions of the user terminal 200 and the information processing device 300 may be realized by one information processing device (for example, one smartphone or tablet terminal), or may be distributed and realized by three or more information processing devices.

[0029] [Hardware configuration of swing diagnosis system] Next, with reference to FIGS. 3 to 5 in addition to FIG. 1, a hardware configuration of the swing diagnosis system 1 will be described.

[0030] <Camera hardware configuration> Fig. 3 is a diagram showing an example of a screen (screen 30) of the display unit 120 of the camera 100. In Fig. 3, auxiliary images 33 to 36 are drawn with dashed lines for the sake of convenience in order to represent points where the live view image 31 is superimposed so as to be transparent.

[0031] As shown in FIG. 1, the camera 100 includes an imaging unit 110, a display unit 120, and a control unit .

[0032] The imaging unit 110 captures and outputs a moving image within a predetermined imaging range based on the camera 100. The imaging unit 110 includes an image sensor (imaging element) such as a Complementary Metal Oxide Semiconductor (CMOS) sensor.

[0033] Display unit 120 displays various screens related to the operation of camera 100. The display unit is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0034] The control unit 130 controls the operation of the camera 100. For example, the control unit 130 controls the imaging unit 110 and the display unit 120.

[0035] The functions of the control unit 130 are realized by any hardware or any combination of hardware and software. For example, the control unit 130 is mainly configured with a computer including a central processing unit (CPU), a memory device, an auxiliary storage device, and an interface device. The memory device is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The auxiliary storage device is, for example, a hard disc drive (HDD), a solid state disc (SSD), an electrically erasable programmable read only memory (EEPROM), or a flash memory. The interface device includes, for example, an interface for internal communication for communicatively connecting with the imaging unit 110 and the display unit 120, and an interface for external communication for communicatively connecting with an external device such as the user terminal 200. This allows the camera 100 to transmit video data to the outside through the interface for external communication, and to download various data and programs used in processing from the outside and install them in the auxiliary storage device. The interface device also includes, for example, an external interface for reading data from a recording medium and writing data to the recording medium. The recording medium is, for example, an SD memory card, which enables the camera 100 to write video data externally through the recording medium, read various data and programs used in processing, and install them in the auxiliary storage device.

[0036] 3, the display unit 120 displays a screen 30 for operating the camera 100 under the control of the control unit 130. In this example, the camera 100 is integrated into a smartphone serving as the user terminal 200.

[0037] The operation screen 30 of the display unit 120 includes a live view image 31, a shooting button image 32, and auxiliary images 33-36.

[0038] The live view image 31 is a real-time captured image of the imaging range output from the imaging unit 110. This allows the photographer to capture a moving image of the user's golf swing motion after determining whether a predetermined range including the subject person is included in the live view image 31. Therefore, the camera 100 can obtain a moving image that appropriately includes the user's golf swing motion. In this example, the live view image 31 is the same as the captured image 20 in FIG. 2.

[0039] The shooting button image 32 is disposed below the live view image 31 on the screen 30. For example, when the shooting button image 32 on the screen 30 is operated via a touch panel or the like, the camera 100 starts and stops shooting a moving image. Specifically, when the shooting button image 32 is operated while a moving image is not being shot, the control unit 130 starts recording the output (captured image) of the imaging unit 110 in an auxiliary storage device or the like, thereby starting shooting of the moving image. Also, when the shooting button image 32 is operated while a moving image is being shot, the camera 100 stops recording the output of the imaging unit 110, thereby stopping shooting of the moving image.

[0040] The auxiliary images 33 to 36 are images for assisting in capturing an appropriate moving image of the user's golf swing, and are displayed on the screen 30 so as to be superimposed on the live view image 31 and to allow the live view image 31 to be seen through.

[0041] The auxiliary image 33 defines the relative positional relationship between the camera 100, the subject, and the target of the ball in the live view image 31. The auxiliary image 33 is an isosceles triangle with the lower end of the live view image 31 as its base. Specifically, a vertex 33A corresponding to the intersection of the isosceles of the isosceles triangle represents the target of the ball in the live view image 31. For example, the target of the ball is a predetermined position on the fairway or a predetermined position on the green. In addition, a left side 33B of the isosceles of the isosceles triangle is a straight line passing through the target of the ball and the position of the subject's feet. The side 33B represents the relative positional relationship between the target of the ball and the subject's feet in the live view image 31. In addition, a right side 33C of the isosceles of the isosceles triangle is a straight line passing through the target of the ball and the position of the ball. One side 33C represents the relative positional relationship between the target of the ball and the ball in the live view image 31. This allows the photographer to align the relative positional relationship between the target of the ball 23 hit by the user 21, the position of the feet of the user 21, and the ball 23 in a predetermined state in the live view image 31, using the auxiliary image 33 as a reference. Therefore, the camera 100 can acquire a moving image in which the positional relationship between the target of the ball, the user as a subject, and the ball is in a predetermined state. Therefore, the information processing device 300 can perform more appropriate swing diagnosis by using the moving image captured by the camera 100.

[0042] Instead of the auxiliary image 33, only two line segments corresponding to the sides 33B and 33C may be displayed as an auxiliary image superimposed on the live view image 31. Also, only one of the two line segments corresponding to the sides 33B and 33C may be displayed superimposed on the live view image 31. Also, a figure including only a line segment corresponding to one of the sides 33B and 33C may be displayed superimposed on the live view image 31. For example, an asymmetric triangle consisting of the side 33B or the side 33C, one side corresponding to a perpendicular line extending from the apex 33A to the bottom end of the live view image 31, and a base corresponding to the bottom end of the live view image 31 may be displayed superimposed on the live view image 31. Also, an auxiliary image of another form, such as the side 33B or the side 33C, which represents the positional relationship between the person of the subject and the target of the ball or the positional relationship between the ball and the target of the ball, may be displayed superimposed on the live view image 31. For example, a wedge-shaped (eg, triangular) auxiliary image that narrows from the ball side to the target for the ball's launch, centered on a line segment corresponding to side 33C, is displayed superimposed on live view image 31.

[0043] Auxiliary image 34 is an image that simulates a target for hitting a ball. Auxiliary image 34 is displayed in accordance with vertex 33A of an isosceles triangle in auxiliary image 33. This makes it easier for the photographer to recognize that vertex 33A of the isosceles triangle is the target for hitting a ball.

[0044] Auxiliary image 35 is a silhouette image simulating the subject (golfer) at address. Auxiliary image 35 is displayed so that the feet of the silhouette are aligned with one side 33B of the isosceles triangle of auxiliary image 33. This makes it easier for the photographer to align the relative positional relationship between the target for launching ball 23 and the position of the feet of user 21 in live view image 31 with auxiliary images 33 and 35 as references. Therefore, camera 100 can more reliably capture a moving image in which the positional relationship between the target for launching ball, the user as the subject, and the ball is in a predetermined state.

[0045] Auxiliary image 36 is a silhouette image simulating a golf club at address. Auxiliary image 36 is displayed so that the head of the silhouette is aligned with one side 33C of the right side of the isosceles triangle of auxiliary image 33. This makes it easier for the photographer to align the relative positional relationship between the target for launching ball 23 and the position of ball 23 in live view image 31 using auxiliary images 33 and 36 as references. Therefore, camera 100 can more reliably capture a moving image in which the positional relationship between the target for launching ball, the user as a subject, and the ball is in a predetermined state.

[0046] Auxiliary images 33 to 36 in Fig. 3 correspond to the case where the subject is a right-handed hitter, and similar auxiliary images may also be prepared to correspond to the case where the subject is a left-handed hitter (left-handed). For example, in the case of a left-handed hitter, auxiliary images obtained by symmetrically inverting auxiliary images 33, 35, and 36 are displayed in live view image 31, and an auxiliary image similar to auxiliary image 34 is displayed at the intersection of the isosceles of an isosceles triangle obtained by symmetrically inverting auxiliary image 33. Whether auxiliary images 33 to 36 corresponding to right-handed hitters or auxiliary images corresponding to left-handed hitters are displayed may be switched by control unit 130 in response to a predetermined operation on camera 100.

[0047] It should be noted that the auxiliary images 33 to 36 may be omitted. Furthermore, in addition to the auxiliary images 33 to 36, other auxiliary images may be displayed superimposed on the live view image 31. For example, an auxiliary image corresponding to a predetermined grid composed of a plurality of vertical lines and a plurality of horizontal lines or a spirit level showing the tilt state of the camera 100 may be displayed superimposed on the live view image 31.

[0048] <Hardware configuration of user terminal> FIG. 4 is a diagram illustrating an example of a hardware configuration of the user terminal 200. As shown in FIG.

[0049] The functions of the user terminal 200 are realized by any hardware or any combination of hardware and software. For example, as shown in Fig. 4, the user terminal 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, a display device 208, and a sound output device 209, which are connected by a bus B2. Also, as described above, when the camera 100 is built into the user terminal 200, the camera 100 may be connected to the bus B2 in the same manner as the other components.

[0050] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB (Universal Serial Bus) memory, and the like. This allows the user terminal 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs for implementing various functions.

[0051] The user terminal 200 may obtain various data and programs used in the processing from an external device via the communication interface 206.

[0052] The auxiliary storage device 202 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD, an SSD, a flash memory, and the like.

[0053] When an instruction to start a program is received, the memory device 203 reads out and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a DRAM or an SRAM.

[0054] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the user terminal 200 in accordance with the programs.

[0055] The communication interface 206 is used as an interface for communicatively connecting to an external device. This allows the user terminal 200 to acquire video data from the camera 100 through the communication interface 206. The user terminal 200 can also communicate with an external device, such as an information processing device 300, through the communication interface 206. The communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected device and the like.

[0056] The input device 207 accepts various inputs from the user.

[0057] The input device 207 includes, for example, an input device that accepts mechanical operation input from a user (hereinafter, referred to as an "operation input device"). The operation input device includes, for example, a button, a toggle, a lever, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, a keyboard, a mouse, and the like.

[0058] The input device 207 may also include a voice input device capable of receiving voice input from a user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0059] The input device 207 may also include a gesture input device capable of receiving a gesture input from a user. The gesture input device includes, for example, a camera capable of capturing an image of a user's gesture.

[0060] The input device 207 may also include a biometric input device capable of accepting biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data including information related to a user's fingerprint or iris.

[0061] The display device 208 displays an information screen and an operation screen for the user. The display device 208 is, for example, a liquid crystal display or an organic EL display.

[0062] The sound output device 209 conveys various kinds of information by sound to the user of the user terminal 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0063] Furthermore, when the functions of the camera 100 are integrated into a user terminal 200 (mobile terminal), the functions of the display unit 120 may be realized by a display device 208, and the functions of the control unit 130 may be realized by an auxiliary storage device 202, a memory device 203, a CPU 204, etc.

[0064] <Hardware configuration of information processing device> FIG. 5 is a diagram illustrating an example of a hardware configuration of the information processing device 300. As shown in FIG.

[0065] The functions of the information processing device 300 are realized by any hardware or a combination of any hardware and software, etc. For example, as shown in Fig. 5, the information processing device 300 includes an external interface 301, an auxiliary storage device 302, a memory device 303, a CPU 304, a high-speed calculation device 305, a communication interface 306, an input device 307, a display device 308, and a sound output device 309, which are connected by a bus B3.

[0066] The external interface 301 functions as an interface for reading data from the recording medium 301A and writing data to the recording medium 301A. The recording medium 301A includes, for example, a flexible disk, a CD, a DVD, a BD, an SD memory card, a USB memory, etc. This allows the information processing device 300 to read various data used in processing through the recording medium 301A, store the data in the auxiliary storage device 302, and install programs that realize various functions.

[0067] The information processing device 300 may obtain various data and programs used in the processing from an external device through the communication interface 306.

[0068] The auxiliary storage device 302 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 302 includes, for example, an HDD, an SSD, a flash memory, and the like.

[0069] When an instruction to start a program is received, the memory device 303 reads and stores the program from the auxiliary storage device 302. The memory device 303 includes, for example, a DRAM or an SRAM.

[0070] The CPU 304 executes various programs loaded from the auxiliary storage device 302 to the memory device 303, and realizes various functions related to the information processing device 300 in accordance with the programs.

[0071] The high-speed calculation device 305 works in conjunction with the CPU 304 and performs calculation processing at a speed relatively higher than that of the CPU 304. The high-speed calculation device 305 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0072] The high speed calculation device 305 may be omitted depending on the required calculation processing speed.

[0073] The communication interface 306 is used as an interface for communicatively connecting to an external device. This allows the information processing device 300 to communicate with an external device such as the user terminal 200 through the communication interface 306. The communication interface 306 may have multiple types of communication interfaces depending on the communication method between the connected device and the like.

[0074] The input device 307 accepts various inputs from the user.

[0075] The input device 307 includes, for example, an operation input device that accepts mechanical operation input from a user. The operation input device includes, for example, a button, a toggle, a lever, a touch panel mounted on the display device 308, a touch pad provided separately from the display device 308, a keyboard, a mouse, and the like.

[0076] The input device 307 includes, for example, a voice input device capable of receiving voice input from a user. The voice input device includes, for example, a microphone capable of collecting the voice of the user.

[0077] The input device 307 includes, for example, a gesture input device capable of receiving a gesture input from a user. The gesture input device includes, for example, a camera capable of capturing an image of a gesture made by the user.

[0078] The input device 307 includes, for example, a biometric input device capable of receiving biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data including information related to a user's fingerprint or iris.

[0079] The display device 308 displays an information screen or an operation screen to the user. The display device 308 is, for example, a liquid crystal display or an organic EL display.

[0080] The sound output device 309 conveys various types of information by sound to the user of the information processing device 300. The sound output device 309 is, for example, a buzzer, an alarm, a speaker, or the like.

[0081] [Functional configuration of swing diagnosis system] Next, the functional configuration of the swing diagnosis system 1 will be described with reference to FIGS.

[0082] FIG. 6 is a functional block diagram showing an example of the functional configuration of the swing diagnosis system 1. FIG. 7 is a diagram for explaining feature points related to a golf swing motion. Specifically, FIG. 7 is an image showing feature points of a subject (user) showing the top state during a golf swing motion. FIG. 8 is a functional block diagram showing an example of the functional configuration of the image extraction unit 3005. FIG. 9 is a diagram for explaining an example of feature amounts related to a head trajectory. In FIG. 9, trajectories BS, DS, and FS of a golf club head corresponding to the take-back, downswing, and follow-through, respectively, in an example of a golf swing motion are drawn. FIG. 10 is a diagram for explaining the relationship between the feature amount (angle difference Δθ) and the head trajectory (angle θ h ) is a diagram showing an example of information indicating a correlation with.

[0083] 6, the user terminal 200 includes an application screen display processing unit 2001, a moving image data acquisition unit 2002, a moving image data transmission unit 2003, and a diagnosis result data acquisition unit 2004. These functions are realized, for example, by loading an application program (hereinafter, for convenience, "swing diagnosis application") installed in the auxiliary storage device 202 into the memory device 203 and executing it on the CPU 204.

[0084] The application screen display processing unit 2001 causes the display device 208 to display a screen related to the swing diagnosis application (hereinafter, "application screen").

[0085] The video data acquisition unit 2002 acquires video data showing a state of a user's golf swing motion from the camera 100. For example, the video data acquisition unit 2002 acquires video data showing a state of a user's golf swing motion from the camera 100 in response to an input from the user on a predetermined application screen using the input device 207. At this time, the video data acquisition unit 2002 may acquire video data that has already been captured by the camera 100, or may acquire the latest video captured by the camera 100 in real time.

[0086] The video data transmission unit 2003 transmits video data showing a user's golf swing motion, acquired by the video data acquisition unit 2002, to the information processing device 300 via the communication interface 206. For example, the video data transmission unit 2003 transmits video data acquired from the camera 100 to the information processing device 300 in response to a user's input on a predetermined application screen using the input device 207.

[0087] The diagnosis result data acquisition unit 2004 acquires reply data including data on the result of the swing diagnosis received from the information processing device 300. The contents of the reply data including the result of the swing diagnosis are displayed on the display device 208 by the application screen display processing unit 2001. This allows the user to check the contents of the reply data including the result of the swing diagnosis.

[0088] 6, the information processing device 300 includes a moving image data acquisition unit 3001, a feature point extraction unit 3003, an image extraction unit 3005, a feature amount calculation unit 3006, a trajectory estimation unit 3008, and a swing diagnosis unit 3009. These functions are realized, for example, by loading a program installed in the auxiliary storage device 302 into the memory device 303 and executing the program in the CPU 304. The information processing device 300 also includes a moving image data storage unit 3002, a feature point data storage unit 3004, and a correlation data storage unit 3007. These functions are realized, for example, by storage areas defined in the auxiliary storage device 302 and the memory device 303.

[0089] The video data acquisition unit 3001 acquires video data showing a golf swing motion by a user, which is received from the user terminal 200. The data acquired by the video data acquisition unit 3001 is stored in the video data storage unit 3002.

[0090] The feature point extraction unit 3003 reads out the video data acquired by the video data acquisition unit 3001 from the video data storage unit 3002, and extracts feature points related to the swing motion of the user as a subject for each frame of the video.

[0091] For example, the feature points related to the user's swing motion include feature points that represent the parts of the club on the image of the frame. As shown in Fig. 7, the feature points related to the parts of the club include, for example, a feature point that corresponds to the head of the club (see the solid white circle).

[0092] Furthermore, the feature points relating to the user's swing motion may include feature points representing the body parts of the user that are the subject of the image of the frame (hereinafter, "physical feature points"). The physical feature points represent the joint positions in the user's skeleton. As shown in FIG. 7, the physical feature points include points on the image that correspond to the head, shoulders, elbows, wrists, waist, knees, ankles, etc. (see black circles).

[0093] Furthermore, as shown in FIG. 7, the feature points relating to the user's swing motion may include a feature point representing a golf ball on the image of the frame (see the dashed white circle).

[0094] The function of the feature point extraction unit 3003 is realized by, for example, an image recognition engine to which image recognition technology such as a known bone detection technology is applied.

[0095] Data relating to the feature points for each frame extracted by the feature point extraction unit 3003 is stored in a feature point data storage unit 3004. The data relating to the feature points includes, for example, data indicating the type of each feature point, data indicating the position of the feature point on the image, and the like.

[0096] The image extraction unit 3005 extracts frame images corresponding to a predetermined timing from among a plurality of frame images included in the video. Specifically, the predetermined timing is, for example, the timing of a predetermined event in a golf swing motion. The predetermined event is, for example, the address, take-back, backswing, top, halfway down, impact, follow-through, and finish. In this example, the image extraction unit 3005 extracts frame images corresponding to the timing of the halfway down and the timing of the follow-through in the golf swing motion.

[0097] 8, image extraction unit 3005 includes action data acquisition unit 3005A, similarity calculation unit 3005D, and event timing estimation unit 3005F. These functions are realized, for example, by loading a program installed in auxiliary storage device 302 into memory device 303 and executing it with CPU 304. Image extraction unit 3005 also includes action data storage unit 3005B, reference action data storage unit 3005C, and similarity data storage unit 3005E. These functions are realized, for example, by storage areas defined in auxiliary storage device 302 and memory device 303.

[0098] The motion data acquisition unit 3005A reads data on feature points for each frame of the video extracted by the feature point extraction unit 3003 from the feature point data storage unit 3004, and calculates and acquires data representing the motion of the subject user for each frame (hereinafter, "motion data"). The motion data includes, for example, data representing the motion of the subject user himself / herself and data representing the motion of the golf club held by the user. The motion data may also include data representing the motion of a golf ball. The motion data acquisition unit 3005A may acquire one type of motion data, or may acquire multiple types of motion data.

[0099] The motion data to be acquired by the motion data acquisition unit 3005A includes data representing the motion of the user's body part. The data representing the motion of the user's body part includes, for example, data representing the joint position of the target body part of the user and data representing the joint angle of the target body part of the user. The data representing the motion of the user's body part may also include data representing the motion speed and motion direction of the target body part of the user. The target body part may be one or more.

[0100] The motion data to be acquired by the motion data acquisition unit 3005A includes data representing the motion of the golf club. For example, the data representing the motion of the golf club includes data representing the position of the head of the golf club and data representing the attitude angle of the shaft of the golf club. The data representing the motion of the golf club may also include data representing the motion speed and motion direction of the head of the golf club.

[0101] Furthermore, the motion data to be acquired by the motion data acquisition unit 3005A may include data representing the position of the golf ball, the motion speed, the motion direction, and the like.

[0102] The motion data acquisition unit 3005A calculates motion data based on, for example, relative positions and postures among multiple feature points, rather than absolute positions and postures on the image. This makes it possible to prevent a situation in which a large difference in motion data occurs due to a difference in the angle of view of the video. Therefore, even in a situation in which a large variation in the angle of view of a video may occur, such as when a user himself takes a video using the camera 100 built into the portable user terminal 200, it is possible to further improve the accuracy of estimating events in a golf swing motion.

[0103] The motion data acquisition unit 3005A may also calculate and acquire motion data normalized by the actual height of the user who is the subject of the video or the length of a predetermined body part. The predetermined body part is, for example, an arm or a leg. Alternatively or in addition to this, the motion data acquisition unit 3005A may also calculate and acquire motion data normalized by a distance equivalent to the height of the user who is the subject of the video or a distance equivalent to a predetermined body part on the frame image. This makes it possible to prevent a situation in which a large difference occurs in the motion data due to a difference in the physical size of the user who is the subject of the video. Therefore, it is possible to further improve the accuracy of estimating events in a golf swing motion.

[0104] The motion data for each frame of the moving image acquired by the motion data acquisition unit 3005A is stored in the motion data storage unit 3005B.

[0105] The similarity calculation unit 3005D calculates and acquires the similarity between the action data acquired by the action data acquisition unit 3005A and the reference action data stored in advance in the reference action data storage unit 3005C for each frame of the moving image.

[0106] The reference motion data is motion data that represents at least one motion of a person, a golf club, and a golf ball when a predetermined event occurs in a golf swing motion. The reference motion data is the same type of motion data as the motion data acquired by the motion data acquisition unit 3005A. When the motion data acquired by the motion data acquisition unit 3005A is of a plurality of types, the reference motion data is prepared in advance as a plurality of types of reference motion data corresponding to each of the plurality of types of motion data. The predetermined event for which the reference motion data is prepared in advance is a predetermined event to be estimated by the event timing estimation unit 3005F described later. In this example, the predetermined event to be estimated by the event timing estimation unit 3005F includes at least halfway down and follow-through among the address, take-back, backswing, top, halfway down, impact, follow-through, and finish, as described above.

[0107] When there are a plurality of predetermined events to be estimated by the event timing estimation unit 3005F, the similarity calculation unit 3005D calculates the similarity in each of the frames of the video for each of the plurality of predetermined events. When there are a plurality of predetermined events to be estimated by the event timing estimation unit 3005F, the similarity calculation unit 3005D obtains the similarity between the motion data and the reference motion data, for example, for each of the plurality of predetermined events, using the same type of motion data. The similarity calculation unit 3005D may obtain the similarity between the motion data and the reference motion data, using different types of motion data between at least some of the plurality of predetermined events. When multiple types of motion data are used, the different types of motion data may include all types of motion data used for each predetermined event being different, and some types of motion data among all types of motion data being different.

[0108] The reference motion data is obtained, for example, by subjecting frame images corresponding to a specified event, extracted from a video of a golf swing motion by a specified subject, to processing similar to that performed by the feature point extraction unit 3003 and motion data acquisition unit 3005A described above.

[0109] The predetermined subject may be one person or may be multiple people. In the latter case, the reference motion data may be an average of the motion data for each subject.

[0110] The similarity between the motion data acquired by the motion data acquisition unit 3005A and the reference motion data is, for example, a correlation value (correlation coefficient) between the two target data. The similarity between the motion data acquired by the motion data acquisition unit 3005A and the reference motion data may be a difference or ratio between the same type of data of the two target data, or may be a total value of the difference or ratio when multiple types of motion data are used. When multiple types of motion data are used, the similarity calculation unit 3005D may set relative weighting between the multiple types of motion data to acquire the similarity between the motion data and the reference motion data. At this time, the same pattern of relative weighting is also set between the multiple types of reference motion data. When there are multiple predetermined events to be estimated by the event timing estimation unit 3005F, and multiple types of motion data that are all the same type are used between the multiple predetermined events, the similarity calculation unit 3005D may change the relative weighting pattern between the multiple types of motion data for each of the multiple predetermined events.

[0111] The similarity data calculated by the similarity calculation unit 3005D is stored in a similarity data storage unit 3005E.

[0112] The event timing estimation unit 3005F estimates the timing of a predetermined event from the time between the first frame and the last frame of the video based on the similarity between frames of the video. For example, when there are multiple predetermined events, the event timing estimation unit 3005F estimates the timing of the predetermined event from the time between the first frame and the last frame of the video based on the similarity between frames of the video for each predetermined event.

[0113] For example, the event timing estimation unit 3005F estimates the timing at which the similarity is maximum as the timing of a predetermined event. In this case, the maximum value of the similarity may be the maximum value among the discrete values ​​of the similarity in time series, or may be the maximum value on a curve approximated from the discrete values ​​of the similarity in time series. In the latter case, the time between the times corresponding to two adjacent frames of the video may be estimated as the timing of the predetermined event.

[0114] Furthermore, when there are a plurality of predetermined events to be estimated, the event timing estimation section 3005F may estimate the timing of the predetermined events in consideration of the temporal relationship between the plurality of events.

[0115] For example, the event timing estimation unit 3005F estimates the timing at which the similarity corresponding to the impact reaches a maximum value in the range after the top timing as the halfway down timing. This is estimated in advance by a method of searching for the timing at which the similarity reaches a maximum value in the entire range from the first frame to the last frame of the image.

[0116] The positions of the user's body parts and the golf club and the posture of the shaft at the time of impact are similar to the positions of the user's body parts and the golf club and the posture of the shaft at the time of address before the top. Therefore, if the timing at which the similarity corresponding to the impact is maximized is searched for over the entire range from the first frame to the last frame of the video, the timing corresponding to the address may be erroneously estimated as the timing of impact.

[0117] In contrast, in this example, the event timing estimation unit 3005F searches for the timing at which the similarity corresponding to the impact is at its maximum value within a range after the top timing. This makes it possible to suppress the occurrence of a situation in which the address timing is erroneously estimated as the impact timing, and improve the estimation accuracy of the impact timing. In addition, since the search range for the maximum similarity is limited, the time required for the impact timing estimation process can be shortened.

[0118] The event timing estimation unit 3005F may also search for the timing at which the similarity corresponding to the half-way down, impact, follow-through, and finish is maximum within a range after the timing of the top. This can improve the estimation accuracy of the timing of the half-way down, impact, follow-through, and finish, as in the case of the impact timing. In addition, since the search range for the maximum similarity is limited, the time required for the estimation process of the timing of the half-way down, impact, follow-through, and finish can be shortened.

[0119] The event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the address, the take-back, and the backswing is maximized within a range before the top timing. This can prevent the occurrence of a situation in which the impact timing or the half-way down timing after the top is erroneously estimated as the address timing, the take-back, or the backswing timing. Therefore, the estimation accuracy of the address timing, the take-back, and the backswing timing can be improved. In addition, since the search range for the maximum similarity is limited, the time required for the estimation process of the address timing, the take-back, and the backswing timing can be shortened.

[0120] In this way, the event timing estimation unit 3005F may first estimate the timing of the top by searching for the timing at which the similarity corresponding to the top is maximum in the entire range from the first frame to the last frame of the video. Then, the event timing estimation unit 3005F may estimate the timing of the address, the take-back, and the backswing by searching for the timing at which the similarity is maximum in the range before the timing of the top. Also, the event timing estimation unit 3005F may estimate the timing of the half-way down, the impact, the follow-through, and the finish by searching for the timing at which the similarity is maximum in the range after the timing of the top. The order of the process of estimating the timing of the address, the take-back, and the backswing and the process of estimating the timing of the half-way down, the impact, the follow-through, and the finish may be arbitrary, and each may be processed in parallel.

[0121] The event timing estimation unit 3005F may estimate the timing of a predetermined event by taking into consideration the tendency of change in time series of the similarity on the time axis of the video.

[0122] For example, it is known that the similarity corresponding to an impact has a minimum value near the timing of the top. Therefore, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the impact is maximum within a range after the timing at which the similarity corresponding to the impact is minimum. This can improve the estimation accuracy of the impact timing. Furthermore, the event timing estimation unit 3005F can estimate the impact timing in parallel with estimating the top timing. Therefore, it is possible to reduce the time required for the estimation process of the timing of a predetermined event.

[0123] Also, for example, the similarity corresponding to the address may maintain a relatively high state from long before the timing of the address, that is, the timing immediately before the club starts to move back. This is because the user is likely to maintain a posture state that is almost the same as the address from the timing before the address. Therefore, if the timing at which the similarity corresponding to the address is maximized is searched for over the entire range from the first frame to the last frame of the video, there is a possibility that the timing before the address will be erroneously estimated as the timing of the address.

[0124] In contrast, in this example, the event timing estimation unit 3005F searches for the timing at which the similarity corresponding to the address reaches a maximum value within a range of a predetermined period T1 before the timing at which the similarity corresponding to the address becomes relatively small relative to a predetermined threshold. The state in which the similarity becomes relatively small relative to the threshold S1th may be a state in which the similarity becomes equal to or smaller than the threshold S1th. Then, the event timing estimation unit 3005F estimates that the timing at which the similarity corresponding to the address reaches a maximum value within that range is the address timing. As a result, by appropriately setting the threshold S1th and the period T1, the timing immediately before the start of the swing motion can be appropriately estimated as the address timing. Therefore, the estimation accuracy of the address timing can be improved.

[0125] Also, for example, the similarity corresponding to the backswing may reach a maximum value at a time point before the address. This is because the user may perform a preparatory movement of moving the golf club to the backswing position before the address. Therefore, if the timing at which the similarity corresponding to the backswing reaches a maximum value is searched for over the entire range from the first frame to the last frame of the video, a timing before the address may be erroneously estimated as the backswing timing.

[0126] In response to this, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the backswing reaches a maximum value within a predetermined period T2 before the timing at which the similarity corresponding to the backswing becomes smaller relative to a predetermined threshold S2th. The event timing estimation unit 3005F then estimates that the timing at which the similarity corresponding to the backswing reaches a maximum value within that range is the timing of the backswing. As a result, by appropriately setting the threshold S2th and the period T2, it is possible to appropriately estimate the timing of the backswing by going back from the timing at which the similarity corresponding to the backswing drops significantly beyond the timing of the backswing. This makes it possible to improve the accuracy of estimating the timing of the backswing.

[0127] Also, for example, the similarity corresponding to the impact may remain relatively high even at the timing of the address and the period immediately before the address. This is because the user's posture, the position of the golf club head, and the posture of the shaft are similar at the address and the impact in a golf swing. Therefore, if the timing at which the similarity corresponding to the impact is maximized is searched for over the entire range from the first frame to the last frame of the video, the address or a timing before the address may be erroneously estimated as the timing of the impact.

[0128] In contrast, in this example, the event timing estimation unit 3005F searches for the timing at which the similarity corresponding to the impact reaches a maximum value in a predetermined period T3 after the timing at which the similarity corresponding to the impact becomes relatively large relative to a predetermined threshold S3th. The state in which the similarity becomes relatively large relative to the threshold S3th may be a state in which the similarity becomes equal to or larger than the threshold S3th. The event timing estimation unit 3005F estimates that the timing at which the similarity corresponding to the impact reaches a maximum value within that range is the timing of the impact. As a result, by appropriately setting the threshold S3th and the period T3, the timing of the impact can be appropriately estimated based on the timing at which the similarity increases again after the similarity has exceeded the timing of the address and decreased. Therefore, the estimation accuracy of the impact timing can be improved.

[0129] Also, for example, the similarity corresponding to the finish may remain relatively high even after the timing of the finish. This is because in a golf swing, the user may maintain the same posture even after the timing of the finish. Therefore, if the timing at which the similarity corresponding to the finish is maximized is searched for over the entire range from the first frame to the last frame of the video, a timing after the finish may be erroneously estimated as the timing of the finish.

[0130] In contrast, in this example, the event timing estimation unit 3005F searches for the timing at which the similarity corresponding to the finish reaches a maximum value within a range of a predetermined period T4 after the timing at which the similarity corresponding to the finish becomes relatively large relative to a predetermined threshold value S4th. Then, the event timing estimation unit 3005F estimates that the timing at which the similarity corresponding to the finish reaches a maximum value within that range is the timing of the finish. As a result, by appropriately setting the threshold value S4th and the period T4, the timing immediately after the swing motion stops can be appropriately estimated as the timing of the finish. Therefore, the estimation accuracy of the timing of the finish can be improved.

[0131] The event timing estimation unit 3005F may also estimate the timing of a predetermined event by focusing on the speed of the golf club head over time in a golf swing motion.

[0132] For example, the speed of the head of the golf club is relatively small at the timing of the address, the take-back, and the backswing, and is approximately zero at the timing of the top. The speed of the head of the golf club accelerates from the top to the halfway down and the impact, and reaches a maximum value immediately after the timing of the top. The speed of the head of the golf club decreases significantly from the follow-through to the finish. Therefore, the timing at which the speed of the head of the golf club is minimum can be set as the pseudo-timing of the top. Therefore, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the address, the take-back, and the backswing is maximum within a range before the timing at which the speed of the head after the start of the motion of the golf club is minimum. Similarly, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the halfway down, the impact, the follow-through, and the finish is maximum within a range after the timing at which the speed of the head after the start of the motion of the golf club is minimum. This can improve the estimation accuracy of a predetermined event before and after the top, and can limit the search range to shorten the processing time related to the estimation of the predetermined event.

[0133] Also, the timing at which the speed of the golf club head is at its maximum can be set as the impact timing in a pseudo manner. Therefore, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the address, take-back, backswing, top, and halfway down is at its maximum within a range before the timing at which the speed of the golf club head is at its maximum. Similarly, the event timing estimation unit 3005F may search for the timing at which the similarity corresponding to the follow-through and finish is at its maximum within a range after the timing at which the speed of the golf club head is at its maximum. This can improve the estimation accuracy of a predetermined event before and after the impact, and can limit the search range to shorten the processing time for estimating the predetermined event.

[0134] Furthermore, as described above, when multiple types of motion data are used, the event timing estimation unit 3005F may estimate the timing of a specified event based on the similarity between the motion data and the reference motion data obtained by setting relative weights among the multiple types of motion data.

[0135] For example, the motion data used for estimating the timing of a specific event may have a degree of importance depending on the type. For example, among the multiple types of motion data, the type of motion data with a high degree of importance is the type of motion data that tends to have a relatively small range of values ​​that can be taken by users at the timing of the specific event of interest, that is, the type of motion data that tends to take relatively close values ​​between users. In other words, among the multiple types of motion data, the type of motion data with a high degree of importance is data that represents the motion of a body part, a part of a golf club, etc. that tends to move similarly depending on the user at the timing of the specific event of interest. On the other hand, among the multiple types of motion data, the type of motion data with a low degree of importance is the type of motion data that tends to have a relatively large range of values ​​that can be taken by users at the timing of the specific event of interest, that is, the type of motion data that tends to take relatively divergent values ​​between users. In other words, among the multiple types of motion data, the type of motion data with a low degree of importance is data that represents the motion of a body part, a part of a golf club, etc. that tends to move differently depending on the user at the timing of the specific event of interest.

[0136] For example, at the timing of the top as a predetermined event, the head of the golf club substantially stops. Therefore, when the timing of the top is estimated, the speed data of the head of the golf club may be defined as the motion data of high importance. On the other hand, at the timing of the top as a predetermined event, the position of the head of the golf club may vary greatly depending on the user. Therefore, when the timing of the top is estimated, the position data of the head of the golf club may be defined as the motion data of low importance.

[0137] In this way, when multiple types of motion data are used, the similarity calculation unit 3005D may set weights between the multiple types of motion data according to the relative importance of the multiple types of motion data, and obtain the similarity between the motion data and the reference motion data. This allows the event timing estimation unit 3005F to more appropriately estimate the timing of a predetermined event based on the similarity obtained by the similarity calculation unit 3005D.

[0138] In addition, when multiple types of motion data are used, the similarity calculation unit 3005D may set weights between the multiple types of motion data according to differences in scale for each of the multiple types of motion data. The scale is the range of change in the value of the motion data in a golf swing motion.

[0139] For example, consider a case in which position and velocity data of a golf club head during a golf swing is used to estimate the timing of a given event.

[0140] The relative change in position of the golf club head during a golf swing is at most a few meters. Meanwhile, the speed of the golf club head during a golf swing varies in a range from 0 m / s to about 50 m / s. Therefore, if the position data and speed data of the golf club head are used as motion data as is, the similarity may be significantly influenced by the head speed data, which is on a larger scale, than the head position data, which is on a smaller scale. As a result, the use of the similarity may result in a case where the timing of a predetermined event is not appropriately estimated.

[0141] In contrast, the similarity calculation unit 3005D may set a small weighting for the speed of the golf club head, which has a relatively large scale, to acquire the similarity between the motion data and the reference motion data. For example, the similarity calculation unit 3005D sets a small weighting for the speed data of the golf club by multiplying the speed data of the golf club by a predetermined positive number smaller than 1, on the premise that the position data of the golf club head is used as is. This allows the similarity calculation unit 3005D to suppress the influence of the speed data of the head on the similarity between the motion data and the reference motion data. As a result, the event timing estimation unit 3005F can appropriately estimate the timing of a predetermined event using the similarity acquired by the similarity calculation unit 3005D.

[0142] In this way, when multiple types of motion data are used, the similarity calculation unit 3005D may set weights between the multiple types of motion data according to the relative scales of the multiple types of motion data to obtain the similarity between the motion data and the reference motion data. This allows the event timing estimation unit 3005F to more appropriately estimate the timing of a predetermined event based on the similarity obtained by the similarity calculation unit 3005D.

[0143] The event timing estimation unit 3005F may estimate the timing of a predetermined event from the time between the first frame and the last frame of the video by searching for a timing at which a predetermined condition related to the motion data corresponding to the predetermined event is satisfied. In this case, for example, a predetermined condition related to the motion data corresponding to the halfway down event is "the head of the golf club and the wrist are at the same height position during the downswing in the golf swing motion." Also, for example, a predetermined condition related to the motion data corresponding to the follow-through event is "the head of the golf club and the front knee (for example, the left knee in the case of right hitting or the right knee in the case of left hitting) are at the same height position during the follow-through in the golf swing motion." Also, the timing of the predetermined event estimated by the event timing estimation unit 3005F may be editable by the user. In this case, the estimation result of the event timing estimation unit 3005F is returned to the user terminal 200. Then, the application screen display processing unit 2001 of the user terminal 200 causes the display device 208 to display the image of the frame of the video corresponding to the estimation result of the timing of the predetermined event received from the information processing device 300 together with a group of frames before and after the video. This allows the user to determine whether the image of the frame corresponding to the estimation result of the timing of the predetermined event is appropriate, and if it is not appropriate, to select an appropriate frame image from the group of frames before and after the event. Therefore, the user terminal 200 returns information about the image of the selected frame to the information processing device 300, so that the event timing estimation unit 3005F of the information processing device 300 can edit the timing of the predetermined event based on the information received from the user terminal 200.

[0144] The image extraction unit 3005 extracts a frame image corresponding to the timing of a predetermined event from among all frame images included in the video data based on the estimation result of the event timing estimation unit 3005F. For example, the image extraction unit 3005 extracts one frame image corresponding to the timing of a predetermined event estimated by the event timing estimation unit 3005F. Furthermore, when the timing of the predetermined event estimated by the event timing estimation unit 3005F corresponds to a timing between images of two frames, the image extraction unit 3005 extracts these two frame images.

[0145] The feature amount calculation unit 3006 calculates and acquires feature amounts relating to the three-dimensional movement state of a predetermined feature point based on the image extracted by the image extraction unit 3005. For example, the feature amount calculation unit 3006 acquires feature amounts relating to the trajectory of the head of a golf club (hereinafter simply referred to as "head trajectory") before and after the impact of a golf swing motion (hereinafter referred to as "impact zone" for convenience). In this case, the predetermined feature point is the head of the golf club.

[0146] For example, as shown in FIG. 9, the feature amount calculation unit 3006 calculates an angle θ d and angle θ f In this example, the angle difference Δθ is calculated as the angle θ f from angle θ d (Δθ=θ f -θ d ).

[0147] Angle θ d is the head position P at the timing of the halfway down in the downswing of the golf swing motion. d The angle θ is the angle between the horizontal line and the line connecting the origin point corresponding to the position of the golf ball and the line connecting the origin point and the horizontal line. f is the head position P at the timing of the follow-through in the golf swing motion fand the position of the origin corresponding to the position of the golf ball, and the angle that the line connecting them forms with the horizontal line.

[0148] The head trajectory represents the direction of movement of the head of the golf club relative to the golf ball in a plan view in the impact zone. In this example, the head trajectory represents the angle θ h Hereinafter, for convenience, the head trajectory (angle θ h ) is positive, and the head trajectory (angle θ h ) is defined as negative. This allows the head trajectory to take a positive value when the user's swing motion has an inside-out tendency, and to take a negative value when the user's swing motion has an outside-in tendency.

[0149] The feature amount calculation unit 3006 calculates the angle θ based on the position of the golf club head and the position of the golf ball on the image at the half-way down timing extracted by the image extraction unit 3005. d When two images corresponding to the half-way down are extracted by the image extraction unit 3005, the head position at the timing of the half-way down is estimated from the head positions in both images, and the angle θ d may be calculated.

[0150] Similarly, the feature amount calculation unit 3006 calculates the angle θ based on the position of the golf club head and the position of the golf ball on the image at the time of follow-through extracted by the image extraction unit 3005. f When two images corresponding to the follow are extracted by the image extraction unit 3005, the head position at the timing of the follow is estimated from the head positions in both images, and the angle θ f may be calculated.

[0151] The trajectory estimation unit 3008 estimates the three-dimensional movement state (e.g., trajectory) of a predetermined feature point based on the feature amount relating to the three-dimensional movement state of a predetermined feature point in a golf swing motion acquired by the feature amount calculation unit 3006. Specifically, the trajectory estimation unit 3008 estimates the three-dimensional movement state of the predetermined feature point using correlation data stored in the correlation data storage unit 3007 based on the feature amount acquired by the feature amount calculation unit 3006. For example, the trajectory estimation unit 3008 estimates the head trajectory (angle θ) using correlation data based on the three-dimensional movement state of the golf club head as the predetermined feature point, i.e., the angle difference Δθ as a feature amount relating to the head trajectory. h ) is estimated.

[0152] The correlation data is data including information expressing a correlation between a feature amount related to a three-dimensional trajectory of a specific feature point in a golf swing motion and a three-dimensional movement state of the specific feature point. For example, the correlation data is data including a conversion formula that converts a feature amount related to a three-dimensional movement state of a specific feature point in a golf swing motion into a three-dimensional movement state. This allows the trajectory estimation unit 3008 to estimate a three-dimensional movement state by substituting the feature amount acquired by the feature amount calculation unit 3006 into the conversion formula.

[0153] For example, as shown in FIG. 10, the head trajectory feature (angle difference Δθ) and the head trajectory (angle θ h The conversion equation expressing the correlation with the α-threshold value (α) is defined in advance as a correlation line (dashed line in the figure) based on multiple actual measurement data of subjects (black circles in the figure).

[0154] The actual measurement data was obtained by simultaneously measuring the subject's golf swing in 3D and capturing a video image with one camera. The head trajectory (angle θ h) and a feature quantity (angle difference Δθ) based on a video image. Three-dimensional measurement of a subject's golf swing motion can be realized by known techniques such as motion capture. In this example, actual measurement data is plotted on a plane with the head trajectory as the horizontal axis and the angle difference Δθ as the vertical axis, and a correlation line is defined based on a plurality of actual measurement data. As a result, the trajectory estimation unit 3008 can estimate the head trajectory from the angle difference Δθ in a form conforming to the actual three-dimensional measurement result by using the conversion formula based on the plurality of actual measurement data in FIG. 10. Therefore, the trajectory estimation unit 3008 can improve the estimation accuracy of the head trajectory based on the two-dimensional image acquired by the camera 100.

[0155] The swing diagnosis unit 3009 performs swing diagnosis based on the estimation result by the trajectory estimation unit 3008, i.e., the estimation result of the three-dimensional moving state (trajectory) of a predetermined feature point. For example, the swing diagnosis unit 3009 evaluates the tendency of inside-out or outside-in in the impact zone in the user's golf swing motion and the degree of the tendency based on the estimation result of the head trajectory by the trajectory estimation unit 3008, and performs swing diagnosis.

[0156] The swing diagnosis unit 3009 may perform swing diagnosis based on other diagnostic indices instead of or in addition to the head trajectory in the impact zone. For example, the swing diagnosis unit 3009 may calculate a diagnostic indices for a golf swing motion based on a frame image corresponding to the occurrence of a predetermined event or a predetermined range of frame images before and after the occurrence of the predetermined event. Diagnostic indices for a golf swing motion include head speed, attack angle, delay, face angle, body center of gravity position, forward lean angle, trajectory difference, etc. Head speed indicates the speed of the head of the golf club. Attack angle indicates the angle of incidence of the club (head) with respect to the golf ball based on the horizontal plane at the time of impact with the golf ball. Delay indicates the time lag between the movement of the arms and the movement of the club head during the downswing. Face angle indicates the orientation of the face surface in the horizontal direction at the time of impact with the golf ball. Body center of gravity position indicates the center of gravity position of the center of mass of each part of the body. Forward lean angle is the forward lean angle of the upper body at the time of address. The trajectory difference represents the difference between the front and rear of the trajectory at a predetermined height of the backswing and the downswing.The swing diagnosis unit 3009 then performs diagnosis on the golf swing motion of the user based on the calculated diagnostic index.For example, the swing diagnosis unit 3009 compares the value of the diagnostic criterion, i.e., the reference value of the diagnostic index, with the calculated value of the diagnostic index to perform diagnosis on the conformity of the user's swing motion to the diagnostic criterion.

[0157] When the swing diagnosis process is completed, the swing diagnosis unit 3009 transmits reply data including data on the result of the swing diagnosis to the user terminal 200 via the communication interface 306. The reply data includes, for example, data on the diagnostic index and data on advice based on the diagnostic index. The reply data may also include data on the joint positions of a plurality of body parts of the user at the timing of a specified event, data on the head position of the golf club, and the like.

[0158] [Swing diagnosis system operation] Next, the operation of the swing diagnosis system 1 will be described with reference to FIG.

[0159] FIG. 11 is a sequence diagram showing an example of the operation of the swing diagnosis system 1.

[0160] As shown in FIG. 11, the user terminal 200 starts up a swing diagnosis application in response to a predetermined input from the user using the input device 207 (step S102).

[0161] After completing the processing of step S102, the video data acquisition unit 2002 acquires video data representing the user's golf swing motion from the camera 100 in response to a specific input from the user on a specific application screen using the input device 207 (step S104).

[0162] After completing the process of step S104, the video data transmission unit 2003 transmits the video data to the information processing device 300 via the communication interface 206 (step S106).

[0163] The video data acquisition unit 3001 acquires the video data transmitted (uploaded) from the user terminal 200 in the process of step S106 (step S107).

[0164] After completing the process of step S107, the feature point extraction unit 3003 extracts feature points related to the swing motion of the subject user for each frame of the video (step S108).

[0165] After completing the process of step S108, the motion data acquisition unit 3005A acquires motion data related to the user's golf swing motion based on the feature points extracted in step S108 for each frame of the video (step S110).

[0166] After completing the process of step S110, the similarity calculation section 3005D calculates the similarity between the action data of each frame of the video and the reference action data for each predetermined event (step S112).

[0167] After completing the process of step S112, the event timing estimation unit 3005F estimates the timing of each predetermined event based on the similarity of each frame for each predetermined event (step S114).

[0168] After completing the process of step S114, the image extraction unit 3005 extracts a frame image corresponding to the timing of the predetermined event estimated in the process of step S114 from among the frame images included in the video (step S116).

[0169] After completion of step S116, the feature amount calculation unit 3006 calculates and acquires feature amounts relating to the three-dimensional movement state of a predetermined feature point in the user's golf swing motion, based on the image of the frame extracted in the processing of step S116 (step S118).

[0170] After completing the process of step S118, the trajectory estimation unit 3008 estimates a trajectory corresponding to the three-dimensional moving state of a predetermined feature point based on the feature amount acquired in the process of step S118 (step S120).

[0171] After completing the process of step S120, the swing diagnosis unit 3009 performs swing diagnosis on the swing motion of the video image data based on the three-dimensional trajectories of the predetermined feature points estimated in the process of step S120 (step S122).

[0172] After completing the process of step S122, the swing diagnosis section 3009 transmits reply data including data on the result of the swing diagnosis to the user terminal 200 (step S124).

[0173] The diagnosis result data acquisition unit 2004 acquires the reply data including the data of the result of the swing diagnosis, transmitted from the information processing device 300 in the process of step S124 (step S125).

[0174] After completing the process of step S125, the application screen display processing unit 2001 causes the display device 208 to display the results of the swing diagnosis (step S126).

[0175] In this manner, the swing diagnosis system 1 can obtain video data showing the state of a golf swing motion by a user, perform swing diagnosis based on the video data, and provide the result of the swing diagnosis to the user via the user terminal 200.

[0176] [Effect] Next, the operations of the information processing device, the information processing method, the program, and the information processing system according to the present embodiment will be described.

[0177] In this embodiment, the information processing device recognizes the positions of feature points of at least one of the person and the golf club in the time series of the moving image based on a moving image showing the golf swing motion of the subject person captured by one camera. The information processing device is, for example, the above-mentioned information processing device 300. The camera is, for example, the above-mentioned camera 100. The information processing device also acquires feature amounts related to the three-dimensional movement state of the feature points based on the recognized positions of the feature points in the time series. Then, the information processing device estimates the three-dimensional movement state of the feature points in the golf swing motion based on the acquired feature amounts.

[0178] In addition, in the information processing method according to the present embodiment, the information processing device recognizes the positions of feature points of at least one of the person and the golf club in the time series of the moving image based on a moving image showing the golf swing motion of the subject person captured by one camera. In addition, in the information processing method, the information processing device acquires feature amounts related to the three-dimensional movement state of the feature points based on the recognized positions of the feature points in the time series. Then, in the information processing method, the information processing device estimates the three-dimensional movement state of the feature points in the golf swing motion based on the acquired feature amounts.

[0179] In this embodiment, the program causes the information processing device to recognize the positions of at least one of feature points of a person and a golf club in a time series of moving images based on a moving image showing a golf swing motion of a person captured by one camera. The program also causes the information processing device to acquire feature amounts related to the three-dimensional movement state of the feature points based on the recognized positions of the feature points in the time series. The program then causes the information processing device to estimate the three-dimensional movement state of the feature points in the golf swing motion based on the acquired feature amounts.

[0180] This allows the information processing device to estimate the three-dimensional movement state of the feature points in a golf swing using two-dimensional images captured by the camera, and therefore the information processing device can grasp the golf swing three-dimensionally with a relatively simple configuration.

[0181] In this embodiment, the moving image may be a moving image captured from behind a subject person.

[0182] This allows the information processing device to estimate the three-dimensional movement state of feature points in a golf swing motion, using a moving image captured from behind the subject.

[0183] In this embodiment, the feature point may include the head of the golf club. The feature amount may be an angle difference between a first angle defined by a head position at a first timing in the downswing of the golf swing and a predetermined origin, and a second angle defined by a head position at a second timing in the follow-through of the golf swing and a predetermined origin. The first angle may be, for example, the above-mentioned angle θ d The second angle is, for example, the above-mentioned angle θ f The angle difference is, for example, the above-mentioned angle difference Δθ. Then, the information processing device may estimate the head trajectory at the time of impact based on the acquired angle difference. The head trajectory is the above-mentioned angle θ h It is.

[0184] This allows the information processing device to estimate the head trajectory in the impact zone during a golf swing. Therefore, the information processing device can grasp the golf swing motion in the impact zone in three dimensions using the two-dimensional image captured by the camera.

[0185] In the present embodiment, the information processing device may acquire data representing the motion of the person and the golf club of each of the multiple images included in the moving image. The information processing device may extract an image corresponding to a first timing from the multiple images based on a similarity between the data of each of the acquired multiple images and first reference data representing the motion of at least one of the person and the golf club corresponding to the first timing. The information processing device may extract an image corresponding to a second timing from the multiple images based on a similarity between the data of each of the acquired multiple images and second reference data representing the motion of at least one of the person and the golf club corresponding to the second timing. The information processing device may acquire the position of the head at each of the first timing and the second timing based on the extracted images corresponding to each of the first timing and the second timing.

[0186] This enables the information processing device to calculate the angle difference as a feature amount related to the head trajectory, based on the head positions at the first timing and the second timing.

[0187] In addition, in this embodiment, the information processing device may estimate the three-dimensional movement state of the feature point from the acquired feature amount by using information representing the correlation between the feature amount and the three-dimensional movement state of the feature point.

[0188] This allows the information processing device to estimate the three-dimensional movement state of the feature points by matching the acquired feature amounts with information indicating the correlation.

[0189] In this embodiment, the information indicating the correlation may be a conversion formula between the feature amount and the three-dimensional movement state of the feature point.

[0190] This allows the information processing device to calculate the three-dimensional movement state of the feature point by substituting the acquired feature amount into the conversion formula.

[0191] In addition, in this embodiment, the information representing the correlation may be information representing the correlation between the three-dimensional movement state of feature points based on 3D measurement when 3D measurement of a golf swing motion and acquisition of a video using a single camera are performed simultaneously for each of a plurality of subjects, and feature amounts based on the video.

[0192] This allows the information processing device to reflect the correlation between the three-dimensional movement state of the feature points obtained by the three-dimensional measurement of the swing motion and the feature amount obtained from the video image of the camera in the estimation result of the three-dimensional movement state of the feature points. Therefore, the information processing device can estimate the three-dimensional movement state of the feature points with higher accuracy.

[0193] In the present embodiment, the information processing system may include the above camera and the above information processing device. Specifically, the camera includes an imaging unit that captures a moving image of a subject person, and a display unit that displays an image of the imaging range of the imaging unit in real time. The imaging unit is, for example, the imaging unit 110 described above. The display unit is, for example, the display unit 120 described above. The display unit may display an auxiliary image for adjusting the relative positional relationship between the camera and a target for hitting a ball by a golf swing motion to a predetermined state, superimposed on the image of the imaging range of the imaging unit in real time. The auxiliary image is, for example, the above auxiliary images 33 to 36. The image of the real time imaging range is, for example, the above live view image 31.

[0194] This allows the information processing device to use video images captured under conditions where the relative positional relationship between the camera and the target of the ball being hit by the golf swing motion is relatively close to a predetermined state, thereby suppressing variation in the relative positional relationship between the camera and the target of the ball being hit by the golf swing motion in the captured video images, and as a result, the information processing device can more accurately estimate the three-dimensional movement state of the feature points.

[0195] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims. [Explanation of symbols]

[0196] 1 Swing Diagnosis System 30 screens 31 Live View Images 32 Shooting button image 33~36 Auxiliary images 33A Vertex 33B One side 33C One Side 100 Cameras 110 Imaging unit 120 Display section 130 Control section 200 User terminals 208 Display device 300 Information processing device 2001 Application screen display processing unit 2002 Video data acquisition unit 2003 Video data transmission unit 2004 Diagnostic result data acquisition department 3001 Video data acquisition unit 3002 Video data storage unit 3003 Feature point extraction unit 3004 Feature point data storage unit 3005 Image Extraction Unit 3005A Operation Data Acquisition Unit 3005B Operation data storage unit 3005C Reference operation data storage unit 3005D Similarity calculation unit 3005E Similarity data storage unit 3005F Event Timing Estimation Unit 3006 Feature Calculation Unit 3007 Correlation data storage unit 3008 Orbit estimation part 3009 Swing Diagnosis Department

Claims

1. Based on a video captured by a single camera and showing a golf swing motion of a subject person, the positions of feature points of at least one of the person and the golf club in a time series of the video are recognized, and based on the positions of the feature points in the recognized time series, feature amounts relating to a three-dimensional movement state of the feature points are acquired, and based on the acquired feature amounts, the three-dimensional movement state of the feature points in the golf swing motion is estimated. Information processing device.

2. The moving image is a moving image captured from behind a subject person. The information processing device according to claim 1 .

3. The features include a golf club head; the characteristic amount is an angle difference between a first angle defined by a position of the head at a first timing in a downswing of a golf swing motion and a predetermined origin, and a second angle defined by a position of the head at a second timing in a follow-through of the golf swing and the predetermined origin, A head trajectory at the time of impact is estimated based on the acquired angle difference. The information processing device according to claim 2 .

4. acquiring data representing the motion of the subject person and the golf club for each of a plurality of images included in the moving image, extracting an image corresponding to the first timing from the plurality of images based on a similarity between the data for each of the acquired plurality of images and first reference data representing the motion of at least one of the person and the golf club corresponding to the first timing, extracting an image corresponding to the second timing from the plurality of images based on a similarity between the data for each of the acquired plurality of images and second reference data representing the motion of at least one of the person and the golf club corresponding to the second timing, and acquiring a position of the head at each of the first timing and the second timing based on the extracted images corresponding to each of the first timing and the second timing. The information processing device according to claim 3 .

5. estimating a three-dimensional movement state of the feature point from the acquired feature amount by using information representing a correlation between the feature amount and a three-dimensional movement state of the feature point; The information processing device according to claim 1 .

6. The information representing the correlation is a conversion formula between the feature amount and a three-dimensional movement state of the feature point. The information processing device according to claim 5 .

7. The information representing the correlation is information representing a correlation between a three-dimensional movement state of the feature points based on the three-dimensional measurement when three-dimensional measurement of a golf swing motion and acquisition of a moving image by one camera are performed simultaneously for each of a plurality of subjects, and the feature amount based on the moving image. The information processing device according to claim 5 .

8. the camera including an imaging unit that captures the moving image of the person as the subject, and a display unit that displays an image of an imaging range of the imaging unit in real time; An information processing device according to any one of claims 1 to 4, the display unit displays an auxiliary image for adjusting a relative positional relationship between the camera and a target for hitting a ball by a golf swing motion to a predetermined state, superimposed on an image in a real-time imaging range of the imaging unit. Information processing system.

9. an information processing device, based on a video captured by a single camera and showing a golf swing motion of a subject person, recognizes positions of feature points of at least one of the person and the golf club in a time series of the video, acquires feature amounts relating to a three-dimensional movement state of the feature points based on the recognized positions of the feature points in the time series, and estimates a three-dimensional movement state of the feature points in the golf swing motion based on the acquired feature amounts; Information processing methods.

10. In the information processing device, Based on a moving image captured by a single camera and showing a golf swing motion of a subject person, the positions of feature points of at least one of the person and the golf club in a time series of the moving image are recognized, a feature amount relating to a three-dimensional movement state of the feature point is acquired based on the recognized positions of the feature point in the time series, and a three-dimensional movement state of the feature point in the golf swing motion is estimated based on the acquired feature amount. program.

Images