System, program, and the like

Abstract

To provide a system and the like that are more excellent than a conventional one, such as a mechanism that provides a user with information on swing.SOLUTION: A system includes: a detection device that performs swing detection being detection of swing of a club by using a sensor; and an analysis device that performs imaging of at least one of a head of the club and a ball and swing analysis being analysis of the swing.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present invention relates to systems, programs, etc.

Background Art

[0002] As background art in this technical field, there is Japanese Patent Application Laid-Open No. 2012-165811 (Patent Document 1), etc. This publication describes, "A golf support system for measuring a golf club swing, which measures head speed, ball speed, meet rate, and estimated flight distance, and compares these measurement results with reference data of preset head speed, ball speed, meet rate, and estimated flight distance to determine pass / fail, and includes control means 21 for determining pass / fail, and notification means 23 for notifying the pass / fail of these measurement results in color based on a signal from the control means 21." (See the abstract).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional mechanisms for communicating communication information have various problems.

[0005] In view of the above problems, one object of the present invention is to provide a system, etc., that is superior to the prior art, such as a mechanism for providing information related to a swing to a user.

[0006] The object of the present invention is not limited to this, and the applicant also has the intention to obtain rights through divisional applications, amendments, etc. for configurations aimed at obtaining effects resulting from parts of the configurations disclosed in this specification, drawings, etc. For example, in this specification, problems obtained by reading parts described as "can" or "is possible" as "is a problem" are disclosed in this specification. The problems are described as independent ones, and the applicant also has the intention to obtain rights through divisional applications, amendments, etc. for configurations for solving each problem alone. Even if a problem is implicitly grasped from the description of the specification, the applicant has the intention to make part of the configuration described in this specification the scope of claims by amendment or divisional application. In addition, configurations for solving problems combining these independent problems are also disclosed, and the applicant has the intention to obtain rights.

Means for Solving the Problems

[0007] In order to solve the above problems, for example, the configuration described in the scope of claims is adopted. This application includes a plurality of means for solving the above problems. For example, a system includes a detection device that uses a sensor to perform swing detection, which is detection of a club swing, and an analysis device that performs at least one of photographing the club head and the ball and swing analysis, which is analysis of the swing.

Effects of the Invention

[0008] According to the present invention, it is possible to provide a system, etc. that is superior to the conventional ones, such as a mechanism for providing information related to a swing to a user. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

[0009] Note that the effects of the invention of the present application are not limited to this, and the effects achieved by the components of the configurations disclosed in this specification, drawings, etc. are also disclosed. The applicant also intends to obtain rights for the configurations achieving such effects through divisional applications, amendments, etc. For example, in this specification, the parts described as "can", "is possible", etc. are descriptions that explicitly indicate the achieved effects. Even without such descriptions as "can", "is possible", etc., there are parts indicating the effects. Further, even without such descriptions, there are effects grasped by the said configuration.

Brief Description of Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Best Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that the embodiments shown below are one of the embodiments for providing the present invention, and the content of the present invention is not limitedly construed based on the following description.

[0012] Also, in each drawing, for components having the same function, the description may be omitted with the addition of reference numerals.

[0013] FIG. 1 is an example of an explanatory diagram for explaining a system 10 etc. connected to a network 100. In the configuration of FIG. 1, the analysis system 10 is composed of a detection terminal 20 and an analysis terminal 30. The analysis system 10 is connected to a user terminal 40 and a management terminal 50 via a network 100. However, as will be described later, the analysis system 10 can function with a configuration in which only the detection terminal 20 and the analysis terminal 30 are communicatively connected.

[0014] The analysis system 10 (hereinafter also referred to as "system 10") is an example of a "system", and analyzes the swing of an instrument by a user, for example, the swing of a golf club, and the ball hit by the swing, for example, a golf ball. In the following description, a golf club and a golf ball will be used as examples for explanation.

[0015] The analysis system 10 has a configuration including at least a detection terminal 20 and an analysis terminal 30. The detection terminal 20 is an example of a "detection device", and the analysis terminal 30 is an example of an "analysis device".

[0016] The detection terminal 20 detects various parameters related to the swing of a golf club by a user and the ball hit by the swing at a golf course, an outdoor practice range, an indoor practice range, etc. Details of the detection terminal 20 will be described later with reference to FIG. 2 and the like.

[0017] The analysis terminal 30 analyzes various parameters related to the swing of a golf club by a user and the ball hit by the swing at a golf course, an outdoor practice range, an indoor practice range, etc. Details of the analysis terminal 30 will be described later with reference to FIG. 3 and the like.

[0018] In the system 10, by linking the detection terminal 20 that detects a swing and the analysis terminal 30 that analyzes the detected (hereinafter sometimes referred to as "detection") swing, functions that cannot be provided only by the detection terminal 20 can be provided to the user.

[0019] In FIG. 1, the system 10 having the detection terminal 20 and the analysis terminal 30 is illustrated as an example connected to a user terminal 40 and a management terminal (model creation terminal) 50 described later via a network 100. However, the system 10 can be used with a simple configuration in which only the detection terminal 20 and the analysis terminal 30 are communicatively connected and are self - contained in the detection terminal 20 and the analysis terminal 30. By doing so, an analysis system 10 that is easy for the user to introduce can be provided.

[0020] The detection terminal 20 and the analysis terminal 30 are preferably configured as portable terminals that can be carried by the user and whose placement location can be easily adjusted. The user can easily connect the detection terminal 20 and the analysis terminal 30, for example, by BLE communication or the like, and easily use them in various places using such a system 10.

[0021] In the system 10 where the detection terminal 20 and the analysis terminal 30 are portable terminals and are wirelessly communicatively connected to each other, the user does not need to use a large - scale swing analysis device or visit a dedicated analysis facility to perform his / her own swing analysis.

[0022] The user can use the system 10 at any location where the available arrangements of the detection terminal 20 and the analysis terminal 30 can be realized (see FIGS. 8, 9, etc.), for example, indoor practice fields, golf courses, gardens, vacant lots, etc., and can check their own swings more closely and inexpensively than swing analysis using conventional large swing analysis devices.

[0023] The user terminal 40 is, for example, a terminal such as a smartphone 40a, a tablet terminal 40b, a PC 40c, a wearable terminal 40d, VR / AR goggles, etc., but is not limited thereto.

[0024] It is preferable that the user can use the user terminal 40 to store (save), confirm, for example, display and play back, the information from the detection terminal 20 and the analysis terminal 30. Furthermore, it is preferable that the user can use the user terminal 40 to operate the detection terminal 20 and the analysis terminal 30.

[0025] The management terminal 50 is, for example, a server operated by the manufacturer or seller of the detection terminal 20, and manages various information obtained, measured, detected, sensed, analyzed, created, etc. by the detection terminal 20 and the analysis terminal 30.

[0026] The management terminal 50 preferably further has a function of generating a machine learning model used in the detection terminal 20 and the analysis terminal 30 as a model creation server. Specific examples of the machine learning model will be described later with reference to FIGS. 19 to 21, etc.

[0027] The network 100 may be a wired network or a wireless network, but in particular, it is preferably a combination of a wired network and a wireless network that enables communication between the detection terminal 20, the analysis terminal 30, the user terminal 40, and the management terminal (model creation terminal) 50.

[0028] Furthermore, the network 100 may be a network that combines a plurality of networks with different standards, specifications, etc. Further, the network 4 may be a network adopted in a stand-alone system independent of other networks.

[0029] The detection terminal 20, the analysis terminal 30, and the management terminal 50 may be, for example, mobile terminals such as smartphones, tablets, mobile phones, and personal digital assistants (PDAs), or wearable terminals such as glasses-type, wristwatch-type, and clothing-type. Further, they may be stationary or portable computers, or servers arranged on the cloud or network. Further, as functions, they may be VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, or MR (Mixed Reality) terminals. Alternatively, they may be a combination of a plurality of these terminals. For example, a combination of one smartphone and one wearable terminal can function logically as one terminal. Further, they may be information processing terminals other than these.

[0030] The detection terminal 20, the analysis terminal 30, and the management terminal 50 each include a processor that executes an operating system, an application, a program, etc., a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), or a flash memory, a communication control unit such as a network card, a wireless communication module, or a mobile communication module, an input device such as a touch panel, a keyboard, a mouse, voice input, or input by motion detection by imaging of a camera unit, and an output device such as a monitor or a display. Note that the output device may be a device or a terminal that transmits information for output to an external monitor, display, printer, device, etc.

[0031] The main memory device stores various programs, applications, etc. (modules), and by the processor executing these programs and applications, each functional element of the entire system is realized. Note that these modules may be implemented in hardware by integration or the like. Also, each module may be an independent program or application, or may be implemented in the form of sub-programs, functions, etc. within a single integrated program or application.

[0032] In this specification, each module is described as the subject (noun) that performs the processing, but in reality, it is the processor that processes various programs, applications, etc. (modules) that executes the processing. The auxiliary storage device stores various databases (DBs). A "database" is a functional element (storage unit) that stores a data set so as to be able to respond to any data operation (for example, extraction, addition, deletion, overwrite, etc.) from a processor or an external computer. The implementation method of the database is not limited, and for example, it may be a database management system, spreadsheet software, or a text file such as XML or JSON.

[0033] Hereinafter, each component, module, etc. of the device described with reference to FIGS. 2 to 5 can be added, omitted, integrated, or substituted according to the embodiment. Also, each component, module, etc. of the device described with reference to FIGS. 2 to 5 may exist in a state where they are distributed and arranged and stored throughout the system with respect to control and processing. Furthermore, FIGS. 2 to 5 are schematic examples of the hardware configuration, and the components, modules, etc. described therein do not mean that all of them are essential components for each terminal or server.

[0034] FIG. 2 is an example of the hardware configuration of the detection terminal 20. The detection terminal 20 is an example of a "detection device" and has the functions described with reference to FIG. 1.

[0035] In the main memory 201 of the detection terminal 20, programs and applications such as a swing detection module 211, a swing information creation module 212, a cooperation module 213, and a display module 214 are stored. By the processor 203 executing these programs and applications, each functional element of the detection terminal 20 is executed.

[0036] In the following description, each process executed by the processor 203 using each of the modules 211 to 214 will be described with each of the modules 211 to 214 as the main subject (subject). Also, the process described with one module as the main subject may be executed in cooperation by the module used as the main subject and other modules.

[0037] Based on the detection signal detected by the detection unit 206, the swing detection module 211 detects information related to the swing of the golf club by the user. The detection unit 206 is an example of a "sensor", and is, for example, a sensor such as a Doppler sensor (hereinafter sometimes referred to as a "sensor").

[0038] Based on the information detected by the swing detection module 211, the swing information creation module 212 creates swing detection information indicating that a swing has been detected, swing timing information, swing parameter information regarding various parameters of the swing, and the like. The swing detection information may particularly indicate that a downswing has been detected.

[0039] The swing detection module 211 can also distinguish between a downswing, an upswing, so-called a backswing, and create different swing detection information for each.

[0040] The cooperation module 213 controls the cooperation between the detection terminal 20 and the analysis terminal 30. For example, the cooperation module 213 establishes communication between the detection terminal 20 and the analysis terminal 30 and controls the established communication.

[0041] The display module 214 displays the swing information created by the swing information creation module 212 and the analysis information created by the analysis terminal 30 on an output device 205 such as a display.

[0042] The auxiliary storage device 202 of the detection terminal 20 stores setting information 221, history information 222, user information 223, statistical information 224, etc. The processor 203 of the detection terminal 20 executes various functions of the detection terminal 20 using at least a part of those pieces of information and the above-mentioned modules 211 to 214, etc.

[0043] FIG. 3 is an example of the hardware configuration of the analysis terminal 30. The analysis terminal 30 is an example of an "analysis device" and has the functions described with reference to FIG. 1.

[0044] The main storage device 301 of the analysis terminal 30 stores programs and applications such as a photographing module 311, a swing analysis module 312, a cooperation module 313, and a display module 314. By the processor 303 executing these programs and applications, each functional element of the analysis terminal 30 is executed.

[0045] In the following description, each process executed by the processor 303 using the modules 311 to 314 will be described with the modules 311 to 314 as the main body (subject). Also, the process described with one module as the main body may be executed in cooperation by the module used as the main body and other modules.

[0046] The photographing module 311 creates photographing information related to the user's swing using a photographing unit 306 such as a camera. The imaging module 311 may be able to create imaging information at a high frame rate, for example, 120 fps or higher, preferably 240 fps or higher.

[0047] Based on the imaging information captured and created by the imaging module 311, the swing analysis module 312 analyzes the swing of the captured user. Specific swing analysis processing by the swing analysis module 312 will be described later with reference to FIGS. 15 to 22 and the like.

[0048] In the configuration of FIG. 3, the swing analysis module 312 includes an image extraction module 315, a coordinate calculation module 316, a trajectory calculation module 317, and a lowest point information / distance information creation module 318.

[0049] The cooperation module 313 controls the cooperation between the detection terminal 20 and the analysis terminal 30. For example, the cooperation module 313 establishes communication between the detection terminal 20 and the analysis terminal 30 and controls the established communication.

[0050] The display module 314 displays the swing information created by the swing information creation module 212 and the analysis information created by the analysis terminal 30 on an output device 305 such as a display.

[0051] The output device 305 of the analysis terminal 30 may be configured to be able to display more detailed information, for example, display more information, display information with higher resolution, etc., compared to the output device 205 of the detection terminal 20.

[0052] In such a configuration, the analysis terminal 30 can represent various information in a richer content, for example, compared to the output display in the detection terminal 20 where the output device 205 is a segment display or the like.

[0053] The auxiliary storage device 302 of the analysis terminal 30 stores setting information 321, history information 322, user information 323, statistical information 324, etc. The processor 303 of the analysis terminal 30 executes various functions of the analysis terminal 30 by using at least a part of that information and each of the above modules 311 to 314 and the like.

[0054] FIG. 4 is an example of the hardware configuration of the user terminal 40. The user terminal 40 has the functions described with reference to FIG. 1. In the main storage device 401 of the user terminal 40, programs and applications such as a user information detection module 411, a user information creation module 412, a cooperation module 413, and a display module 414 are stored. By executing these programs and applications by the processor 403, each functional element of the user terminal 40 is executed.

[0055] In the following description, each process executed by the processor 403 using each of the modules 411 to 414 will be described with each of the modules 411 to 414 as the main subject (subject). Further, the process described with one module as the main subject may be executed in cooperation by the module that is the main subject and other modules.

[0056] The user information detection module 411 detects information about individual users from the information obtained from the detection terminal 20, the analysis terminal 30, and the like. The user information detection module 411 further detects biometric information of the user and performs an association such as the time when the biometric information was obtained and the time related to the information obtained from the detection terminal 20, the analysis terminal 30, and the like.

[0057] The user information creation module 412 creates user information from the detected information about the user, thereby facilitating information management for each user based on, for example, purpose, progress, physical condition, etc. of the information obtained from the detection terminal 20, the analysis terminal 30, and the like.

[0058] The cooperation module 413 controls the cooperation between the user terminal 40 and the detection terminal 20 and the cooperation between the user terminal 40 and the analysis terminal 30. For example, the cooperation module 413 establishes and controls communication between the user terminal 40 and the detection terminal 20, and between the user terminal 40 and the analysis terminal 30.

[0059] The display module 414 displays the swing information created by the swing information creation module 212 and the analysis information created by the analysis terminal 30 on an output device 405 such as a display.

[0060] The auxiliary storage device 402 of the user terminal 40 stores setting information 421, history information 422, user information 423, statistical information 424, and the like. The user can store the information stored in the detection terminal 20 or the analysis terminal 30 in the auxiliary storage device 402 of the user terminal 40 via the network 100 or a storage medium described in FIG. 6 below, such as a microSD (registered trademark) card. The processor 403 of the user terminal 40 executes various functions of the user terminal 40 using at least a part of the information and the above-described modules 411 to 414.

[0061] FIG. 5 is an example of the hardware configuration of the management terminal 50. The management terminal (model creation terminal) 50 has the functions described with reference to FIG. 1. The main storage device 501 of the management terminal 50 stores programs and applications such as a statistical information creation module 511, a model generation module 512, a correction information creation module 513, and a display module 514. By executing these programs and applications by the processor 503, each functional element of the management terminal 50 is executed.

[0062] In the following description, each process executed by the processor 503 using the modules 511 to 514 will be described with the modules 511 to 514 as the main subject (subject). Also, the process described with one module as the main subject may be executed in cooperation by the module used as the main subject and other modules.

[0063] The statistical information creation module 511 creates statistical information based on the information acquired from the detection terminal 20, the analysis terminal 30, the user terminal 40, etc. The statistical information creation module 511 may further have a function of providing the created statistical information to each user, and a function of creating information regarding a practice menu or the like for each user based on the created statistical information.

[0064] The model generation module 512 creates and updates a machine learning model used in the analysis process by the swing analysis module 312 of the analysis terminal 30. In the configuration of FIG. 5, the model generation module 512 has an image extraction model generation module 515 and a coordinate calculation model generation module 516. Details of the machine learning model created by the model generation module 512 will be described later with reference to FIGS. 20, 21, etc.

[0065] The correction information creation module 513 creates correction information used in the analysis process by the swing analysis module 312 of the analysis terminal 30. The correction information is information regarding the installation environment of the analysis terminal 30, for example, the orientation and inclination of the installed analysis terminal 30, the change in brightness due to the brightness of the installation environment of the analysis terminal 30 and the flickering of the lighting, the shape and color of the objects and background existing in the installation environment of the analysis terminal 30, etc.

[0066] The display module 514 displays various information created by the statistical information creation module 511, the model generation module 512, and the correction information creation module 513 on an output device 505 such as a display.

[0067] The auxiliary storage device 502 of the management terminal 50 stores setting information 521, shooting information 522, user information 523, statistical information 524, swing information 528, etc. The processor 503 of the management terminal 50 executes various functions of the management terminal 50 using at least a part of those information and the above-mentioned modules 511 to 514, etc.

[0068] In the configuration of FIG. 5, the shooting information 522 is a moving image or a still image. In the auxiliary storage device 502, as the shooting information 522, target information 525, annotation information 526, shooting environment information 527, etc. are stored.

[0069] The target information 525 is information about the target captured in the shooting information 522. The annotation information 526 is information for identifying, specifying, classifying, etc. the target captured in the shooting information 522. The shooting environment information 527 is information regarding the state of the environment in which the shooting information 522 was shot.

[0070] The management terminal 50 stores, as the shooting information 522, a moving image or a still image for generating a machine learning model, such as an image extraction model or a coordinate calculation model, which will be described with reference to FIGS. 20 and 21.

[0071] For example, if those moving images or still images are moving images or still images shot at an illuminance level of about 500 lx to 1000 lx, it is possible to generate a machine learning model based on information shot in an environment similar to the play environment of the user 800 described later.

[0072] FIG. 6 is an example of an explanatory diagram for explaining the names and functions of the respective parts of the detection terminal 20. The names and functions of the respective parts of the detection terminal 20 are as described in FIG. 6. The relationship between each component described in FIG. 6 and each component described in FIG. 2 will be briefly explained below.

[0073] The "liquid crystal screen" is an example of the output device 205. The four "buttons" located below the liquid crystal screen are an example of the input device 204. The "detection sensor" is an example of the detection unit 206.

[0074] FIG. 7 is an example of an explanatory diagram for explaining a specification example of the detection terminal 20. The specifications of the detection terminal 20 described in FIG. 7 are an example of the detection terminal 20 and are not limited thereto. The relationship between each component described in FIG. 7 and each component described in FIG. 2 will be briefly described below.

[0075] The specifications and information regarding "Bluetooth" (registered trademark) are examples of information related to the operation of the cooperation module 213. The specifications and information regarding the "microwave sensor" are examples of information related to the detection unit 206, which is an example of a "sensor".

[0076] The specifications and information regarding the "head speed display range", "ball speed display range", "meat rate display range", and "average value calculation function" are examples of information related to the display on the display module 214.

[0077] FIG. 8 is a first example of a schematic diagram of the system 10. FIG. 8(A) is an example of a schematic diagram showing a specific example of the usage state of the system 10 by the user 800.

[0078] FIG. 8(B) is an example of a schematic diagram showing a first arrangement example in the height direction between the detection terminal 20 and the golf ball 820. FIG. 8(C) is an example of a schematic diagram showing a second arrangement example in the height direction between the detection terminal 20 and the golf ball 820.

[0079] The golf club 810 described in FIG. 8 is an example of a "club". The club head 811 described in FIG. 8 is an example of a "head". The golf ball 820 described in FIG. 8 is an example of a "ball".

[0080] In the schematic diagram of FIG. 8(A), the specific arrangements of the detection terminal 20 and the analysis terminal 30 are illustrated. The detection terminal 20 is arranged behind in the hitting direction D1 for hitting the golf ball 820.

[0081] Furthermore, the analysis terminal 30 has the lens of the photographing unit 306 arranged facing the golf club 810 or the golf ball 820. In the example of FIG. 8(A), the camera arranged on the side opposite to the main display of the analysis terminal 30, that is, the so-called rear camera, is directed toward the golf club 810 or the golf ball 820.

[0082] Thereby, the analysis terminal 30 placed in front of the user 800 who is a player, for example, a smartphone, is directed toward the ball. And the detection terminal 20 is arranged on the opposite side of the golf ball 820 across the position of the club, preferably with the lens of the camera positioned on the lower side (ground side). "Placed in front of the user 800" means, for example, a state in which the analysis terminal 30 is arranged at a position where the user 800 photographed by the photographing module 311 of the analysis terminal 30 overlaps at least partially with the center of the photographing screen, for example, the center point or the center line.

[0083] The detection terminal 20 and the analysis terminal 30 arranged as described above are associated (communicate) to realize the functions described later. The detection terminal 20 can use microwaves (μ waves) to detect and analyze the swing and send the detection results to the analysis terminal 30. The analysis terminal 30 also uses this to execute various processes.

[0084] The detection terminal 20 emits a measurement wave 850 toward the golf club 810 or the golf ball 820. The detection terminal 20 uses the measurement wave 850 to detect the start of the swing of the golf club 810, the swing speed of the golf club 810, the speed of the golf ball 820, and the like.

[0085] The detection terminal 20 photographs the golf club 810 and the golf ball 820 and outputs the measurement wave 850 based on the photographed imaging information. The detection terminal 20 uses the measurement wave 850 to detect the start of the swing of the golf club 810, the swing speed of the golf club 810, the speed of the golf ball 820, and the like.

[0086] The detection terminal 20 outputs a communication wave 860 for communication with the analysis terminal 30. The communication wave 860 may conform to the Bluetooth communication standard described with reference to FIG. 7, preferably the communication standard of Bluetooth Low Energy (registered trademark). The communication between the detection terminal 20 and the analysis terminal 30 will be described later with reference to FIG. 15 and the like.

[0087] Regarding the installation of the detection terminal 20 and the analysis terminal 30, for example, calibration and the degree of proper installation, it is advisable to describe them in the instruction manuals of the detection terminal 20 and the analysis terminal 30. The instruction manuals may be realized by a medium (e.g., paper medium) included with the detection terminal 20, but may also be electronic data in a format that can be displayed on an information display terminal such as the analysis terminal 30. The content described in the instruction manuals and the like includes, for example, brightness, the distance between the analysis terminal 30 and the golf ball 820, the orientation (left, right, up, down, corners) of the analysis terminal 30, the type of ball, the type of club, etc.

[0088] In order to address the situation where the analysis process cannot be smoothly executed due to the difference in images during learning, for example, the inability to smoothly recognize the ball head, it is advisable to also describe in the instruction manuals and the like the method of calibration and the degree of proper installation for each recognition accuracy according to the camera specifications and algorithms.

[0089] The system 10 may further include a fixture for appropriately guiding and fixing the positional relationship between the detection terminal 20 and the golf ball 820, the positional relationship between the analysis terminal 30 and the golf ball 820, the positional relationship between the detection terminal 20 and the analysis terminal 30, or the positional relationship among the detection terminal 20, the analysis terminal 30, and the golf ball 820. The fixture may be a member referred to as a bracket, mounting member, pedestal, or the like.

[0090] In a configuration where the system 10 has such a fixture, even if the recognition accuracy is greatly affected by the installation situation, the user can perform a swing analysis satisfactorily. Additionally or alternatively, the analysis terminal 30 may display a guide or the like in the preview.

[0091] FIG. 8(B) shows an arrangement state in which the arrangement height of the detection terminal 20 and the stationary height of the golf ball 820 are substantially aligned. FIG. 8(C) shows an arrangement state in which the arrangement height of the detection terminal 20 is lower than the stationary height of the golf ball 820.

[0092] When comparing the arrangement state of FIG. 8(B) and the arrangement state of FIG. 8(C), it is preferable that the arrangement state of FIG. 8(B) has less interference with the measurement wave 850 emitted from the detection terminal 20 by the mat 830 and the accuracy of swing detection is further improved.

[0093] FIG. 9 is a second example of a schematic diagram of the system 10. FIG. 9 shows another example of the specific arrangement method of the detection terminal 20 and the analysis terminal 30. FIGS. 9(A) and 9(B) are images of the system 10 in which the detection terminal 20 and the analysis terminal 30 are arranged, taken from different viewpoints. Regarding each component shown in FIG. 9, the description of FIG. 8 is incorporated by reference, and repeated descriptions are omitted.

[0094] FIG. 10 is an example of an explanatory diagram for explaining the arrangement adjustment of the analysis terminal 30. FIG. 10(A) is an example of a top view for explaining the arrangement adjustment of the analysis terminal 30. FIG. 10(B) is an example of a front view for explaining the arrangement adjustment of the analysis terminal 30. FIG. 10(C) is an example of a side view for explaining the arrangement adjustment of the analysis terminal 30.

[0095] As shown in FIG. 10(A), when the analysis terminal 30 is arranged within a range of lateral displacement of -5° to +5° from the state of capturing the stationary golf ball 820 in front, it is possible to obtain imaging information that can be analyzed with higher accuracy.

[0096] As shown in FIG. 10(B), the analysis terminal 30 may be arranged at a height position of 4 cm to 8 cm with respect to the floor surface 1000 on which the analysis terminal 30 is arranged, and may be arranged within a range of an inclination angle of -5° to +5° with respect to the floor surface 1000, so that imaging information that can be analyzed with higher accuracy can be obtained.

[0097] As shown in FIG. 10(C), when the analysis terminal 30 is arranged within a range of a depression angle of 0° to +5° with respect to the floor surface 1000 on which the analysis terminal 30 is arranged, imaging information that can be analyzed with higher accuracy can be obtained, and it is also easy to prevent the reflection of others other than the user 800 in the imaging information, so it is good. Also, the reason for arranging the analysis terminal 30 within the range of the depression angle is to prevent surreptitious photography and misidentification of surreptitious photography. That is, it is also based on the idea of arranging the analysis terminal 30 in a posture in which the imaging direction of the imaging unit 306 (camera) is a downward depression angle in order to prevent surreptitious photography and misidentification of surreptitious photography. Also, as described above, preventing surreptitious photography and misidentification of surreptitious photography, and measures against the reflection of surrounding people are important when the user uses the analysis terminal 30. Therefore, the analysis terminal 30 (for example, the display module 314) displays a consent screen when starting an application for swing analysis, such as the imaging module 3111 or the swing analysis module 312, or when the imaging unit 306 takes an image after the application is started. The consent screen may be a screen for confirming to the user that he / she agrees to prevent surreptitious photography and misidentification of surreptitious photography, and also to consider the reflection of surrounding people. The analysis terminal 30 may, for example, display a message indicating prevention of surreptitious photography and misidentification of surreptitious photography, and consideration of the reflection of surrounding people on the consent screen, and further display a soft button marked "Agree". The analysis terminal 30 may perform imaging for swing analysis on the condition that this soft button is operated by the user.

[0098] The analysis terminal 30 may have correction information created and updated by the management terminal 50 as, for example, setting information 321. The swing analysis module 312 can correct changes in calculated values that may occur according to the inclination of the arranged analysis terminal 30, such as changes in calculated values of various coordinates, trajectories, and the lowest point described later, using such correction information and the values of the inclination sensor of the analysis terminal 30. The inclination of the arranged analysis terminal 30 is an example of the "inclination of the device itself". Such correction is a process for accurately calculating various calculated values even if the analysis terminal 30 is installed in a tilted state from the assumed posture. In this way, for example, even if the user does not precisely adjust the posture of the analysis terminal 30, or even if the user deliberately tilts the analysis terminal 30 for preventing accidental shooting, the analysis terminal 30 can accurately calculate various calculated values.

[0099] In addition, the analysis terminal 30 uses the imaging unit 306 that functions as a so-called external camera to image the club head and the ball. Therefore, the screen of the analysis terminal 30 cannot be seen by the user who swings using the club. Therefore, the analysis terminal 30 preferably has a function of informing the user that the terminal itself is correctly installed and correctly measured. For example, the analysis terminal 30 (processor 303) causes a screen for informing the user that the terminal itself is correctly installed and correctly measured to be displayed on the display of the output device 305. A mirror having a mirror surface on the display side may be installed in front of this display. In this way, the user can see the display of the display reflected on the mirror surface by looking at the mirror surface of the mirror. The mirror is preferably configured integrally with the above-described fixture. In particular, when the fixture is installed in front of the user with the analysis terminal 30 fixed to the fixture, the position and orientation of the mirror are preferably set so that the mirror surface can be seen from the user. It is particularly preferable that the user can adjust the position and posture of the mirror with respect to the fixture.

[0100] The swing analysis module 312 can perform the above-described correction using a table or function that defines the values of the inclination sensor and the corresponding correction values. In another configuration, the swing analysis module 312 can also perform the above-described correction by switching or combining a plurality of machine learning models according to the situation using, for example, a machine learning model trained as follows.

[0101] Furthermore, the swing analysis module 312 may perform the above-described correction by switching or combining the method using the above-described table or function and the method using the above-described machine learning model.

[0102] The swing analysis module 312 may correct the change in the calculated value that may occur according to the inclination of the arranged analysis terminal 30 by inputting the value of the inclination sensor into a learning model trained using videos or images taken with different arrangement adjustments with respect to, for example, the lateral deviation described in FIG. 10(A), the inclination angle described in FIG. 10(B), and the depression angle described in FIG. 10(C).

[0103] FIG. 11 is an example of an explanatory diagram for explaining the conditions for the lowest point analysis by the analysis terminal 30 and the like. The details of the conditions for the lowest point analysis by the analysis terminal 30 and the like are as described in FIG. 11.

[0104] FIG. 12 is an example of the display screen 1200 of the detection terminal 20. The display screen 1200 corresponds to the output device 205 described in FIG. 2 and the liquid crystal screen described in FIG. 6.

[0105] FIG. 12(A) is an example of the first display of the display screen 1200 during swinging. FIG. 12(B) is an example of the second display of the display screen 1200 during swinging. FIG. 12(C) is an example of the display of the display screen 1200 during putting. FIGS. 12(D) to 12(F) are examples of the display of the display screen 1200 during approach. The details of each parameter shown in FIG. 12 will be described later with reference to FIG. 13.

[0106] The display module 214 can alternately switch between, for example, the first display of the display screen 1200 during the swing in FIG. 12(A) and the second display of the display screen 1200 during the swing in FIG. 12(B), and display them on the display screen 1200.

[0107] The second display of the display screen 1200 during the swing in FIG. 12(B) is an example of "display based on swing analysis information". Similarly, the display obtained by switching between the first display of the display screen 1200 during the swing in FIG. 12(A) and the second display of the display screen 1200 during the swing in FIG. 12(B) is also an example of "display based on swing analysis information".

[0108] In the switching display as described above, even when the detection terminal 20 has a configuration where, for example, the output device 205 is a segment display or the like and the display range and the number of display items on its display screen 1200 are limited, various information can be provided to the user 800. For example, the detection terminal 20 can also provide information that cannot be provided by the detection terminal 20 alone to the user 800.

[0109] FIG. 13 is an example of an explanatory diagram for explaining the content of the information displayed on the display screen 1200 of the detection terminal 20. Regarding the display method of each parameter shown in FIG. 13, refer to each display screen 1200 of the detection terminal 20 shown in FIG. 12. Details of the information and the like displayed on the display screen 1200 of the detection terminal 20 are as described in FIG. 13.

[0110] FIG. 14 is a first example of an explanatory diagram for explaining the communication connection state between the detection terminal 20 and the analysis terminal 30. FIG. 14 explains the communication connection state conforming to the Bluetooth standard between the detection terminal 20 and the analysis terminal 30. In FIG. 14, the detection terminal 20 is a peripheral device and the analysis terminal 30 is a central device. However, the communication connection of the system 10 is not limited thereto.

[0111] The cooperation module 213 of the detection terminal 20 advertises to the analysis terminal 30 at time point T1. The cooperation module 313 of the analysis terminal 30 sends a connection request to the detection terminal 20 at time point T2.

[0112] After the connection between the detection terminal 20 and the analysis terminal 30 is successful and the connection is established, the detection terminal 20 and the analysis terminal 30 perform initial settings. The cooperation module 213 of the detection terminal 20 sends information regarding the swing timing to the analysis terminal 30 at time point T3. The information regarding the swing timing is an example of swing detection information.

[0113] The cooperation module 213 of the detection terminal 20 sends information regarding the measurement data to the analysis terminal 30 at time point T4. The measurement data is data regarding, for example, the head speed of the club head 811 or the ball speed of the golf ball 820. The measurement data may be, for example, other data detected by the detection unit 206. The measurement data may be referred to as detection data or detection information. The cooperation module 313 of the analysis terminal 30 sends information regarding the swing lowest point distance difference to the detection terminal 20 at time point T5. The information regarding the swing lowest point distance difference is an example of swing analysis information.

[0114] The cooperation module 313 of the analysis terminal 30 makes a history count request to the detection terminal 20 at time point T6. The cooperation module 213 of the detection terminal 20 sends information regarding the history count to the analysis terminal 30 at time point T7.

[0115] The cooperation module 313 of the analysis terminal 30 makes a history data request to the detection terminal 20 at time point T8. The cooperation module 213 of the detection terminal 20 sends information regarding the history data to the analysis terminal 30 at time point T9.

[0116] FIG. 15 is a second example of an explanatory diagram for explaining the communication connection state between the detection terminal 20 and the analysis terminal 30. The processing of the analysis terminal 30 shown in FIG. 15 may also be referred to as processing when the analysis terminal 30 operates in the swing analysis mode, or swing analysis processing. Note that, in the figure of FIG. 15, each step 1510d to 1560d of the detection terminal 20 and each step 1510a to 1540a of the analysis terminal 30, which are described in parallel, for example, step 1510d of the detection terminal 20 and step 1510a of the analysis terminal 30, do not necessarily need to be performed simultaneously in terms of time.

[0117] When the user sets up the system 10 and turns on the power button of the detection terminal 20 as described with reference to FIGS. 8 to 10, the detection terminal 20 enters the standby state for swing detection (S1510d). Also, when the user turns on the analysis function of the analysis terminal 30, which is, for example, a smartphone, etc., the analysis terminal 30 starts shooting for creating shooting information for analysis (S1510a).

[0118] Note that, in the example of FIG. 15, the detection terminal 20 and the analysis terminal 30 are already in a communication-connected state at the stage of step S1510d and step S1510a. For the establishment of the communication connection state, refer to the description of FIG. 14.

[0119] Hereinafter, steps S1520d to S1560d by the detection terminal 20 shown in FIG. 15 will be described. The swing detection module 211 of the detection terminal 20 in the standby state for swing detection detects the swing of the golf club 810 by the user 800 by, for example, the method described later with reference to FIG. 16 (S1520d).

[0120] When the swing detection module 211 detects a swing, the swing information creation module 212 creates swing detection information (S1530d). The swing detection information is information or a signal indicating that a swing is being performed or has been performed.

[0121] The cooperation module 213 of the detection terminal 20 transmits swing detection information to the analysis terminal 30 at time point T10. For example, the time point T10 when transmitting the swing detection information may be a time point delayed by about 0.5 seconds, for example, from the time point when the swing is detected or the time point when the swing detection information is created.

[0122] This delay is caused by, for example, the computing power of the detection terminal 20, the time required for communication cooperation between the cooperation module 213 of the detection terminal 20 and the cooperation module 313 of the analysis terminal 30, etc.

[0123] The above-mentioned delay and lag are caused by, for example, the computing power of the detection terminal 20, the cooperation state between the cooperation module 213 of the detection terminal 20 and the cooperation module 313 of the analysis terminal 30, etc.

[0124] The swing detection module 211 measures and detects various swing parameter information related to the detected swing, and the swing information creation module 212 calculates various swing parameter information based on the information measured and detected by the swing detection module 211 (S1540d).

[0125] The swing parameter information is various information displayed on the display screen 1200 described using FIG. 12, and specifically, head speed, ball speed, flight distance, meet rate, etc.

[0126] The detection terminal 20 may perform the measurement or calculation of the swing parameter information (S1540d) simultaneously with or in parallel with the detection of the swing (S1520d). The detection terminal 20 may perform the measurement or calculation of the swing parameter information (S1540d) before or in parallel with the creation of the swing detection information (S1530d).

[0127] The display module 314 of the detection terminal 20 displays the swing parameter information on the display screen 1200 as described using FIG. 12 (S1550d). Furthermore, when the display module 314 of the detection terminal 20 acquires the swing analysis information from the analysis terminal 30 at time point T13, as described with reference to FIG. 12, on the display screen 1200, the swing parameter information (FIG. 12(A)) and the swing analysis information (FIG. 12(B)) are switched and displayed, for example, periodically or in response to an operation by the user (S1560d).

[0128] Hereinafter, steps S1520a to S1540a performed by the analysis terminal 30 shown in FIG. 15 will be described. The cooperation module 313 of the analysis terminal 30 receives the swing detection information transmitted from the detection terminal 20 at time point T10 at time point T11.

[0129] Time point T10 is, for example, the time point when the cooperation module 213 of the detection terminal 20 starts the transmission process of the swing detection information. Time point T11 is, for example, the time point when the cooperation module 313 of the analysis terminal 30 completes the reception process of the swing detection information.

[0130] Therefore, the time point T11 when the cooperation module 313 of the analysis terminal 30 receives the swing detection information may be a time point that is delayed in time from time point T10 due to, for example, the computing capabilities of the detection terminal 20 and the analysis terminal 30, the time required for communication cooperation, etc.

[0131] When the swing analysis module 312 of the analysis terminal 30 acquires the swing detection information at time point T11, it starts the swing analysis based on the shooting information shot by the shooting module 311 (S1520a).

[0132] When the swing analysis module 312 of the analysis terminal 30 acquires the swing detection information at time point T11, the shooting module 311 may stop shooting the shooting information. The shooting module 311 that has stopped shooting may resume shooting the shooting information, for example, when the swing analysis process ends or when new swing detection information is acquired from the detection terminal 20.

[0133] In this way, the analysis terminal 30 can easily capture the imaging information used for analyzing the swing by the user 800 while saving the computing power allocated to the imaging module 311, the storage area for the imaging information, and the like.

[0134] Note that the imaging module 311 may set the timing for stopping the capture of imaging information, for example, after a predetermined time has elapsed from the time point T11 when the swing detection information is acquired. Even with such a configuration, when the imaging module 311 that has stopped imaging finishes the swing analysis process or acquires new swing detection information from the detection terminal 20, it may resume the capture of imaging information as described above.

[0135] This predetermined time may be set as a time sufficient for capturing the swing from the acquisition of the swing detection information to the end of the swing. For example, the predetermined time is a value set based on the time set by the user, the average value of the set swings, and the like.

[0136] When the cooperation module 313 of the analysis terminal 30 acquires the swing detection information at the time point T11, the swing analysis module 312 executes swing analysis using the imaging information captured by the imaging module 311 (S1530a). Details of the swing analysis will be described with reference to FIGS. 19 to 25.

[0137] With such a configuration, the swing analysis module 312 can use the imaging information captured before the time point T11 when the swing detection information is acquired for swing analysis.

[0138] Additionally or alternatively, the swing analysis module 312 can also use the imaging information captured before the time point T11 when the swing detection information is acquired and the imaging information captured after the time point T11 when the swing detection information is acquired for swing analysis.

[0139] The imaging information captured before the time point T11 when the swing detection information was acquired is an example of "imaging information captured before the acquisition of swing detection information". The imaging information captured after the time point T11 when the swing detection information was acquired is an example of "imaging information captured after the acquisition of swing detection information".

[0140] For example, the swing detected by the detection terminal 20 may already have ended at the time point T11 when the analysis terminal 30 acquired the swing detection information. The analysis terminal 30 can perform an analysis of the swing detected by the detection terminal 20 using the imaging information captured before the time point T11 when the swing detection information was acquired.

[0141] Thereby, for example, at the time point T11, even if the golf club 810, the club head 811, the golf ball 820, etc. have gone out of the imaging range by the imaging module 311, the analysis terminal 30 can perform an analysis of the swing detected by the detection terminal 20.

[0142] The swing analysis module 312 outputs the result of the swing analysis as swing analysis information and ends the swing analysis process. The cooperation module 313 of the analysis terminal 30 transmits the swing analysis information to the detection terminal 20 at the time point T12.

[0143] The swing analysis module 312 stores the swing analysis information in the auxiliary storage device 302, for example, as history information 322 (S1540a). Note that the swing analysis module 312 may store the swing analysis information (S1540a) earlier in time than the time point T12 or approximately simultaneously with the time point T12.

[0144] FIG. 16 is a first example of an explanatory diagram for explaining the lowest point analysis by the analysis terminal 30. FIG. 16 is an example in which the imaging information of the swing captured by the imaging module 311 of the analysis terminal 30 is displayed as a series of photographs.

[0145] The swing detection module 211 of the detection terminal 20 detects the swing speed of the swing performed within the detection range 1600 of the detection terminal 20 by the user 800, and the hitting speed of the golf ball 820 hit thereby. The detection range 1600 of the detection terminal 20 schematically shows the irradiation range of the microwave (μ-wave) irradiated by the detection terminal 20.

[0146] The swing detection module 211 of the detection terminal 20 further detects the golf club 810 and the club head 811 being swung within the downswing detection range 1610. The swing detection module 211 detects the club head 811 passing through the downswing detection range 1610 based on Doppler information (hereinafter also referred to as a Doppler signal) representing, for example, the Doppler shift or the Doppler effect.

[0147] In a configuration where the swing detection module 211 detects a swing based on a Doppler signal, when the swing detection module 211 detects a predetermined wave specific to the downswing of the swing, for example, a wave with a Doppler signal period of less than 620 μs, that is, a speed of 10 m / s or more, for a predetermined number, for example, 32 waves in a row, it determines that it is the swing timing of the downswing of the swing.

[0148] As described above, when the swing detection module 211 detects that the golf club 810 or the club head 811 is being swung, or detects that the golf club 810 or the club head 811 has been swung, the cooperation module 213 transmits a swing detection signal to the analysis terminal 30 via the network 100 within a predetermined time, for example, within 0.1 seconds. The swing detection signal is an example of swing detection information.

[0149] When the swing analysis module 312 of the analysis terminal 30 receives a swing detection signal from the detection terminal 20, it performs swing analysis based on the shooting information captured by the shooting module 311.

[0150] The detection terminal 20 may notify, for example, the swing-down of the golf club 810 to the analysis terminal 30. The analysis terminal 30 may perform image analysis for a period according to the swing-down timing to recognize the golf ball 820 and the golf club 810 and perform swing analysis.

[0151] Based on the swing analysis, the analysis module 312 analyzes, for example, the distance between the lowest point of the swing and the stationary golf ball 820 as described with reference to FIGS. 17 and 18.

[0152] FIG. 17 is a second example of an explanatory diagram for explaining the lowest point analysis by the analysis terminal 30. In FIG. 17, a display example is shown in which the shooting information taken by the shooting module 311 of the analysis terminal 30 is displayed on an output device 305 such as a display.

[0153] FIGS. 17(A) to 17(D) are extracted from a video capturing a series of swings detected by the swing detection module 211 of the detection terminal 20. FIG. 17(A) is an example of an image capturing the stationary golf ball 820 before being hit.

[0154] FIGS. 17(B) and 17(C) are examples of images capturing the stationary golf ball 820 before being hit and the club head 811 during the detected swing. FIG. 17(D) is an example of an image capturing the hit golf ball 820 and the club head 811 during the detected swing.

[0155] FIG. 17(E) is a display example of information calculated by the swing analysis described later. In FIG. 17(E), coordinate information, trajectory information, distance information, etc. related to FIGS. 17(A) to 17(D) are displayed.

[0156] In FIGS. 17(A) to 17(D), the analysis module 312 demarcates the golf ball 820 and the club head 811 in the image using the first identification frame 1700 and the second identification frame 1710, respectively.

[0157] Regarding the first identification frame 1700 and the second identification frame 1710, the "identification frame" may be paraphrased as the so-called "bounding box". The still images and moving images demarcated using the first identification frame 1700 and the second identification frame 1710 are examples of "still images based on swing analysis information" and "moving images based on swing analysis information".

[0158] "Demarcation" may be read as "identification", "recognition", "classification", "categorization", "segmentation", "annotation", etc. The specific method of demarcation will be described with reference to FIGS. 20 to 22 and the like.

[0159] Based on the swing analysis by the analysis module 312, the display module 314 displays a plurality of coordinate information 1720 related to the club head 811 identified in the shooting information in FIG. 17(E). The first head coordinate 1720a is the coordinate related to the club head 811 identified in FIG. 17(B).

[0160] The second head coordinate 1720b is the coordinate related to the club head 811 identified in FIG. 17(C). The third head coordinate 1720c is the coordinate related to the club head 811 identified in FIG. 17(D). The first head coordinate 1720a, the second head coordinate 1720b, and the third head coordinate 1720c are examples of "head coordinates that are the coordinates of the head".

[0161] The display module 314 displays, in FIG. 17(E), based on the swing analysis by the analysis module 312, further, the distance information 1740 representing the distance between the trajectory 1730 of the club head 811 and the golf ball 820 from its lowest point, which is "-5 cm". In this example, when the lowest point is located on the front side of the golf ball 820 with respect to the swing direction (i.e., before impact), the display module 314 displays the distance using the "-" (minus) sign, and when the lowest point is located on the back side of the golf ball 820 (i.e., after impact), it displays the distance using the "+" (plus) sign.

[0162] While the display module 314 displays the distance based on the distance information 1740 using numbers, it is advisable not to display the evaluation of the quality of the swing based on the distance information 1740. Thus, the analysis terminal 30 may preferably be configured not to provide a function of displaying the evaluation based on the distance information 1740. This is because in golf, there is also the aspect that where the best lowest point is depends on a person (e.g., style).

[0163] The trajectory 1730 is an example of "a graph based on swing analysis information" or "the trajectory of the head". The distance information 1740 is an example of "a numerical value based on swing analysis information" or "distance information regarding the distance between the lowest point coordinates and the ball coordinates". These pieces of information will be described more specifically with reference to FIG. 18.

[0164] FIG. 18 is a third example of an explanatory diagram for explaining the lowest point analysis by the analysis terminal 30. FIG. 18(A) is an explanatory diagram for explaining FIG. 17(E) more specifically. FIG. 18(B) is an explanatory diagram for explaining the distance between the lowest point and the golf ball 820.

[0165] FIG. 18(A) is obtained by adding the horizontal coordinate axis x and the vertical coordinate axis y to FIG. 17(E). The lowest point 1800 of the trajectory 1730 calculated by the analysis module 312 has coordinates (x1, y1) as the coordinates of the lowest point 1800.

[0166] Although not shown in FIG. 18(A), the stationary golf ball before hitting has golf ball coordinates (x2, y2) as the coordinates of the golf ball. The golf ball coordinates (x2, y2) are illustrated in FIG. 18(B).

[0167] FIG. 18(B) illustrates the analysis length, which is the length between the lowest point 1800 of the calculated trajectory 1730 and the golf ball 820. The analysis length may be defined, for example, as the length between x1, which is the x coordinate of the coordinates (x1, y1) of the lowest point 1800, and x2, which is the x coordinate of the golf ball coordinates (x2, y2) of the golf ball 820 in the stationary state before hitting.

[0168] The coordinates (x1, y1) of the lowest point 1800 are an example of "the lowest point coordinates that are the coordinates of the lowest point of the trajectory". The golf ball coordinates (x2, y2) are an example of "the ball coordinates that are the coordinates of the ball".

[0169] The analysis module 312 calculates the distance information 1740 as x2 - x1. For example, in FIGS. 17(E) and 18(A), a "- (minus)" is attached to the analysis length. This means that the lowest point coordinates (x1, y1) of the lowest point 1800 are located behind (the left side of the drawing) the golf ball coordinates (x2, y2) with respect to the hitting direction D1.

[0170] Information about where the lowest point of the swing head is located relative to the golf ball is very important for the user 800. Based on the distance information 1740 for each swing, the user 800 can recognize the state of their own swing and practice much more efficiently than before.

[0171] FIG. 19 is an example of a swing analysis flow 1900. In addition, each of the steps S1910 to S1950 described below may be executed in any order or in parallel, as long as there is no technical conflict.

[0172] When the analysis terminal 30 receives a swing detection signal from the detection terminal 20, it starts the swing analysis flow 1900. The swing analysis module 312 of the detection terminal 20 acquires a video or an image as shooting information captured by the shooting module 311 (S1910).

[0173] Based on the acquired shooting information, the swing analysis module 312 extracts an image using an image extraction model, which is a machine learning model (S1920). The extraction of the image will be described in more detail with reference to FIGS. 20 and 22.

[0174] Note that "image extraction" may be referred to as "frame extraction". A "frame" is, for example, 240 individual frames (images) captured in one second in a video with a frame rate of 240 fps.

[0175] In the image extraction step (S1920), an image may be extracted together with information regarding which image (frame) in the target video file it is. The resolution of the extracted image may be, for example, about 1920x1080.

[0176] Based on the extracted image, the swing analysis module 312 calculates the coordinates of objects in the extracted image, such as the club head 811 and the golf ball 820, using a coordinate calculation model, which is a machine learning model (S1930).

[0177] The extracted image (hereinafter also referred to as the extracted frame, frame image, or frame) is an example of "shooting information" or "shooting information for analysis". The calculation of the coordinates will be described in more detail with reference to FIGS. 21 and 24.

[0178] Based on the calculated coordinates, the swing analysis module 312 calculates the distance between the ball coordinates and the lowest point of the head (S1940). The calculation of the distance to the lowest point will be described in more detail with reference to FIG. 25.

[0179] The cooperation module 313 outputs information regarding the calculated distance between the ball coordinates and the lowest point of the head to the detection terminal as swing analysis information (S1950), and ends the swing analysis flow 1900.

[0180] The above-described analysis length, the lowest point coordinates (x1, y1) of the lowest point 1800, the golf ball coordinates (x2, y2), and the trajectory 1730 are examples of "swing analysis information". The analysis terminal 30 transmits all or part of this information to the detection terminal 20.

[0181] FIG. 20 is an example of an image extraction model generation flow 2000. The image extraction model is generated, for example, by the image extraction model generation module 515 of the model generation module 512 of the management terminal 50.

[0182] The image extraction model generated by the image extraction model generation flow 2000 described in FIG. 20 can be used in the image extraction step (S1920) using the image extraction model described with reference to FIG. 19.

[0183] In the auxiliary storage device 502 of the management terminal 50, videos of the swing being photographed, images of the club head and the golf ball being photographed, etc. are stored as photographing information. These videos and images are photographed under different photographing conditions.

[0184] The photographing conditions can be specified based on parameters such as the distance to a stationary golf ball, the angle of view, the background color, the objects present in the background, the brightness and color of the ambient light, the frame rate, and the installation state of the photographing device such as the tilt angle and the depression angle, etc.

[0185] The image extraction model generation module 515 acquires, for example, the above-described plurality of shooting information from the auxiliary storage device 502 (S2010), and adds annotation information to the club head or golf ball captured in each piece of shooting information (S2020).

[0186] The image extraction model generation module 515 may be configured to add annotation information (S2020) based on an input operation of a developer or the like who operates the management terminal 50 using an input device 504 such as a mouse or a keyboard for each moving image or image displayed on an output device 505 such as a display.

[0187] The image extraction model generation module 515 may be configured to add annotation information (S2020) based on, for example, the target information 525 and annotation information 526 stored in the auxiliary storage device 502.

[0188] The image extraction model generation module 515 learns the moving image or image to which the annotation information is added (S2030), generates or updates an image extraction model that extracts an image of the club head or golf ball captured from the moving image of the swing being shot (S2040), outputs the generated or updated image extraction model (S2050), and ends the image extraction model generation flow 2000.

[0189] FIG. 21 is an example of a coordinate calculation model generation flow 2100. The coordinate calculation model is generated, for example, by the coordinate calculation model generation module 516 of the model generation module 512 of the management terminal 50.

[0190] The coordinate calculation model generated by the coordinate calculation model generation flow 2100 described in FIG. 21 can be used in the coordinate calculation step (S1930) using the coordinate calculation model described with reference to FIG. 19.

[0191] In the auxiliary storage device 502 of the management terminal 50, shooting information 522 such as videos of swings, images of club heads and golf balls taken, etc. is stored. These videos and images are taken under different shooting conditions.

[0192] The shooting conditions can be specified based on parameters such as the distance to a stationary golf ball, the angle of view, the background color, the objects present in the background, the brightness and color of the ambient light, the frame rate, the installation state of the shooting device such as the tilt angle and the depression angle, etc.

[0193] In the auxiliary storage device 502 of the management terminal 50, furthermore, as target information 525, the types and dimensions of golf clubs and golf balls captured in the shooting information, coordinates based on an arbitrary point, for example, the lower right vertex of the shooting information, such as the coordinates of a stationary golf ball, the coordinates of the club head during a swing, etc. may be stored.

[0194] The coordinate calculation model generation module 516 acquires shooting information with annotation information added from, for example, the auxiliary storage device 502 (S2110). The shooting information with annotation information added may, for example, utilize the shooting information created in the step of adding annotation information for the head and ball in the image extraction model generation flow 2000 described with reference to FIG. 20 (S2020).

[0195] The coordinate calculation model generation module 516 acquires coordinate information based on annotation information, for example, coordinate information based on an identification frame or a bounding box surrounding the club head or the golf ball (S2120).

[0196] Note that the coordinate information acquired by the coordinate calculation model generation module 516 in the coordinate information acquisition step (S2120) may be defined based on feature points such as the center point of the identification frame or the bounding box, the upper vertex, the midpoint of the upper side, etc.

[0197] By defining the coordinates as described above, for example, even when the club head or golf ball is in a state of contacting or being buried in the floor surface, ground, or lawn, it becomes easier to define the coordinates, particularly the coordinates that specify the horizontal positional relationship of the club head or golf ball.

[0198] The coordinate calculation model generation module 516 learns the video or image with annotation information and the coordinate information related thereto (S2130), generates or updates a coordinate calculation model for calculating the coordinates of the photographed club head or golf ball (S2140), outputs the generated or updated coordinate calculation model (S2150), and ends the coordinate calculation model generation flow 2100.

[0199] FIG. 22 is an example of an image extraction flow 2200 using an image extraction model. The image extraction flow 2200 using an image extraction model may also be referred to as a "ball search phase". The image extraction flow 2200 and the ball search phase are examples of "processing for detecting a ball".

[0200] In the ball search phase, the image extraction module 315 searches for the timing at which the ball is hit by the swing, for example, the timing at which the ball disappears from the photographed information such as the extracted image, with respect to the photographed swing in the photographed information.

[0201] Hereinafter, an example in which the image extraction module 315 of the swing analysis module 312 executes the image extraction flow 2200 using an image extraction model as the frame extraction step (S1920) using a frame extraction model will be described.

[0202] The image extraction module 315 acquires a video or image as photographed information photographed by the photographing module 311 (S2210), and inputs the acquired photographed information into the image extraction model (S2220). Note that the image extraction model is, for example, a learning model generated by the image extraction model generation flow 2000 described with reference to FIG. 20.

[0203] The swing analysis module 312 can create an integrated video file by arranging and integrating a plurality of video files in time series. Note that the integrated video file is an example of "shooting information".

[0204] The swing analysis module 312 provides, as shooting information, the video files shot by the shooting module 311 and the integrated video file created by integrating those video files to the image extraction module 315.

[0205] The image extraction module 315 inputs, as shooting information, a video of a predetermined time, for example, a plurality of video files of about 0.1 to 0.5 seconds, or an integrated video file obtained by integrating those videos, to the image extraction model (S2220).

[0206] For example, the image extraction module 315 inputs a plurality of video files so that the total time length is about 0.5 to 1.0 seconds or more, or inputs an integrated video file with a time length of about 0.5 to 1.0 seconds or more (S2220).

[0207] Hereinafter, a configuration in which the swing analysis module 312 executes swing analysis processing using, as an example, three video files of 0.25 seconds will be described. Note that in such a configuration, the shooting module 311 may create one video file every 0.25 seconds as shooting information and save at least the latest three video files.

[0208] Note that the shooting module 311 may perform shooting settings, for example, frame rate, time length setting and adjustment of one video file, based on the information stored in the user information 323, for example, the swing speed of the user. Thereby, the swing analysis module 312 can perform analysis processing using shooting information suitable for each individual user.

[0209] The image extraction module 315 integrates three 0.25-second video files to create one integrated video file of 0.75 seconds. Hereinafter, an example of processing using the "integrated video file" will be described. The "video file" and the "integrated video file" can be files in any format.

[0210] Preferably, the "video file" and the "integrated video file" are files in a format that can be played by any video playback application, for example, files in formats such as MP4 (.mp4), M4V (.m4v), MOV (.mov), AVI (.avi), WMV (.wmv), MPEG2 (.mpg), MKV (.mkv), FLV (.flv), ASF (.asf), etc.

[0211] The image extraction module 315 executes the swing extraction process by the image extraction model (S2230), and extracts, based on the shooting information, an image that captures the club head 811 during the swing and the stationary golf ball 820 to be hit from, for example, the integrated video file created as described above.

[0212] The image extraction module 315 may directly extract the image that captures the club head during the swing and the golf ball to be hit as the frame image of the integrated video file, or may extract it in a state where the golf ball 820 and the club head 811 are annotated (identified, classified) by a predetermined method.

[0213] The state of "annotating (identifying, classifying) the golf ball 820 and the club head 811 by a predetermined method" means, for example, a state where a frame line of a geometric shape, such as a rectangle, is assigned to the golf ball 820 and the club head 811 in the image, or a state where an edge line is assigned to the golf ball 820 and the club head 811 in the image as a boundary.

[0214] The image extraction module 315 performs annotation processing on the extracted image as necessary, and outputs it as an image in which the golf ball 820 or the club head 811 is annotated (S2240), and ends the image extraction flow 2200 using the image extraction model.

[0215] Note that the annotated image is, for example, an image in which the club head or the hit golf ball during the swing is bounded, identified, recognized, classified, class-divided, segmented, etc. by, for example, a rectangular frame or a bounding box.

[0216] As described above, in the ball search phase, the image extraction module 315 uses an image extraction model, which is a machine learning model, to extract a "frame capturing the swing". For this purpose, the image extraction module 315 first obtains a "frame image" of the club head and the golf ball from the shooting information captured by the shooting module 311 of the analysis terminal 30.

[0217] After that, the image extraction module 315 takes in the obtained frame image into an image extraction model, which is a machine learning model, and extracts a "frame capturing the swing". The image extraction module 315 uses the image extraction model to execute a process of searching for the timing when the ball disappears from a plurality of consecutive "frames capturing the swing" extracted.

[0218] In order to determine the timing when the ball disappears, the image extraction module 315 can analyze the frame image of the integrated video file one by one, for example, from the first or the last frame of the integrated video file.

[0219] Alternatively or selectively, the image extraction module 315 can efficiently reduce the computational load by analyzing the timing when the ball disappears according to the method described below.

[0220] The image extraction module 315 may determine the presence or absence of the golf ball 820 from a predetermined frame of the integrated video file, for example, a frame 80 to 120 frames before the last frame of the integrated video file in terms of time.

[0221] The image extraction module 315 determines, for example, from a frame 96 frames before the last frame of the integrated video file in terms of time (hereinafter referred to as the "first search frame"), the presence or absence of the golf ball 820 within the first search frame.

[0222] In the following description, both the state where "the golf ball 820 is photographed within the frame" and the state where "the golf ball 820 photographed in the frame can be discriminated" are described as "the golf ball 820 is photographed within the frame".

[0223] Similarly, in the following description, both the state where "the golf ball 820 is not photographed within the frame" and the state where "the golf ball 820 photographed in the frame cannot be discriminated" are described as "the golf ball 820 is not photographed within the frame".

[0224] In the first example of ball search, when the image extraction module 315 determines that the golf ball 820 is photographed in the first search frame, the image extraction module 315 selects frames behind in terms of time at a predetermined frame interval from the first search frame, and repeats the process of determining the presence or absence of the golf ball 820 within the frame. The predetermined frame interval is, for example, an interval every 12 frames.

[0225] For example, the image extraction module 315 performs the process of determining the presence or absence of the golf ball 820 within the frame in order of 96 frames before, 84 frames before, 72 frames before, ··· from the last frame of the integrated video file in terms of time.

[0226] In the following description, an example will be described where the integrated video file is a video file with a length of 0.75 seconds at 240 fps. Such an integrated video file has a total of 180 frames, and the last frame is the 180th frame from the beginning of the integrated video file in terms of time.

[0227] Similarly, the frame 96 frames before the last frame (the first search frame) is the 84th frame from the beginning of the integrated video file in terms of time, the frame 84 frames before the last frame is the 96th frame from the beginning of the integrated video file, and the frame 72 frames before the last frame is the 108th frame from the beginning of the integrated video file.

[0228] When the image extraction module 315 determines that the golf ball 820 has disappeared in the frame 72 frames before the last frame of the integrated video file (the 108th frame from the beginning of the integrated video file), the image extraction module 315 determines the presence or absence of the golf ball 820 in the frames in order of 73 frames before (the 107th frame from the beginning of the integrated video file), 74 frames before (the 106th frame from the beginning of the integrated video file), 75 frames before (the 105th frame from the beginning of the integrated video file),... in terms of time from the last frame of the integrated video file.

[0229] When the image extraction module 315 determines that the stationary golf ball 820 exists in the frame 75 frames before (the 105th frame from the beginning of the integrated video file), for example, the frame 75 frames before the last frame of the integrated video file (the 105th frame from the beginning of the integrated video file) is determined to be "the frame in which the stationary golf ball 820 before the impact by the club head 811 is photographed (hereinafter also referred to as the 'frame just before impact')".

[0230] Similarly, the image extraction module 315 determines that the frame 74 frames before the last frame of the integrated video file in terms of time (the 106th frame from the beginning of the integrated video file) is a "frame in which the stationary golf ball 820 is not photographed (hereinafter also referred to as the 'frame immediately after impact')".

[0231] And it can be determined that the time point of the impact between the club head 811 and the golf ball 820 (hereinafter referred to as the 'impact time point') is between the frame immediately before impact (in this example, the 105th frame from the beginning of the integrated video file) and the frame immediately after impact (in this example, the 106th frame from the beginning of the integrated video file).

[0232] In the configuration where the image extraction module 315 performs ball search as described above, the impact time point can be estimated more efficiently compared to the process of analyzing frame images one by one from the beginning.

[0233] Hereinafter, a second example of ball search will be described with reference to FIG. 23. FIG. 23 is an example of an explanatory diagram for explaining the computational load in the swing analysis process. In the second example of ball search, the image extraction module 315 uses the binary search method to determine the timing when the ball disappears.

[0234] The computational load in the search process for the image extraction module 315 to determine the presence or absence of the golf ball 820 from the frames of the integrated video file can be significantly reduced compared to the linear search process of analyzing one by one.

[0235] For example, as will be described later, first search for two frames before and after the golf ball 820 changes from "present" to "absent" within the frame of the integrated video file. If the search algorithm determines the frame number for detecting the presence or absence of the ball using, for example, binary search instead of looking at each frame one by one from the beginning, the computational complexity becomes of the order of log(n) (dashed line), and the computational complexity can be significantly reduced compared to the case of searching all frames. This example will be described in more detail below.

[0236] Similar to the above description, an example will be described where the integrated video file is a video file with a length of 0.75 seconds at 240 fps. Also, here, similar to the first example of ball search, an example will be described for determining the presence or absence of the golf ball 820 starting from the first search frame, which is the frame 96 frames before the last frame of the integrated video file in terms of time.

[0237] When the image extraction module 315 determines that the golf ball 820 is captured in the first search frame, the image extraction module 315 analyzes the frame 48 frames after the first search frame in terms of time (hereinafter referred to as the "second search frame") and determines the presence or absence of the golf ball 820 in the second search frame.

[0238] Conversely, when the image extraction module 315 determines that the golf ball 820 is not captured in the first search frame, the image extraction module 315 analyzes the frame 48 frames before the first search frame in terms of time (hereinafter referred to as the "third search frame") and determines the presence or absence of the golf ball 820 in the third search frame.

[0239] As described above, the first search frame is the frame 96 frames before the last frame of the integrated video file, and is the 84th frame from the beginning of the integrated video file in terms of time. The frame 48 frames after the first search frame (the 84th frame from the beginning) is the frame 48 frames before the last frame of the integrated video file in terms of time (the 132nd frame from the beginning). The frame that is 48 frames before the first search frame (the 84th frame from the beginning) in terms of time is the frame that is 132 frames before the last frame of the integrated video file in terms of time (the 36th frame from the beginning of the integrated video file).

[0240] Similarly, the image extraction module 315 finally determines, according to the binary search method, the frame in which the stationary golf ball 820 before the impact by the club head 811 was photographed (the frame immediately before impact) and the frame in which the stationary golf ball 820 was not photographed (the frame immediately after impact), in the same way as the first example of ball search.

[0241] It can be determined that the impact point is between the consecutive "frame immediately before impact" and "frame immediately after impact". In such a ball search, the impact point can be estimated more efficiently compared to the process of analyzing each frame image one by one or the process according to the first example of ball search.

[0242] As described above, the image extraction module 315 identifies the "frame immediately before impact" photographed immediately before the impact point in the shooting order and the "frame immediately after impact" photographed immediately after the impact point in the shooting order.

[0243] The image extraction module 315 extracts a predetermined number of consecutive frames, for example 16 frames, before and after the impact point. For example, the image extraction module 315 extracts a total of 8 frames that are consecutive in time before the frame at the impact point and are the "frame immediately before impact" and the 8 frames consecutive in time before it, and a total of 8 frames that are consecutive in time after the frame at the impact point and are the "frame immediately after impact" and the 8 frames consecutive in time after it.

[0244] It is advisable that the image extraction module 315 can appropriately change or set the total number of frames to be extracted before and after in terms of time. For example, when the image extraction module 315 identifies two frames before and after the ball disappearance from the integrated video file, it may recognize only the golf head positions in each of several frames before and after these two frames in terms of time, and calculate the lowest point based on these positions.

[0245] In addition, if the total number of frames to be extracted is 240 fps, for example, if it is the normal swing speed of a club, it may be about 8 frames in total. The image extraction module 315 compares the preset swing speed of a general golf player stored as the setting information 321 with the swing speed of the user, and may change the total number of frames to be extracted, fps, shooting time, etc. for those with slower or faster swing speeds.

[0246] In both the first example and the second example of the ball search, the image extraction module 315 can identify the "frame of the impact moment". The "frame of the impact moment" refers to, for example, the frame in which the club head 811 and the golf ball 820 are in contact, the frame in which the center position of the club head 811 and the center position of the golf ball 820 are closer than a predetermined distance, for example, the length of the radius of the ball, the frame in which the identification frame of the golf ball 820 overlaps with the identification frame of the club head 811 by a predetermined ratio or more, for example, 50% or more, etc.

[0247] When the image extraction module 315 identifies the "frame of the impact moment", in addition to the "frame of the impact moment", it extracts a total of 17 frames, including a total of 8 frames that are temporally continuous before the "frame of the impact moment" and a total of 8 frames that are temporally continuous after the "frame of the impact moment".

[0248] The image extraction module 315 provides the club head 811 and the golf ball 820 captured in the extracted frames, and the frame image annotated, to the coordinate calculation module 316.

[0249] Note that the annotation process for the club head 811 and the golf ball 820 captured in the extracted frame may be executed by the coordinate calculation module 316 that has acquired the frame image. The above-mentioned extracted frame, frame image, frame, etc. are examples of "shooting information" and "shooting information for analysis".

[0250] FIG. 24 is an example of a coordinate calculation flow 2400 using a coordinate calculation model. The coordinate calculation flow 2400 using a coordinate calculation model may also be referred to as a "head classification phase". The coordinate calculation flow 2400 and the head classification phase are examples of "processing for detecting a head".

[0251] Hereinafter, an example in which the coordinate calculation module 316 of the swing analysis module 312 executes a coordinate calculation flow 2400 using a coordinate calculation model as a coordinate calculation step (S1930) using a coordinate calculation model will be described.

[0252] The coordinate calculation module 316 takes in the acquired "frame capturing a swing" into a coordinate calculation model that is a machine learning model, and acquires the "coordinates within the image" of the "golf head" and the "golf ball".

[0253] For example, the coordinate calculation module 316 acquires (S2410) an image annotated with the club head 811 and the golf ball 820 acquired from the image extraction module 315, and inputs the acquired annotated image into the coordinate calculation model (S2420). Note that the coordinate calculation model is, for example, a machine learning model generated by the coordinate calculation model generation flow 2100 described with reference to FIG. 21.

[0254] The coordinate calculation module 316 executes coordinate calculation processing according to the coordinate calculation model (S2430), outputs the coordinates of the club head 811 during the swing and the stationary golf ball 820 calculated from the annotated image (S2440), and ends the coordinate calculation flow 2400 using the coordinate calculation model.

[0255] As described above, in the head classification phase, the coordinate calculation module 316 uses the coordinate calculation model, which is a machine learning model, to obtain the "coordinates within the image" of the "golf head" and the "golf ball".

[0256] FIG. 25 is an example of a calculation flow 2500 for the distance between the ball coordinates and the lowest point of the head. The calculation flow 2500 for the distance between the ball coordinates and the lowest point of the head may also be referred to as the "lowest point calculation phase".

[0257] The trajectory calculation module 317 calculates the "lowest point" using an approximation curve, such as a function of a spline curve, etc., from the "coordinates within the image" of the "golf head" obtained from the coordinate calculation module 316, and calculates the distance from the "coordinates within the image" of the "golf ball" and the "lowest point".

[0258] For example, the trajectory calculation module 317 obtains the coordinate information 1720 (1720a, 1720b, 1720c) of the golf ball 820 and the club head 811 from the coordinate calculation module 316 (S2510).

[0259] The trajectory calculation module 317 calculates the swing trajectory 1730 from the obtained coordinate information 1720 (1720a, 1720b, 1720c) of the club head 811 (S2520).

[0260] Based on the plurality of coordinate information 1720 (1720a, 1720b, 1720c) of the club head 811, the trajectory calculation module 317 can calculate a trajectory 1730 using an arbitrary approximation curve, such as a function of a spline curve or the like. In addition to the coordinate information, the trajectory calculation module 317 may further calculate the trajectory 1730 using the physical information of the user, the dimensional information of the club used, and the like.

[0261] Regarding the coordinate information 1720 (1720a, 1720b, 1720c), please also refer to the description of FIG. 17. Regarding the trajectory 1730, please also refer to the description of FIG. 17.

[0262] The lowest point information - distance information creation module 318 calculates the lowest point 1800 (x1, y1) of the trajectory 1730 calculated by the trajectory calculation module 317. Regarding the lowest point 1800 (x1, y1), please also refer to the description of FIG. 18.

[0263] The lowest point information - distance information creation module 318 further calculates the distance between the coordinates (x2, y2) of the golf ball and the lowest point 1800 (x1, y1) of the swing (S2530). More specifically, the lowest point information - distance information creation module 318 calculates, as the analysis distance, the analysis length 1820 between the lowest point 1800 (x1, y1) and the coordinates 1810 (x2, y2) of the golf ball 820 based on the calculated lowest point 1800 (x1, y1) and the coordinates 1810 (x2, y2) of the golf ball 820 (S2530).

[0264] The analysis length 1820 may be the length between the calculated lowest point 1800 (x1, y1) and the coordinates 1810 (x2, y2) of the golf ball 820, but it is better that it is the length between the x - coordinate x1 of the calculated lowest point 1800 (x1, y1) on the x - axis of the coordinate system and the x - coordinate x2 of the coordinates 1810 (x2, y2) of the golf ball 820 on the x - axis of the coordinate system.

[0265] By defining it as the length between the x - coordinate x1 of the lowest point 1800 (x1, y1) where the analysis length 1820 is calculated and the x - coordinate x2 of the x - axis of the coordinates 1810 (x2, y2) of the golf ball 820, the user can clearly understand how far forward or backward the position of the lowest point 1800 (x1, y1) of their swing is relative to the golf ball 820.

[0266] The swing analysis module 312 collaborates with the cooperation module 313 and the display module 314 to execute output processing such as transmitting the calculated analysis length 1820 to the detection terminal 20, displaying the calculated analysis length 1820 on the output device 505, and storing it in the auxiliary storage device 502 (S2540).

[0267] The analysis terminal 30 may display information regarding the progress of the processing being executed on the output device 505 at each step (S1910, S1920 (S2210 - S2240), S1930 (S2410 - S2440), S1940 (S2510 - S2540)) of the swing analysis flow 1900.

[0268] FIG. 34 is a first example of an explanatory diagram for explaining the display screen 3400 of the analysis terminal 30 during swing analysis. FIG. 34 illustrates the display screen 3400 of the analysis terminal 30 that displays the progress information 3410 (3410a - 3410c) of the processing during the execution of the swing extraction process (S2220) by the image extraction model described with reference to FIG. 22, for example.

[0269] The analysis terminal 30 first detects the ball and displays the progress information 3410 (3410a) regarding the process on the output device 505. In FIG. 34(A), the progress information 3410 (3410a) of the search process for searching for the presence or absence of the ball from the integrated video file described with reference to FIG. 22 is displayed using the total number of frames for the search process and the number of the currently processed frame.

[0270] In FIG. 34(A), as progress information 3410(3410a), text information of "Searching for ball" and "73 / 96" is overlaid (superimposed) on the video being shot by the shooting module 311 and displayed. Here, "73 / 96" indicates that, for example, the search process for the golf ball 820 has advanced up to the 73rd frame out of the planned number of 96 frames to be processed.

[0271] In FIG. 34(B), as progress information 3410(3410b), text information of "Analyzing" is overlaid on the video being shot by the shooting module 311 and displayed. Here, "Analyzing" can indicate, for example, that a process of adding annotation information of the golf ball 820 to the extracted frames is being executed, or that an analysis process such as "a frame in which the stationary golf ball 820 before the impact by the club head 811 is shot (frame immediately before impact)", "a frame in which the stationary golf ball 820 is not shot (frame immediately after impact)", or the impact time point is being executed.

[0272] When the golf ball 820 can be detected, the analysis terminal 30 may display a progress screen in phases of analysis while detecting the club head 811. In FIG. 34(C), as progress information 3410(3410c), text information of "Head analyzing" is overlaid on the video being shot by the shooting module 311 and displayed. Here, "Head analyzing" can indicate, for example, that a process of adding annotation information of the club head 811 to the extracted frames is being executed.

[0273] In the example of FIG. 34, the analysis terminal 30 can update and display the type and content of the progress information 3410(3410a to 3410c) on the display screen 3400 according to the currently executing process. As a result, the user can better grasp the progress of the analysis process being performed on the analysis terminal 30.

[0274] FIG. 35 is a second example of an explanatory diagram for explaining the display screen 3400 of the analysis terminal 30 during swing analysis. FIG. 35 illustrates the display screen 3400 of the analysis terminal 30 that displays the progress information 3410 (3410d, 3410e) during the execution (S2220) of the swing extraction process by the image extraction model described with reference to FIG. 22, for example.

[0275] In FIG. 35(A), as the progress information 3410 (3410d), text information "Ball detection impossible" is overlaid on the video being captured by the imaging module 311 and displayed. Here, "Ball detection impossible" indicates that, for example, the golf ball 820 has not been detected, identified, or discriminated from the acquired integrated video file.

[0276] The analysis terminal 30 may further display, as the progress information 3410 (3410d), information regarding the elimination of "Ball detection impossible", such as the arrangement of the analysis device such as the inclination, the imaging environment such as the brightness, and the target information such as the color of the ball.

[0277] In FIG. 35(B), as the progress information 3410 (3410e), text information "Head detection impossible" is overlaid on the video being captured by the imaging module 311 and displayed. Here, "Head detection impossible" indicates that, for example, the club head 811 has not been detected, identified, or discriminated from the acquired integrated video file.

[0278] The analysis terminal 30 may further display, as the progress information 3410 (3410e), information regarding the elimination of "Head detection impossible", such as the arrangement of the analysis device such as the inclination, the imaging environment such as the brightness, and the target information such as the shape and color of the club.

[0279] If the head cannot be detected, the analysis terminal 30 may display "Head detection impossible" in an overlay until the next head detection trigger is applied and display "Continue". This makes it easier for the user to grasp the current processing status of the analysis terminal 30. Additionally or alternatively, the analysis terminal 30 may continuously display information regarding "detection impossible" for either the ball or the head until the next head detection trigger is obtained, or until the next swing detection information is obtained.

[0280] In the above example, while progress information 3410 (3410d, 3410e) such as "Ball detection impossible" or "Head detection impossible" is displayed on the display screen 3400 of the output device 505 of the analysis terminal 30, on the detection terminal 20, for example, the switching display between the first display in FIG. 12(A) and the second display in FIG. 12(B) on the display screen 1200 of the detection terminal 20 is not performed. Since there is no screen switching, it can be understood that detection was impossible without looking at the display screen 3400 of the output device 505 of the analysis terminal 30.

[0281] Furthermore, in the above example, after performing the detection process (ball search phase) of the golf ball 820 from the acquired integrated video file by the analysis terminal 30, the analysis terminal 30 performs the detection process (head classification phase) of the club head 811 and displays separate progress information 3410 (3410d, 3410e) regarding each process that is not proceeding normally. This makes it easier for the user to guess where the cause is that the swing analysis process is not being performed normally.

[0282] [Examples regarding the abnormal progress of the detection process] When the detection process of the golf ball 820 or the detection process of the club head 811 does not proceed normally, the analysis terminal 30 may store information indicating the content of the error (hereinafter referred to as "error information"). The processor 303 may cause the auxiliary storage device 302 to store the error information by, for example, the method included in the history information 322. The error information may include, for example, information specifying which detection process did not proceed normally in addition to the fact that the error occurred. When the analysis terminal 30 displays the history of swing analysis based on the history information 322, in addition to the history of the swing analysis results specified by the history information 322, the history of the case where the detection process did not proceed normally based on the error information (that is, the history of the occurrence of the error) may be displayed. The analysis terminal 30 may, for example, display the history of the case where the detection process proceeded normally and the history of the case where it did not proceed in chronological order on the same screen, but the history of the case where the detection process proceeded normally and the history of the case where it did not proceed may be displayed on different screens, respectively.

[0283] At the detection terminal 20 as well, it is advisable to perform a display when the detection process of the golf ball 820 or the detection process of the club head 811 is not proceeding normally. The analysis terminal 30 may send information indicating that the detection process of either the golf ball 820 or the club head 811 is not proceeding normally (for example, an error code) to the detection terminal 20. In other words, the analysis terminal 30 may send information indicating the cause of the inability to detect the lowest point to the detection terminal 20. The detection terminal 20 may perform a display indicating that the detection process is not proceeding normally based on the information received from the analysis terminal 30. As described above, the detection terminal 20 cannot express information in as rich content as the analysis terminal 30. Therefore, at the detection terminal 20, it is advisable to display an error code using the output device 205. The error code may be information that expresses in characters which of the detection processes of the golf ball 820 and the club head 811 is not proceeding normally. For example, in the case where the output device 205 has a segment display, the processor 203 may use the segment display to display, as the error code, characters indicating which detection process is abnormal. The characters may be, for example, alphabets, numbers, or a combination thereof. The error code may be different information for each of "ball detection impossible", "head detection impossible", and "lowest point detection impossible".

[0284] FIG. 38 is a diagram showing a display screen 2600 of an error code displayed by the detection terminal 20. As shown in FIG. 38, in addition to the segment for displaying the characters "lowest point" using the segment display, the alphabetic characters "Err" meaning the occurrence of undetectability (that is, an error) and a number indicating which of ball detection impossible, head detection impossible, and lowest point detection impossible it is (in the example of FIG. 38, "02") may be displayed.

[0285] FIG. 39 is a diagram for explaining error displays on the analysis terminal 30 and the detection terminal 20. In the diagram shown in FIG. 39, the correspondence between an error code (Err code), an error name (Err name), error details (Err details), and a countermeasure method is shown. The error code is the error code displayed on the detection terminal 20 described above. The error name is the name of the error. The error name may be the text information displayed on the analysis terminal 30 described in FIG. 35. The error details are the details of the error, and may be the conditions for determining that an error has occurred on the analysis terminal 30. The countermeasure method indicates the actions that the user should take when an error occurs. When an error occurs on the analysis terminal 30, the analysis terminal 30 may display this countermeasure method on the same screen as the screen for displaying the text information described in FIG. 35, or on a different screen (for example, the next screen after the screen for displaying the text information).

[0286] In relation to the success or failure of the detection process of the ball and the head, the inventor discovered the following problems and considered the following solutions. (1) Problem that the ball or head cannot be detected under backlight As a solution, the analysis terminal 30 may a. perform backlight detection and, when it is backlit, display a message indicating that it is highly likely that detection is not possible because it is backlit in the first place. For example, similar to the above-mentioned "head detection impossible", the analysis terminal 30 may perform an overlay display of the character string "measurement impossible due to backlight" on the camera image. Also, the analysis terminal 30 may display both a character string indicating the cause showing the difference between "backlight", "front light", "insufficient light quantity", and "excessive light quantity", and a character string indicating the result of "measurement impossible" within one screen. Alternatively, as another solution, b. when the ball cannot be detected, the detection terminal 20 may perform a predetermined first display, such as flashing the LED (an example of a light-emitting unit) of the detection terminal 20 or the segment display of the detection terminal 20 in a first pattern. Alternatively, as another solution, c. when the ball has been detected but the head cannot be detected, the detection terminal 20 (processor 203) may perform a predetermined second display, such as flashing the LED of the detection terminal 20 or the segment display of the detection terminal 20 in a second pattern different from the first pattern.

[0287] (2) Problem that the position of the ball (also referred to as height) and size cannot be detected As a solution, the analysis terminal 30 may overlay and display on the image (camera video) captured by the imaging unit 306 a grid of the position of the ball (including the height of the ball) and the size of the ball where detection is successful (i.e., recommended). Alternatively, as another solution, the analysis terminal 30 may be provided with a calibration mode in which a single photo is taken before actually swinging the club to test whether the ball can be detected. At this time, the analysis terminal 30 may also measure the brightness and the degree of backlight and inform the user whether the state is good enough for measurement.

[0288] (3) The analysis terminal 30 may be provided with a function of narrowing down the search area of the ball to a predetermined range and overlaying and displaying on the camera image a range that is the same as or slightly narrower than the predetermined range as the "range where the ball should be placed". Based on the inventor's experience, depending on the image recognition method, if the search range is not narrowed down, a ball in an extremely strange position may be recognized as a ball. For example, it is based on the fact that when a clock mark imitating an analog clock with a circular shape and a white background is displayed in the image to be subjected to image recognition, this clock mark was recognized as a ball.

[0289] (4) In the process of detecting the ball, the analysis terminal 30 may have a function of filtering by circularity. Based on the inventor's experience, if circularity is not considered, an object with a small degree of circularity in the camera image (for example, the shoe part of the user's foot) may be misrecognized as a ball. Circularity is an index for evaluating whether the shape of the contour is close to a circle. Circularity takes a value in the range of, for example, "0" to "1", and the closer the value is to "1", the closer the contour is to a circular (perfect circle) shape. For example, the analysis terminal 30 may detect the ball from the camera image on the condition that the circularity of the object is equal to or greater than a predetermined threshold value.

[0290] (5) When the analysis terminal 30 detects an object recognized from a camera image as a ball, it may be configured to set an upper limit and a lower limit for the number of pixels in the camera image, and detect only objects between the upper limit and the lower limit as balls. This is because when the positional relationship between the analysis terminal 30 and the user is determined to a certain extent, the image of the ball shown in the camera image should fall within a certain range of pixel numbers. By doing so, the possibility of detecting an object that is not a ball as a ball can be reduced compared to the case where the upper limit and the lower limit of the pixel number are not specified.

[0291] (6) The analysis terminal 30 may be provided with a function of allowing the user to input the ball position in the camera image. By doing so, the possibility of misrecognizing an object (for example, the pattern of the user's clothes) located at a position where the ball cannot exist in the camera image as a ball can be suppressed compared to the case where the ball position is not specified. The analysis terminal 30 may be configured to display the camera image and allow the user to input the ball position using, for example, a touch panel. The user may input, for example, the ball search area described in (3) above as the ball position, or may input an area where the ball can be placed in the camera image (for example, an area where the mat is shown, or an area where the ground is shown).

[0292] (7) The analysis terminal 30 may have a function of displaying an image (for example, a marker) indicating the area (location) recognized as a ball in the camera image. At this time, the analysis terminal 30 may change the display mode (in particular, the color is preferable) of the image indicating the area recognized as a ball according to the degree of certainty (that is, the confidence level) that the area recognized as a ball is actually a ball. By doing so, even when a plurality of objects are recognized as balls from the camera image, the camera image can be displayed such that the actual ball has a confidence level of 1st and is green, and the heel of the left shoe has a confidence level of 2nd and is blue. Thus, the certainty of the ball can be grasped from the camera image.

[0293] (8) The analysis terminal 30 may detect the "ball" in real time or at high speed after recording by a first process that is a light process, while detecting the "head" by a second process that is heavier than the first process. The first process may be a process using image processing, and may be a process that utilizes the fact that a golf ball usually appears as a round and white object in a camera image. In particular, the first process may convert the camera image to grayscale, binarize the converted image, and further use Hough transform to detect circular objects. The second process may be a process using machine learning, and in particular, may be a process such as inferring using a learned model of the head such as deep learning.

[0294] (9) It is possible that the lower side of the ball may not be reflected in the camera image, such as when part of the ball is buried in the grass. Therefore, the analysis terminal 30 may detect the ball by a process considering a semi-circle (only the upper side). For example, when an object with a low degree of circularity is recognized, the degree of match of the upper half of the object may also be checked to determine whether it is a ball.

[0295] (10) When the analysis terminal 30 recognizes the ball from camera images of a plurality of frames, it may determine as the ball those that are in the same range in a predetermined plurality of frames among those recognized as the ball.

[0296] (11) When starting the process of detecting the ball, the analysis terminal 30 may display at least any one of the following messages. As the messages, in particular, messages such as "Please use a white ball", "Is the ball a prominent color (brightness) compared to the surroundings?", and "It becomes difficult to recognize if the ball is not prominent" may be used.

[0297] [Examples Regarding Definition of Coordinates] Regarding the definition of coordinates, initially, the inventor considered recognizing the club head from a camera image and using the coordinates of the center of the bounding box after such recognition as the coordinates of the feature points. However, it was found that since the shape of the head as seen from the camera (i.e., the imaging unit 306) changes, the error in the coordinates of the center may become large. As a cause of the change in the shape of the head, for example, it is conceivable that the imaging direction of the camera changes depending on the installation method of the analysis terminal 30. Therefore, the inventor considered that the following should be done at present.

[0298] (1) The coordinate calculation model generation module 516 includes the fitting metal fitting at the root of the head of the club in the bounding box. Then, the coordinate calculation model generation module 516 acquires the coordinate information of the center point of the portion between the shaft and the root of the head, which can almost be said to be the shaft portion (in other words, uses the center coordinates of the shaft). By doing so, the inventor obtained the finding that the accuracy of the X coordinate of the lowest point is improved.

[0299] (2) In the above embodiment, the analysis terminal 30 calculates the value of the Y coordinate of the lowest point, but it may also be configured not to calculate the value of the Y coordinate of the lowest point or not to use the value of the Y coordinate of the lowest point.

[0300] (3) As an example of the case of using the value of the Y coordinate of the lowest point, the inventor considered that it may be as follows. The inventor thought that if the distance between the center coordinate of the shaft and the lowest point of the head, which was described in the above [Modification of coordinate definition], was input, it should be possible to more reliably determine "top" and "duff" in the swing. When using the value of the Y coordinate of the lowest point, it is possible to know (A) how far the lowest point of the head dives below the horizontal line passing through the lowest point of the ball, or (B) how much above the horizontal line passing through the center point of the ball the head passes. Then, depending on the amount of the head's dive, (I) in the case of (A), and if the lowest point of the swing arc is on the right side of the ball, it becomes "duff", (II) in the case of (A), and if the lowest point of the swing arc is on the left side of the ball, it becomes " turf", and (III) in the case of (B), it should be possible to more reliably determine that it becomes completely "top", the inventor thought.

[0301] As an example, the processor 303 (swing analysis module 312) of the analysis terminal 30 has a function of receiving an input of the distance between the center coordinate of the shaft and the lowest point of the head. The processor 303 may directly receive an input of the value of this distance, for example, but may also receive an input of information for specifying this distance, such as the club manufacturer and model number. The processor 303 may receive an input from the user using the input device 304, but may also receive an input from an external device with which the analysis terminal 30 can communicate. The processor 303 (swing analysis module 312) may determine whether it is "top" or "duff" based on the received distance between the center coordinate of the shaft and the lowest point of the head, the Y coordinate of the lowest point, and the positional relationship between the lowest point of the swing arc and the center point of the ball. The processor 303 may make this determination based on which of the above (I) to (III) it corresponds to. The processor 303 may output this determination result by a predetermined method, and for example, like the information of the lowest point, it may be output by methods such as display (for example, display within the same screen as the information of the lowest point), recording, etc.

[0302] [Example Regarding Display of Lowest Point] In the above-described embodiment, in the analysis terminal 30, when the lowest point of the display module 314 is located on the front side of the golf ball 820 with respect to the swing direction (that is, before impact), the distance is displayed using the "-" (minus) sign, and when the lowest point is located on the back side of the golf ball 820 (that is, after impact), the distance is displayed using the "+" (plus) sign. Instead of this, the display module 314 may express on which side (right or left) of the ball the lowest point is, as seen by the user (the batter), using the signs "L" and "R". "L" is the first letter of "Left" indicating left, and "R" is the first letter of "Right" indicating right.

[0303] FIG. 40 is a diagram showing a batter, who is the user, as seen from above, and shows the relationship between the position of the batter, the hitting direction of the ball (denoted by enclosing "B" in a circle in FIG. 40), and the sign ("L" or "R"). As shown in FIG. 40, when the user is a right-handed batter and a left-handed batter, the left and right of the hitting direction of the ball as seen by the user are reversed, so the signs "L" and "R" are also reversed. Specifically, when the user is a right-handed batter, when the lowest point is located on the front side of the golf ball, the lowest point is located on the right side of the golf ball, and when the lowest point is located on the back side of the golf ball, the lowest point is located on the left side of the golf ball. When the user is a left-handed batter, when the lowest point is located on the front side of the golf ball, the lowest point is located on the left side of the golf ball, and when the lowest point is located on the back side of the golf ball, the lowest point is located on the right side of the golf ball.

[0304] To determine whether the user is a right-handed or left-handed hitter, the analysis terminal 30 (for example, the swing analysis module 312) may have a function of receiving input of information indicating whether the user is a right-handed or left-handed hitter using the input device 304. The analysis terminal 30 may enable the user to set their dominant hand, for example, from a "Dominant Hand Setting" menu associated with a lower layer of the "Settings" menu. The analysis terminal 30 may determine whether the user is a right-handed or left-handed hitter based on this dominant hand setting. Then, in the analysis terminal 30, when the user is a right-handed hitter, the display module 314 may display the distance using "R" when the lowest point is located in front of the golf ball 820, and display the distance using "L" when the lowest point is located behind the golf ball 820. When the user is a left-handed hitter, the display module 314 may display the distance using "L" when the lowest point is located in front of the golf ball 820, and display the distance using "R" when the lowest point is located behind the golf ball 820. Also in the detection terminal 20, it may be possible to represent on which side (right or left) of the ball the lowest point is located as seen from the user (hitter) using the signs "L" and "R". Under the configuration where the output device 205 uses a 7-segment display, it may be possible to display the characters "L" and "R" (for example, "r" in lowercase).

[0305] Figure 41 is a diagram showing a history screen of swing analysis that the analysis terminal 30 displays based on the history information 322. As shown in Figure 41, the analysis terminal 30 divides the area within the display screen into a plurality of areas in the column direction, and displays the analysis results of the swing in each of the plurality of areas. As shown in Figure 41, as the analysis results of the swing, in addition to the head speed, ball speed, estimated carry distance, and meet rate, the lowest point is displayed using the signs "L" or "R". Note that for the calculation of the head speed, ball speed, estimated carry distance, and meet rate, known methods before the filing of this application may be used.

[0306] [Feedback Function] FIG. 42 is a diagram for explaining the feedback function realized by the analysis system 10. In the swing analysis mode, the analysis terminal 30 has a function of transmitting (also referred to as uploading or making a feedback post) a swing video, which is a video taken during swing analysis, and analysis information, which is information indicating the analysis result of the swing, to the management terminal 50. The management terminal 50 receives the swing video and the analysis information from the analysis terminal 30. The management terminal 50 updates (for example, retrains) the machine learning model based on the received swing video and analysis information. With such a feedback function, the swing video and analysis information used in the actual swing analysis are utilized to improve the accuracy of the machine learning model.

[0307] As an example, the analysis terminal 30 may display a soft button (soft button marked with "!") for making a feedback post on the screen that displays the swing video shown in FIG. 42(A). When the analysis terminal 30 receives an operation of selecting the soft button, as shown in FIG. 42(B), it displays a screen for confirming with the user whether to agree to transmit the swing video and the analysis information to the management terminal 50. When receiving an operation of selecting the soft button marked with "Post", the analysis terminal 30 transmits the swing video and the analysis information to the management terminal 50 (that is, makes a feedback post). When the feedback post is completed, the management terminal 50 issues and stores a feedback number, which is an identification number unique to this feedback post, and further transmits the issued feedback number to the analysis terminal 30. When the analysis terminal 30 receives the feedback number from the management terminal 50, it displays the received feedback number as shown in FIG. 42(C). The analysis terminal 30 may further have a function of reporting detailed situations at the time of inquiry based on this feedback number. For example, the analysis terminal 30 may display an input field for the inquiry number and an input field for text for reporting the situation during swing analysis, and transmit the inquiry number input by the user and the content of the report on the situation during swing analysis in these input fields to the management terminal 50. The user (for example, the administrator) of the management terminal 50 can utilize this inquiry number and the report content for improving the accuracy of the machine learning model and the like.

[0308] [Camera mode selection function] The analysis terminal 30 may have a camera mode selection function. The camera mode indicates the mode of shooting using the imaging unit 306 which is a camera. FIG. 43 is a diagram for explaining the camera mode selection function of the analysis terminal 30. To cope with various usage environments of the analysis terminal 30 by the user, the analysis terminal 30 may have a function of selecting the usage environment. For example, when performing swing analysis for the first time, the analysis terminal 30 may accept the selection of the usage environment. For example, as shown in FIG. 43(A), the analysis terminal 30 may display a screen for allowing the user to select any one of "outdoor (direct sunlight)", "shaded daytime practice range", or "indoor nighttime practice range" as the usage environment. The analysis terminal 30 may select the usage environment based on the user's input. The analysis terminal 30 may perform camera settings based on the selected usage environment. The camera settings may be set to be optimal for swing analysis. The analysis terminal 30 may set the ISO sensitivity and shutter speed according to the usage environment based on the relationship shown in FIG. 43(B) for example. Also, as shown in FIG. 43(C), the analysis terminal 30 may display a soft button for changing the setting of the camera mode, and may have a function of changing the setting of the camera mode even after the camera is activated.

[0309] [Approach practice function] As described above, the detection terminal 20 may have an approach practice function. FIG. 26 is an example of an explanatory diagram for explaining the approach practice mode. The details of the approach practice function (approach practice mode) are as described in FIG. 26.

[0310] The display module 214 can switch and display the above-described swing information display function and the approach practice function, for example, in response to a switching operation by the user. FIG. 27 is an example of an explanatory diagram for explaining the switching operation to the approach practice mode. The details of the switching operation to the approach practice mode are as described in FIG. 27.

[0311] The detection terminal 20 may have a short / middle approach practice function as an approach practice function.

[0312] FIG. 28 is an example of a first explanatory diagram for explaining a practice operation in the short / middle approach mode. FIG. 28 is an example of a second explanatory diagram for explaining a practice operation in the short / middle approach mode. FIG. 30 is an example of a third explanatory diagram for explaining a practice operation in the short / middle approach mode.

[0313] The practice operation in the short / middle approach mode is as described in FIGS. 28 to 30. In the short / middle approach mode, the user can confirm the estimated flight distance of an approach shot to a randomly set distance between 30 and 100 yards for short and between 100 and 170 yards for middle.

[0314] The detection terminal 20 may have an approach practice function for any distance, in addition to or instead of, the short / middle approach practice function illustrated in FIGS. 28 to 30.

[0315] FIG. 31 is an example of a first explanatory diagram for explaining a practice operation in the approach mode for any distance. FIG. 32 is an example of a second explanatory diagram for explaining a practice operation in the approach mode for any distance. FIG. 33 is an example of a third explanatory diagram for explaining a practice operation in the approach mode for any distance.

[0316] The practice operation in the approach mode for any distance is as described in FIGS. 31 to 33. In the approach mode for any distance, the user can confirm the estimated flight distance of an approach shot to an arbitrarily set distance.

[0317] [Explanation of matters related to SDGs] By introducing the analysis system 10, it is expected to contribute to some of the Sustainable Development Goals (SDGs) as follows.

[0318] <SDG3: Good health and well-being for all> Measurements in sports competitions such as golf are useful for monitoring the fitness and performance of users. According to the analysis system 10, through appropriate training and the pursuit of performance, the health and well-being of individuals can be improved.

[0319] <SDG4: Quality education for all> By using the analysis system 10, it can be expected that the training and performance evaluation of sports for users will be improved. As a result, high-quality education and training will be provided to users, enabling the improvement of users' skills.

[0320] [Variant example] (1) As described above, when the analysis terminal 30 is installed in an inclined manner, a deviation occurs at the lowest point of the swing trajectory. Therefore, a method for correcting (modifying) this deviation is proposed. FIG. 36 shows a correction method obtained from the derivative coefficient of the third-order approximation formula being equal to 0. The analysis terminal 30 may calculate the lowest point of the swing trajectory with the deviation corrected by performing each process in the order of "1" to "5" described in FIG. 36. In the process of "4" in FIG. 36, the reason for obtaining the approximation formula of the third-order approximation formula is based on the inventor's idea that the swing orbit is not symmetric about the left and right and does not necessarily become a beautiful parabola. In order to make the curve (approximate curve) specified by the approximation formula closer to the actual swing orbit, if a second-order approximation is performed, as shown in FIG. 37(A), it can be seen that the deviation between the parabolic approximate curve and the coordinates of the head is large. This is an example of performing a second-order approximation on a point where the original parabola is tilted by -10 degrees, and it can be seen that the deviation is large in the second-order approximation (specifically, the contribution rate = 0.9837). In this example, because there is a tendency to consciously stretch the head longer to the left side of the ball and shift the body weight to the left foot and be conscious of the handfast, the curvature tends to be larger on the left side of the ball. Conversely, the inventor thought that a person who hits with a swing orbit with a contribution rate close to 1 in the second-order approximation formula may even be said to have a poor swing. On the other hand, FIG. 37(B) shows the approximation formula and the approximate curve when a third-order approximation is performed, and the figure shows the approximation formula and the approximate curve when the contribution rate = 0.996. It can be expected that increasing the degree of the approximation formula will make the contribution rate closer to 1, but when making a fourth-order approximation, it is necessary to find the solution of a cubic equation, and the calculation may become extremely complicated. Therefore, considering the balance between suppressing the deviation between the approximate curve and the swing orbit and the calculation amount, the inventor thought that a third-order approximation should be used. However, if the problem of the calculation amount can be solved, it is also possible to adopt a higher-order approximation formula to suppress the deviation between the approximate curve and the swing orbit. The inventor thought that up to the fifth-order approximation is possible because there is a formula for the solution up to the fourth-order equation.

[0321] (2) The analysis terminal 30 may execute the swing analysis process according to the following procedures (2-1) to (2-5). (2-1) Analysis Preparation (Video Shooting) The analysis terminal 30 performs video shooting using the shooting unit 306 and creates video data of the swing timing. When starting the analysis screen, the analysis terminal 30 starts video shooting, and the resolution of the video shooting is preferably 1920x1080. The analysis terminal 30 stops recording 0.1 seconds after the swing timing from the detection terminal 20. The analysis terminal 30 cuts out 0.512 seconds from the end of the video (0.5 seconds + 0.012 seconds for countermeasures against frame dropout). The number of frames of the video is 124 frames. Thereafter, the analysis terminal 30 acquires and updates the inclination every 3 seconds. The inclination is reflected in the acquired coordinates when calculating the lowest point. (2-2) Analysis Preparation (Generation of Analysis Image) The analysis terminal 30 generates an analysis image from the shot video. The analysis terminal 30 converts the video into frame images. The resolution of the frame images is 640x540, in jpeg format, and preferably 124 images. When converting to frame images, the analysis terminal 30 may black out the areas where detection processing is not desired. The areas where detection processing is not desired may be set in advance, for example, or may be set by the user. (2-3) Ball Detection Processing The analysis terminal 30 detects the ball position coordinates and identifies the frame position immediately before impact from the analysis image. For example, the analysis terminal 3- sets the detection tag as Golfball and the detection tag confidence threshold as 0.6. The analysis terminal 30 performs image recognition of the ball by binary search on all frame images and acquires the coordinates. The analysis terminal 30 sets the last detected frame as the ball detection frame. The analysis terminal 30 determines the practice swing. The determination of the practice swing may be made based on the fact that the ball could not be detected in the first frame and the ball was detected in the 124th frame. The analysis terminal 30 calculates the magnification from pixels (px) to millimeters (mm) based on the golf ball size (about 43 mm). When the ball cannot be detected, the analysis terminal 30 determines that the ball detection is impossible. (2-4) Head Analysis Processing The analysis terminal 30 detects the head coordinates from the image for analysis, before and after the frame immediately before the impact detected by the ball detection process. The analysis terminal 30 sets the detection tags as Golfhead1, Golfhead2, Golfhead3, Golfhead6, Golfhead7, and Golfhead8. The detection tag certainty threshold is set to 0.2. The analysis terminal 30 performs image detection of the head, and acquires the head coordinates, for the frames before and after the ball detection frame (frames -6 to 8). When two or more tags are detected in the same frame image, the analysis terminal 30 may adopt the tag with the higher certainty. In addition, the analysis terminal 30 may exclude detection targets that are not moving. In this way, for example, false detection of objects such as shoes and chairs can be excluded. The analysis terminal 30 excludes the head that is too close to the ball coordinates. When the score of the head is 3 points or less, the analysis terminal 30 may determine that the head cannot be detected. (2-5)Lowest point calculation The analysis terminal 30 calculates the lowest point based on the ball position coordinates and the head coordinates by the following procedure. The analysis terminal 30 tilts the coordinates according to the tilt of the terminal. The tilt at the time when the video is stopped is preferably used as the tilt. The analysis terminal 30 obtains a quadratic function by the least squares method based on the head coordinates. If the result of the cubic function described below shows that the distance of the lowest point exceeds 10 cm, the analysis terminal 30 may use this quadratic function as an insurance when the head coordinates are 4 points. If the head coordinates are 5 points or more, the analysis terminal 30 obtains a cubic function by the least squares method. When obtaining the cubic function, if the coefficient of determination (R2) is 0.9 or less, the analysis terminal 30 deletes the coordinates farthest from the graph and repeats the recalculation until it becomes 0.9 or more. The "furthest coordinate" is preferably the coordinate whose y coordinate is farthest from the graph. If the analysis terminal 30 cannot obtain the quadratic function or the cubic function, it determines that the lowest point cannot be detected. The analysis terminal 30 obtains the coordinates of the obtained graph when it reaches its maximum. The analysis terminal 30 multiplies the difference between the x coordinate of the ball and the x coordinate of the maximum (i.e., (x coordinate of the ball - x coordinate of the maximum)) by a magnification factor to obtain the distance (cm) of the actual lowest point.

[0322] (3) The analysis terminal 30 may have a function of recording in the storage means an approximate expression (for example, various functions) indicating the swing described so far and an approximate curve specified by the approximate expression. It is preferable to use the auxiliary storage device 302 as the storage means, but other storage means may also be used. The analysis terminal 30 may have a function of causing the display means to display the approximate curve indicating the swing described so far. As the display means, it is preferable to use the display of the output device 305, but other display means may also be used. In particular, the analysis terminal 30 may display the approximate curve superimposed on the camera image (that is, perform overlay display).

[0323] (4) In the above-described embodiment, the analysis terminal 30 obtained an approximate curve indicating the swing trajectory using the function of the spline curve. Functions other than the function of the spline curve may be used. Further, the analysis terminal 30 may be provided with a function of selecting a function to be used from a plurality of different functions. The analysis terminal 30 may have a function of selecting a function regardless of the user's selection, that is, a function of automatically selecting a function. The analysis terminal 30 may select a function based on the state of other sensors, for example, the inclination of a G sensor (an example of an acceleration sensor). In this case, the analysis terminal 30 may change the function depending on, for example, the club number. Alternatively, the analysis terminal 30 may change the function to be selected depending on the states of a plurality of frames in which the head could be recognized (for example, the number of recognized frames, the degree of subsampling of the frames). Alternatively, the analysis terminal 30 may have a function of allowing the user to manually select a function. The analysis terminal 30 may have a function of associating and storing information specifying the used function with information based on the swing trajectory (for example, the position of the lowest point several centimeters before and after). The analysis terminal 30 may store information on the swing history, such as "the 10th swing, +5 cm, calculated using function No. 1".

[0324] (5) The analysis terminal 30 was installed so as to photograph the club and the ball with the user from almost the horizontal direction in the front, but it is not limited to this. Although various installation methods of the analysis terminal 30 can be considered, for example, the analysis terminal 30 may be installed so as to photograph the head of the club or the ball from directly above at the position of the user or in the vicinity thereof using a predetermined fixture.

[0325] [Appendix] In addition, for the part described as, for example, A may be a1, a2 ···, and an, A may be specified as including one or more of a1, a2 ··· and an, or A may be specified as not including one or more of a1, a2 ··· and an. Therefore, there is also an intention to obtain rights for inventions excluding at least some of the enumerated elements, such as "A (excluding a1)", "A (excluding a1 and a2)".

[0326] For the part described in the form of A includes B and B includes C, it may be configured such that A includes C. For the part where the process is described as performing B after A and then performing C after B, it may be configured to perform C after A. The part described as the process to be executed after the condition judgment also discloses the part that executes the process part without assuming the condition judgment, and there is also an intention to obtain rights for these. For the part described in the form of P includes Q, it may be configured to include an intervening substance or an intervening process between P and Q. There is also an intention to obtain rights for these.

[0327] Note that the scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed in this specification. Among the present invention, the configuration for which a patent is sought is specified in the appended claims, but even a configuration that is not currently specified in the claims has an intention to make the configuration disclosed in this specification the claims in the future.

[0328] The invention of the present application is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modification examples may be arbitrarily selected and combined. Also, any constituent element of each embodiment or modification example may be arbitrarily combined with any constituent element described in the means for solving the invention or a constituent element embodying any constituent element described in the means for solving the invention. Regarding these, there is also an intention to obtain rights in amendments or divisional applications of the present application. Even if there are descriptions such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. Configurations other than these cases and times are also disclosed, and there is an intention to obtain rights. Also, the parts described in a sequential manner are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and there is an intention to obtain rights.

[0329] Also, due to a change to a design registration application, there is an intention to obtain rights for the overall design or partial design. Although the drawings depict the entire apparatus in solid lines, the drawings include not only the overall design but also partial designs claimed for a part of the apparatus. For example, not only can a part of the members of the apparatus be a partial design, but the drawings also include a partial design for a part of the apparatus regardless of the members. As a part of the apparatus, it may be a part of the members of the apparatus or a part of the member. Regarding the overall design, of course, there is an intention to claim as rights a partial design in which an arbitrary part of the solid-line part of the drawing is made into a dashed-line part. Also, regarding the modules, members, parts, etc. inside the housing of the apparatus, those shown in the drawings are all independently subject to transactions, and similarly, there is an intention to make a change to a design registration application and claim as rights.

[0330] (1) The system may include a detection device that uses a sensor to perform swing detection, which is the detection of the swing of a club, and an analysis device that performs at least one of photographing the head of the club and the ball and swing analysis, which is the analysis of the swing. By doing so, by executing swing detection and swing analysis on separate terminals, the computational load can be distributed. Thus, in conjunction with the detection device, the analysis results obtained by photographing the club or ball can be provided to the user. Also, since the detection device and the analysis device can be arranged at separate positions, various usage modes of the system can be provided to the user at low cost.

[0331] (2) The detection device may transmit swing detection information related to the swing detection to the analysis device, and the analysis device may start the swing analysis using the photographed information based on the swing detection information. By doing so, based on the swing detection information transmitted by the detection device, the analysis device can start the swing analysis at an appropriate timing when the swing detection information is received while distributing the computational load related to the swing detection process. Even if the time synchronization between the detection device and the analysis device is not accurate, a system that executes a more efficient swing analysis process can be provided to the user.

[0332] (3) The analysis device may transmit swing analysis information calculated by the swing analysis to the detection device, and the detection device, or the detection device and the analysis device, may perform a display based on the swing analysis information. By doing so, the detection device can display analysis information related to the detected swing based on the analysis results obtained by photographing the club or ball in conjunction with the detection device, and thus can provide the analysis information, contributing to improving the quality of the user's club swing and golf play. Also, it is possible to obtain analysis information regarding parameters that cannot be measured or calculated by the detection device alone due to spatial limitations such as the arrangement position, in terms of computational power and measurement angle. Furthermore, even when the display unit of the analysis device is not visible, the user can confirm the analysis results by the detection device.

[0333] (4) The detection device may display a numerical value based on the swing analysis information, and the analysis device may display at least one of a still image, a moving image, and a graph based on the swing analysis information. By doing so, since the detection device and the analysis device display different information based on the same swing analysis information, it is possible to provide a system that can display information to the user in various ways, for example, by varying the amount of information to be displayed.

[0334] (5) The analysis device may extract analysis shooting information used for the swing analysis from the shooting information taken before acquiring the swing detection information, or from the shooting information taken before acquiring the swing detection information and the shooting information taken after acquiring the swing detection information. By doing so, since the analysis device extracts the analysis shooting information used for the swing analysis from the shooting information taken before acquiring the swing detection information, even if the swing has already ended at the time when the analysis device acquires the swing detection information due to a delay in communication between the detection device and the analysis device or the influence of the user's high swing speed, it becomes easier to perform a swing analysis using the shooting information taken during the swing, and it is possible to provide a system that performs a swing analysis with higher reliability for the user.

[0335] (6) The swing detection information may indicate that the downswing of the golf swing has been detected. By doing so, even if the downswing of the swing has ended at the time when the analysis device acquires the swing detection information, it becomes easier to perform a swing analysis using the shooting information taken during the downswing, for example, the shooting information taken around the time of impact between the club head and the ball during the downswing, and it is possible to provide a system that performs a swing analysis with higher reliability for the user.

[0336] (7) The analysis device performs a process of detecting the ball from the imaging information, and when the ball is detected, it performs a process of detecting the head. When the process of detecting the ball is being performed, the analysis device displays the progress of the process of detecting the ball. When the process of detecting the head is being performed, the analysis device displays the progress of the process of detecting the head. This is preferable. By doing so, the analysis device separately performs the progress of the ball search process and the head edge creation process, and displays the progress of each process separately. Therefore, it becomes easier for the user to specifically grasp the progress of the currently executing analysis process.

[0337] (8) When the head is not detected, the analysis device preferably continuously displays that the head cannot be detected until the next head detection process is started. By doing so, when the head is not detected, the analysis device continuously displays information indicating that the head cannot be detected until the next head detection process is started. Therefore, it becomes easier for the user to specifically grasp the status of the currently executing analysis process, for example, whether the analysis process is proceeding smoothly or has stopped due to a detection error.

[0338] (9) The analysis device preferably calculates the head coordinates, which are the coordinates of the head, and the ball coordinates, which are the coordinates of the ball, from the imaging information for analysis. By doing so, even if the swing has already ended when the analysis device acquires the swing detection information, the analysis device calculates the head coordinates and the ball coordinates using the imaging information captured during the swing. Therefore, it becomes easier to perform swing analysis based on the positional relationship between the head and the ball, and it becomes easier for the user to obtain specific information about the positional relationship between the head that moves at high speed during the swing and the ball that is hit, which is difficult to visually confirm from a video or the like taken by the user himself / herself.

[0339] (10) The analysis device preferably calculates the trajectory of the head based on the head coordinates. In this way, even if the swing has already ended by the time the analysis device acquires the swing detection information, the analysis device calculates the trajectory of the head using each coordinate calculated based on the imaging information captured during the swing. Therefore, it becomes easier for the user to obtain specific information about the trajectory of the head that moves at high speed during the swing, which is difficult to visually confirm from the videos or continuous photos taken by the user himself / herself.

[0340] (11) Preferably, the analysis device calculates the lowest point coordinates, which are the coordinates of the lowest point of the trajectory, based on the calculated trajectory. In this way, even if the swing has already ended by the time the analysis device acquires the swing detection information, the analysis device calculates the lowest point on the trajectory of the head during the swing using each coordinate and the trajectory calculated based on the imaging information captured during the swing. Therefore, even if the actual lowest point of the head during the swing is not captured in the imaging information, it becomes easier to perform an analysis based on the positional relationship between the lowest point of the head and the position of the ball, and it becomes easier for the user to obtain specific information about the positional relationship between the lowest point of the head that moves at high speed during the swing and the position of the ball to be hit, which is difficult to visually confirm from the videos taken by the user himself / herself.

[0341] (12) Preferably, the analysis device detects the inclination of its own device and calculates the lowest point coordinates based on the trajectory of the swing corrected according to the detected inclination. In this way, the analysis device calculates the lowest point coordinates based on the corrected swing trajectory according to the inclination. Therefore, the user can set the system relatively flexibly and it becomes easier to use the system more casually.

[0342] (13) Preferably, the analysis device calculates distance information regarding the distance between the lowest point coordinates and the ball coordinates, the analysis device transmits the distance information to the detection device, and the detection device displays the distance information. In this way, the analysis device calculates the distance information between the lowest point of the head and the ball, and the detection device displays the distance information. Thus, it becomes easier for the user to grasp, as analysis results, information about how far the lowest point of the swing is from the ball in the front-back direction, which is difficult to visually confirm from videos or continuous photos taken by the user himself / herself.

[0343] (14) It is preferable that the analysis device has a function of storing or displaying at least one of the distance information and the trajectory. In this way, since the analysis device stores or displays the distance information and the trajectory, the user can also confirm, with the analysis device, the distance information and the trajectory that were numerically confirmed by the detection device during practice.

[0344] (15) It is preferable that a program for realizing the above functions in the system is provided. In this way, it is possible to provide swing analysis information to the user of the system according to each function.

[0345] (16) The detection device is a detection device that uses a sensor to perform swing detection, which is the detection of the swing of the club, and transmits swing detection information related to the swing detection to an analysis device that performs at least one of photographing at least one of the head of the club and the ball and swing analysis, which is the analysis of the swing, and acquires swing analysis information related to the swing analysis from the analysis device, and performs display based on the swing analysis information. In this way, the detection device transmits the swing detection information of the detected swing to the analysis device, acquires the swing analysis information of the detected swing, and performs display based on it. Thus, in cooperation with the analysis device, it is possible to provide the user with analysis results obtained by photographing the club or the ball, and since the detection device and the analysis device can be arranged at separate positions, it is possible to provide the user with more diverse usage modes of the detection device. Also, the user can use the detection device with more diverse information such as the analysis results by the analysis device displayed.

[0346] It is preferable to provide a program for causing a detection device to realize the function described in (16) above. In this way, it is possible to provide analysis information of the swing to the user of the detection device according to each function.

[0347] (18) An analysis device is an analysis device that performs at least one of photographing a club head and a ball and swing analysis of the swing of the club, and obtains swing detection information related to the swing detection from a detection device that performs swing detection of the swing using a sensor, starts the swing analysis using the photographed imaging information based on the obtained swing detection information, and transmits swing analysis information related to the swing analysis to the detection device. In this way, since the analysis device starts the swing analysis based on the obtained swing detection information and transmits the swing analysis information related to the swing analysis to the detection device, it is possible to provide the detection device with the analysis result obtained by photographing the club or the ball in cooperation with the detection device, and since the detection device and the analysis device can be arranged at separate positions, the user can arrange the analysis device at a position where more accurate swing analysis can be performed according to his / her play environment without worrying about the accuracy of swing detection.

[0348] (19) It is preferable to provide a program for causing an analysis device to realize the function described in (18) above. In this way, it is possible to provide analysis information of the swing to the user of the analysis device according to each function.

[0349] The inventions described in (1) to (14) above can be arbitrarily combined. For example, it may be configured to add at least a part of the configuration of at least one of the inventions after (2) to all or a part of the configuration of the invention described in (1). In particular, it is preferable to use an invention in which at least a part of the configuration of at least one of the inventions after (2) is added to the invention described in (1). Also, any configuration can be extracted from the inventions described in (1) to (14), and the extracted configurations can be combined. The applicant of this application intends to obtain rights for inventions including these configurations. Also, even if there is a description such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. These show examples of better configurations, and the applicant also intends to obtain rights for configurations that are not these cases or times. Also, the order in the descriptions with an order is not limited to this order. The applicant also discloses configurations in which some parts are deleted or the order is changed, and intends to obtain rights for them.

Explanation of Signs

[0350] 10…System, 20…Detection Terminal (Detection Device), 30…Analysis Terminal (Analysis Device), 40…User Terminal, 50…Management Terminal, 211…Swing Detection Module, 212…Swing Information Creation Module, 311…Shooting Module, 312…Swing Analysis Module

Claims

1. A detection device that uses a sensor to perform swing detection, which is the detection of the swing of a club, and An analysis device that performs at least one of photographing the head of the club and the ball, and swing analysis, which is the analysis of the swing. A system comprising the above.

2. The detection device transmits swing detection information related to the swing detection to the analysis device, The analysis device starts the swing analysis using the photographed imaging information based on the swing detection information. The system according to claim 1.

3. The analysis device transmits swing analysis information calculated by the swing analysis to the detection device, The detection device, or the detection device and the analysis device, perform a display based on the swing analysis information. The system according to claim 2.

4. The detection device displays a numerical value based on the swing analysis information, The analysis device displays at least one of a still image, a moving image, and a graph based on the swing analysis information. The system according to claim 3.

5. The analysis device Extracts analysis imaging information used for the swing analysis from the imaging information photographed before the acquisition of the swing detection information, Or Extracts analysis imaging information used for the swing analysis from the imaging information photographed before the acquisition of the swing detection information and the imaging information photographed after the acquisition of the swing detection information. From The system according to any one of claims 2 to 4.

6. The swing detection information indicates that the downswing of the golf swing has been detected. The system according to claim 5.

7. The analysis device performs a process of detecting the ball from the shooting information, and when the ball is detected, performs a process of detecting the head. When the process of detecting the ball is being performed, the analysis device displays the progress of the process of detecting the ball. When the process of detecting the head is being performed, the analysis device displays the progress of the process of detecting the head. The system according to claim 6.

8. When the head is not detected, the analysis device continuously displays that the head is not detectable until the next head detection process is started. The system according to claim 7.

9. The analysis device calculates the head coordinates, which are the coordinates of the head, and the ball coordinates, which are the coordinates of the ball, from the shooting information for analysis. The system according to claim 8.

10. The analysis device calculates the trajectory of the head based on the head coordinates. The system according to claim 9.

11. The analysis device calculates the lowest point coordinates, which are the coordinates of the lowest point of the trajectory, based on the calculated trajectory. The system according to claim 10.

12. The analysis device detects the inclination of its own device. The analysis device calculates the lowest point coordinates based on the trajectory of the swing corrected according to the detected inclination. The system according to claim 11.

13. The analysis device calculates distance information regarding the distance between the lowest point coordinates and the ball coordinates. The analysis device transmits the distance information to the detection device. The detection device displays the distance information. The system according to claim 12.

14. The analysis device has a function of storing or displaying at least one of the distance information and the trajectory. The system according to claim 13.

15. A program for causing a system to implement each function according to any one of Claims 1 to 4.

16. A detection device that uses a sensor to perform swing detection, which is detection of a club swing, transmits swing detection information related to the swing detection to an analysis device that performs at least one of photographing the club head and the ball and swing analysis of the swing, and acquires swing analysis information related to the swing analysis from the analysis device and performs display based on the swing analysis information.

17. A program for causing a detection device to implement the function according to Claim 16.

18. An analysis device that performs at least one of photographing the club head and the ball and swing analysis of the club swing, acquires swing detection information related to the swing detection from a detection device that uses a sensor to perform swing detection of the swing, starts the swing analysis using the photographed imaging information based on the acquired swing detection information, and transmits swing analysis information related to the swing analysis to the detection device.

19. A program for causing an analysis device to implement the function according to Claim 18.

Images