Information processing device, information processing method and program

Abstract

To provide an information processing device, information process method, and program for improving convenience and practicality of golf swing support.SOLUTION: A target direction receiving unit 221 receives a flight direction of a ball position before a user's golf swing in order to define it as a target direction. An image data processing unit 223 specifies a position of the target direction in a captured image corresponding to image data acquired from a head device 100 as an imaging device by an image data acquisition unit 222. A support information generation unit 224 generates positional relation information between the user's toes and a club face as information for supporting the user's golf swing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Systems for assisting golf swings are known. For example, Patent Document 1 discloses a golf swing analysis system that uses a photographing device such as a digital video camera to photograph a player making a golf swing and compares the downward swing angle of the club head with a reference angle, thereby enabling the player to grasp the correct swing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7011030 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 requires the user to adjust the position and tilt of the device before recording the swing, which makes it less convenient and practical for users who actually carry the device around golf grounds.

[0005] The present invention has been made in consideration of the above-described circumstances, and has an object to provide an information processing device, an information processing method, and a program that improve the convenience and practicality of golf swing assistance. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the information processing device of the present invention comprises a receiving means for receiving the flight direction of the ball position before a user makes a golf swing, a specifying means for specifying the position of a target direction in an image taken by an imaging device carried by the user, corresponding to the flight direction of the ball position received by the receiving means, and a generating means for generating positional relationship information between the user's toes and the club face based on the position of the target direction specified by the specifying means. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the convenience and practicality of golf swing assistance. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a head device and an external device according to an embodiment. [Figure 2] 1A and 1B are diagrams showing examples of how the head device according to the embodiment is worn; [Figure 3] 10A and 10B are diagrams illustrating an example of the operation of a projection unit in the head device according to the embodiment. [Figure 4] 10 is a flowchart illustrating an example of support information projection processing according to the embodiment. [Figure 5] 10A and 10B are diagrams showing an example of a target direction setting screen according to an embodiment, respectively; [Figure 6] FIG. 10 is a diagram showing an example of an address time image according to the embodiment; [Figure 7] 10A and 10B are diagrams illustrating examples of projection of appropriate positions according to an embodiment. [Figure 8] FIG. 10 is a diagram showing an example in which a difference between a target position and a fly ball position is set according to a modified example. [Figure 9] 10A and 10B are diagrams illustrating a plurality of phases in a golf swing according to a modified example. [Figure 10] FIG. 10 is a diagram showing an example of a graph display according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, a golf swing support system including a head device 100 and an external device 200 is configured as an embodiment of the present invention. The head device 100 is worn on the head of a player, and functions as an example of an imaging device carried by the user, a projection device worn by the user, and a terminal device combining these. For example, the head device 100 may be worn on the side of the head of a player performing a golf swing, and may be worn in any desired manner, such as ear hook type, eyeglass type, hat type, headset type, headband type, integrated into a hat type, or any other suitable manner. The head device 100 shown in FIG. 1 includes a control unit 111, a storage unit 112, a communication unit 113, an imaging unit 114, and a projection unit 115. These units are connected via a bus line BL1.

[0010] The control unit 111 can be configured to include a processor such as a CPU (Central Processing Unit). The processor may be configured to include an MPU (Micro-Processing Unit), an MCU (Micro-Controller Unit), or one or both of them instead of or in addition to a CPU. The control unit 111 executes various programs in the head device 100, enabling software-based information processing to be specifically realized using hardware. The storage unit 112 can be configured to include, for example, a semiconductor memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a flash memory, or any other storage device. The storage unit 112 stores various programs executed by the processor of the control unit 111 in the head device 100, data used for various processes, and the like.

[0011] The communication unit 113 can be configured using a communication integrated circuit (IC) or the like that can transmit and receive radio frequency signals. The communication unit 113 may be configured to include an antenna, an amplifier circuit, a tuner, a coder and decoder (CODEC), etc. For example, the communication unit 113 may be capable of communicating with a short-range wireless communication device, a mobile communication network, a wireless local area network (LAN), or any other device or system capable of wireless communication via a telecommunications network or transmission medium, and may be capable of transmitting and receiving wireless signals in accordance with a predetermined communication protocol.

[0012] The imaging unit 114 is configured with one or more cameras capable of capturing images of the line of sight of the player wearing the head device 100 and the target direction, which is the ideal ball flight direction resulting from the golf swing. As an example, the imaging unit 114 can be configured to include a first camera 114A capable of capturing images of the user's line of sight and a second camera 114B capable of capturing images in a direction perpendicular to the first camera. In this case, the first camera 114A has a first angle of view capable of capturing images of the user's feet, including the ball position, when the user addresses the golf swing. The second camera 114B can capture images of the left side of the user wearing the head device 100, i.e., the direction of the ball, when the user addresses the golf swing. The second camera 114B can capture images of the right side of the user wearing the head device 100, i.e., the direction of the ball, when the user wears the head device 100, is a right-handed player. As a result, the second camera 114B has a second angle of view capable of capturing images of the front area of ​​the golf course, including the target direction where the ball will be hit, when the user addresses the golf swing.

[0013] The projection unit 115 may be configured to include a laser pointer or projector, or other optical equipment capable of projecting any light beam or image. For example, a projection unit 115 configured using a line laser laser pointer includes an actuator that allows it to rotate in a predetermined direction relative to the main body of the head device 100. In this case, on the main body of the head device 100, the direction from the focal point of the first camera 114A toward the lens is defined as the X axis, the direction from the lens of the second camera 114B toward the focal point is defined as the Z axis, and the direction perpendicular to these is defined as the Y axis. The laser pointer output device is attached so as to be rotatable around the Y axis and the Z axis.

[0014] The external device 200 is preferably an information processing device that can be carried by the same user as the head device 100 and has higher processing performance than the head device 100. For example, the external device 200 may be a smartphone, a tablet terminal, a wearable terminal, or any other information processing device. The external device 200 shown in FIG. 1 includes a control unit 211, a storage unit 212, a communication unit 213, an input unit 214, and an output unit 215. These units are connected via a bus line BL2.

[0015] The control unit 211 can be configured to include a processor similar to that of the control unit 111. It is desirable that the processor of the control unit 211 has higher processing performance than the processor of the control unit 111. The processor of the control unit 211 may be configured to include, in addition to a CPU, any of a GPU (Graphics Processing Unit), a VDP (Video Display Processor), or any other DSP (Digital Signal Processor), thereby enabling high-speed image processing. The control unit 211 executes various programs in the external device 200, thereby enabling information processing by software to be specifically realized using hardware. The storage unit 212 can be configured to include a storage device similar to that of the storage unit 112. The storage unit 212 stores various programs executed by the processor of the control unit 211 in the external device 200, data used for various processes, and the like.

[0016] The communication unit 213 can be configured using a communication IC similar to that of the communication unit 113. The communication unit 213 may perform wireless or wired communication with the communication unit 113 of the head device 100 and transmit image data, control data, and the like. The communication unit 213 also has a GPS (Global Positioning System) signal acquisition function. The communication unit 213 can receive GPS signals, which are radio waves emitted from GPS satellites, and generate location information indicating the current location of the external device 200. The communication unit 213 may be capable of receiving radio waves corresponding to a single GPS technology as the GPS signal, or may be capable of receiving radio waves corresponding to multiple GNSS (Global Navigation Satellite System) technologies.

[0017] The input unit 214 can be configured using an input unit that detects actions that become input operations by the user of the external device 200 and supplies input information corresponding to the detection results to the control unit 211. For example, the input unit 214 may be configured to include a touch panel and may be capable of detecting actions that become various touch operations by the user. The input unit 214 may also be configured using a push button that can detect a press operation by the user. The input unit 214 may be capable of inputting image data generated by capturing an image of the periphery of the external device 200 using a camera to the control unit 211 and storing the image data in the storage unit 212.

[0018] The output unit 215 can be configured using an output unit that outputs various types of information, etc. For example, the output unit 215 may be configured to output information by display using a display unit such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0019] Next, a description will be given of the functional configuration of the control unit 211 included in the external device 200. The control unit 211 reads out a control program stored in the ROM of the storage unit 212 and uses the RAM as a work area, thereby functioning as a target direction receiving unit 221, an image data acquisition unit 222, an image data processing unit 223, a support information generation unit 224, and a control command supply unit 225.

[0020] The target direction receiving unit 221 receives the flight direction of the ball position before the user's golf swing in order to determine it as the target direction. In this case, the camera of the input unit 214 or the first camera 114A or the second camera 114B of the imaging unit 114 provided in the head device 100 captures an image of the ball's expected flight area, including the flight direction to be determined as the target direction, and the target direction can be received using the acquired image data. When determining the target direction from the image data, image matching using feature points is performed. More specifically, when selecting feature points, image matching can be performed based not only on the direction but also on the center point of the expected flight direction and feature points surrounding it, to determine the direction. The target direction receiving unit 221 is an example of a receiving means.

[0021] The image data acquisition unit 222 controls the communication unit 213 to acquire image data representing a captured image from the head device 100 as an imaging device. The captured image from the head device 100 may include one or both of a first image captured by the first camera 114A of the imaging unit 114 and a second image captured by the second camera 114B of the imaging unit 114. In other words, it is sufficient that the image data to be acquired can be selected according to the processing content. The image data acquisition unit 222 supplies the first image and the second image corresponding to the image data acquired from the head device 100 to the image data processing unit 223.

[0022] The image data processing unit 223 performs any image recognition processing including part or all of the following, such as pattern matching using a template, shape detection using edge detection or Hough transform, image matching and object recognition using a shift-invariant algorithm such as SIFT (Scale-Invariant Feature Transform), using the image data acquired by the image data acquisition unit 222. In addition, the image data processing unit 223 may perform any image processing such as image analysis, editing, processing, and generation using various types of image data.

[0023] For example, the image data processing unit 223 identifies multiple types of positions corresponding to the address determination before the golf swing, such as the ball position, the club face position, and the user's toe position in the first image captured by the first camera 114A, among the captured images corresponding to the image data acquired from the head device 100. The image data processing unit 223 also identifies a position corresponding to the flight direction of the ball position accepted before the golf swing, such as the position of the target direction in the second image captured by the second camera 114B, among the captured images corresponding to the image data acquired from the head device 100. The image data processing unit 223 is an example of an identifying means.

[0024] The support information generation unit 224 generates information such as positional relationship information between the user's toes and the club face as information to support the user's golf swing. In this case, the positional relationship information is generated based on the position of the target direction in the second image identified by the image recognition processing of the image data processing unit 223. The storage unit 212 stores material data for generating the positional relationship information corresponding to the position of the target direction in the second image identified by the image recognition processing of the image data processing unit 223. The support information generation unit 224 generates positional relationship information including a projection image by combining and processing the material data corresponding to the position of the target direction. The support information generation unit 224 is an example of a generation means.

[0025] The control command supply unit 225 supplies control commands for various operations to the head device 100 based on the information generated by the support information generation unit 224. For example, a control command for projecting the appropriate positions of the user's toes and the club face corresponding to the positional relationship information generated by the support information generation unit 224 is supplied to the head device 100 by communication using the communication unit 213. This control command may include image data indicating the projection image. In response to the control command supplied from the external device 200, the control unit 111 of the head device 100 controls a laser pointer or the like of the projection unit 115 to project a position support image including a projection image indicating the appropriate position onto the ground including the areas around the positions of the user's toes, the ball, and the golf club.

[0026] In the example of wearing the head device 100 shown in Fig. 2, the first camera 114A of the imaging unit 114 has a first angle of view corresponding to the user's line of sight when the user (player) addresses the ball before swinging, and is able to capture an image of the area around the user's feet. At this time, the second camera 114B of the imaging unit 114 is arranged in a direction perpendicular to the first camera 114A and has a second angle of view, and is able to capture an image of the area in front of the golf course, which is the expected ball flight area after the golf swing. In addition, the projection unit 115 projects a projection image onto the ground surface including the area around the user's feet, thereby performing a projection operation corresponding to golf swing support information, such as the appropriate positions of the user (player's) toes and club face, and the target direction of the ball.

[0027] 3, for example, the laser pointer of the projection unit 115 is rotatably driven by an actuator. In this case, the laser pointer of the projection unit 115 only needs to be rotatable within a first rotation range around the Y axis indicated by arrow A01 and a second rotation range around the Z axis indicated by arrow A02. Note that in the main body of the head device 100, the long side direction is the X axis, the short side direction is the Y axis, and the target direction to which the ball is hit is the Z axis.

[0028] Next, with reference to FIG. 4, the support information projection process according to the embodiment of the present invention will be described. The support information projection process is initiated when the head device 100 and the external device 200 are able to communicate with each other. First, the target direction of the ball to be hit by the user's golf swing is accepted (step S101). This target direction is determined based on the flight direction of the ball position before the user's golf swing. Before starting the golf swing, the user sets the flight direction to be used as the target direction of the ball position on a target direction accepting screen such as that shown in FIG. 5. The target direction accepting screen accepts the flight direction to be used as the target direction using image data acquired by imaging. In this case, examples of acquiring image data by imaging may include a first acquisition example using the first camera 114A of the imaging unit 114 included in the head device 100, a second acquisition example using the second camera 114B of the imaging unit 114 included in the head device 100, and a third acquisition example using the camera of the input unit 214 included in the external device 200, or some or all of these examples.

[0029] In a first acquisition example, the user faces the front of the golf course, and image data captured by the first camera 114A corresponding to the user's line of sight is transmitted from the communication unit 113 to the external device 200. In a second acquisition example, the user faces the ball at the address position where he or she will perform a golf swing, and image data captured by the second camera 114B corresponding to the user's left side is transmitted from the communication unit 113 to the external device 200. In these first and second acquisition examples, the image data acquired through communication by the communication unit 213 in the external device 200 is supplied to the control unit 211. In a third acquisition example, the user holds the external device 200, for example, and points the camera of the input unit 214 toward the front of the golf course. At this time, the image data captured by the camera of the input unit 214 is input to the control unit 211.

[0030] These image data are acquired by the image data acquisition unit 222 of the control unit 211. The image data processing unit 223 of the control unit 211 performs image processing using the image data acquired by the image data acquisition unit 222, thereby causing the display unit of the output unit 215 to display a target direction acceptance screen including the target area image that is the imaging result. At this time, an icon display C01 for setting the target direction and a confirmation button display C02 for completing the setting of the target direction are added along with the golf course image display B01 shown in FIG. 5(A). As shown in FIG. 5(B), it is sufficient that the user performs an input operation using a finger D01 on a touch panel or the like at the input unit 214 of the external device 200, and can change the position of the icon display C01 by sliding the icon display C01, for example.

[0031] When the target direction is indicated by the icon display C01, setting of the target direction is completed in response to a tap operation of the confirmation button display C02, for example. Instead of using the confirmation button display C02, setting of the target direction may be completed when a press operation of a push button provided on the head device 100 is detected, or when a nodding action of the user wearing the head device 100 is detected from the image capture results of the imaging unit 114. Instead of when a nodding action of the user is detected, setting of the target direction may be completed when a predetermined action as a target direction setting completion action is detected. When setting of the target direction is completed, the target direction receiving unit 221 of the control unit 211 supplies image data including the golf course image display B01 and the icon display C01 to the image data processing unit 223 as received data. In this way, the flight direction of the ball position before the user's golf swing is received to be used as the target direction.

[0032] After receiving the target direction of the ball position in step S101 of FIG. 4, an address motion determination is performed (step S102). The address motion determination is a determination of whether or not a motion to determine an address has been performed when the user makes a golf swing. In this case, the control unit 211 of the external device 200 acquires image data representing a captured image from the first camera 114A of the imaging unit 114 provided in the head device 100 using the image data acquisition unit 222. The image data processing unit 223 determines whether or not an address motion has been performed using the image data acquired by the image data acquisition unit 222. For example, the image data processing unit 223 determines that an address motion has been performed when the user's toes, the ball, and the club face are stationary and a preset stationary determination time, such as one second, has elapsed. Alternatively, the image data processing unit 223 determines that an address motion has been performed when a motion of hitting the ground twice with the club face is detected. Alternatively, the address motion may be determined to have been performed when any preset address determination motion is detected.

[0033] Following step S102, image data at address is acquired (step S103). At this time, the control unit 211 of the external device 200 acquires image data representing captured images from the first camera 114A and the second camera 114B of the imaging unit 114 provided in the head device 100, via the image data acquisition unit 222. The image data processing unit 223 uses the image data acquired by the image data acquisition unit 222 to determine whether or not the assistance condition is met. The assistance condition includes a first assistance condition corresponding to the first image and a second assistance condition corresponding to the second image. The first assistance condition is met when the user's toes, the ball, and the club face can be detected within the imaging range of the first image captured by the first camera 114A. Automatic detection using object recognition may be used to detect these by image recognition. Alternatively, stickers corresponding to the toes, the ball, and the club face may be attached, and it may be detected that the color or pattern of the stickers in the image is met.

[0034] The second assistance condition is met when the target direction received in step S101 is included within the capture range of the second image captured by the second camera 114B. This can be detected by image recognition by comparing the image data acquired in step S103 with the image data acquired in step S101 to detect identical or similar edges. For example, the address-time image G01 shown in FIG. 6 includes a predetermined tilt corresponding to the user's address motion before a golf swing. The image data processing unit 223 may detect the target position included within the capture range of the second image by image matching, including tilt adjustment of the second image. Furthermore, if the ball target direction is received in step S101 using the first camera 114A in the first acquisition example or the camera in the input unit 214 of the external device 200 in the third acquisition example, the image may be cropped around the icon display C01, and the target position may be detected and set by image matching, including tilt adjustment of the second image.

[0035] In response to the establishment of the support condition in step S103, an appropriate position is estimated (step S104). In the control unit 211 of the external device 200, the image data processing unit 223 estimates the appropriate position based on the image data acquired by the image data acquisition unit 222 in step S103. Using this estimation result, the support information generation unit 224 generates positional relationship information that serves as support information for the golf swing. In this case, matching is performed between the first image captured by the first camera 114A and the second image captured by the second camera 114B. For example, the image data processing unit 223 uses the image data from the second camera 114B among the image data acquired in step S103 to identify the position of the ball flight direction, which is the target direction, in the second image. The first camera 114A is positioned in a specific positional relationship with the second camera 114B, in a direction perpendicular to the second camera 114B. This specific positional relationship allows a straight line L10 corresponding to the target direction to be estimated and drawn within the first image captured by the first camera 114A. At this time, the straight line L10 is superimposed so as to pass through the center position of the ball in the first image, thereby identifying the position of the target direction in the first image.

[0036] In this embodiment, the correct position is achieved when the club face is perpendicular to the target direction. In this case, in the first image captured by the first camera 114A, a line L11 parallel to the line L10 is drawn with the toe of the left foot as a reference point for the user's toes, which were also detected in step S103. In the case of a left-handed player, a line L11 parallel to the line L10 is drawn with the toe of the right foot as a reference point. This estimates the correct position of the user's toes. Then, in the first image captured by the first camera 114A, a line L12 perpendicular to the lines L10 and L11 is drawn from the center position of the club face, which was also detected in step S103. This estimates the correct orientation of the club face. The support information generator 224 can generate positional relationship information between the user's toes and the club face, which are identified through these estimates.

[0037] After the estimation in step S104, the appropriate positions are projected (step S105). In the control unit 211 of the external device 200, the control command supply unit 225 generates a control command for projecting the appropriate positions of the user's toes and the club face in accordance with the positional relationship information generated by the support information generation unit 224 in step S104. The generated control command is supplied to the head device 100 by communication using the communication unit 213. The control unit 111 of the head device 100 controls the projection unit 115 in accordance with the control command to project, onto the ground, the positional relationship information including the positions of the user's toes and the ball, the direction of the club face, and a projection image showing the appropriate positions thereof at the address of the golf swing, thereby visualizing the positional relationship information. After the positional support information is generated in step S104, it may be immediately projected in accordance with the generated control command.

[0038] The projection example of the appropriate position shown in FIG. 7 includes the toe position P01 of the user's left foot, the toe position P02 of the user's right foot, the ball position P10, and the club face position P11. In relation to these, a straight line L10 corresponding to the target direction is projected to pass through the center of the ball at the ball position P10. A straight line L11 parallel to the straight line L10 is projected to pass through the toe position P01 of the user's left foot. A straight line L12 perpendicular to the straight lines L10 and L11 is projected to pass through the center of the club face at the club face position P11. Since the straight line L12 is perpendicular to the straight line L10 corresponding to the target direction, the club face and the straight line L12 are parallel, i.e., the club face and the target direction are perpendicular, resulting in the appropriate position.

[0039] Known methods for visualizing the proper position at address in a golf swing include placing an alignment stick on the ground or attaching it to a golf club or the user's body, or performing swing analysis after a golf shot using a camera or other method. These methods require installation and retrieval, making them difficult to implement while playing on the golf field. Furthermore, post-shot swing analysis cannot indicate the proper position at address in a golf swing. In contrast, the head device 100 and external device 200 of the present invention eliminate the need for installation and retrieval, and can easily indicate the proper position at address while playing on the golf field.

[0040] In step S104 of the support information projection process shown in FIG. 4 , when the target direction is estimated using the line L10 in the first image, the support information generation unit 224 may configure a first adjustment unit that adjusts the positional relationship information between the user's toes and the club face by using pre-stored flight information to calculate, as a statistical quantity, the difference between the target position set by the user's input operation and the flight position, which is the ball position after the user's golf swing. The difference statistical process that calculates and records the difference between the target position and the flight position may, for example, read image data pre-stored in the storage unit 212 of the external device 200, display a corresponding golf course map image on the display unit of the output unit 215, and obtain the coordinates of the ball target position. Furthermore, the control unit 211 of the external device 200 obtains position information at the time of target setting using a GPS signal received via the communication unit 213. Having set the ball target position, the user performs a golf swing at that position and hits the ball. After the shot, the user moves to the flight position. When the player moves to the fly ball position, position information after the actual hit fly ball is acquired using the GPS signal received by the communication unit 213. In this case, the difference in the flight distance, directionality, etc. is calculated based on the position information at the time of target setting and the setting contents of the hit ball target position, and the calculation results are stored in the storage unit 212 as fly ball information.

[0041] In a golf course map M01 shown in FIG. 8 , a line segment T01 corresponding to the distance to a ball target position S01 is set, and a line segment T02 corresponding to the distance to an actual hit ball position S02, which indicates the ball's position after the user's golf swing, is acquired. Then, a difference E0 in the directionality between the line segments T01 and T02 can be acquired. The distance calculation also takes into account elevation differences. If measurable wind direction and wind speed are available, they may also be reflected. If the line segment T01 corresponding to the ball target position S01 is set in a direction that includes obstacles such as trees or a pond, the image data processing unit 223 may present a detour ball route relative to the target direction on a display unit of the output unit 215. The support information generating unit 224 may also adjust the positional relationship information between the user's toes and the club face based on the detour ball route.

[0042] The imaging unit 114 of the head device 100 may continuously acquire images of the user during the golf swing, and estimate the head position. This position estimation process for estimating the head position is initiated in response to a determination similar to step S102 in FIG. 4 when the head device 100 and the external device 200 are in a state where they can communicate with each other, and image data at the address of the golf swing is acquired using the first camera 114A of the imaging unit 114 of the head device 100. At this time, the position, size, etc. of the ball are recorded as the initial position in the first image acquired by the first camera 114A, for example. The position of the ball is recorded for detecting displacement in the left / right and up / down directions. The size of the ball is recorded for detecting depth.

[0043] As shown in FIG. 9 , to estimate the head position by the position estimation process, images corresponding to multiple phases X01 to X10 of the user's golf swing are continuously acquired. Of these, the first phase X01 is the stage at which the address motion is performed. The second phase X02 is the stage at which the backswing starts and the golf club shaft becomes horizontal. The third phase X03 is the stage at which the backswing progresses and the right arm becomes horizontal. The fourth phase X04 is the stage at which the backswing reaches the top position and is completed. The fifth phase X05 is the stage at which the downswing starts and the right arm becomes horizontal. The sixth phase X06 is the stage at which the downswing progresses and the golf club shaft becomes horizontal. The seventh phase X07 is the impact stage at which the club face hits the ball. The eighth phase X08 is the stage at which the follow-through starts and the golf club shaft becomes horizontal. The ninth phase X09 is the stage at which the follow-through progresses and the left arm becomes horizontal. The tenth phase X10 is the finish stage at which the entire golf swing is completed.

[0044] Using images acquired corresponding to these multiple phases X01 to X10, the position and size of the ball relative to its initial position are calculated, and the amount of head position movement is estimated at each stage. The estimation results may be combined with flight ball information to calculate the relationship between the user's head position and the flight ball direction. The flight ball direction may be the difference between the target position set by the user's input operation and the ball position after the user's golf swing. For example, the relationship between the head angle H0 at impact in the seventh phase X07 and the difference E0 corresponding to the difference between the target direction and the flight ball direction may be calculated and illustrated in a graph. The example graph display Y01 shown in FIG. 10 presents the calculation results of the head angle H0 and the difference E0 by combining the estimation results of the amount of head position movement and the flight ball information.

[0045] The relationships for the other stages may be similarly identified, and the relationship between the timing at which the head angle H0 begins to change and the difference E0 or flight distance may be calculated and illustrated in a graph. Positional relationship information between the user's toes and the club face may be adjusted based on the relationship between the user's head position and the ball flight direction or difference E0. In the control unit 211 of the external device 200, the image data acquisition unit 222 may be an example of an image acquisition means that continuously acquires images corresponding to multiple phases of the user's golf swing. The image data processing unit 223 may be an example of a first estimation means that estimates the user's head position using the images continuously acquired by the image data acquisition unit 222. The support information generation unit 224 may be an example of a first presentation means that calculates the relationship between the user's head position estimated by the image data processing unit 223 and the ball flight direction or difference E0, and presents the calculation result on a display unit of the output unit 215, for example. In addition, the support information generation unit 224 may be an example of a second adjustment means that adjusts the positional relationship information between the user's toes and the club face based on the relationship between the user's head position estimated by the image data processing unit 223 and the ball flight direction or difference E0.

[0046] The imaging unit 114 of the head device 100 may continuously capture images of the user's golf swing to estimate the club trajectory, which is the path of the golf club. The trajectory estimation process for estimating the club trajectory begins in the same manner as the position estimation process. Image data at address is acquired using the first camera 114A of the imaging unit 114 of the head device 100. At this time, the club face is detected as the initial position of the golf club using object recognition or a Hough transform. This detection result is recorded along with the initial position of the position estimation process. Next, similar to the position estimation process, images corresponding to multiple phases X01 to X10 are continuously captured. Using these images, the amount of change in head position is estimated, and external parameters are used to compensate for the position change of the images due to the movement of the head position. In this case, alignment is performed using the impact time of the seventh phase X07 as a reference. The trajectory of the golf club head is then estimated using the inter-frame difference between each image, etc. The estimation result may be combined with flight information to calculate the relationship between the club trajectory and the flight direction of the ball. For example, the relationship between the estimated club trajectory and the difference E0 corresponding to the deviation between the target direction and the ball flight direction may be calculated and graphed. The positional relationship information between the user's toes and the club face may be adjusted based on the relationship between the club trajectory and the ball flight direction or the difference E0. The club trajectory may be estimated using consecutive images taken before and after the impact in the seventh phase X07, including the impact, to estimate the ball's impact position, speed, lateral angle, and club face angle and speed. For example, the club trajectory near the impact in the seventh phase X07 may be estimated by calculating and interpolating the center of a detection rectangular region for detecting the shape of the club head. In the detection rectangular region, the upper contour of the club face, which is the top line of the club head, and the lower contour of the club face, which is the leading edge, may be detected using edge detection, Hough transform, etc. to estimate the angle of the club face.

[0047] In the control unit 211 of the external device 200, the image data processing unit 223 may be an example of a second estimation means that estimates a club trajectory, including a trajectory of the head of the golf club, using images corresponding to multiple phases of the user's golf swing that are continuously acquired by the image data acquisition unit 222. The support information generation unit 224 may be an example of a second presentation means that calculates the relationship between the club trajectory estimated by the image data processing unit 223 and the ball flight direction or the difference E0, and presents the calculation result on a display unit or the like of the output unit 215. The support information generation unit 224 may also be an example of a third adjustment means that adjusts positional relationship information between the user's toes and the club face based on the relationship between the club trajectory estimated by the image data processing unit 223 and the ball flight direction or the difference E0.

[0048] The present invention requires the user to wear the head device 100, and does not require positioning of the external device 200, so it can visualize the correct address posture relative to the target and present the trajectories of the club and ball at impact, without any limitations on usage. Furthermore, the system presents an appropriate address position using flight ball information indicating the difference between the hit target position and the actual hit flight ball position, and calculates the relationship with the flight ball direction from the estimated results of the head position and club trajectory during the golf swing. This makes it easier to recognize trends and areas for improvement in golf swings, which tend to vary from person to person, and provides appropriate support for the user's golf swing.

[0049] The imaging unit 114 included in the head device 100 is not limited to being configured to include the first camera 114A and the second camera 114B, and may be configured with a single camera having a wide angle of view, such as a camera capable of panoramic photography using a fisheye lens. The positional relationship between the first camera 114A and the second camera 114B is not limited to being orthogonal to each other, and may be any positional relationship that is preset as a specific positional relationship.

[0050] When estimating the appropriate position in step S104 of the support information projection processing shown in FIG. 4, any conditions, such as the type of golf club, the distance to the target position, the lie of the ball, the inclination of the address position, the wind direction, the wind speed, the trajectory of the hit ball (slice, fade, draw, hook), or some or all of these, may be set, and the estimation result may be adjusted in response to each condition. In the example of the projection of the appropriate position shown in FIG. 7, the line L11 does not have to be parallel to the line L10, and the line L12 does not have to be perpendicular to the lines L10 and L11. The estimation result may be adjusted in response to each condition. The line L11, which is an example of the second line, may be set to be parallel to the line L10, which is an example of the first line, or to have a first angle other than 0° with the line L10, which is an example of the first line. The line L12, which is an example of the third line, may be set to be perpendicular to the line L10, which is an example of the first line, and the line L11, which is an example of the second line, or to have a second angle other than 90° with the line L10, which is an example of the first line, and the line L11, which is an example of the second line.

[0051] Recognizing shape and position using image data, and estimating and determining relationships between directions, positions, and other factors may be performed in part or in whole using a trained model based on machine learning. The trained model may be obtained by machine learning using any machine learning model. The trained model is generated by machine learning of information about any golf swing, including, for example, a target direction, a golf club type, a flight distance, a ball flight position, and captured images received through a user's input operation. By supplying input information, including the target direction and captured images received through a user's input operation, to such a trained model, it may be possible to generate positional relationship information between the user's toes and the club face.

[0052] In the above embodiment, high processing performance is required for processing image data, etc., so the external device 200 is used in addition to the head device 100. On the other hand, if sufficient processing performance can be obtained, the functions of the external device 200 may be integrated into the head device 100. In this way, the functions of the head device 100 and the external device 200 may be realized as a single information processing device, or as multiple information processing devices connected to each other so as to be able to communicate with each other. The image data, material data, calculation result data, and estimation result data may be stored in their entirety in the storage unit 212 of the external device 200, or may be stored in part or in their entirety in an external storage device via a telecommunications network and may be obtainable by communication processing using the communication unit 213.

[0053] The head device 100 and the external device 200 are not limited to those in which the control units 111, 211 execute the control programs stored in the memory units 112, 212, and for example, the functional configuration realized by some or all of the control programs may be realized by hardware using an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other electrical circuits.

[0054] At least a part of the information processing devices constituting the head device 100 and the external device 200 may be realized as a processing unit provided on a cloud as a computer using a processor. For example, in the control unit 211 of the external device 200, part or all of the image processing by the image data processing unit 223 is not limited to being realized by the control unit 211 executing a control program stored in the storage unit 212, but may be realized as a program itself or as a recording medium on which the program is recorded.

[0055] The above embodiments are illustrative, and the present invention is not limited to these. Various embodiments are possible within the scope of the invention as defined in the claims. The components described in the embodiments and variations can be freely combined. Furthermore, inventions equivalent to the inventions defined in the claims are also included in the present invention. In addition, even if the components of the invention defined in the claims have the same names as the components described in the above embodiments, they are not limited to the components described in the above embodiments themselves, and can be modified or applied as appropriate. [Explanation of symbols]

[0056] 100...head device, 111, 211...control unit, 112, 212...storage unit, 113, 213...communication unit, 114...imaging unit, 114A...first camera, 114B...second camera, 115...projection unit, 200...external device, 214...input unit, 215...output unit, 221...target direction receiving unit, 222...image data acquisition unit, 223...image data processing unit, 224...support information generation unit, 225...control command supply unit, P01, P02...toe position, P10...ball position, P11...club face position, L10 to L12...straight line

Claims

1. A receiving means for receiving a flight direction of a ball position before a user's golf swing; an identification means for identifying a position of a target direction in an image captured by an imaging device carried by the user, in accordance with the flight direction of the ball position accepted by the acceptance means; generating means for generating positional relationship information between the user's toes and a club face based on the position of the target direction identified by the identifying means; Information processing device.

2. a supply means for supplying a control command corresponding to the positional relationship information generated by the generation means to a projection device worn by the user, to project a position assistance image onto the ground. The information processing device according to claim 1 .

3. the specifying means obtains, from a terminal device in which the imaging device is combined with the projection device, a first image captured by a first imaging unit that captures an image in the line of sight of the user and a second image captured by a second imaging unit that is in a specific positional relationship with the first imaging unit, and specifies the position of the target direction in the second image; The information processing device according to claim 2 .

4. The identification means After specifying the position of the target direction in the second image, a first line, which is the target direction estimated based on the specified positional relationship, is superimposed in the first image so as to pass through the center of the ball position; In the first image, using one of the toes of the user as a reference, estimating a correct position of the toe of the user using a second straight line that is parallel to the first straight line or has a first angle with the first straight line; In the first image, a correct orientation of the club face is estimated based on a third line that is perpendicular to the first line and the second line or has a second angle with the first line and the second line at a center position of the club face; the generating means generates positional relationship information between the toes of the user identified by the estimation by the identifying means and the club face. The information processing device according to claim 3 .

5. a first adjusting means for adjusting positional relationship information between the user's toes and the club face by calculating, as a statistical quantity, a difference between a target position set by the user's input operation and a ball position after the user's golf swing, using previously accumulated flight ball information; The information processing device according to claim 1 .

6. a first estimation means for estimating a head position of the user using images continuously acquired corresponding to a plurality of phases during the user's golf swing; a first presentation means for calculating a relationship between the estimation result by the first estimation means and the difference and presenting the calculation result; and second adjustment means for adjusting positional relationship information between the user's toes and the club face based on the relationship between the estimation result by the first estimation means and the difference. The information processing device according to claim 5 .

7. a second estimation means for estimating a trajectory of a head of a golf club using images continuously acquired corresponding to a plurality of phases during the user's golf swing; a second presentation means for calculating a relationship between the estimation result by the second estimation means and the difference and presenting the calculation result; and third adjustment means for adjusting positional relationship information between the user's toes and the club face based on the relationship between the estimation result by the second estimation means and the difference. The information processing device according to claim 5 .

8. An information processing method by an information processing device, a receiving step of receiving a flight direction of a ball position before a user's golf swing; a specifying step of specifying a position of a target direction in an image captured by an imaging device carried by the user, corresponding to the flight direction of the ball position accepted in the accepting step; a generating step of generating positional relationship information between the user's toes and a club face based on the position of the target direction identified in the identifying step. Information processing methods.

9. Computer, a receiving means for receiving a flight direction of a ball position before a user's golf swing; an identification means for identifying a position of a target direction in an image captured by an imaging device carried by the user, in accordance with the flight direction of the ball position accepted by the acceptance means; a generating means for generating positional relationship information between the user's toes and the club face based on the position of the target direction identified by the identifying means; A program that functions as a

Images