PROGRAM, DEVICE, SYSTEM, AND METHOD FOR PROPOSING TREATMENT FOR TREATMENT SUBJECT

Abstract

To provide a treatment proposing program capable of providing a proposal on a treatment to a treatment entity such as a player of a ball game performing the treatment for a treatment object such as a ball.SOLUTION: A treatment proposing program causes a computer to function as: treatment position estimation means for estimating a treatment position of a treatment for a treatment object using a treatment position estimation model trained by correct answer data on the treatment position, which is a position where the treatment is performed on the basis of at least one acquired information of movement information related to the movement or relative displacement of the treatment object and action information related to the action that has caused the movement or the relative displacement; treatment mode estimation means for estimating a treatment mode of the treatment using a treatment mode estimation model trained by correct answer data of the treatment mode, which is a mode for performing the treatment, on the basis of the estimated treatment position; and treatment proposal generation means for generating and outputting treatment proposal information related to the treatment position and the treatment mode of the treatment on the basis of the estimated treatment position and treatment mode.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for proposing treatment for a treatment target. [Background technology]

[0002] In recent years, research and development of AR (Augmented Reality) technology has been actively pursued. AR not only provides humans with information beyond reality, but is also effective in developing and improving human capabilities. For example, the application of AR technology is being promoted in various sports fields.

[0003] For example, Patent Document 1 discloses a technology that displays an arrow on the goal hoop indicating the target for shooting to a basketball player wearing an AR display. Furthermore, Patent Document 2 discloses a technology that uses a HUD (Head Up Display) to display the spot where a soccer player should kick the ball. This technology allows a player to see in his or her field of vision the direction in which the ball should be shot.

[0004] Furthermore, for example, Non-Patent Document 1 discloses a technology that uses a Kalman filter to predict the future trajectory of a moving ball and displays the predicted trajectory on an HMD (Head-Mounted Display), which allows the user to enhance their perception of the ball's movement.

[0005] Furthermore, for example, Non-Patent Document 2 discloses a technique for predicting the landing point on a ping-pong table of an oncoming ball using information on the posture of an opponent in ping-pong. Specifically, in this technique, a CNN (Convolutional Neural Network) model is used to generate 10 upper-body joint position images from an RGB image of the opponent, and then an LSTM (Long Short-Term Memory) model is used to predict the landing point of the ball. Finally, a mark is projected onto the predicted landing point on the ping-pong table by a projector installed on the ceiling.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] As described above, conventionally, technologies for predicting the future position and trajectory of a ball or the like and presenting the prediction information to a user have advanced. However, simply providing such prediction information is, for example, difficult to sufficiently contribute to the improvement of human capabilities.

[0009] For example, in ball games, even if the future position of the ball and the target position for scoring are presented using the technologies related to Patent Documents 1 and 2 and Non-Patent Documents 1 and 2, it does not necessarily enable the user to perform an optimal or suitable response action. That is, for example, even if the predicted trajectory and future position of the ball are displayed on the front display, usually, a user in training (especially a beginner-level competitor) cannot perform an optimal or suitable response to the ball.

[0010] Therefore, an object of the present invention is to provide a treatment proposal program, device, system, and method capable of providing a proposal related to the treatment to a treatment subject that performs treatment on a treatment target such as a ball, for example, a competitor in a ball game.

Means for Solving the Problems

[0011] According to the present invention, a treatment position estimation means is provided for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data for the treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to the movement or relative displacement of the treatment target and motion information related to the motion that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment suggestion program is provided that causes a computer to function as a

[0012] In the treatment suggestion program according to the present invention, the treatment position estimation means calculates an amount relating to a difference between an acquired subject treatment position, which is a treatment position of a treatment performed by a treatment subject on the treatment target, and an estimated treatment position, and newly estimates a treatment position of the treatment based on the amount relating to the difference; The treatment mode estimation means newly estimates a treatment mode of the treatment based on the newly estimated treatment position; It is also preferable that the treatment proposal generating means newly generates the treatment proposal information based on the newly estimated treatment position and treatment mode.

[0013] In one embodiment of the treatment suggestion program according to the present invention, the treatment position estimation model is trained using the correct answer data including the treatment time point, which is the time point at which the treatment is performed, and the treatment position estimation means also estimates the treatment time point of the treatment, It is also preferable that the treatment proposal generating means generates the treatment proposal information relating to the treatment time point of the treatment based on the estimated treatment time point as well.

[0014] Furthermore, in the above embodiment, the treatment position estimation means calculates an amount related to a difference between the acquired subject treatment time, which is the treatment time of the treatment performed by the treatment subject on the treatment target, and the estimated treatment time, and newly estimates the treatment position and treatment time of the treatment based on the amount related to the difference; The treatment mode estimation means newly estimates a treatment mode of the treatment based on the newly estimated treatment position; It is also preferable that the treatment proposal generating means newly generates the treatment proposal information based on the newly estimated treatment position, treatment time point, and treatment mode.

[0015] In another embodiment of the treatment suggestion program according to the present invention, the treatment mode estimation model is trained by the correct answer data including a treatment speed, which is a speed at which the treatment is performed, and the treatment mode estimation means estimates a treatment mode including the treatment speed of the treatment, It is also preferable that the treatment proposal generating means generates the treatment proposal information related to the treatment speed of the treatment based on the estimated treatment mode including the treatment speed of the treatment.

[0016] Furthermore, as yet another embodiment of the treatment suggestion program according to the present invention, the treatment mode estimation means calculates an amount relating to a difference between an acquired subject treatment mode, which is a treatment mode of a treatment performed by a treatment subject on the treatment target, and an estimated treatment mode, and newly estimates a treatment mode of the treatment based on the amount relating to the difference; It is also preferable that the treatment proposal generating means newly generates the treatment proposal information based on the newly estimated treatment mode.

[0017] In addition, in the above embodiment, it is also preferable that the treatment mode estimation means newly estimates the treatment mode of the treatment based on the acquired treatment result information, which is information related to the treatment target after the treatment performed by the treatment subject.

[0018] Furthermore, in the treatment suggestion program according to the present invention, it is also preferable that the treatment mode estimating means estimates the treatment mode of the treatment based on a subject pre-treatment mode, which is a mode of the treatment subject before performing the treatment.

[0019] Furthermore, as a specific example of treatment proposal information according to the present invention, it is also preferable that the generated treatment proposal information is at least one of information relating to an image displayed to the treatment subject, information relating to audio output to the treatment subject, and information sensed by the treatment subject's sense of touch.

[0020] Furthermore, as a specific example to which the present invention is applied, the processing target is a ball used in a ball game, and the action information is information related to an action of an opponent in the ball game with respect to the ball, The action is an action by the player taking action to affect the ball that has moved from the opponent, The manner of action is the position of the player or the position of the equipment used by the player when performing the action to affect the ball, It is also preferable that the treatment suggestion information is information relating to the treatment position, which is the position at which the ball should be acted upon, and the posture of the player or the posture of the tool that should be taken when performing the action to act upon the ball.

[0021] According to the present invention, there is also provided a treatment position estimation means for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data of a treatment position, which is a position where treatment is performed, based on at least one of movement information related to a movement or relative displacement of a treatment target and motion information related to a motion that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment suggestion device is provided having:

[0022] According to the present invention, there is further provided a treatment position estimation means for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data of a treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to a movement or relative displacement of a treatment target and movement information related to a movement that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment recommendation system is provided having:

[0023] According to the present invention, furthermore, a step of estimating a treatment position for the treatment target using a treatment position estimation model trained with correct data of the treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to the movement or relative displacement of the treatment target and motion information related to the motion that caused the movement or relative displacement; A step of estimating a treatment mode of the treatment based on the estimated treatment position using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A computer-implemented treatment suggestion method is provided, comprising: [Effects of the Invention]

[0024] According to the treatment proposal program, device, system, and method of the present invention, a proposal for treatment can be provided to a treatment subject who will perform treatment on a treatment target. [Brief explanation of the drawings]

[0025] [Figure 1]1 is a functional block diagram showing a functional configuration of an embodiment of a treatment suggestion device according to the present invention. [Figure 2] FIG. 1 is a schematic diagram for explaining an embodiment of a treatment position estimation model according to the present invention. [Figure 3] 10A and 10B are schematic diagrams for explaining specific examples of treatment proposal information according to the present invention. [Figure 4] 10A and 10B are schematic diagrams for explaining specific examples of treatment proposal information according to the present invention. [Figure 5] 10A and 10B are schematic diagrams for explaining another embodiment of the treatment proposal information generation process according to the present invention. [Figure 6] FIG. 10 is a schematic diagram for explaining yet another embodiment of the treatment proposal information generation process according to the present invention. [Figure 7] 1 is a flow chart that schematically illustrates one embodiment of a treatment suggestion method according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0027] [Treatment suggestion device] FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a treatment suggestion device according to the present invention.

[0028] The AR (Augmented Reality) glasses 1 shown in Figure 1, which are one embodiment of the treatment suggestion device according to the present invention, are worn by a player playing a ball game (table tennis in Figure 1) in this embodiment, and are a device that provides "treatment suggestion information" to the player.

[0029] Here, the provision of this "treatment proposal information" to this athlete may be, for example, (a) An image showing an "action position" where an action is to be taken on the table tennis ball 2, which is the "action target," for example, an image of the ball 2 showing the position where it should be hit with the racket 32, is displayed on the display 105 as augmented reality; (b) An image showing a "treatment mode" that is a mode of treatment at the above-mentioned "treatment position," for example, an image showing the posture of the racket 32 ​​that should be taken when hitting the ball 2, is displayed on the display 105 as augmented reality. This may also be the case.

[0030] Incidentally, the "treatment suggestion information" may consist only of an image showing the "treatment mode" (b) above (for example, an image showing the posture of the racket 32), but by also including an image showing the "treatment position" (a) above (for example, an image of the treatment target (ball 2)), more suitable suggestions can be made.

[0031] In this embodiment, the display 105 that displays the "treatment suggestion information" is a transparent display. A player wearing the AR glasses 1 can view the real competition environment, such as the ball 2, the opponent, and the table tennis table, through the display 105. Furthermore, the player can view virtual images (a) showing the "treatment position" (image of the ball 2) and (b) showing the "treatment manner" (image showing the posture of the racket 32) displayed on the display 105 within the field of view of this competition environment, and receive guidance related to the competition while playing.

[0032] Alternatively, the display 105 may be non-transparent and display a camera image of the playing environment generated by the camera 103, and may also display the virtual image described above in conjunction with this.

[0033] In any case, the AR glasses 1 (treatment suggestion device) perform the function of providing the "treatment suggestion information" described above, specifically, (A) A treatment position estimation unit 111 that estimates the "treatment position" of a treatment for the "treatment object" (ball 2 in FIG. 1 ) using a "treatment position estimation model" trained with correct answer data of the "treatment position" based on acquired movement information (e.g., time-series position data of ball 2) related to the movement of the "treatment object" (ball 2 in FIG. 1 ); (B) a treatment mode estimation unit 112 that estimates a "treatment mode" (e.g., a posture that the racket 32 ​​should take) of a treatment against a "treatment target" (ball 2) based on the estimated "treatment position" and using a "treatment mode estimation model" trained with correct answer data of the "treatment mode"; (C) a treatment proposal generating unit 113 that generates and outputs “treatment proposal information” (for example, the images (a) and (b) above) related to the “treatment position” and “treatment mode” of the treatment for the “treatment target” (ball 2) based on the estimated “treatment position” and “treatment mode”; It has the following characteristics.

[0034] In this way, the AR glasses 1 detect the "target" (e.g., ball 2) (A) "treatment position", for example, the position where the ball 2 to be treated should be hit, and (a) "Handling manner", for example, the posture that the player should take when handling the racket 32 By estimating this, it is possible to generate, output, and provide a proposal for the action (e.g., the images (a) and (b) above) to the action subject (e.g., the player) who will take action against the "action target" (ball 2) (e.g., the act of hitting ball 2 and returning it). This allows the player, in the example of Figure 1, to improve their returning skills in table tennis by referring to or using the "action proposal information" as a model.

[0035] Here, the movement information (A) above may be information relating to the relative displacement of the "target for treatment." For example, in a downhill skiing competition, if the "target for treatment" is a pole set up on the course, the movement information used to estimate the "treatment position," which is the position where the ski edge is used (the position where the curve begins), may be information relating to the relative displacement of the pole, which is the "target for treatment," such as time-series position (relative position) data of the pole in the athlete's field of vision or time-series data of the distance to the pole.

[0036] Furthermore, the information used to estimate the "treatment position" in (A) above is not limited to movement information related to the movement or relative displacement of the "treatment target." Specifically, the "treatment position" may be estimated using movement information (e.g., time-series position data of the racket 31) related to the movement that caused the movement or relative displacement (e.g., the movement of the opponent using the racket 31 in FIG. 1) alone or together with the movement information.

[0037] Furthermore, the "action suggestion information" in (C) above is not limited to information related to an image displayed to the action subject (the player wearing the AR glasses in FIG. 1), as described above. For example, it can also be information related to audio output to the action subject (the player), such as "(Ball 2) is coming low to the right!" output from the speaker 106 (via the output control unit 122).

[0038] It may also be information sensed by the tactile sense of the person providing treatment (the athlete), for example, a vibration pulse output from the vibration device 107 (via the output control unit 122) indicating that the treatment position is high (a suitable position for smashing). Furthermore, it is also preferable to provide information that combines the above-mentioned images, sounds, and vibration-related information as "treatment suggestion information."

[0039] Furthermore, the fields to which the AR glasses 1 can be applied are not limited to table tennis. For example, the AR glasses 1 can also be applied to ball games such as soccer, baseball, basketball, and volleyball. (a) The "subject to treatment" is a "ball" used in a ball game, (b) Movement information is information related to the movement of the "ball," and action information is information related to the opponent's action against the "ball," (c) The treatment is an action taken by the player wearing the AR Glasses 1 to affect the "ball" that has moved from the opponent. (d) The "treatment position" is the position where an action is to be taken on the "ball," and the "treatment manner" is the position that the player (wearing the AR glasses 1) should take, or the position that the player's equipment should take, when taking an action to affect the "ball." (e) "Proposed Action Information" is information relating to the position where an action should be taken on the "ball" (action position) and the position of the player (wearing AR Glasses 1) or the position of the equipment to be used (action position) when taking an action to take on the "ball." It is also preferable.

[0040] Furthermore, the fields in which the AR glasses 1 can be applied are not limited to ball games. That is, the AR glasses 1 can also be used in non-ball sports such as boxing (where, for example, the opponent's fist is the target) and various skiing and snowboarding competitions (where, for example, a pole on the course is the target). The AR glasses 1 can also be used as a training device in various training activities, such as training for handling machinery and heavy equipment, and flight simulation training, by appropriately setting the target. Furthermore, the AR glasses 1 can also be applied in performing arts fields such as theater and dance, where it is necessary to perform appropriate performances in accordance with the movements of other performers who are the target of treatment.

[0041] Furthermore, it is also possible to adopt an embodiment in which at least one of the above-mentioned (A) treatment position estimation unit 111 and the above-mentioned (B) treatment mode estimation unit 112, which are functional components of the AR glasses 1, is a functional component of another external device, such as an external server or cloud server, and estimation information is transmitted from this other device to the AR glasses having the above-mentioned (C) treatment proposal generation unit 113, for example, via a wireless communication network. In this case, the other device and the AR glasses having the treatment proposal generation unit 113 constitute a treatment proposal system according to the present invention.

[0042] Naturally, the treatment proposal device / system according to the present invention is not limited to a wearable terminal (configuration including) equipped with a display such as AR glasses. That is, depending on the field of application, it can also be a variety of display devices (configuration including) (transparent or non-transparent) installed in the environment where the treatment target or treatment subject is present. It can also be a simulator device (configuration including) equipped with a display. Furthermore, if the "treatment proposal information" is only audio information or tactile (vibration) information, it may be a device / system without a display (equipped with a speaker, vibration device, etc.).

[0043] [Device configuration, treatment proposal program and method] The functional configuration of the AR glasses 1 will be described in detail below. As also shown in the functional block diagram of Fig. 1, the AR glasses 1 of this embodiment include a communication interface (IF) 101, an acceleration / angular velocity sensor 102, a camera 103, a microphone 104, a display 105, a speaker 106, a vibration device 107, and a processor / memory. Here, the processor / memory stores an embodiment of a treatment suggestion program according to the present invention, has computer functions, and performs treatment suggestion processing by executing this treatment suggestion program.

[0044] Furthermore, for this reason, the treatment proposal device according to the present invention is not limited to the AR glasses (1) as described above, but can also be other display-equipped wearable terminals such as HMDs (Head-mounted Displays), various display devices such as HUDs (Head-Up Displays), and even simulator devices, which are equipped with one embodiment of the treatment proposal program according to the present invention.

[0045] The processor memory also includes, as functional components, a treatment position estimation unit 111, a treatment mode estimation unit 112, a treatment proposal generation unit 113, an input control unit 121, and an output control unit 122. These functional components can be considered to be functions implemented by an embodiment of a treatment proposal program according to the present invention stored in the processor memory. The processing flow shown by connecting the functional components of the AR glasses 1 (treatment proposal device) with arrows in the functional block diagram of FIG. 1 can also be understood as an embodiment of a treatment proposal method according to the present invention.

[0046] Hereinafter, the above-described functional components of this embodiment will be described in detail using the specific example shown in FIG. 1. In this specific example, a table tennis practice match is being held using a table tennis table installed in a table tennis court, and the target to be treated is a ball 2. The table tennis court is also equipped with at least one camera 4 that can capture images of (a) the ball 2 as the target to be treated, (b) the opponent (the player's arms, legs, head, etc.), (c) the racket 31 used by the opponent, (d) the player wearing the AR glasses 1 as the subject of treatment (the player's arms, legs, head, etc.), and (e) the racket 32 ​​used by the player, and can wirelessly transmit the captured image data to the AR glasses 1. Here, by applying known image recognition technology to this captured image data, it becomes possible to identify and specify the above items (a) to (e), determine the positions and velocities of the above items (a) to (e), and even track the positions of the above items (a) to (e). Furthermore, a microphone 5 is also installed that can collect various sounds during the table tennis practice match and transmit the audio data wirelessly to the AR glasses 1.

[0047] In this example, a contact sensor, such as a piezoelectric sensor, is provided between the racket body and the rubber of racket 31 and racket 32, capable of wirelessly transmitting the detection result of the event of hitting ball 2 to AR glasses 1. Incidentally, a racket equipped with such a piezoelectric sensor is disclosed in, for example, non-patent document: Takahiro Yamashita and Takeshi Kobayashi, “Smart Table Tennis Racket Using a Rubber Mounted Ultrathin Piezoelectric Sensor Array,” Sensors and Materials, Vol. 33, No. 3, pp. 1081-1089, 2021.

[0048] Furthermore, the rackets 31 and 32 are provided with an attitude sensor, such as an acceleration sensor and angular velocity (gyro) sensor unit, on the handle of the racket body, that can wirelessly transmit the detection results of the attitude and position (6 DoF) of the racket to the AR glasses 1. Incidentally, the AR glasses 1 of this specific example receive information from various sensors, including the camera 4 and microphone 5, via a wireless LAN (Local Area Network), Bluetooth, or the like, through a communication interface 101.

[0049] <Treatment position estimation means> Similarly, in the functional block diagram of FIG. 1, the treatment position estimation unit 111 (a) Obtain "time-series position data of ball 2," which is information on the movement of ball 2 after the opponent hits it with racket 31, (b) The acquired time-series position data is input into the “action position estimation model”, and as its output, “action position data” relating to the position where action should be taken (hit and return) on ball 2 is generated.

[0050] Here, the "time-series position data of ball 2" in (a) above can be generated, for example, from image data generated by camera 4 or camera 103 during the time a predetermined time has elapsed since the contact sensor of racket 31 detected the hit of ball 2, using a known image recognition process (with ball 2 as the recognition target). Also, the hit of ball 2 can be detected using a known voice recognition process from voice data generated by microphone 5 or microphone 104. Incidentally, in this embodiment, the generation process of the "time-series position data of ball 2", which includes such image recognition process and voice recognition process, is performed by input control unit 121.

[0051] Next, the above (b) "treatment position estimation model" will be described in detail with reference to Fig. 2. Fig. 2 is a schematic diagram for explaining one embodiment of the treatment position estimation model according to the present invention.

[0052] As shown in FIG. 2, the treatment position estimation model 111M of this embodiment is (a) Time-series position data {P t=1 , P t=2 , P t=3 , P t=4 , P t=5 an encoding unit 111Ma including an embedding layer that sequentially generates embedding representation vectors from each of the {} and a Long Short-Term Memory (LSTM) cell that sequentially takes in the generated embedding representation vectors and sequentially generates hidden state vectors that are internal representations of position information; (b) An LSTM cell that takes in the hidden state vector last output from the encoding unit 111Ma and uses the embedding vector (or null) from the previous point in time to sequentially generate an embedding vector of the estimated position, and from the generated embedding vector, the processing position data {P t=T^-2 , P t=T^-1 , P t=T^ , P t=T^+1 , P t=T^+2} and an output layer that sequentially outputs The number of pieces of input data (time-series position data) and output data (time-series treatment position data) (five in this embodiment) is set arbitrarily in advance.

[0053] Here, the action position estimation model 111M is trained by learning data including many pairs of actually measured time-series position data (x1, y1, z1) to (x5, y5, z5) of the ball 2 and actually measured hit position data (x, y, z, T), which is correct action position data. Here, the correct action position data (x, y, z, T) indicates that the actual hit of the ball 2 was made at position (x, y, z) at time T. That is, in this embodiment, the action position estimation model 111M uses correct data that also includes the action time (T), which is the time when the action is made, to output action position data {P t=T^-2 , P t=T^-1 , P t=T^ , P t=T^+1 , P t=T^+2}, the position data P at the middle point (the third point in this embodiment) t=T^ It is trained so that (x^, y^, z^, T^) is the estimated treatment location.

[0054] Therefore, the treatment position estimation unit 111 of this embodiment uses the position data P output from the treatment position estimation model 111M. t=T^ Based on (x^, y^, z^, T^), estimated treatment position data (x^, y^, z^) is generated.

[0055] <Treatment mode estimation means> Returning to the functional block diagram of Figure 1, the action mode estimation unit 112 of this embodiment inputs the action position data (x^, y^, z^) generated by the action position estimation unit 111 into the "action mode estimation model", and generates, as its output, "action mode data" relating to the mode to be taken when taking action on the ball 2 (hitting and returning the ball).

[0056] In the specific example of FIG. 1, the "treatment mode data" includes the posture (3DoF) that the racket 32 ​​should take and the position (x r , y r , z r ) can be used as posture and position (6DoF) data. It is also possible to adopt only posture (3DoF) data relating to the posture that the racket 32 ​​should take as the action mode data. The posture (3DoF) data can be, for example, angular coordinate data (θ r , θ p , θ y ) can be used.

[0057] Next, the above-mentioned "action mode estimation model" will be described. The "action mode estimation model" is a machine learning model that uses action position data (x^, y^, z^) as input (explanatory variables) and posture / position (6DoF) data as output (target variables). In this embodiment, initial position data of the racket 32, which is the state before the action is made by the player (action subject) wearing the AR glasses 1 (action subject), for example, position data of the racket 32 ​​at the time when the racket 31 hits the ball 2 (x ri , y ri , z ri ) are also used as inputs (explanatory variables) to the "treatment mode estimation model."

[0058] The "action mode estimation model" also includes posture and position (6DoF) data, as well as action speed data, which is the speed at which the player (action subject) wearing the AR glasses 1 performs the action, such as action speed data v r It is also preferable to output ^ as the objective variable.

[0059] Furthermore, since the "action mode estimation model" in this embodiment also uses the initial position data of the racket 32, which is the subject pre-action mode, as an input (explanatory variable), it is also preferable to train it to output predetermined "more specific action mode information" as a response variable. Here, this "more specific action mode information" can be information such as "smash" or "backhand" in the case of table tennis, as shown in the specific example of FIG. 1. In this case, the "action mode estimation model" is trained to output, for example, "backhand" if the initial position of the racket 32 ​​is on the right and the action position of the ball 2 is on the left, or to output, for example, "smash" if both the initial position of the racket 32 ​​and the action position of the ball 2 are on the right.

[0060] Furthermore, "more specific action mode information" may be information such as "guard" or "movement to dodge a punch" in the case of boxing, for example. In this case, the "action mode estimation model" is trained to output, for example, "guard" if the initial position of the boxer's (the person performing the action) glove and the action position of the opponent's fist are closer than a predetermined distance, or to output, for example, "movement to dodge a punch" if they are farther apart than a predetermined distance.

[0061] In any case, the "treatment mode estimation model" of this embodiment is configured by, for example, a fully-connected DNN (Fully-connected Deep Neural Network) algorithm, and is trained with learning data including many pairs of explanatory variable data and correct answer target variable data. Note that when the "treatment mode estimation model" also estimates (outputs) treatment speed data, the learning data used for the training includes treatment speed data when the treatment was actually performed as correct answer data.

[0062] The action mode estimation unit 112 of this embodiment generates the posture and position (6DoF) data that the racket 32 ​​should take, which is the estimated action mode data, based on the posture and position (6DoF) data output from such an "action mode estimation model." Alternatively, the posture (3DoF) data that the racket 32 ​​should take may be generated as the estimated action mode data.

[0063] Furthermore, when the "action mode estimation model" also estimates (outputs) action speed data, it is also preferable that the action mode estimation unit 112 generates action speed data as estimated action mode data, for example, action speed data relating to the speed that the racket 31 should take at the time of hitting the ball 2. Furthermore, when the "action mode estimation model" also estimates (outputs) "more specific action mode information" as described above, the action mode estimation unit 112 may generate "more specific action mode data" as estimated action mode data, for example, data relating to the specific mode that the player (subject of action) should take at the time of hitting the ball 2, such as a "smash" or a "backhand."

[0064] <Means for generating treatment proposals> Similarly, in the functional block diagram of FIG. 1, the treatment proposal generation unit 113 of this embodiment includes: (a) treatment position data estimated (generated) by the treatment position estimation unit 111, and (b) Treatment mode data (posture and position (6DoF) data, treatment speed data, etc.) estimated (generated) by the treatment mode estimation unit 112 Based on this, "treatment suggestion information" relating to the treatment position and treatment mode of the treatment is generated and output.

[0065] Here, a typical example of the treatment suggestion information to be generated is image information in which an "image of ball 2 or a mark indicating the treatment position" is placed at a position corresponding to the treatment position data in (a) above, and further, an "image showing the state of racket 32" performing treatment in a manner and position corresponding to the treatment manner data in (b) above is superimposed on this "image of ball 2 or mark indicating the treatment position."

[0066] In this embodiment, this action proposal information as image information is output to the display 105 via the output control unit 122, where it is displayed as a virtual reality (i.e., at a position in virtual space that corresponds to the position in real space to be displayed (as the destination of coordinate conversion)). Furthermore, the timing for providing (displaying) the action proposal information may be set to "promptly (as soon as possible)," for example, as soon as the action position data (a) above or the action mode data (b) above is generated. In this way, the action proposal information is provided promptly, allowing the athlete (the person performing the action) time to take action in accordance with the provided action proposal information.

[0067] As described above, when the treatment position estimation model is trained using correct data that also includes the treatment time point, which is the time point at which the treatment is performed, and the treatment position estimation unit 111 also estimates the treatment time point of the treatment, the treatment proposal generation unit 113 may generate treatment proposal information related to the treatment time point based on the estimated treatment time point.

[0068] For example, if the estimated time (which is the time of action) at which the racket 32 ​​should hit the ball 2 is earlier than a predetermined reference time by more than a predetermined amount, the action suggestion information may include, for example, a text message such as "Hurry!" displayed at the top of the screen of the display 105. It is also possible to use alarm sound information (output from the speaker 106) that indicates this message or vibration information (output from the vibration device 107) that indicates this message.

[0069] Furthermore, as described above, when the treatment mode estimation model is trained using correct data that also includes the treatment speed, which is the speed at which the treatment is performed, and the treatment mode estimation unit 112 estimates the treatment mode that also includes the treatment speed of the treatment, the treatment proposal generation unit 113 may generate treatment proposal information that also relates to the treatment speed based on the estimated treatment mode that also includes this treatment speed.

[0070] For example, if the estimated action speed (one of the action modes) to be taken when hitting the ball 2 with the racket 32 ​​is greater than a preset reference speed by a predetermined amount, the action suggestion information may include, for example, a text message such as "Smash!" or "Hard!" displayed at the top of the screen of the display 105. Alternatively, the action suggestion information may be alarm sound information (output from the speaker 106) that means this message, or vibration information (output from the vibration device 107) that means this message.

[0071] Furthermore, it is also preferable to present the bat swing speed (treatment speed) as a treatment mode (for example, displaying an image of a speed gauge indicating the speed) to a batter (treatment subject) wearing the AR glasses 1 during baseball batting practice as treatment suggestion information including the treatment speed. Here, for example, it is preferable to grasp the maximum actual speed of the bat swing acquired up to that point (for example, 27 m / sec), and if the speed of the bat swing (treatment speed) to be presented is, for example, 30 m / sec and exceeds this maximum value (27 m / sec), to present treatment suggestion information that sets this maximum value (27 m / sec) as the treatment speed to be taken in the bat swing. This makes it possible to avoid situations where the batter (treatment subject) attempts an unreasonable swing as much as possible.

[0072] Furthermore, as described above, when the action style estimation model is trained with ground truth data that also includes "more specific action style information," and the action style estimation unit 112 estimates an action style that also includes "more specific action style information," the action proposal generation unit 113 may generate action proposal information that includes a more specific action style, for example, a suggestion for a "smash" or a suggestion for a "backhand," based on the estimated action style that also includes this "more specific action style information." Here, for example, the presentation of action proposal information that includes a suggestion for a "smash" may be canceled (as it is difficult for the player to hit a smash) if the maximum value of the actual speed of the racket 32 ​​acquired up to that point is less than a predetermined threshold value (for example, 20 m / sec).

[0073] Figures 3 and 4 are schematic diagrams for explaining specific examples of action proposal information according to the present invention. The specific example in Figure 3 is an example of a table tennis practice match, similar to Figure 1, while the specific example in Figure 4 is an example of soccer goalkeeper practice.

[0074] First, in the specific example of a table tennis practice match shown in Figures 3(A) and (B), (a) After the opponent hits ball 2 with racket 31, the position of ball 2 is estimated from the movement information (time-series position data) of ball 2, (b) An “image of ball 2” is displayed on the screen of the display 105 of the AR glasses 1 (FIG. 1) at a position (within the screen space) corresponding to the estimated treatment position, (c) The posture and position (6 DoF) of the racket 32 ​​as the action mode is estimated from the estimated action position (and further from the measured initial position of the racket 32 ​​(FIG. 1)), (d) On the screen of the display 105 of the AR glasses 1, an "image of the racket 32" is displayed, which has a posture and position (within the screen space) that corresponds to the estimated posture and position (6DoF).

[0075] Here, the "image of ball 2" in (b) above and the "image of racket 32" in (d) above are "reaction guidance images" that serve as treatment suggestion information for the player (treatment subject) wearing the AR glasses 1. In fact, by looking at the "image of ball 2" in (b) above, the player can know in advance the position of the "image of ball 2" where the ball 2, which will soon be coming, should be hit. Also, by looking at the "image of racket 32" in (d) above, the player can know in advance what posture and position the racket 32 ​​should be in when swinging out the ball 2, which will be coming out at the position of the "image of ball 2." In other words, the player can see that the racket 32 ​​should be swinging out in a posture and position similar to that of the "image of racket 32."

[0076] Incidentally, in the above (a) of this specific example, instead of or in addition to the movement information (time-series position data) of the ball 2 after the hit, the action position of the ball 2 may be estimated using the motion information of the racket 31 before the hit (time-series position data of the racket 31). Specifically, for example, these time-series data as explanatory variables may be input in a predetermined order to the encoding unit 111Ma (FIG. 2) of the action position estimation model 111M, and the action position of the ball 2 as the objective variable may be extracted from the decoding unit 111Mb (FIG. 2).

[0077] Next, a specific example of a soccer goalkeeper's practice will be described using Figures 4(A) to (C). In this example, as shown in Figure 4(A), the goalkeeper, who is the subject of treatment, wears the AR glasses 1. Then, as shown in Figures 4(B) and (C), (a) The position where the ball 2 will be taken is estimated from the motion information (time-series position data) of the opponent's foot (for example, the toe or heel) before the opponent kicks the ball 2 as a shot. (b) An "image of ball 2" is displayed on the screen of the display 105 of the AR glasses 1 at a position (within the screen space) that corresponds to the estimated treatment position, and a sound indicating this display position, such as "Top right!", is output from the speaker 106 of the AR glasses 1 (FIG. 1). (c) From the estimated treatment position (and the measured initial position of the goalkeeper), the posture and position of the goalkeeper (6DoF) are estimated as the treatment mode, (d) On the screen of the display 105 of the AR glasses 1, an "image of the goalkeeper" is displayed in a posture and position (within the screen space) that corresponds to the estimated posture and position (6DoF).

[0078] Here, the "image of ball 2" and the audio, for example, "Top right!" in (b) above, as well as the "image of the goalkeeper" in (d) above, are response guidance images and audio as action suggestion information for the goalkeeper (action subject) wearing the AR glasses 1. In fact, by looking at the "image of ball 2" in (b) above and hearing the audio, for example, "Top right!" in (b) above, the goalkeeper can know in advance the position where he should stop ball 2, which will soon be coming towards him (the position where he should touch ball 2), from the position indicated by the "image of ball 2" and the audio. In addition, by looking at the "image of the goalkeeper" in (d) above, he can know in advance what posture and position he should take in relation to ball 2, which will be coming towards the position of the "image of ball 2." In other words, he knows that he should intercept ball 2 (go and touch ball 2) with a posture and position similar to that of the "image of the goalkeeper."

[0079] Incidentally, in the above (a) of this specific example, instead of or in addition to the movement information (time-series position data) of the opponent's foot (e.g., toe or heel) before the kick, the treatment position of the ball 2 may be estimated using movement information of the ball 2 after the kick (time-series position data of the ball 2). As a modification, the technology disclosed in Non-Patent Document 2 may be applied to estimate the treatment position of the ball 2 using movement information of the opponent's leg before the kick (time-series data of leg joint position images). Specifically, a CNN (Convolutional Neural Network) model is used to generate time-series data of leg joint position images from image data of the opponent moment by moment before the kick, and then an LSTM model is used to estimate the treatment position of the ball 2 from this time-series data.

[0080] [Embodiments for Coordinating Treatment Proposals] Hereinafter, another embodiment of the treatment proposal information generation process described above, specifically, an embodiment in which treatment proposal information is adjusted (modified) based on the content of the actual treatment performed by the treatment subject, will be described.

[0081] Returning to the functional block diagram of FIG. 1, the treatment position estimation unit 111 of this embodiment: (a) The acquired “subject action position” is the action position of the action taken by the player (action subject) wearing the AR glasses 1 on the ball 2 (action target) after receiving the action proposal information, (b) The treatment position previously estimated (which was the basis for the treatment proposal information in (a) above) It is also preferable to calculate a quantity related to the difference between the two, for example, RMSE (Root Mean Squared Error), and newly estimate the treatment position of the treatment based on this quantity related to the difference (RMSE).

[0082] In this case, the treatment mode estimation unit 112 newly estimates the treatment mode of the treatment based on this newly estimated treatment position, and then the treatment proposal generation unit 113 newly generates treatment proposal information based on these newly estimated treatment positions and treatment modes.

[0083] Furthermore, the treatment position estimation unit 111 of this embodiment (a) The acquired "time of subject action" is the time of the action taken by the player (subject of action) wearing the AR glasses 1 on the ball 2 (target of action) after receiving the action proposal information, (b) The treatment time previously estimated (which was the basis for the treatment proposal information in (a) above) It is also preferable to calculate a quantity related to the difference, for example, RMSE, and newly estimate the treatment position and treatment time point of the treatment based on this quantity related to the difference (RMSE).

[0084] In this case, the treatment mode estimation unit 112 newly estimates the treatment mode of the treatment based on this newly estimated treatment position, and then the treatment proposal generation unit 113 newly generates treatment proposal information based on these newly estimated treatment position, treatment time point, and treatment mode.

[0085] Furthermore, the treatment mode estimation unit 112 of this embodiment (a) The acquired “subject action mode” is the action mode of the action taken by the player (action subject) wearing the AR glasses 1 on the ball 2 (action target) after receiving the action proposal information; (b) The previously estimated treatment mode (the basis for the treatment proposal information in (a) above) It is also preferable to calculate a quantity related to the difference between the two, for example, RMSE, and newly estimate the treatment mode of the treatment based on this quantity related to the difference (RMSE). In this case, the treatment proposal generator 113 newly generates treatment proposal information based on this newly estimated treatment mode as well.

[0086] Furthermore, it is also preferable that the action mode estimation unit 112 of this embodiment newly estimates the action mode of the action based on the acquired “action result information,” which is information related to the ball 2 (action target) after the action taken by the player (action subject) wearing the AR glasses 1. In this case, the action proposal generation unit 113 newly generates action proposal information based on this newly estimated action mode.

[0087] 5 and 6 are schematic diagrams for explaining another embodiment of the treatment proposal information generation process described above.

[0088] First, according to FIG. 5(A), a reaction guidance image (images of the ball 2 and racket 32) is displayed as action suggestion information to an athlete (subject to take action) wearing AR glasses 1 (FIG. 1), and the athlete (subject to take action) sees this and uses it as reference to swing the racket 32 ​​to hit the ball 2. Here, the displayed reaction guidance image is an image at t=36 (=T^). Meanwhile, the actual position where the ball 2 was hit (subject action position) is behind (closer to the subject to take action) the position of this reaction guidance image. Therefore, the actual time when the ball 2 was hit (subject action time) is also a time later than t=36 (=T^).

[0089] Also in FIG. 5(B), a reaction guidance image (images of the ball 2 and racket 32) is displayed to the player (subject to take action) as action suggestion information, and the player (subject to take action) sees this and uses it as reference to swing the racket 32 ​​to hit the ball 2. Here, the displayed reaction guidance image is an image at t=40 (=T^). On the other hand, the actual position where the ball 2 was hit (subject action position) is forward (closer to the opponent) than the position of this reaction guidance image. Therefore, the actual time when the ball 2 was hit (subject action time) is also a time before t=40 (=T^).

[0090] In this way, depending on the balance between what the reaction guidance image requires, the player's actual ability and condition, and the player's intended return shot, it is quite possible that a difference will arise between the action position and action time presented by the reaction guidance image and the actual position (main action position) and time (main action time) at which the ball 2 is hit. Therefore, in this embodiment, the following equation is used: (1) RMSE=(Σ i=1 n (Y i -Y^ i ) 2 / n) 0.5 The RMSE, which is a quantity related to this difference, is calculated using the above formula, and the difference is compensated for using this RMSE.

[0091] In the above formula (1), n ​​is the number of times that a pair of the presentation of the response guidance image and the action (response) performed in response to the image was performed, and Y^ i is the time point related to the i-th reaction guidance image, and Y i is the time point (subject action time) related to the action (reaction) by the athlete (action subject) who saw the i-th reaction guidance image.

[0092] The treatment position estimation unit 111 (FIG. 1) of this embodiment newly estimates the treatment position (and in this embodiment, the treatment time point) for generating the (n+1)th reaction guidance image based on the RMSE value calculated for the presentation of n reaction guidance images. For example, if the RMSE value is 2 and (Y i -Y^ i ) is a positive value (when the actual reaction is slower than (time = )2 on average), the treatment position estimation unit 111 outputs the third output P (default) in the decoding unit 111Mb (of the treatment position estimation model 111M) (FIG. 2). t=T^ Instead, the 5th (=3+2) output P t=T^+2 is extracted and used as the new (corrected) estimate of the treatment position and treatment time.

[0093] In this case, for example, if the (default) estimated value at the time of treatment is 36 (= T^), as shown in Figure 5(C), the reaction guidance image at time t = 38 (= 36 + 2) (the treatment position of ball 2 is P t=T^+2 An image in which the position of the target is displayed is displayed. The player (subject to the action) who tends to be slow in reaction by about (time = )2 will be more likely to see this and move the racket 32 ​​more appropriately. In this case, it is also preferable to calculate the RMSE value for the past n actions from the n+2th action onwards and estimate a new (corrected) action position and time in the same manner as above.

[0094] As a modification, it is also possible to estimate a new (corrected) treatment position and treatment time point in the same manner as above, based on the amount of difference (RMSE) between the treatment positions. In this case, Y^ in the above equation (1) can be used to estimate the new (corrected) treatment position and treatment time point. i is the y-coordinate value of the treatment position (the position coordinate axis in the direction from the other party to the treatment subject is the y-coordinate axis) of the treatment position related to the i-th reaction guidance image, and Y i can be set as the y coordinate value of the subject treatment position of the athlete (treatment subject) who viewed the i-th reaction guidance image.

[0095] Next, an embodiment will be described in which a new (modified) treatment mode is estimated based on the measure of difference (RMSE) between treatment modes.

[0096] First, as shown in Figure 6(A), a reaction guidance image (image of racket 32) is displayed as action suggestion information to the player (subject to treatment) wearing the AR glasses 1 (Figure 1), and the player (subject to treatment) sees this and, using it as a reference, swings out the racket 32 ​​to hit the ball 2. Here, the posture of the racket 32 ​​in the reaction guidance image differs from the posture of the racket 32 ​​when it is actually swinging out, and as a result, the ball 2 hit by the player (subject to treatment) actually goes outside the court of the table tennis table. In other words, the presentation of guidance can be said to be "successful" because the ball was returned, but the actual return shot was "inaccurate" because it went outside the court.

[0097] In this way, depending on the balance between the content required by the reaction guidance image and the player's actual ability and condition, as well as the content of the return shot intended by the player, it is quite possible that a difference will arise between the action mode presented by the reaction guidance image and the action mode (the actual posture of the racket 32, the main action mode) when the ball 2 is actually hit. Therefore, in this embodiment, the following equation is used: (2) RMSE pose =(Σ i=1 n (Z i -Z^ i ) 2 / n) 0.5 Using the RMSE (RMSE pose ) and use the RMSE to compensate for this difference.

[0098] In the above formula (2), n is the number of times that the response guidance image was presented and the action (response) was taken after seeing it, and Z^ i is the posture information of the racket 32 ​​in the i-th reaction guidance image, for example, the roll angle θ of the racket 32 ​​(about the axis extending from the handle to the tip of the racket) r^ value. Also Z i is the actual posture information of the racket 32 ​​that is deployed by the player (action subject) who has seen the i-th reaction guidance image, for example, the roll angle θ (about the axis extending from the handle to the tip of the racket) r It can be a value.

[0099] The treatment mode estimation unit 112 (FIG. 1) of this embodiment calculates the RMSE calculated as above. pose Based on the value, a new (modified) action mode (e.g., the roll angle θ of the racket 32) is determined. r For example, the roll angle θ of the racket 32 ​​in the presented reaction guidance image is estimated. r Z value k As the above (Z i -Z^ i ) is positive and the actual roll angle θ of the racket 32 r If is too large, for example, (3) Z k+1 =Z k -RMSE pose / 2 The roll angle θ of the racket 32 ​​in the reaction guidance image presented next is r Z k+1 Determine.

[0100] Then, the determined Z k+1 If the information (action result information) regarding ball 2 (target of action) after the player (action subject) who saw the reaction guidance image regarding ball 2 returns the ball (after action is taken) is "accurate" (ball 2 has entered the court), this Z k+1 When a predetermined number of correct data have been collected in this way, the correct data are used to sequentially train the "treatment mode estimation model" used by the treatment mode estimation unit 112.

[0101] This successively trained "action style estimation model" makes it possible to estimate a new action style that takes into account the tendencies of the player's (the player making the action) actual action style. This also makes it possible, for example as shown in Figure 6(B), for the posture of the racket 32 ​​in the newly displayed reaction guidance image (action suggestion information) to roughly match the actual posture of the racket 32, thereby increasing the likelihood that the actual return shot will land in the opponent's court, i.e., the return shot will be "accurate."

[0102] Incidentally, the above-mentioned action result information for the ball 2 hit by the player (action subject), such as information that "ball 2 has gone out of the court," may be generated, for example, when a sound recognized as "the sound of ball 2 bouncing on the court" is not collected by microphone 5 (FIG. 1) or microphone 104 (FIG. 1) within a predetermined time after the hit event of racket 32 ​​is detected by the piezoelectric sensor. Conversely, if such a sound is collected within the predetermined time, action result information that "ball 2 has entered the court" can be generated.

[0103] In the above specific examples, the actual posture of the racket 32 ​​is used as the actual action mode of the player (action subject), but for example, the actual posture of the goalkeeper himself may be used as the actual action mode of the soccer goalkeeper described in FIG. 4. In this case, the actual posture of the goalkeeper himself can be measured by, for example, the acceleration / angular velocity sensor 102 (FIG. 1), which is a posture sensor. Furthermore, whether it is the posture of the racket 32 ​​or the posture of the goalkeeper, these postures can also be determined by, for example, analyzing image data captured by an externally installed camera 4 using known image recognition technology.

[0104] Furthermore, it is also possible to use the speed of the racket 32 ​​as the actual action style of the player (action subject). Here, the action style estimated by the "action style estimation model" includes speed information (of the racket 32), and the displayed reaction guidance image (action suggestion information) may include this speed information (for example, an image of a speed gauge indicating the magnitude of the relevant speed).

[0105] In this case, Z^ in the above equation (2) i is the speed related to the speed information of the i-th reaction guidance image, and Z i is the actual speed of the racket 32 ​​sent out by the player (treatment subject) who has seen the i-th reaction guidance image, and RMSE as a quantity related to the difference in speed. pose This also makes it possible to calculate the speed of the displayed reaction guidance image (action suggestion information) so that it roughly matches the actual speed of the racket 32, in the same way as above, and thereby increase the likelihood that the actual return shot will land in the opponent's court, i.e., the return shot will be "accurate."

[0106] [How to propose treatment] 7 is a flowchart illustrating an embodiment of a treatment suggestion method according to the present invention. Specifically, this embodiment generates new treatment suggestion information, i.e., "updated treatment suggestion information," using the actual treatment position and treatment mode of the subject.

[0107] (S11) The initial movement information of the object to be processed (for example, ball 2) is acquired. (S12) Based on the acquired initial movement information, the treatment position of the treatment target is estimated. (S13) Based on the estimated treatment position, a treatment mode for the treatment target is estimated. (S14) Based on the estimated treatment position and treatment mode, treatment proposal information for the treatment target is generated and presented. (S15) The actual treatment position (subject treatment position) of the treatment target by the treatment subject (for example, the athlete) who received the treatment proposal information is obtained. (S16) The actual treatment mode (subject treatment mode) of the treatment subject that received the treatment proposal information for the treatment target is acquired.

[0108] (S17) It is determined whether the number of times that treatment proposal information has been presented, and therefore steps S11 to S16 have been performed, is less than a predetermined number of times. If it is less than the predetermined number of times, the process returns to step S11, and steps S11 to S16 are repeated. On the other hand, if it has reached the predetermined number of times, the process proceeds to the next step S21.

[0109] (S21) The initial movement information of the treatment target is obtained. (S22) A new treatment position of the treatment target is estimated based on the acquired initial movement information and the RMSE values ​​of the subject treatment positions acquired a predetermined number of times (most recently). (S23) RMSE relating to the estimated new treatment position and the subject treatment state for a predetermined number of times (recent) acquired so far pose Based on the values, a new treatment mode for the treatment target is estimated. (S24) Based on the estimated new treatment position and new treatment mode, "updated treatment proposal information" for the treatment target is generated and presented. (S25) The actual processing position of the processing target (subject processing position) by the processing subject that received the "update processing proposal information" is acquired. (S26) The actual treatment mode (subject treatment mode) for the treatment target by the treatment subject that received the "update treatment proposal information" is acquired.

[0110] (S27) It is determined whether the number of times "update action proposal information" has been presented, and therefore steps S21 to S26 have been performed, is less than a predetermined number of times. If it is less than the predetermined number of times, the process returns to step S21, and steps S21 to S26 are repeated. On the other hand, if it has reached the predetermined number of times, the process of generating and presenting action proposal information is terminated.

[0111] As explained in detail above, according to the present invention, by estimating the position and manner of treatment for an object to be treated (for example, a ball in a ball game), it is possible to generate, output, and provide a proposal for the treatment, i.e., treatment proposal information, to an entity (for example, a player) who will be performing a treatment for the object to be treated (ball) (for example, the act of hitting the ball). Furthermore, if the present invention is applied to a ball game, for example, a player of the ball game can improve his or her ability in the ball game by referring to or using the provided treatment proposal information as a model.

[0112] Furthermore, by applying the treatment proposal technology of the present invention to sports movements classes at schools or private sports schools, training sessions for movements in professional sports, and even work practices or training sessions involving movements, it is possible to encourage more children and adults to improve their movements in sports or work.In other words, the present invention can contribute to achieving Goal 4 of the United Nations Sustainable Development Goals (SDGs), "Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all," and Goal 8, "Promote inclusive and sustainable economic growth, employment and decent work for all."

[0113] With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical spirit and perspective of the present invention may be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents. [Explanation of symbols]

[0114] 1. AR glasses (treatment suggestion device) 101 Communication Interface (IF) 102 Acceleration and angular velocity sensor 103, 4 camera 104, 5 microphones 105 Display 106 Speaker 107 Vibration Device 111 Treatment position estimation unit 112 Treatment mode estimation unit 113 Treatment proposal generation unit 121 Input control section 122 Output control section 2 Balls (treatment subject) 31, 32 rackets

Claims

1. a treatment position estimation means for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data of a treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to a movement or relative displacement of the treatment target and motion information related to a motion that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment suggestion program that causes a computer to function as follows.

2. the treatment position estimation means calculates a difference between an acquired subject treatment position, which is a treatment position of a treatment performed by a treatment subject on the treatment target, and an estimated treatment position, and newly estimates a treatment position of the treatment based on the difference; The treatment mode estimation means newly estimates a treatment mode of the treatment based on the newly estimated treatment position, The treatment proposal generating means generates new treatment proposal information based on the newly estimated treatment position and treatment mode. The treatment suggestion program according to claim 1 .

3. The treatment position estimation model is trained by the correct answer data including the treatment time point, which is the time point at which the treatment is performed, and the treatment position estimation means also estimates the treatment time point of the treatment, The treatment proposal generating means generates the treatment proposal information related to the treatment time point of the treatment based on the estimated treatment time point. The treatment suggestion program according to claim 1 .

4. The treatment position estimation means calculates a difference between an acquired subject treatment time, which is a treatment time of a treatment performed by a treatment subject on the treatment target, and an estimated treatment time, and newly estimates a treatment position and treatment time of the treatment based on the difference. The treatment mode estimation means newly estimates a treatment mode of the treatment based on the newly estimated treatment position, The treatment proposal generating means generates new treatment proposal information based on the newly estimated treatment position, treatment time point, and treatment mode.

4. The treatment suggestion program according to claim 3.

5. The treatment mode estimation model is trained by the correct answer data including a treatment speed, which is a speed at which the treatment is performed, and the treatment mode estimation means estimates a treatment mode including the treatment speed of the treatment, The treatment proposal generating means generates the treatment proposal information related to the treatment speed of the treatment based on the estimated treatment mode including the treatment speed of the treatment. The treatment suggestion program according to claim 1 .

6. The treatment mode estimation means calculates a difference between an acquired subject treatment mode, which is a treatment mode of a treatment performed by a treatment subject on the treatment target, and an estimated treatment mode, and newly estimates the treatment mode of the treatment based on the difference. The treatment proposal generating means generates new treatment proposal information based on the newly estimated treatment mode.

6. The treatment suggestion program according to claim 1, wherein the treatment suggestion program is a program for suggesting a treatment to be performed by a user.

7. The treatment suggestion program according to claim 6, characterized in that the treatment mode estimation means newly estimates the treatment mode of the treatment based on acquired treatment result information, which is information related to the treatment target after the treatment performed by the treatment subject.

8. 6. The treatment suggestion program according to claim 1, wherein the treatment mode estimation means estimates the treatment mode of the treatment based also on a subject pre-treatment mode, which is a mode of the treatment subject before performing the treatment.

9. A treatment proposal program as described in any one of claims 1 to 5, characterized in that the treatment proposal information generated is at least one of information related to an image displayed to the treatment subject, information related to audio output to the treatment subject, and information sensed by the treatment subject's sense of touch.

10. the processing target is a ball used in a ball game, and the action information is information related to an action of an opponent in the ball game with respect to the ball; The action is an action by the player taking action to affect the ball that has moved from the opponent, The manner of action is the position of the player or the position of the equipment used by the player when performing the action to affect the ball, The proposed action information is information relating to the action position, which is the position where the action should be applied to the ball, and the player's posture or the posture of the tool that should be taken when performing the action to apply the action to the ball.

6. The treatment suggestion program according to claim 1, wherein the treatment suggestion program is a program for suggesting a treatment to be performed by a user.

11. a treatment position estimation means for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data of a treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to a movement or relative displacement of the treatment target and motion information related to a motion that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment suggestion device comprising:

12. a treatment position estimation means for estimating a treatment position for a treatment target using a treatment position estimation model trained with correct data of a treatment position, which is a position where treatment is performed, based on at least one of acquired movement information related to a movement or relative displacement of the treatment target and motion information related to a motion that caused the movement or relative displacement; a treatment mode estimation means for estimating a treatment mode of the treatment based on the estimated treatment position, using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; a treatment proposal generating means for generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; A treatment suggestion system comprising:

13. A step of estimating a treatment position for the treatment target using a treatment position estimation model trained with correct data of the treatment position, which is the position where the treatment is performed, based on at least one of acquired movement information related to the movement or relative displacement of the treatment target and motion information related to the motion that caused the movement or relative displacement; A step of estimating a treatment mode of the treatment based on the estimated treatment position using a treatment mode estimation model trained with correct answer data of the treatment mode, which is a mode of performing the treatment; generating and outputting treatment proposal information relating to the treatment position and treatment mode of the treatment based on the estimated treatment position and treatment mode; 10. A computer-implemented method for treatment suggestion, comprising:

Images