Information processing device, learned model generation device, method, and program

Abstract

To easily estimate an intention of a player's action from the player's action observed while sports are being played.SOLUTION: An information processing device 24 acquires action information indicating a player' action while sports are being played. The information processing device 24 generates information indicating an intention of the player's action by inputting the player's action information to a learned model that outputs information indicating an intention of the player' action when action information is input. The information processing device 24 outputs information indicating the intention of the player's action.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a trained model generation device, a method, and a program. [Background technology]

[0002] Conventionally, a motion analysis method capable of teaching the launch direction of a ball is known (for example, see Patent Document 1). This motion analysis method includes an acquisition step of acquiring a target direction of the ball, an estimation step of estimating the launch direction of the ball from the state of the golf club or the golfer, and a comparison information generation step of comparing the target direction and the launch direction to generate comparison information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-46987 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a person's behavior is observed, the behavior may reflect some intention. For example, a person's behavior while playing a sport may reflect some intention. For this reason, there are cases where we want to know what intention a person had in their behavior while playing a sport.

[0005] However, the technology disclosed in Patent Document 1 only estimates the direction of a hit ball from the state of the golf club or the golfer, and does not take into consideration the estimation of the intention of the action.

[0006] The present disclosure has been made in consideration of the above points, and aims to easily estimate the intention of a player's action from the player's action observed while a sport is being played. [Means for solving the problem]

[0007] In order to achieve the above-mentioned objective, the information processing device of the present disclosure is an information processing device that includes an acquisition unit that acquires behavioral information representing a player's behavior while a sport is being played, a generation unit that generates information representing the player's behavioral intention corresponding to the behavioral information acquired by the acquisition unit by inputting the behavioral information acquired by the acquisition unit into a trained model that outputs information representing the player's behavioral intention when the behavioral information is input, and an output unit that outputs the information representing the player's behavioral intention generated by the generation unit.

[0008] In addition, the trained model generation device of the present disclosure is a trained model generation device that includes a training acquisition unit that acquires training data in which behavioral information representing the behavior of a training player and information representing the intention of the training player's behavior while a sport is being played are associated with each other, and a learning unit that generates a trained model that outputs information representing the intention of the training player's behavior when the behavioral information is input, by machine learning a training model based on the training data acquired by the training acquisition unit.

[0009] In addition, the information processing method disclosed herein is an information processing method in which a computer executes a process to acquire behavioral information representing a player's behavior while a sport is being played, input the acquired behavioral information to a trained model that outputs information representing the player's behavioral intention when the behavioral information is input, thereby generating information representing the player's behavioral intention corresponding to the acquired behavioral information, and outputting the generated information representing the player's behavioral intention.

[0010] In addition, the trained model generation method disclosed herein is a trained model generation method in which a computer executes a process to acquire training data in which actions representing the actions of a training player while a sport is being played are associated with information representing the intention of the training player's actions, and a training model is machine-learned based on the acquired training data to generate a trained model that outputs information representing the intention of the player's actions when the action information is input.

[0011] In addition, the information processing program disclosed herein is an information processing program for causing a computer to execute a process of acquiring behavioral information representing a player's behavior while a sport is being played, inputting the acquired behavioral information into a trained model that outputs information representing the player's behavioral intention when the behavioral information is input, thereby generating information representing the player's behavioral intention corresponding to the acquired behavioral information, and outputting the generated information representing the player's behavioral intention.

[0012] In addition, the trained model generation program of the present disclosure is a trained model generation program for causing a computer to execute a process of acquiring training data in which behavioral information representing the behavior of a training player and information representing the intention of the training player's behavior while a sport is being played are associated, and a training model is machine-learned based on the acquired training data, thereby generating a trained model that outputs information representing the intention of the training player's behavior when the behavioral information is input. Effect of the Invention

[0013] According to the information processing device, trained model generation device, method, and program disclosed herein, the intention of a player's actions can be easily estimated from the player's actions observed while a sport is being played. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining an overview of the present embodiment. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of the trained model generation device according to the present embodiment. [Diagram 3] FIG. 1 is a block diagram showing a schematic configuration of a trained model generation device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram for explaining learning data in the present embodiment. [Diagram 5] FIG. 1 is a diagram for explaining a trained model in this embodiment. [Figure 6] 1 is a block diagram showing a schematic configuration of an information processing device according to an embodiment of the present invention. [Figure 7] 1 is a block diagram showing a hardware configuration of an information processing device according to an embodiment of the present invention. [Figure 8] 1 is a flowchart showing the flow of a trained model generation process in this embodiment. [Figure 9] 4 is a flowchart showing a flow of information processing in the present embodiment. [Figure 10] FIG. 1 is a diagram for explaining the present embodiment. [Figure 11] FIG. 13 is a diagram for explaining the results of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensions and ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0016] <Overview of the embodiment> When a person's behavior is observed, it may be possible that the behavior reflects some intention. Therefore, when a person performs a certain action, there are cases where it is desirable to know what intention was behind that action.

[0017] Therefore, in this embodiment, a known machine learning model is trained to generate a trained model that outputs information representing the intention of a person's behavior when behavioral information, which is information observed when a person performs an action, is input. Then, in this embodiment, the behavioral information of a target person is input to the trained model, and information representing the intention of the person's behavior is output from the trained model. This makes it possible to easily estimate the intention of a person's behavior from the person's behavior.

[0018] In addition, the behavioral information in this embodiment is information observed when a person performs an action. For example, the behavioral information includes information on a state in which a person is stationary and information on a state in which a person is moving. Specifically, for example, the behavioral information includes information on a person's posture, facial expression, gaze information, or various physical quantities that are considered to have arisen from the action. Information on various physical quantities that are considered to have arisen from the action includes, for example, information on sound, vibration, pressure, etc. More specifically, for example, information on various physical quantities includes information on a person's footsteps, wind noise, or vibrations and pressure changes of air and other media caused by them. In addition, the behavioral information includes information observed from a local part of a person in addition to information observed from the whole person. For example, information on a specific part such as the posture, myoelectricity, or electrooculography of a person's hand is also included.

[0019] In this embodiment, the case where the intention of the player's action is estimated from the action information that represents the player's action while playing a sport will be described as an example. In this embodiment, the case where posture information is used as an example of the action information will be described as an example, but the present invention is not limited to this. For example, the player's facial expression information, line of sight information, footstep information, wind noise information, etc. may be used.

[0020] Fig. 1 is a diagram for explaining an overview of this embodiment. As shown in Fig. 1, in this embodiment, the intention of a player's action is estimated based on posture information P1, P2 of the player while the sport is being played. Fig. 1 shows a scene during a table tennis match, which is an example of a competitive ball sport.

[0021] In sports, a player's decision is to select an action, and the selected action is expressed as a physical result. For example, in ball sports, the direction and rotation of the ball controlled by the player are expressed as a physical result.

[0022] In conventional sports analysis, the actions of players and their results are mainly analyzed. For example, in ball sports, the direction and rotation of the ball controlled by the player are statistically analyzed. However, the direction and rotation of the ball are merely the result of the player's actions and do not necessarily match the results intended by the player.

[0023] In contrast, it is considered that the posture of a player while playing a sport contains various pieces of information. It is considered that the posture of a player contains information that the player is performing a certain action or intends to perform a certain action, and this information is useful for various analyses. Therefore, in this embodiment, the intention of the player's action is estimated based on the posture information of the player while playing a sport.

[0024] Specifically, in this embodiment, a known machine learning model is trained to generate a trained model that outputs information representing the intention of the player's action when posture information of the player while a sport is being played is input. Then, in this embodiment, posture information of the player while a target sport match is being played is input to the trained model, and information representing the intention of the player's action is output from the trained model. This makes it possible to easily estimate the intention of the player's action from the posture of the player while a sport is being played.

[0025] The specific details will be explained below.

[0026] [Trained model generation device] Fig. 2 is a block diagram showing a hardware configuration of the trained model generation device 10 according to the present embodiment. As shown in Fig. 2, the trained model generation device 10 has a CPU (Central Processing Unit) 42, a memory 44, a storage device 46, an input / output I / F (Interface) 48, a storage medium reading device 50, and a communication I / F 52. Each component is connected to each other via a bus 54 so as to be able to communicate with each other.

[0027] The storage device 46 stores a trained model generation program for executing each process described below. The CPU 42 is a central processing unit, and executes various programs and controls each configuration. That is, the CPU 42 reads the program from the storage device 46 and executes the program using the memory 44 as a working area. The CPU 42 controls each of the above configurations and performs various calculation processes according to the program stored in the storage device 46.

[0028] The memory 44 is made up of a RAM (Random Access Memory) and temporarily stores programs and data as a working area. The storage device 46 is made up of a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including an operating system, and various data.

[0029] The input / output I / F 48 is an interface for inputting and outputting data. In addition, an input device for performing various inputs, such as a keyboard or a mouse, and an output device for outputting various information, such as a display or a printer, may be connected. A touch panel display may be used as the output device to function as the input device.

[0030] The storage medium reader 50 reads data stored in various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray Disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0031] The communication I / F 52 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0032] Next, the functional configuration of the trained model generation device 10 will be described. As shown in Fig. 3, the trained model generation device 10 functionally includes a learning acquisition unit 12 and a learning unit 16. A learning data storage unit 14 and a trained model storage unit 18 are provided in a predetermined storage area of ​​the trained model generation device 10. Each functional configuration is realized by the CPU 42 reading each program stored in the storage device 46, expanding it in the memory 44, and executing it.

[0033] The learning data storage unit 14 stores learning data. The learning data in this embodiment is data in which posture information of a learning player and information representing the intention of the learning player's action while a sport is being played are associated with each other. The posture information of a player is, for example, information representing the player's skeleton as shown in FIG. 1. The posture information is, for example, composed of position information of each joint of the player and lines connecting each joint. For example, by using a known image processing algorithm, it is possible to extract information representing the player's skeleton as shown in FIG. 1 from images at each time of video data while a sport is being played.

[0034] Fig. 4 is an example of learning data stored in the learning data storage unit 14. As shown in Fig. 4, posture information of one learning player and information representing the intention of the action of one learning player are associated with each other and stored in the learning data storage unit 14.

[0035] The information representing the intention of the learning player's actions is set in advance by an expert with knowledge of the sport. Specifically, the expert watches a video of a certain sport being played. The expert then estimates what intention of the player's actions is represented by posture information of the player shown in the video at each time, and adds information representing the intention of the player's actions to the posture information of the player at each time. More specifically, the expert sets information representing whether the posture information of the player is offensive or defensive as information representing the intention of the player's actions.

[0036] In some cases, the person who sets the information expressing the intention of the behavior does not have to be an expert. For example, when the work of setting the information expressing the intention of the behavior is performed by multiple people (so-called annotators), it is sufficient that the knowledge that enables consistent labeling regarding the setting of the intention of the behavior is shared among the multiple people, and that the multiple people can perform labeling with similar quality.

[0037] The trained model storage unit 18 stores trained models generated by processing described later. Fig. 5 is a diagram for explaining the trained model of this embodiment. As shown in Fig. 5, when a player's posture information is input, the trained model of this embodiment outputs a score representing the degree to which the player is aggressive and a score representing the degree to which the player is defensive, as information representing the player's intention of action.

[0038] For example, a player's play is aggressive, meaning that the player is actively trying to score points and has a high probability of scoring through that play. Also, a player's play is defensive, meaning that the player is trying to prevent a loss of points and has a low probability of losing points through that play. For example, if the score representing the degree to which the player is aggressive is 0.8 and the score representing the degree to which the player is defensive is 0.2, it is determined that the posture information of the player is more aggressive.

[0039] For example, the position and speed of a ball during a ball sport are directly observed information as a result of a player's actions. In contrast, the intention of a player's actions cannot be directly observed, and therefore requires the intervention of an expert with knowledge of the sport in question.

[0040] Therefore, in this embodiment, an expert with knowledge of the sport adds information representing the intention of the player's action to the posture information of the player. Furthermore, in this embodiment, a trained model is generated based on learning data generated by the expert in which the posture information of the player and information representing the intention of the player's action are associated. This trained model is a model that outputs information representing the intention of the player's action when posture information of the player while the sport is being played is input. By generating such a trained model, it becomes possible to easily estimate the intention of the player's action from the posture of the player while the sport is being played.

[0041] The learning data acquisition unit 12 acquires learning data. Specifically, the learning data stored in the learning data storage unit 14 is read out.

[0042] The learning unit 16 generates a trained model that outputs information representing the intention of the player's action when the player's posture information is input by performing machine learning on a known learning model based on the learning data acquired by the learning acquisition unit 12. As the learning model, for example, a known neural network model can be used. For example, as the known neural network model, a Multi Layer Perceptron (MLP) model and a Long Short Term Memory (LSTM) model can be used. Any machine learning algorithm may be used when generating the learning model and the trained model.

[0043] Then, the learning unit 16 stores the generated trained model in the trained model storage unit 18.

[0044] [Information processing device] Fig. 6 is a block diagram showing a schematic configuration of an information processing system 20 according to the present embodiment. As shown in Fig. 6, the information processing system 20 includes a camera 22 and an information processing device 24. The information processing device 24 according to the present embodiment estimates the intention of a player's action at each time of a video captured by the camera 22, using the trained model generated by the trained model generation device 10.

[0045] The camera 22 captures video while the sport is being played, and outputs the obtained video data to the information processing device 24.

[0046] Fig. 7 is a block diagram showing a hardware configuration of the information processing device 24 according to this embodiment. As shown in Fig. 7, the information processing device 24 has a CPU (Central Processing Unit) 62, a memory 64, a storage device 66, an input / output I / F (Interface) 68, a storage medium reading device 70, and a communication I / F 72. Each component is connected to each other via a bus 74 so as to be able to communicate with each other.

[0047] The storage device 66 stores control programs for executing each process described below. The CPU 62 is a central processing unit, and executes various programs and controls each component. That is, the CPU 62 reads the programs from the storage device 66 and executes the programs using the memory 64 as a working area. The CPU 62 controls each component and performs various arithmetic processes according to the programs stored in the storage device 66.

[0048] The memory 64 is made up of a RAM (Random Access Memory) and temporarily stores programs and data as a working area. The storage device 66 is made up of a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system, and various data.

[0049] The input / output I / F 68 is an interface for inputting and outputting data. In addition, an input device for performing various inputs, such as a keyboard or a mouse, and an output device for outputting various information, such as a display or a printer, may be connected. A touch panel display may be used as the output device to function as the input device.

[0050] The storage medium reader 70 reads data stored in various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray Disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0051] The communication I / F 72 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0052] Next, the functional configuration of the information processing device 24 will be described. As shown in Fig. 7, the information processing device 24 functionally includes an acquisition unit 28, a generation unit 30, and an output unit 32. A data storage unit 26 and a trained model storage unit 27 are provided in a predetermined storage area of ​​the information processing device 24. Each functional configuration is realized by the CPU 62 reading each program stored in the storage device 66, expanding it in the memory 64, and executing it.

[0053] The data storage unit 26 stores video data captured by the camera 22.

[0054] The trained model storage unit 27 stores the trained model generated by the trained model generation device 10.

[0055] The acquisition unit 28 acquires posture information of the player while the target sport is being played. Specifically, the acquisition unit 28 reads the video data stored in the data storage unit 26, and extracts posture information of the player captured at each time point in the video data by using a known image processing algorithm.

[0056] The generation unit 30 generates information representing the intention of the player's action corresponding to the posture information by inputting the posture information of the player acquired by the acquisition unit 28 into the trained model stored in the trained model storage unit 27. Specifically, the generation unit 30 generates information representing the intention of the player's action at each time by inputting the posture information of the player at each time into the trained model stored in the trained model storage unit 27. The trained model outputs a score representing the degree to which the player is aggressive and a score representing the degree to which the player is defensive, as shown in FIG. 5.

[0057] The output unit 32 outputs the information generated by the generation unit 30 that indicates the player's intention of action.

[0058] The user checks the information indicating the player's intention regarding his / her actions output from the output unit 32, and analyzes what intention the player had regarding his / her actions at each time shown in the video.

[0059] Next, the operation of the trained model generation device according to this embodiment will be described.

[0060] First, an expert watches multiple videos of a sport being played, and labels the posture of a player at each time with information that represents the intention of the player's action. For example, if the sport in question is a competitive ball sport such as table tennis, the expert labels the posture of each player with information that represents the intention of the player's action. This generates learning data in which information on the posture of a player during a sport is associated with information that represents the intention of the player's action.

[0061] A user inputs the learning data generated by an expert to the trained model generation device 10. When the learning data is input to the trained model generation device 10, the learning data is stored in the learning data storage unit 14.

[0062] Then, when the trained model generation device 10 receives a predetermined instruction signal, the CPU 42 of the trained model generation device 10 reads out the trained model generation program from the storage device 46, deploys it in the memory 44, and executes it. As a result, the CPU 42 functions as each functional component of the trained model generation device 10, and the trained model generation process shown in FIG. 8 is executed.

[0063] In step S100, the learning data acquisition unit 12 acquires the learning data stored in the learning data storage unit .

[0064] In step S102, the learning unit 16 generates a trained model that outputs information representing the intention of the player's behavior when the player's posture information is input, by machine learning a known learning model based on the learning data acquired in step S100.

[0065] In step S104, the learning unit 16 stores the trained model generated in step S102 in the trained model storage unit 18.

[0066] Next, the operation of the information processing system 20 according to this embodiment will be described.

[0067] When the trained model generated by the trained model generation device 10 is input to the information processing device 24, the trained model is stored in the trained model memory unit 27 of the information processing device 24.

[0068] Next, a video of the sport of interest is captured by the camera 22, and the video data is input to the information processing device 24. The information processing device 24 stores the input video data in the data storage unit 26.

[0069] Then, when the information processing device 24 receives a predetermined instruction signal, the CPU 62 of the information processing device 24 reads out the information processing program from the storage device 66, deploys it in the memory 64, and executes it. As a result, the CPU 62 functions as each functional component of the information processing device 24, and the information processing shown in FIG. 9 is executed.

[0070] In step S200, the acquisition unit acquires, from the video data stored in the data storage unit , posture information of the player at each time.

[0071] In step S202, the generation unit 30 reads out the trained model stored in the trained model storage unit 27.

[0072] In step S204, the generation unit 30 inputs the posture information of the player acquired in step S200 into the trained model read out in step S202, thereby generating information representing the intention of the player's action.

[0073] In step S206, the output unit 32 outputs, as a result, the information representing the intention of the behavior generated in step S204.

[0074] The user checks the information indicating the player's intention regarding his / her actions output from the output unit 32, and analyzes what intention the player had regarding his / her actions at each time shown in the video.

[0075] As described above, the information processing device according to this embodiment acquires posture information, which is an example of behavioral information of a player while a sport is being played. The information processing device inputs the acquired posture information to a trained model that outputs the intention of the player's behavior when the posture information is input, thereby generating information representing the intention of the player's behavior corresponding to the acquired posture information. Then, the information processing device outputs the generated information representing the intention of the player's behavior. This makes it possible to easily estimate the intention of a person's behavior from information observed while the person is performing an action. Specifically, it is possible to easily estimate the intention of a player's behavior from the player's behavior while a sport is being played.

[0076] Furthermore, the trained model generation device according to this embodiment acquires training data in which posture information of a training player is associated with information representing the intention of the training player's action while a sport is being played. The trained model generation device generates a trained model that outputs information representing the intention of the player's action when posture information of a player is input, by machine learning the training model based on the acquired training data. This makes it possible to obtain a training model for easily estimating the intention of the player's action from the posture of the player while a sport is being played.

[0077] As described above, the learning data used to generate the trained model is created by an expert with knowledge of the sport. This makes it possible to add information that cannot be directly observed, such as the intention of the player's actions while the sport is being played, to the player's posture information, and makes it possible to estimate the intention of the player's actions from the player's posture while the sport is being played.

[0078] Furthermore, by utilizing the information indicating the intention of the player's actions obtained by this embodiment, it becomes possible to perform sports play analysis and game strategy planning. Note that this embodiment can be applied not only to real sports, but also to virtual sports, for example, and similarly enables play analysis and game strategy planning. For example, it becomes possible to analyze how to combine offensive and defensive plays to win a game.

[0079] Furthermore, by using the information representing the intention of the player's actions, it is possible to evaluate plays and matches, and to perform analysis to improve the player's level. Therefore, the information representing the intention of the player's actions can also be used in sports training and coaching. For example, it is possible to determine whether a player is properly practicing offensively or defensively.

[0080] In addition, by using the information that indicates the intention of the player's actions, it is possible to analyze the degree of agreement with Joseki in the target situation, etc. In addition, by using the information that indicates the intention of the player's actions, it is also possible to analyze situations in which one player is advantageous or disadvantageous, etc.

[0081] In addition, the information about the intentions of players' actions can be used to create highlight videos for sports commentary. For example, by using the information about the intentions of players' actions, it is possible to create a video that links together only scenes of offensive plays.

[0082] For example, by using information that indicates the intention of a player's actions in a robot such as a table tennis robot, it will be possible for the robot to gain a deeper understanding of the human player. Also, by understanding the patterns of attack and defense, it will be possible to understand what situations will excite the audience.

[0083] The present disclosure is not limited to the above-described embodiment and examples, and various modifications and applications are possible without departing from the gist of the present invention.

[0084] For example, in the above embodiment, the intention of a player's action is estimated from the posture information of the player while playing a sport, but the present invention is not limited to this. For example, the intention of a person's action may be estimated from the action information observed from the person while performing other actions.

[0085] In addition, in the above embodiment, the target sport is table tennis, which is a competitive ball sport, but the present invention is not limited to this and may be any sport.

[0086] In the above embodiment, the case where the posture information of the player is extracted from the video data during the sport is being played is described as an example, but the present invention is not limited to this. Any method may be used as long as the posture information of the player can be obtained, and various methods such as motion capture or a vibration detection method for detecting the movement of a person can be adopted.

[0087] In the above embodiment, the case where information representing the intention of the player's action is estimated from only the posture information of the player has been described as an example, but the present invention is not limited to this. For example, a trained model may be generated so that ball position information can be input to the trained model. This is because the ball position information is considered to affect the intention of the player's action. In this case, the trained model is configured to output information representing the intention of the player's action when the player's posture information and ball position information are input. In this way, by having the trained model learn the ball position information as well, it becomes possible to accurately estimate information representing the intention of the player's action.

[0088] In the above embodiment, the target sport is described as one-on-one table tennis, but the present invention is not limited thereto. The target sport may be a competitive team sport in which a team has multiple players. For example, in the case of table tennis, the sport may be doubles table tennis. Other competitive team sports in which a team has multiple players may be soccer, rugby, baseball, and the like. In this case, the acquisition unit 28 acquires posture information of each of the multiple players belonging to one team, and the generation unit 30 generates information representing the intention of each of the multiple players' actions by inputting the posture information of each of the multiple players into the trained model. This makes it possible to estimate information representing not only the intention of the action of one player but also the intention of the team.

[0089] In addition, it is believed that the posture information of players who are not directly playing the ball, such as hitting the ball, influences the intention of the player who is hitting the ball. For this reason, for example, in a 2-on-2 doubles table tennis match, by training the posture information of two players on one team into a trained model, it becomes possible to estimate the information that represents the intention of the players' actions with greater accuracy.

[0090] The learning data in the above embodiment may include data in which posture information of the learning player when viewed from a plurality of different angles is associated with information representing the intention of the learning player's action. The intention of a player's action during a game should not change regardless of the angle from which the player is viewed. By using data in which posture information of the learning player when viewed from a plurality of different angles is associated with information representing the intention of the learning player's action as learning data, it is possible to increase the number of learning data, and a trained model for estimating the intention of the player's action with higher accuracy can be obtained. For the same reason, data in which the player's posture information is inverted may be included in the learning data.

[0091] In addition, the trained model of the above embodiment has been described as outputting only information representing the intention of the player's actions, but is not limited thereto. For example, the trained model may be configured to further output a score probability corresponding to the posture information of a player when the posture information of the player is input. In this case, the score probability is output together with information on whether each player is offensive or defensive, so that it is possible to analyze the relationship between the intention of the player's actions and the score probability. In addition, for example, as in the example described later, the trained model may be configured to output information representing the intention of the player's actions and the label information of the player when the posture information of the player is input. In this case, it is possible to obtain information such as which player has what intention of action.

[0092] In addition, various processors other than the CPU may execute each process executed by reading the software (program) in the above embodiment. Examples of the processor in this case include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. Each process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). More specifically, the hardware structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements.

[0093] In the above embodiment, the programs are stored (installed) in advance in a storage device, but the present invention is not limited to this. The programs may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, a Blu-ray disc, or a USB memory. The programs may be downloaded from an external device via a network. EXAMPLES

[0094] In this embodiment, the target sport is singles table tennis, and posture information of the player at each time point is extracted from video data during the match, and the player's intention to act is estimated based on the posture information.

[0095] (1. Assigning correct labels) First, correct labels were assigned to the player's posture information at each time point in the video data for the five games. These correct labels indicate whether the intention of the player's action is "offensive (O)" or "not attacking (X)" (offensive / defensive label). Furthermore, the correct label for the intention of that action is also assigned the label of which player it is, "Player A (A)" or "Player B (B)" (player label). For this reason, by combining these, a total of four labels (OA, OB, XA, XB) were prepared.

[0096] Strictly speaking, the correct label in this example indicates "whether the play is offensive (or not) in order to gain points." Since it is thought that a deep knowledge of tactics as well as the rules of table tennis is required to judge the offensive / defensive labels, two university students (one male and one female) who have been involved in table tennis as players since junior high school were asked to serve as annotators of the correct labels. For this reason, the correct labels in this example were assigned by an expert.

[0097] (2. Extraction of posture information) We used known techniques to extract the player's pose information from the video data. Specifically, we used Kapao (William McNally, Kanav Vats, Alexander Wong, John McPhee, "Rethinking Keypoint Representations: Modeling Keypoints and Poses as Objects for Multi-Person Human Pose Estimation", https: / / arxiv.org / abs / 2111.08557) and DeepSort (Nicolai Wojke, Alex Bewley, Dietrich Paulus, "Simple Online and Realtime Tracking with a Deep Association Metric", https: / / arxiv.org / abs / 1703.07402) to extract pose information.

[0098] (3. Classifier Design) In this embodiment, two models based on a Multi Layer Perceptron (MLP) model and a Long Short Term Memory (LSTM) model were created to estimate offensive and defensive positions. As the MLP model, a model configuration in which two fully connected layers are stacked was adopted. Note that in the data set used for input in this embodiment, a time series exists regarding the actions of the players. Therefore, an LSTM model was also created to take the time series into account.

[0099] Fig. 10 is an outline of the design of the classifier of this embodiment. As shown in Fig. 10, time series data A of the player's posture information is extracted from the image frame IM at each time of the video data, and input to the MLP model or LSTM model M. The MLP model or LSTM model M outputs a combination of a player label R1 and an offensive / defensive label R2. The MLP model or LSTM model is trained so that the error between the prediction result P, which is this combination, and the correct label C is reduced.

[0100] (4. Experimental Results) The following table shows the experimental results of this embodiment. The results shown in the following table are the prediction accuracy of each model obtained by using the known leave one-game out. The average prediction accuracy of the five games was 0.515 for the MLP model and 0.536 for the LSTM model. In addition, in the following table, min is the prediction accuracy of the game with the lowest prediction accuracy among the five games, and max is the prediction accuracy of the game with the highest prediction accuracy among the five games. Looking at the average prediction accuracy of the five games, it was confirmed that the LSTM model had slightly higher prediction accuracy.

[0101] [Table 1]

[0102] Also, Fig. 11 shows another experimental result of this embodiment. Fig. 11 shows a heat map of the confusion matrix of five games, where Men's 1, 2 represent men's games, Women's 1, 2 represent women's games, and Cutman represents games in which the player is a cutman. As shown in Fig. 11, it can be seen that the correct answer label by the expert (Actual in Fig. 11) and the prediction result by the model (Predict in Fig. 11) are somewhat consistent.

[0103] (Additional Note) The following additional notes are given regarding aspects of the present disclosure.

[0104] (Appendix 1) an acquisition unit that acquires behavior information representing a player's behavior while the sport is being played; a generation unit that generates information representing an intention of the player's action corresponding to the action information acquired by the acquisition unit by inputting the action information acquired by the acquisition unit into a trained model that outputs information representing an intention of the player's action when the action information is input; an output unit that outputs information representing an intention of the player's action generated by the generation unit; An information processing device including

[0105] (Appendix 2) the trained model is a model that has been machine-learned in advance based on training data in which behavioral information of a training player is associated with information representing an intention of the behavior of the training player, The information representing the intention of the player's action for learning is information preset by a skilled person having knowledge of the sport. 2. An information processing device according to claim 1.

[0106] (Appendix 3) The sport is a competitive sport, the information representing the intention of the learning player's action is information representing whether the intention of the learning player's action is offensive or defensive; When the action information is input, the trained model outputs a degree to which the player's action intention is aggressive and a degree to which the player's action intention is defensive. 3. An information processing device according to claim 2.

[0107] (Appendix 4) The sport is a ball sport, The acquisition unit further acquires position information of a ball, The trained model outputs information representing the player's intention to act when the action information and the ball position information are input. The information processing device according to any one of Supplementary Note 1 to Supplementary Note 3.

[0108] (Appendix 5) The sport is a competitive team sport in which each team has multiple players, The acquisition unit acquires the behavioral information of each of a plurality of players belonging to one team, The generation unit generates information representing the intention of each of the actions of the multiple players by inputting the action information of each of the multiple players to the trained model. The information processing device according to any one of Supplementary Note 1 to Supplementary Note 4.

[0109] (Appendix 6) the acquiring unit acquires behavioral information of the player at each time from a video of the sport being played, the generation unit generates information representing the player's intention of action at each time by inputting action information of the player at each time to the trained model; the output unit outputs information representing the player's intention of action at each time. The information processing device according to any one of Supplementary Note 1 to Supplementary Note 5.

[0110] (Appendix 7) the learning data is data in which the behavioral information of the learning player when viewed from a plurality of different angles is associated with information expressing the intention of the behavior of the learning player; 3. An information processing device according to claim 2.

[0111] (Appendix 8) The trained model further outputs a score probability corresponding to the behavioral information when the behavioral information is input. The information processing device according to any one of Supplementary notes 1 to 7.

[0112] (Appendix 9) an acquisition unit for acquiring learning data in which behavioral information representing the behavior of a learning player and information representing the intention of the behavior of the learning player are associated with each other while a sport is being played; a learning unit that performs machine learning on a learning model based on the learning data acquired by the learning acquisition unit, thereby generating a trained model that outputs information representing the player's intention to act when the action information is input; A trained model generating device comprising:

[0113] (Appendix 10) Obtaining behavioral information representing the actions of players while a sport is being played; inputting the acquired action information into a trained model that outputs information representing the player's intention to act when the action information is input, thereby generating information representing the player's intention to act corresponding to the acquired action information; outputting the generated information representing the player's intention to act; An information processing method in which processing is performed by a computer.

[0114] (Appendix 11) acquiring learning data in which actions representing the actions of a learning player and information representing the intention of the actions of the learning player are associated with each other while a sport is being played; A learning model is machine-learned based on the acquired learning data to generate a trained model that outputs information representing the intention of the player's action when the action information is input. A method for generating trained models in which processing is performed by a computer.

[0115] (Appendix 12) Obtaining behavioral information representing the actions of players while a sport is being played; inputting the acquired action information into a trained model that outputs information representing the player's intention to act when the action information is input, thereby generating information representing the player's intention to act corresponding to the acquired action information; outputting the generated information representing the player's intention to act; An information processing program for causing a computer to execute processing.

[0116] (Appendix 13) acquiring learning data in which behavioral information representing the behavior of a learning player and information representing the intention of the behavior of the learning player are associated with each other while a sport is being played; A learning model is machine-learned based on the acquired learning data to generate a trained model that outputs information representing the intention of the player's action when the action information is input. A trained model generation program for allowing a computer to execute processing. [Explanation of symbols]

[0117] 10 Trained model generator 12 Learning acquisition section 14 Learning data storage unit 16 Learning Department 18 Trained model memory 24 Information processing equipment 28 Acquisition Department 30 Generation part 32 Output section

Claims

1. an acquisition unit that acquires behavior information representing a player's behavior while the sport is being played; a generation unit that generates information representing an intention of the player's action corresponding to the action information acquired by the acquisition unit by inputting the action information acquired by the acquisition unit into a trained model that outputs information representing an intention of the player's action when the action information is input; an output unit that outputs information representing an intention of the player's action generated by the generation unit; An information processing device comprising:

2. the trained model is a model that has been machine-learned in advance based on training data in which behavioral information of a training player is associated with information representing an intention of the behavior of the training player, The information representing the intention of the player's action for learning is information preset by a skilled person having knowledge of the sport. The information processing device according to claim 1 .

3. The sport is a competitive sport, the information representing the intention of the learning player's action is information representing whether the intention of the learning player's action is offensive or defensive; When the action information is input, the trained model outputs a degree to which the player's action intention is aggressive and a degree to which the player's action intention is defensive. The information processing device according to claim 2 .

4. The sport is a ball sport, The acquisition unit further acquires position information of a ball, The trained model outputs information representing the player's intention to act when the action information and the ball position information are input.

4. The information processing device according to claim 1.

5. The sport is a competitive team sport in which each team has multiple players, The acquisition unit acquires the behavioral information of each of a plurality of players belonging to one team, The generation unit generates information representing the intention of each of the actions of the multiple players by inputting the action information of each of the multiple players to the trained model.

4. The information processing device according to claim 1.

6. the acquiring unit acquires behavioral information of the player at each time from a video of the sport being played, the generation unit generates information representing the player's intention of action at each time by inputting the player's action information at each time to the trained model; the output unit outputs information representing the player's intention of action at each time.

4. The information processing device according to claim 1.

7. the learning data is data in which the behavioral information of the learning player when viewed from a plurality of different angles is associated with information expressing the intention of the behavior of the learning player; The information processing device according to claim 2 .

8. The trained model further outputs a score probability corresponding to the behavioral information when the behavioral information is input.

4. The information processing device according to claim 1.

9. an acquisition unit for acquiring learning data in which behavioral information representing the behavior of a learning player and information representing the intention of the behavior of the learning player are associated with each other while a sport is being played; a learning unit that performs machine learning on a learning model based on the learning data acquired by the learning acquisition unit, thereby generating a trained model that outputs information representing the player's intention to act when the action information is input; A trained model generating device comprising:

10. Obtaining behavioral information representing the actions of players while a sport is being played; inputting the acquired behavioral information into a trained model that outputs information representing the intention of the player's behavior when the behavioral information is input, thereby generating information representing the intention of the player's behavior corresponding to the acquired behavioral information; outputting the generated information representing the player's intention to act; An information processing method in which processing is performed by a computer.

11. acquiring learning data in which behavioral information representing the behavior of a learning player and information representing the intention of the behavior of the learning player are associated with each other while a sport is being played; A learning model is machine-learned based on the acquired learning data to generate a trained model that outputs information representing the intention of the player's action when the action information is input. A method for generating trained models in which processing is performed by a computer.

12. Obtaining behavioral information representing the actions of players while a sport is being played; inputting the acquired behavioral information into a trained model that outputs information representing the intention of the player's behavior when the behavioral information is input, thereby generating information representing the intention of the player's behavior corresponding to the acquired behavioral information; outputting the generated information representing the player's intention to act; An information processing program for causing a computer to execute processing.

13. acquiring learning data in which behavioral information representing the behavior of a learning player and information representing the intention of the behavior of the learning player are associated with each other while a sport is being played; A learning model is machine-learned based on the acquired learning data to generate a trained model that outputs information representing the intention of the player's action when the action information is input. A trained model generation program for allowing a computer to execute processing.

Images