Information processing device, information processing method, and program

Abstract

The present invention acquires useful information for evaluating performance of a player or a team. This information processing device includes: an acquisition means that acquires image data including at least one of at least one player, at least one coach, and at least one referee as at least one subject; and an inference means that executes pose inference processing relating to at least one of the at least one subject with reference to the image data, in association with each of a plurality of regions in a competition area indicated by the image data.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] In recent years, there has been progress in technologies for improving the performance of athletes and teams. For example, Patent Literature 1 describes a technology for acquiring motion data of athletes during exercise and evaluating the degree of completion of their play or performance.

[0003] Japanese Patent Application Publication No. 2018-068516

[0004] In order to evaluate the performance of a player or a team, it is also important to know in which area of ​​the playing area the player's play is taking place. Even the same play by the same player may have different contributions to the player's or team's performance depending on the area in which the play is taking place. Furthermore, the movements and positions of not only the player but also the coach and referee can affect the player's or team's performance. The technology disclosed in Patent Document 1 leaves room for improvement in these respects.

[0005] The present disclosure has been made in consideration of the above problems, and one exemplary purpose thereof is to provide a technology that can acquire useful information for evaluating the performance of athletes and teams.

[0006] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects, and an estimation means for performing a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes an information processing device acquiring image data including one or more players, one or more coaches, and one or more referees as one or more subjects, and performing a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

[0008] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, the program causing the computer to acquire image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects, and to perform a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

[0009] According to one exemplary aspect of the present disclosure, one exemplary effect is provided in which a technology can be provided that acquires useful information for evaluating the performance of an athlete or a team.

[0010] FIG. 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram illustrating a flow of an information processing method according to the present disclosure. FIG. 3 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. FIG. 4 is a block diagram illustrating an application example of the information processing system according to the present disclosure. FIG. 5 is a block diagram illustrating an application example of the information processing system according to the present disclosure. FIG. 6 is a diagram illustrating an application example of the information processing system according to the present disclosure. FIG. 7 is a diagram illustrating an example of processing by the information processing system according to the present disclosure. FIG. 8 is a diagram illustrating an example of processing by the information processing system according to the present disclosure. FIG. 9 is a block diagram illustrating a configuration of a computer functioning as an information processing device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Information Processing Device) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11 and an estimation unit 12.

[0014] (Acquisition Unit 11) The acquisition unit 11 acquires image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects.

[0015] (Estimation Unit 12) The estimation unit 12 performs a pose estimation process with reference to the image data acquired by the acquisition unit 11, and performs the pose estimation process for at least one of the one or more subjects. In other words, the estimation unit 12 applies the pose estimation process to the image data (or one or more subjects included in the image data). Here, the specific algorithm of the pose estimation process does not limit this exemplary embodiment, but as an example, the pose estimation process may include a two-dimensional or three-dimensional skeleton estimation process. Furthermore, the pose estimation process may use an algorithm based on machine learning or a rule-based algorithm. Furthermore, the pose estimation process may include an estimation process with reference to a human skeleton structure model. Furthermore, the pose estimation process may be configured to include a movement detection process, or may be configured to be executed as part of the movement detection process.

[0016] The estimation unit 12 identifies each of a plurality of regions in the competition area indicated by the image data and executes the pose estimation process in association with each of the plurality of regions. The algorithm used by the estimation unit 12 to identify each of the plurality of regions is not limited to this exemplary embodiment, and may be, for example, an algorithm based on machine learning. The estimation unit 12 may manage (save) the results of the pose estimation process in association with identifiers for identifying the plurality of regions, or may manage (save) the results in association with coordinate information (position information) corresponding to the plurality of regions in the competition area. Note that, although each of the plurality of regions may be referred to as a "tile" in this specification, this name does not limit the matters described herein.

[0017] (Effects of information processing device) As described above, the information processing device 1 is configured to acquire image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects, and to perform a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of multiple areas in the competition area indicated by the image data.

[0018] According to the above configuration, it is possible to obtain useful information for evaluating the performance of athletes and teams.

[0019] (Flow of Information Processing Method) Next, the flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes an acquisition process (acquisition step) S11 and an estimation process (estimation step) S12. The information processing method S1 is executed, as an example, in the information processing device 1 described above.

[0020] (Step S11) In step S11, the acquisition unit 11 of the information processing device 1 acquires image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects.

[0021] (Step S12) In step S12, the estimation unit 12 of the information processing device 1 performs a pose estimation process with reference to the image data acquired by the acquisition unit 11 in step S11, the pose estimation process being related to at least one of the one or more subjects. The estimation unit 12 also identifies each of a plurality of areas in the competition area indicated by the image data, and performs the pose estimation process in association with each of the plurality of areas. In this step, the estimation unit 12 may manage (save) the results of the pose estimation process in association with identifiers for identifying the plurality of areas, or may manage (save) the results in association with coordinate information (position information) corresponding to the plurality of areas in the competition area. Other specific processes performed by the estimation unit 12 have been described above, and therefore will not be described here.

[0022] (Effects of information processing method) As described above, the information processing method S1 is configured to acquire image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects, and to perform a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of multiple areas in the competition area indicated by the image data.

[0023] According to the above configuration, it is possible to obtain useful information for evaluating the performance of athletes and teams.

[0024] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0025] (Problems with the Conventional Art) First, we will explain the problems with the conventional art. In the world of sports, even small mistakes or incorrect strategies can determine victory or defeat. For example, volleyball matches are fast-paced, and teams are made up of multiple players with different roles. Whether a team wins a match depends on whether they attack, block, or receive the ball. Therefore, it is important for team managers and coaches to detect problems in the performance, strategy, and tactics of individual players. However, because volleyball matches are fast-paced and each player's performance differs, it is not easy to analyze data in real time or after a match or training session.

[0026] Furthermore, coaches' analysis of team performance is typically based primarily on an overall image of the team (images and visual information). Even if a coach can identify gaps in performance (e.g., a decline in performance) among certain players, it is difficult to grasp the overall team's performance at once. Furthermore, when a manager focuses on the team's performance, it becomes difficult to recognize and analyze the opposing team. As a result, data that should be accumulated is lost, and the individual performance of players and the opposing team's tactics and dynamics are not understood. Coaches make assumptions about how to improve their team's performance, but the actual data they should refer to is missing. This has resulted in no confirmation or performance analysis of why a team lost a game or why they faced such difficulties that led to a complete league defeat.

[0027] Although the configuration according to the present exemplary embodiment is intended to improve training in sports activities, primarily volleyball, as an example, the configuration according to the present exemplary embodiment can also be applied to other indoor and outdoor sports activities. For example, the configuration according to the present exemplary embodiment can be applied to one-on-one individual sports in which one player competes against another player, or to team sports in which a team of players competes against another team.

[0028] The sports activity targeted in this exemplary embodiment may be performed indoors or outdoors, based on specific rules, for example. Athletes play in a specific area of ​​limited size, such as a court, under the guidance of a coach or trainer. In this exemplary embodiment, the area in which athletes play is referred to as the playing area. For example, the playing area includes the court and the area surrounding the court where athletes can play. For example, in the case of volleyball, the playing area may include the volleyball court, courtside, serving area, etc. Athletes may also be referred to as players.

[0029] In recent years, there has been an increasing demand for improving the performance of athletes and teams in such sports activities. Configurations according to the present exemplary embodiment are provided, as an example, to address such demands. In the present exemplary embodiment, useful information is acquired and provided to support the training and coaching of athletes and teams, as described below. For example, coaches can utilize data recorded in the present exemplary embodiment to improve the performance of athletes and teams. For example, in the case of volleyball, the present exemplary embodiment analyzes player data recorded via a camera and processed with machine learning techniques to identify gaps in performance and other obstacles that hinder game success. For example, a volleyball game is fast-paced and relies on various strategies and tactics, such as attacking and blocking. The configurations according to the present exemplary embodiment provide useful information to coaches who wish to optimize the performance of each player and the coordination of play within a team.

[0030] Using the configuration according to this exemplary embodiment, a team coach can supervise the training of the team and individual players using data recorded and analyzed, for example, by machine learning. The coach can refer to the recorded data and modify or change the team's strategy and tactics. As an example, in a volleyball match, the subjects of recording and analysis may be, but are not limited to, 12 players, including six players from the team and six players from the opposing team. The subjects of recording and analysis may also include multiple substitute players, or may include one or more people who may be involved in discussions, such as a manager, coach, trainer, or referee. Furthermore, managers, coaches, trainers, etc. may be collectively referred to as coaches.

[0031] An arrangement according to this exemplary embodiment records player or team performance and processes the recordings, for example, using machine learning techniques, to enable managers and coaches to analyze overall team and individual player performance, including analysis of opposing teams and their strategies and performance.

[0032] The configuration according to the exemplary embodiment also records the performance of an athlete or team, and the recorded data can be played back in mixed reality or virtual reality. In this way, playing back the data in virtual reality can be used to train and coach individual athletes or entire teams. The configuration according to the exemplary embodiment can also help coaches improve the performance of athletes or teams and better prepare for upcoming matches.

[0033] (Configuration of Information Processing System) Next, the configuration of the information processing system 100A according to this exemplary embodiment will be described with reference to FIGS. 3 to 5. FIG. 3 is a block diagram showing the configuration of the information processing system 100A. As shown in FIG. 3, the information processing system 100A includes, as an example, an information processing device 1A, an information processing device 2A, a camera 17, a microphone 18, and an input / output device 19. Note that, as an example, the information processing device 2A has the same configuration as the information processing device 1A described below.

[0034] (Camera 17) Camera 17 captures an image of a target person and supplies image data including the target person in its angle of view to information processing device 1A or other devices included in information processing system 100A. As an example, camera 17 may be composed of one or more cameras installed in the competition area. For example, camera 17 may be composed of a front camera that captures images of the competition area from the front, a rear camera that captures images of the competition area from the opposite side to the front camera, and other cameras that capture images of important locations in the competition area.

[0035] The camera 17 may also include a camera attached to the body of an athlete, coach, or referee. As an example, the camera 17 may include a camera attached to or above the head of an athlete, coach, or referee. Such a camera is suitable for capturing images of the entire playing area, and by acquiring image data from such a camera, the information processing device 1A can record and analyze the entire playing area.

[0036] (Microphone 18) The microphone 18 collects sounds in the target area and supplies sound data containing the sounds to the information processing device 1A or other devices included in the information processing system 100A. As an example, the microphone 18 may be composed of one or more microphones installed in the competition area. For example, the microphone 18 may be composed of a front microphone that collects sounds in the competition area from the front, a back microphone that collects sounds in the competition area from the opposite side of the front microphone, and other microphones that collect sounds from important locations in the competition area.

[0037] The microphone 18 may also include a microphone attached to the body of an athlete, coach, or referee. As an example, the microphone 18 may include a microphone attached to the head of an athlete, coach, or referee. Such a microphone is suitable for collecting sounds from a specific subject, and by acquiring sound data from such a microphone, the information processing device 1A can suitably perform recording and analysis related to the subject.

[0038] (Input / output device 19) The input / output device 19 receives instructions from a user, supplies information indicating the received instructions to the information processing device 1A or other devices, and presents information obtained from the information processing device 1A or other devices to the user.

[0039] As an example, the input / output device 19 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. For example, the input / output device 19 visually presents information acquired from the information processing device 1A or other devices to the user via a touch panel. The input / output device 19 may also be configured to display a GUI (Graphical User Interface) on the touch panel and acquire instructions from the user as input to the GUI.

[0040] Furthermore, the input / output device 19 may be configured to include, for example, a head-mounted display or an eyeglass-type display, and to present information acquired from the information processing device 1A or another device to the user as virtual reality or mixed reality by displaying the information on the head-mounted display or the eyeglass-type display. More specifically, the input / output device 19 may be configured to present a digital copy of the subject generated by the estimation unit 12 (described later) as virtual reality or mixed reality including the digital copy.

[0041] 3, the information processing device 1A includes a control unit 10A, a storage unit 15A, and a communication unit 16A. The information processing device 1A may also include a configuration equivalent to the input / output device 19 described above.

[0042] (Communication unit 16A) The communication unit 16A communicates with devices external to the information processing device 1A. As an example, the communication unit 16A communicates with the camera 17, the microphone 18, the input / output device 19, and the information processing device 2A. The communication unit 16A transmits data supplied from the control unit 10A to the input / output device 19 and the information processing device 2A, and supplies data received from the camera 17, the microphone 18, the input / output device 19, and the information processing device 2A to the control unit 10A.

[0043] (Storage Unit 15A) The storage unit 15A stores various types of data referenced by the control unit 10A and various types of data generated by the control unit 10A. As an example, the storage unit 15A stores the following: Input data IN referenced by the control unit 10A Pose estimation result PE, which is an example of a processing result by the control unit 10A Analysis result RR, which is an example of a processing result by the control unit 10A Learned model LM used in processing by the control unit 10A

[0044] Here, the input data IN includes, for example, image data including people or other objects in the playing area and its surroundings within the field of view, more specifically, the input data IN includes image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects. The input data IN may also include sound data representing sounds collected by the microphone 18.

[0045] The pose estimation result PE includes information indicating the results of the pose estimation process performed by the estimation unit 12, which will be described later. As an example, this pose estimation process is performed in association with each of a plurality of areas (area 1, area 2, ...) in the competition area indicated by the image data. Therefore, as shown in Figure 3, the pose estimation result PE includes, for example: an estimation result PE1 for area 1; an estimation result PE2 for area 2; etc.

[0046] The analysis result RR includes information indicating the results of the analysis process performed by the estimation unit 12, which will be described later. For example, the analysis result RR may include the results of an analysis process related to at least one of the following: The performance of at least one of the one or more subjects, with reference to the results of the pose estimation process; and The performance of a team associated with the one or more subjects, with reference to the results of the pose estimation process. Furthermore, for example, the analysis process is performed for each of multiple areas (area 1, area 2, ...) in the competition area indicated by the image data. Therefore, as shown in FIG. 3 , the analysis result RR includes: An analysis result RR1 for area 1; An analysis result RR2 for area 2, etc.

[0047] As will be described later, the analysis result RR may also include gap information GI. Here, the gap information GI is, for example, information including at least one of: information regarding the difference between the performance of the subject and the performance of the comparison; and information regarding the difference between the performance of the team and the performance of the comparison.

[0048] As will be described later, the analysis result RR may also include information including at least one of tactics and strategies (strategy / tactics information SI).

[0049] The model LM, for example, has multiple parameters learned based on a deep learning algorithm, and outputs one or multiple estimation results by referring to the input data IN. Here, the estimation process using the model LM includes, for example, a pose estimation process and an analysis process that refers to the pose estimation process.

[0050] (Control Unit 10A) As shown in FIG. 3, the control unit 10A includes an acquisition unit 11, an estimation unit 12, and an output unit 13.

[0051] (Acquisition Unit 11) The acquisition unit 11 acquires the above-mentioned input data IN. Here, the input data IN includes image data captured by the camera 17, and including at least one or more players, one or more coaches, and one or more referees as one or more subjects. The acquisition unit 11 may also acquire sound data collected by the microphone 18 as the input data IN. The acquisition unit 11 may also be configured to acquire instructions from a user input via the input / output device 19.

[0052] (Estimation Unit 12) As shown in FIG. 3, the estimation unit 12 includes a pose estimation unit 12-1 and an analysis unit 12-2.

[0053] (Pose estimation unit 12-1) The pose estimation unit 12-1 detects each of one or more subjects included in the image data acquired by the acquisition unit 11. For example, each of the detected subjects is assigned a different identifier (subject ID), and the results of each process described below are associated with the subject ID.

[0054] Furthermore, the pose estimation unit 12-1 performs a pose estimation process with reference to the image data acquired by the acquisition unit 11, and the pose estimation process is related to at least one of the one or more subjects. In other words, the pose estimation unit 12-1 applies the pose estimation process to the image data (or one or more subjects included in the image data). Here, the specific algorithm of the pose estimation process does not limit this exemplary embodiment, but as an example, the pose estimation process may include a two-dimensional or three-dimensional skeleton estimation process. The pose estimation process may also use an algorithm based on machine learning. Specifically, it may be a process using the learned model LM described above. The pose estimation process may also use a rule-based algorithm. It may also include an estimation process with reference to a human skeleton structure model.

[0055] Furthermore, the pose estimation unit 12-1 identifies each of a plurality of areas in the competition area indicated by the image data and executes the pose estimation process in association with each of the plurality of areas. Here, the algorithm used by the pose estimation unit 12-1 to identify each of the plurality of areas is not limited to this exemplary embodiment, but may be, for example, an algorithm based on machine learning. More specifically, the above-described trained model LM may be used.

[0056] Furthermore, the pose estimation process by the pose estimation unit 12-1 may be configured to include a process of detecting the subject's movement using a machine learning model (for example, a learned model LM) in each of the multiple regions. Here, for example, the movement detection algorithm may be executed by referring to a series of sequential processing results of the pose estimation process, or may be executed independently (in parallel) of the pose estimation process. Specifically, the movement detection algorithm does not limit this exemplary embodiment. Furthermore, the pose estimation process by the pose estimation unit 12-1 may include a process of classifying athletes according to the results of the pose estimation process or the movement detection process.

[0057] In addition, the pose estimation result by the pose estimation unit 12-1 may be configured to include sound data indicating the sound collected by the microphone 18 in the scene being processed, or processed sound data obtained by applying a predetermined processing to the sound data.

[0058] Furthermore, the pose estimation unit 12-1 may store the results of the pose estimation process in the storage unit 15A in association with identifiers for identifying the plurality of areas, or in association with coordinate information (position information) corresponding to the plurality of areas in the competition area. Note that, although each of the plurality of areas may be referred to as a tile in this specification, this name does not limit the matters described in this specification.

[0059] (Analysis Unit 12-2) The analysis unit 12-2 executes an analysis process with reference to the result of the pose estimation process. As an example, the analysis unit 12-2 executes an analysis process with reference to the result of the pose estimation process, with reference to the performance of at least one of the one or more subjects and the performance of a team related to the one or more subjects.

[0060] Here, the analysis result by the analysis unit 12-2 may be configured to include information regarding which of the multiple areas in the playing area indicated by the image data is more important. For example, such importance determination is performed with reference to at least one of the pose estimation process and the motion detection process described above. Furthermore, in the case of ball games, importance determination may be performed based on the degree (frequency) at which the ball is detected in each area. For example, an area in which the ball is detected more frequently than other areas may be determined to be more important.

[0061] The analysis unit 12-2 may also generate a digital copy of the subject by referring to at least one of the results of the pose estimation process and the movement detection process. Here, the digital copy is, for example, a data set that represents (or simulates) at least one of the pose and movement of the subject in two or three dimensions, and is, for example, presented to the subject or user as virtual reality or mixed reality via the input / output device 19.

[0062] The analysis results by the analysis unit 12-2 may include at least one of the following: Information on the difference between the subject's performance and a comparison performance, and Information on the difference between the performance of the team to which the subject belongs and a comparison performance. Here, the comparison performance may be an ideal performance for the subject or the team, or a provisional performance to approach such ideal performance.

[0063] As an example, the performance for comparison may be configured to be set in advance by a user such as a coach or trainer with reference to: - a number of past performances of the subject - a number of past performances of the team to which the subject belongs, etc. Alternatively, a learning process may be applied to the above-mentioned model LM or other machine learning model with reference to: - a number of past performances of the subject - a number of past performances of the team to which the subject belongs, and the learned model may output the performance for comparison.

[0064] Furthermore, the analysis results by the analysis unit 12-2 may include at least one of information regarding the strategy or tactics of the team to which the subject belongs or of other teams, and information to support the coach's decision-making.

[0065] Here, the information about the strategy or tactics of the team and the information for supporting the coach's decision-making may be output by a machine-learned model. As an example, the above-mentioned model LM or another machine-learning model may be trained in advance with reference to: - a plurality of past performances of the subject; - a plurality of past performances of the team to which the subject belongs; - a plurality of past performances of the team against which the subject is currently playing, etc., and the trained model may be input with: - a current performance of the subject; - a plurality of current performances of the team to which the subject belongs; - a plurality of current performances of the team against which the subject is currently playing, etc., to output information about the strategy or tactics of the team.

[0066] The analysis results by the analysis unit 12-2 may also include advice on at least one of the following: what the subject should improve; and what the team to which the subject belongs should improve. For example, such advice may be generated by a process similar to or identical to the process for generating information about strategy or tactics described above. Furthermore, advice from a coach or trainer may also be included as part of the advice.

[0067] The analysis process by the analysis unit 12-2 may include a process of detecting the subject's movement using a machine learning model (for example, the learned model LM) in each of the multiple regions. The analysis process by the analysis unit 12-2 may also include a process of classifying athletes based on the results of the pose estimation process or the movement detection process. The movement detection algorithm has been described above, so a detailed description thereof will be omitted here.

[0068] In addition, the analysis results by the analysis unit 12-2 may be configured to include sound data indicating the sound collected by the microphone 18 in the scene being processed, or processed sound data obtained by applying a predetermined processing to the sound data.

[0069] (Output unit 13) The output unit 13 outputs at least one result of the pose estimation process and the analysis process performed by the estimation unit 12. More specifically, the output unit 13 supplies at least one result of the pose estimation process and the analysis process performed by the estimation unit 12 to the input / output device 19 and controls the input / output device 19 to output the result. The output unit 13 also stores at least one result of the pose estimation process and the analysis process performed by the estimation unit 12 in the storage unit 15A.

[0070] As partially described above, the results of at least one of the pose estimation process and analysis process by the estimation unit 12 may be output, for example, via a touch panel, or as virtual reality or mixed reality via a head-mounted display or eyeglass-type display.

[0071] (Information Processing Device 2A) As an example, the information processing device 2A has a configuration similar to that of the information processing device 1A described above. However, the user of the information processing device 2A may be different from the user of the information processing device 1A. As an example, the information processing device 1A may be installed locally or on-premise (for example, in a facility having a competition area), and the information processing device 2A may be installed remotely (in a remote location). When grounded in this way, the information processing device 1A can exchange information with athletes, teams, coaches (including managers and trainers), referees, etc. participating in the competition, and the information processing device 2A can exchange information with other coaches (including managers and trainers), other referees, etc. located in remote locations.

[0072] (Application Example 1 of Information Processing System 100A) Next, Application Example 1 of the information processing system 100A will be described with reference to FIG. 4. FIG. 4 is a diagram showing an application example of the information processing system 100A. As shown in FIG. 4, in this example, at least two cameras (17-1, 17-2) and at least two microphones (18-1, 18-2) are installed in the competition area CA. Also, as shown in FIG. 4, the competition area CA is composed of, for example, an area CA1 where player P1 plays and an area CA2 where player P2 competes. Also, as shown in FIG. 4, area CA1 is composed of three areas, and area CA2 is also composed of three areas. Each of these multiple areas is identified by the above-mentioned estimation unit 12, and each area is associated with at least one of the following: At least one of pose estimation results and analysis results related to the players in that area At least one of pose estimation results and analysis results related to the team in that area.

[0073] 4, an input / output device 19 is installed on the premises and used by an operator or coach who is a user on the premises. The user can input instructions to the information processing device 1A and view the pose estimation and analysis results performed by the information processing device 1A via the input / output device 19.

[0074] 4, the information processing device 2A is placed in a remote location and used by an operator or coach who is a user at the remote location. The user can input instructions to the information processing device 2A or the information processing device 1A, and view the pose estimation and analysis results performed by the information processing device 2A or the information processing device 1A.

[0075] Note that communication between the devices constituting information processing system 100A is carried out over a wide area network such as 5G, Beyond 5G, or 6G, for example. In such a configuration, there is almost no communication delay, so even a coach in a remote location can check the analysis results in real time, allowing them to give instructions to athletes or teams in real time and storing the analysis results in a database (e.g., storage unit 15A) for future evaluation.

[0076] As described above, the playing area CA is made up of multiple areas (tiles, sectors). As an example, in the case of volleyball, the court has sidelines, end lines, and other lines. These lines are predetermined and have lengths determined by the rules of volleyball. These lines define different areas on the court. The information processing system 100A (particularly the estimation unit 12) recognizes the court and its lines from the video recorded by the camera 17, and recognizes the areas defined by these lines as tiles. Then, for each tile, the estimation unit 12 recognizes the actions and performance of each player or team.

[0077] Identifying tiles is important in the processing of the information processing system 100A. Some tiles are likely to result in points, and some tiles are likely to result in points being lost. Performance at these tiles determines the outcome of the match. The information processing system 100A (particularly the estimation unit 12) analyzes the performance of the player or team at each tile, and the analysis results are reported to the coach. For example, if the information processing system 100A (particularly the estimation unit 12) analyzes that many spikes are being made at a particular tile, this indicates that the defense at that particular tile is insufficient. In such a case, the estimation unit 12 generates analysis results including, for example, the following: - Many spikes are being made at the particular tile - Defense at that particular tile should be improved, and presents these results to the coach.

[0078] (Application Example 2 of Information Processing System 100A) Next, application example 2 of the information processing system 100A will be described with reference to FIG. 5 . FIG. 5 is a diagram showing application example 2 of the information processing system 100A. As shown in FIG. 5 , in this example, the camera 17 includes a front camera ("Front" in the group of cameras shown in FIG. 5 ), a rear camera ("Back" in the group of cameras shown in FIG. 5 ), and side cameras ("Side A" and "Side B" in the group of cameras shown in FIG. 5 ). Furthermore, the microphones 18 include a front microphone ("Front" in the group of microphones shown in FIG. 5 ), a rear microphone ("Back" in the group of microphones shown in FIG. 5 ), and side microphones ("Side A" and "Side B" in the group of microphones shown in FIG. 5 ).

[0079] The video data captured by the cameras and the sound data collected by the microphones constitute input data IN. The input data IN is input to the learned model LM (AI in FIG. 5 ) of the estimation unit 12 of the information processing device 1A (or 2A). An operator or coach, who is the user of the information processing device 1A (or 2A), inputs instructions to the information processing device 1A (or 2A) via the input / output device 19 (GUI in FIG. 5 ) and visually views the analysis result information output by the information processing device 1A (or 2A). The analysis results include, for example, the following, as described above: pose estimation results for the subject; the subject's performance; the performance of the team to which the subject belongs; information regarding the strategies or tactics of the team to which the subject belongs or other teams; information to support the coach's decision-making; areas that the subject needs to improve; and areas that the subject's team needs to improve. Furthermore, this information is presented in association with multiple areas (tiles) that make up the competition area. Furthermore, this information is presented to the user as mixed reality or virtual reality via a head-mounted display or eyeglass-type display provided in the input / output device 19.

[0080] In this example, as described above, the camera 17 is configured with an array of multiple cameras, such as a front camera, a rear camera, and a side camera. Images captured by these cameras are then recorded. The estimation unit 12 can track and record the movements of each player (athlete) by referring to these images. Furthermore, the estimation unit 12 can track and record the coordination of the entire team by analyzing the image data acquired by the camera 17. It is also preferable to install cameras on the court in order to observe the entire court at once.

[0081] In this example, as described above, the microphone 18 is configured as an array of multiple microphones, including front microphones, back microphones, and side microphones. Sound collected by these microphones is recorded. The coordination and conversations of the players (athletes) are recorded for real-time analysis and future use. In this way, by referencing audio data in addition to video data, the information processing system 100A is suitable for analyzing the real-time coordination of the entire team and for coaches to analyze team performance.

[0082] (Application Example 3 of Information Processing System 100A) Next, Application Example 3 of the information processing system 100A will be described with reference to FIG. 6. FIG. 6 is a diagram showing Application Example 3 of the information processing system 100A. As described above, the information processing system 100A monitors and analyzes the court in real time using multiple cameras. In the case of volleyball, as shown in FIG. 6, the playing area is made up of the court CA, its surrounding area, and one or more pieces of equipment for playing the game (a net ("Net" in FIG. 6) and poles for supporting the net).

[0083] In the case of volleyball, the court on the front side of the net is made up of a front zone ("Front zone" in FIG. 6) CA1-1 and a back zone ("Back zone" in FIG. 6) CA1-2. In the case of volleyball, the playing area also includes a serving area ("Serving area" in FIG. 6) CA1-3. The same applies to the court on the other side of the net.

[0084] In this example, a front camera ("Camera (front)" in FIG. 6) is used to monitor the court on the front side of the net, and a back camera ("Camera (back)" in FIG. 6) is used to monitor the court on the other side of the net. With reference to the video captured by the front camera, the estimation unit 12 mainly performs: processing related to each player on the court on the front side of the net; and area classification processing with reference to the sidelines, attack lines, end lines, etc. that define the court on the front side of the net. Similarly, with reference to the video captured by the back camera, the estimation unit 12 mainly performs: processing related to each player on the court on the other side of the net; and area classification processing with reference to the sidelines, attack lines, end lines, etc. that define the court on the other side of the net.

[0085] In this example, a camera ("Camera (Serving, front)" in FIG. 6) 17-2 is provided to monitor the serving area CA1-3 in the playing area on the front side of the net. Referring to the video captured by the camera, the estimation unit 12 mainly analyzes the following: The player's movements and performance in the serving area CA1-3. The same applies to the serving area in the playing area on the other side of the net.

[0086] In this example, an upper camera ("Camera (Top)" in FIG. 6) is placed above the competition area CA. By referring to the video captured by the upper camera, the estimation unit 12 can analyze the movements and performance of each player and each team in the entire competition area.

[0087] (Processing Example 1 by Information Processing System 100A) FIG. 7 is a diagram illustrating processing example 1 by the information processing system 100A. More specifically, FIG. 7 is a diagram illustrating an example of area identification by the information processing system 100A. As described above, the estimation unit 12 can identify multiple areas constituting the playing area and identify and analyze the actions and performance of each player (each team) in each tile. In the example illustrated in FIG. 7 , the court is divided into multiple areas by multiple vertical lines VL and multiple horizontal lines HL. Here, the multiple vertical lines VL include sidelines ("Side lines" in FIG. 7 ), and the multiple horizontal lines HL include end lines ("End lines" in FIG. 7 ), attack lines ("Attack lines" in FIG. 7 ), and center lines ("Center lines" in FIG. 7 ). As illustrated in FIG. 7 , the serve area is also defined by vertical and horizontal lines.

[0088] The estimation unit 12 of the information processing system 100A identifies multiple areas (tiles) defined by the above-mentioned multiple lines. More specifically, it identifies areas z1, z2, z3, z4, z5, and z6 on the court on the near side of the net. Here, areas z5, z6, and z1 constitute the above-mentioned back zone CA1-2, and areas z2, z3, and z4 constitute the above-mentioned front zone CA1-1. In this way, the area identification process by the estimation unit 12 may be performed hierarchically.

[0089] In the case of volleyball, area z5 is an area where hit balls are concentrated, and is an area where processing by the estimation unit 12 is particularly important. The estimation unit 12 detects more events of hit balls landing in this area than in other areas. Based on the detection results, the estimation unit 12 may then analyze area z5 as being more important than other areas. The estimation unit 12 also refers to images captured by camera 17 to further analyze the movements of each player on and off the court.

[0090] (Processing example 2 by information processing system 100A) Fig. 8 is a diagram showing processing example 2 by information processing system 100A. More specifically, Fig. 8 is a diagram showing the results of real-time classification processing for each athlete and each area by information processing system 100A with reference to image data IN. Fig. 8 is also an example of output information output by input / output device 19.

[0091] FIG. 8 shows multiple boundaries (tile boundaries) representing the results of region division (tiling) performed by the information processing system 100A. In FIG. 8, vertical lines VL1, VL2, etc. and horizontal lines HL1, HL2, etc. are shown as such tile boundaries. Each region (tile) is defined by these boundaries. FIG. 8 also shows boundary boxes BB1 to BB8 as examples of the identification results performed by the information processing system 100A for each of the multiple athletes. The estimation unit 12 can simultaneously perform the identification process for each tile, as well as the detection process and pose estimation process for the athletes present in that tile.

[0092] (Processing Example 3 by Information Processing System 100A) Fig. 9 is a diagram showing Processing Example 3 by the information processing system 100A. More specifically, Fig. 9 is a diagram showing the results of real-time pose estimation processing for each athlete by the estimation unit 12.

[0093] In this example, the estimation unit 12 extracts an image of the player P1 from the images captured by the camera 17 and applies a pose estimation process to the image of the player P1 to generate a digital copy DC1 of the player P1. The digital copy DC1 is composed of vertices (black circles in FIG. 9 ) corresponding to the joints of the player P1 and links connecting the vertices.

[0094] The pose estimation process by the estimation unit 12 is performed online in real time, and can be performed for one athlete or for multiple athletes simultaneously.

[0095] (Processing Example 4 by Information Processing System 100A) FIG. 10 is a diagram illustrating Processing Example 4 by the information processing system 100A. More specifically, FIG. 10 illustrates a real-time processing result for player P1 within the competition area performed by the estimation unit 12. FIG. 10 also illustrates an example of output information output by the input / output device 19. As illustrated in FIG. 10 , the estimation unit 12 executes a pose estimation process for player P1 in real time. The estimation unit 12 also identifies a time transition of the pose estimation result for player P1. In other words, the estimation unit 12 recognizes where and how the player's pose is changing. In the example illustrated in FIG. 10 , the estimation unit 12 recognizes that player P1 moves through areas z1, z6, and z5 in chronological order. The estimation unit 12 may store the movement path MT of player P1 in the memory unit 15A and perform analysis processing with reference to the movement path MT.

[0096] (Processing Example 5 by Information Processing System 100A) FIG. 11 is a diagram showing Processing Example 5 by the information processing system 100A. FIG. 11 is also an example of output information output by the input / output device 19. The upper part of FIG. 11 is a diagram showing the results of real-time pose estimation processing by the estimation unit 12 for one or more athletes in the competition area. As shown in the upper part of FIG. 11, the estimation unit 12 can recognize not only the pose of a single athlete, but also the poses of multiple athletes at once in real time. Specifically, the upper part of FIG. 11 shows the result (digital copy) DC1 of the pose estimation processing for the first athlete and the result (digital copy) DC2 of the pose estimation processing for the second athlete.

[0097] The lower part of Fig. 11 is a diagram showing the result of the real-time pose estimation process on the referee performed by the estimation unit 12. Specifically, the lower part of Fig. 11 shows the result (digital copy) DC1 of the pose estimation process on the referee.

[0098] For example, the estimation unit 12 may execute the various analysis processes described above by referring to the result of the pose estimation process for the referee, together with or instead of the result of the pose estimation process for each player. The output unit 13 stores the result of the pose estimation process by the estimation unit 12 in the memory unit 15A or presents it to the user via the input / output device 19.

[0099] (Effects of Information Processing System 100A) The information processing system 100A configured as described above acquires image data including one or more players, one or more coaches, and one or more referees as one or more subjects, and performs a pose estimation process for at least one of the one or more subjects by referencing the image data, in association with each of multiple areas in the competition area indicated by the image data. This configuration makes it possible to acquire useful information for evaluating the performance of athletes and teams. Furthermore, the estimation unit 12 of the information processing system 100A executes the various analytical processes described above, thereby outputting useful analysis results.

[0100] As described above, the information processing system 100A can be suitably applied to volleyball, as an example.

[0101] As an example, the information processing system 100A may support recording of volleyball training sessions and apply AI-based techniques (e.g., processing by the estimator 12) to assist a team coach in analyzing their performance. The recorded data may be provided to the coach in real time and / or as playback information. Furthermore, because the recorded data is analyzed (e.g., by the estimator 12) with the aid of AI-based techniques, analysis time is reduced compared to manual analysis by the coach. The AI-based techniques of the present invention recognize movements in each sector of the volleyball court, providing new data points and insights for the coach to further analyze, which can be easily shared and analyzed with the players.

[0102] AI-based coaching support also improves each player's performance and endurance, reducing injuries. This is because data is recorded, analyzed, and accessible to coaches. Coaches use the recorded data to make suggestions to each athlete. The data is then consumed and players are instructed on how to improve their endurance and avoid injuries. This results in improved endurance for the entire team and fewer injured players, who could play a key role in a game. Each player's blocking and pitching skills are analyzed and improved based on the coach's recommendations.

[0103] Furthermore, the information processing system 100A records and analyzes the performance of the opposing team. By comparing this data with the team's own performance, it is possible to determine whether there are any tactics that are incorrect. This information can be used by coaches to fine-tune their own team's strategies, such as pitching and blocking.

[0104] In this way, the information processing system 100A, as an example, provides a basis for improving the performance of an entire volleyball team. Because competition in a volleyball league is fierce, such data is highly valuable for coaches to provide accurate instructions to individual players. As a result, important factors such as teamwork among players, endurance, and motivation to win a game are improved.

[0105] (Other Application Examples) The application of the information processing system 100A is not limited to volleyball, but can also be used as a coaching system for any team sport, such as handball, soccer, rugby, baseball, or basketball. This allows for real-time recording of individual players and the entire team. For example, even when applied to soccer, where the team size is 11 players, totaling 22 players, plus a referee, the information processing system 100A can identify these numerous players in real time. By recording each player's movements, pass selection, defensive and offensive skills, and analyzing them using AI-based technology (for example, analysis by the estimation unit 12), the information processing system 100A can assist managers and coaching teams.

[0106] Furthermore, the present invention may be fully embodied in a virtual environment using at least one of mixed reality (MR) and virtual reality (VR). For example, the input / output device 19 may present the analysis results to the user in at least one of mixed reality (MR) and virtual reality (VR) modes.

[0107] Additionally, recorded multimodal data, including video and audio, can be acquired by the information processing system 100A, and the analysis results obtained by referencing the data can be used for playback in a virtual environment, which can be used for gaming purposes as well as training sessions for athletes to better understand team interactions.

[0108] As an example, individual sports such as golf are also interesting and can be recorded and analyzed by the information processing system 100A. For example, the information processing system 100A can record and analyze performance and decision-making such as strategy, course selection, swing, walking, and hiking, and compare the performance and decision-making with control performance and decision-making. This can also contribute to the health care of the subject.

[0109] Additionally, as partially described above, the information processing system 100A can be used to analyze the performance of opposing teams. Recording and analyzing the opposing team's tactics and combinations can provide coaches with new ideas on how to optimize their team's performance, such as focusing on blocking or pitching. Each game is different and therefore requires a different strategy, which can be easily selected with the help of the information processing system 100A.

[0110] The information processing system 100A can also be used to analyze the referee's performance as needed. From the recorded data and AI-based analysis (e.g., analysis by the estimation unit 12), new ideas and discoveries can be obtained about how teams should select strategies, how they should cooperate, or how they should avoid mistakes. For example, if a referee exhibits a particular bias, the information processing system 100A can also analyze how to mitigate that bias when selecting a strategy.

[0111] [Example of implementation by software] Some or all of the functions of the information processing devices 1, 1A, 2A, and input / output device 19 (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0112] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 12. Figure 12 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0113] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0114] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0115] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0116] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0117] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0118] (Appendix A1) An information processing device comprising: an acquisition means for acquiring image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects; and an estimation means for performing a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

[0119] (Supplementary Note A2) The information processing device according to Supplementary Note A1, wherein the estimation means further performs an analysis process on at least one of performances of the one or more subjects and performances of a team related to the one or more subjects, with reference to a result of the pose estimation process.

[0120] (Appendix A3) The information processing device according to Appendix A2, further comprising: an output unit that outputs a result of the analysis process performed by the estimation unit.

[0121] (Supplementary Note A4) The information processing device according to Supplementary Note A3, wherein at least one of the pose estimation process and the analysis process includes a process of detecting a movement of the subject using a machine learning model in each of the plurality of regions.

[0122] (Supplementary Note A5) The information processing device according to Supplementary Note A4, wherein the result of the analysis process by the estimation means includes information regarding which of the plurality of regions has a higher importance.

[0123] (Appendix A6) The information processing device described in Appendix A4, wherein the estimation means generates a digital copy of the subject by referring to the result of the pose estimation process, and the output means presents the result of the analysis process as virtual reality or mixed reality including the digital copy of the subject.

[0124] (Appendix A7) The information processing device described in any one of Appendices A2 to A6, wherein the results of the analysis process include at least one of information regarding the difference between the subject's performance and a comparison performance, and information regarding the difference between the team's performance and a comparison performance.

[0125] (Appendix A8) The information processing device according to any one of Appendices A2 to A6, wherein the results of the analysis process include at least one of information regarding the strategy or tactics of the team, and information for supporting the coach's decision-making.

[0126] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0127] (Appendix B1) An information processing method including: an acquisition process in which at least one processor acquires image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects; and an estimation process in which the at least one processor references the image data and executes a pose estimation process for at least one of the one or more subjects in association with each of a plurality of areas in a competition area indicated by the image data.

[0128] (Supplementary Note B2) In the information processing method described in Supplementary Note B1, in the estimation process, the at least one processor further performs an analysis process on at least one of the performance of the one or more subjects and the performance of a team related to the one or more subjects, with reference to a result of the pose estimation process.

[0129] (Supplementary Note B3) The information processing method according to Supplementary Note B2, further comprising an output process in which the at least one processor outputs a result of the analysis process performed by the estimation process.

[0130] (Supplementary Note B4) The information processing method according to Supplementary Note B3, wherein the at least one processor performs at least one of the pose estimation process and the analysis process, and the at least one processor performs a process of detecting the subject's movement in each of the plurality of regions using a machine learning model.

[0131] (Supplementary Note B5) The information processing method according to Supplementary Note B4, wherein the result of the analysis process by the estimation process includes information regarding which of the plurality of regions has a higher importance.

[0132] (Appendix B6) An information processing method as described in Appendix B4, wherein, in the estimation process, the at least one processor generates a digital copy of the subject by referring to the result of the pose estimation process, and in the output process, the at least one processor presents the result of the analysis process as virtual reality or mixed reality including the digital copy of the subject.

[0133] (Appendix B7) An information processing method described in any one of Appendices B2 to B6, wherein the results of the analysis process include at least one of: information regarding the difference between the subject's performance and a comparison performance; and information regarding the difference between the team's performance and a comparison performance.

[0134] (Appendix B8) An information processing method described in any one of Appendices B2 to B6, wherein the results of the analysis process include at least one of information regarding the team's strategy or tactics, and information to support the coach's decision-making.

[0135] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0136] (Appendix C1) A program that causes a computer to function as an information processing device, the information processing program causing the computer to function as: an acquisition means that acquires image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects; and an estimation means that, by referring to the image data, executes a pose estimation process for at least one of the one or more subjects in association with each of a plurality of areas in a competition area indicated by the image data.

[0137] (Appendix C2) The information processing program according to Appendix C1, wherein the estimation means further performs an analysis process on at least one of the performances of the one or more subjects and the performance of a team related to the one or more subjects, with reference to a result of the pose estimation process.

[0138] (Supplementary Note C3) The information processing program according to Supplementary Note C2, which causes the computer to function as an output process that outputs a result of the analysis process by the estimation means.

[0139] (Supplementary Note C4) The information processing program according to Supplementary Note C3, wherein at least one of the pose estimation process and the analysis process includes a process of detecting a movement of the subject using a machine learning model in each of the plurality of regions.

[0140] (Supplementary Note C5) The information processing program according to Supplementary Note C4, wherein the result of the analysis process by the estimation means includes information regarding which of the plurality of regions has a higher importance.

[0141] (Appendix C6) An information processing program as described in Appendix C4, wherein the estimation means generates a digital copy of the subject by referring to the result of the pose estimation process, and the output means presents the result of the analysis process as virtual reality or mixed reality including the digital copy of the subject.

[0142] (Appendix C7) An information processing program described in any one of Appendices C2 to C6, wherein the results of the analysis process include at least one of: information regarding the difference between the subject's performance and a comparison performance; and information regarding the difference between the team's performance and a comparison performance.

[0143] (Appendix C8) An information processing program described in any one of Appendices C2 to C6, wherein the results of the analysis process include at least one of information regarding the team's strategy or tactics, and information to support the coach's decision-making.

[0144] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0145] (Appendix D1) An information processing device comprising at least one processor, the at least one processor performing an acquisition process of acquiring image data including at least one of one or more players, one or more coaches, and one or more referees as one or more subjects, and an estimation process of performing a pose estimation process of at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

[0146] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0147] (Appendix D2) The information processing device described in Appendix D1, wherein in the estimation process, the at least one processor further performs an analysis process on at least one of the performance of the one or more subjects and the performance of a team related to the one or more subjects, with reference to a result of the pose estimation process.

[0148] (Supplementary Note D3) The information processing device according to Supplementary Note D2, wherein the at least one processor executes an output process that outputs a result of the analysis process performed by the estimation process.

[0149] (Supplementary Note D4) The information processing device according to Supplementary Note D3, wherein at least one of the pose estimation process and the analysis process includes a process of detecting a movement of the subject using a machine learning model in each of the plurality of regions.

[0150] (Supplementary Note D5) The information processing device according to Supplementary Note D4, wherein the result of the analysis process by the estimation process includes information regarding which of the plurality of regions has a higher importance.

[0151] (Appendix D6) The information processing device described in Appendix D4, wherein in the estimation process, the at least one processor generates a digital copy of the subject by referring to the result of the pose estimation process, and in the output process, the at least one processor presents the result of the analysis process as virtual reality or mixed reality including the digital copy of the subject.

[0152] (Appendix D7) An information processing device described in any one of Appendices D2 to D6, wherein the results of the analysis process include at least one of information regarding the difference between the subject's performance and a comparison performance, and information regarding the difference between the team's performance and a comparison performance.

[0153] (Appendix D8) An information processing device according to any one of appendices D2 to D6, wherein the results of the analysis process include at least one of information regarding the strategy or tactics of the team, and information to support the coach's decision-making.

[0154] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0155] (Appendix E1) A non-transitory recording medium having recorded thereon an information processing program that causes a computer to function as an information processing device, the information processing program causing the computer to execute an acquisition process for acquiring image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects, and an estimation process for performing a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of a plurality of areas in the competition area indicated by the image data.

[0156] REFERENCE SIGNS LIST 1, 1A, 2A ... Information processing device 100A ... Information processing system 17 ... Camera 18 ... Microphone 19 ... Input / output device 10A ... Control unit 11 ... Acquisition unit 12 ... Estimation unit 13 ... Output unit

Claims

1. An information processing device comprising: an acquisition means for acquiring image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects; and an estimation means for performing a pose estimation process for at least one of the one or more subjects by referencing the image data, in association with each of a plurality of areas in a competition area indicated by the image data.

2. The information processing device according to claim 1, wherein the estimation means further performs an analysis process on at least one of the performance of the one or more subjects and the performance of a team related to the one or more subjects, with reference to the results of the pose estimation process.

3. The information processing device according to claim 2, further comprising output means for outputting the results of the analysis processing by the estimation means.

4. The information processing device according to claim 3, wherein at least one of the pose estimation process and the analysis process includes a process of detecting the subject's movement in each of the plurality of regions using a machine learning model.

5. The information processing device according to claim 4, wherein the result of the analysis process by the estimation means includes information regarding which of the plurality of areas is more important.

6. The information processing device described in claim 4, wherein the estimation means generates a digital copy of the subject by referring to the results of the pose estimation process, and the output means presents the results of the analysis process as virtual reality or mixed reality including the digital copy of the subject.

7. An information processing device according to any one of claims 2 to 6, wherein the results of the analysis process include at least one of information regarding the difference between the subject's performance and a comparison performance, and information regarding the difference between the team's performance and a comparison performance.

8. An information processing device according to any one of claims 2 to 6, wherein the results of the analysis processing include at least one of information regarding the strategy or tactics of the team, and information to support the coach's decision-making.

9. An information processing method in which an information processing device acquires image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects, and performs a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of multiple areas in the competition area indicated by the image data.

10. A program that causes a computer to function as an information processing device, the program causing the computer to acquire image data including at least one or more players, one or more coaches, and one or more referees as one or more subjects, and execute a pose estimation process for at least one of the one or more subjects by referring to the image data, in association with each of multiple areas in the competition area indicated by the image data.

Images