Action determination system, action determination method, and program

Abstract

Provided are an action determination system, an action determination method, and a program, whereby the orientation of a player can be determined with high precision.　The action determination system comprises: a video acquisition unit that acquires video information captured from above or obliquely above a stadium by an imaging device; a sensor information acquisition unit that acquires sensing information from a sensor worn by a player; an action determination unit that determines, on the basis of the video information and the sensing information, an action including the orientation of the body of the player; and an output unit that outputs the determination result.

Description

Motion determination system, motion determination method, and program

[0001] An embodiment of the present invention relates to a movement determination system, a movement determination method, and a program for determining a movement of a player.

[0002] In recent years, the importance of players performing their movements appropriately has been recognized in various sports, including ball games. For example, Patent Document 1 discloses an information processing device that includes an acquisition unit that acquires image data captured from the air by an unmanned aerial vehicle, and a direction identification unit that analyzes the image data to identify the direction in which a target depicted in the image data is facing.

[0003] International Publication No. 2019 / 239694

[0004] In order to coach athletes or implement competitive tactics more effectively, it is necessary to obtain more accurate information about the movements of athletes during competition, including the orientation of their bodies. However, the information processing device in Patent Document 1 is not designed to determine the orientation of athletes' bodies. Furthermore, in sports involving fast-moving movements, it may be difficult to accurately determine the orientation of athletes using only video.

[0005] An object of one embodiment of the present invention is to provide a movement determination system, a movement determination method, and a program that can determine the orientation of a player in a competition with high accuracy.

[0006] (1) The movement determination system includes an image acquisition unit that acquires image information captured by a camera from above or diagonally above the stadium, a sensor information acquisition unit that acquires sensing information from a sensor worn by the player, a movement determination unit that determines the movement of the player, including the orientation of the player's body, based on the image information and the sensing information, and an output unit that outputs the determination result.

[0007] As a result, the movement determination system of this embodiment can provide more accurate information regarding the movements of players during a game, including the body orientation of the player, in order to more effectively coach the player or implement competition tactics. Furthermore, in a game in which the players move quickly, for example, it is difficult to accurately determine the player's movements using video alone, but the movement determination system of this embodiment determines the player's body orientation based on sensing information obtained by sensors worn by the player in addition to video information, so that the player's body orientation can be determined with high accuracy even in a game in which the players move quickly.

[0008] (2) The sensor information acquisition unit acquires sensing information from a pair of compass sensors attached to the player's shoulders, hips, front and back of the torso, or shoes on both feet.

[0009] This allows the motion determination system to more accurately identify the body orientation.

[0010] (3) The movement determination system includes a player identification unit that identifies players appearing in the captured video based on video information or based on both the video information and the sensing information.

[0011] The movement determination system of this embodiment identifies each player even in a sport involving multiple players, and can therefore determine the orientations of multiple players with high accuracy.

[0012] (4) The sensing information acquired by the sensor information acquisition unit includes sensor position information from the sensor attached to the player and sensor identification information of the sensor, and the player identification unit determines the position of the player based on the sensor position information and the sensor identification information, and identifies the player appearing in the captured video based on the player position information of the determined player.

[0013] In aerial footage, multiple players may overlap, making it impossible to capture images of each player. The movement determination system of this embodiment acquires detection information from position detection sensors worn by the players and player identification information associated with the sensors to identify the players appearing in the captured images. Therefore, the movement determination system of this embodiment can accurately determine the orientation of each of multiple players even when multiple players overlap in aerial footage.

[0014] (5) The player identification unit identifies each player or identifies personal attributes including the team they belong to, the referee, and the goalkeeper based on at least one of the information such as the color of their clothing or their uniform number appearing in the acquired video.

[0015] As a result, the motion determination system of this embodiment can identify the team to which a player belongs, the referee, the goalkeeper, etc. in sports such as soccer, rugby, American football, hockey, basketball, or yacht racing, and can provide more useful information for coaching players or for more effective sports tactics.

[0016] (6) The movement determination system includes a display control unit that displays the player's identification information and body orientation information on the video.

[0017] As a result, the movement determination system of this embodiment can visually provide information on the body orientation determined for each player.

[0018] (7) The movement determination system includes an individual performance determination unit that determines the performance of each identified player based on information including the determined body orientation.

[0019] As a result, the movement determination system of this embodiment can provide more useful information for coaching athletes or for implementing competition tactics more effectively.

[0020] (8) The movement judgment system includes a display control unit that displays the identification information of the player and the judgment information of the performance judgment on the video.

[0021] As a result, the movement determination system of this embodiment can visually provide more useful information for coaching athletes or for more effectively implementing tactics for competitions.

[0022] (9) The motion determination system includes an offensive / defensive state determination unit that determines offensive players and defensive players based on the offensive / defensive state, and an organizational performance determination unit that determines the organizational performance of multiple offensive players or the organizational performance of multiple defensive players.

[0023] For example, in sports such as soccer, whether or not an offense or defense is organized is very important. The movement determination system of this embodiment can provide more useful information by determining the offensive and defensive states and determining the offensive or defensive performance.

[0024] (10) The organizational performance judgment unit judges whether the offensive player is unmarked based on the identification information of the team to which the identified player belongs, the position of the offensive player, and the judgment results of the position, speed, and body orientation of the defensive player.

[0025] For example, in sports such as soccer, it is very important to determine whether an offensive player has lost track of the marker. The motion determination system of this embodiment can determine whether an offensive player has lost track of the marker and provide more useful information.

[0026] (11) The organizational performance judgment unit judges whether the defensive formation has collapsed based on the identification information of the teams to which the identified players belong, the positions of the offensive players, and the positions and body orientations of the defensive players.

[0027] For example, in sports such as soccer, it is very important to determine whether an organized defensive formation has collapsed. The movement determination system of this embodiment can determine whether the defensive formation has collapsed and provide more useful information.

[0028] (12) The organizational performance determination unit determines the offensive team's passing route based on the team identification determination of each player, the positions and body orientations of the offensive players, and the positions of the defensive players.

[0029] For example, in sports such as soccer, determining the offensive pass route is extremely important. The motion determination system of this embodiment can determine the offensive pass route and provide more useful information.

[0030] (13) The organizational performance determination unit detects spaces into which the offensive team should run based on the team identification determination of each player and the positions and body orientations of multiple defensive players.

[0031] For example, in a sport such as soccer, it is very important for the offensive team to make a run in. The motion determination system of this embodiment can detect when the offensive team should make a run in and provide more useful information.

[0032] (14) The action determination system includes a display control unit that highlights on the video the players or positions that are off-marker, out of formation, passing route, or related to space, as determined by the organizational performance determination unit.

[0033] As a result, the movement determination system of this embodiment can visually provide an easy-to-understand picture of players or positions related to a break in marking or formation, passing routes, or spaces.

[0034] (15) The action determination system includes a tag information receiving unit that receives a tag input for video information corresponding to a play of interest among the video information.

[0035] In this way, the motion determination system of this embodiment may accept manual tagging input for video when a user, such as a coach, determines a play to be noteworthy. This allows players, managers, coaches, or managers to review the play to be noted later. Furthermore, managers can reduce the effort required for scoring.

[0036] (16) The action determination system includes a tagging unit that tags video when the organizational performance determination unit determines that a player has lost his mark, his formation has been broken, a passing route has been created, or there is space.

[0037] In this case, the video of when a player loses his mark, breaks formation, or judges a passing route or space is automatically tagged, making it easier for players, coaches, or managers to analyze notable plays when reviewing them later.

[0038] According to one embodiment of the present invention, the orientation of a player in a competition can be determined with high accuracy.

[0039] 10(A) is a diagram showing an example of the front of a uniform, and FIG. 10(B) is a diagram showing an example of the back of a uniform. A user interface screen showing an example display of the determination results. FIG. 10(B) is a functional block diagram of the control unit 12 according to Modification 1. FIG. 10(B) is a flowchart showing the operation of the control unit 12 according to Modification 1. FIG. 10(B) is a flowchart showing the operation of player identification. A conceptual diagram showing the result of uniform determination. A conceptual diagram showing the result of number determination. A conceptual diagram showing the result of player identification. A flowchart showing the operation according to another example of player identification. A conceptual diagram showing correspondence information between players and worn sensors. A conceptual diagram showing the result of position determination using video. A conceptual diagram showing the result of player identification. A flowchart showing the operation according to another example of player identification. A functional block diagram of the control unit 12 according to Modification 2. A flowchart showing the operation of the control unit 12 according to Modification 2. 12 is a diagram showing an example of an image displayed on the display 11. FIG. 13 is a functional block diagram of the control unit 12 according to Modification 3. FIG. 14 is a flowchart showing the operation of the control unit 12 according to Modification 3. FIG. 15 is a functional block diagram of the control unit 12 according to Modification 4. FIG. 16 is a flowchart showing the operation of the control unit 12 according to Modification 4. FIG. 17 is an example of an image displayed on the display 11. FIG. 18 is a functional block diagram of the control unit 12 according to Modification 5. FIG. 19 is a flowchart showing the operation of the control unit 12 according to Modification 5. FIG. 19 is a diagram showing an example of a display when the organizational performance determination unit 1281 determines that a player has fallen off the mark. FIG. 20 is a diagram showing an example of a display when the organizational performance determination unit 1281 determines that the defensive formation has collapsed. FIG. 21 is a diagram showing an example of a display when the organizational performance determination unit 1281 determines a passing route for the offensive team. FIG. 22 is a diagram showing an example of a display when the organizational performance determination unit 1281 detects space into which the offensive team should run. FIG. 23 is a functional block diagram of the control unit 12 according to Modification 6.40(A) is a schematic diagram of an image capturing device 2A supported by a plurality of physical lines and movable in a planar direction, and FIG. 40(B) is a schematic diagram of an image capturing device 2B supported by rails and movable in a planar direction. A block diagram showing the overall configuration of a motion determination system according to an application example. A diagram showing a display example when the motion determination system of this embodiment is applied to a yacht or windsurfing race. A diagram showing a display example when the motion determination system of this embodiment is applied to a yacht or windsurfing race. FIG. 44(A) is a schematic plan view of a snowboard, and FIG. 44(B) is a schematic plan view of skis.

[0040] 1 is a block diagram showing the overall configuration of a motion determination system, which includes an information processing device 1, an image capturing device 2, and a sensor 3. A controller unit 101 is connected to the information processing device 1.

[0041] The information processing device 1 is an example of a movement determination device. The information processing device 1 is a general-purpose personal computer, smartphone, or tablet computer used by a user. The information processing device 1 is used by a user who provides the movement determination system of this embodiment as a business. The imaging device 2 captures video from above or diagonally above the stadium. The imaging device 2 is also used by a user who provides the movement determination system of this embodiment as a business. The sensor 3 is worn on the body of an athlete in a competition.

[0042] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 includes a display 11, a control unit 12, a flash memory 13, a RAM 14, a communication I / F 15, and a touch panel 16. The display 11, the control unit 12, the flash memory 13, the RAM 14, the communication I / F 15, and the touch panel 16 are connected to one another via a bus.

[0043] The control unit 12 has one or more processors such as a CPU, an MPU, or a DSP, and controls the operation of the information processing device 1. The control unit 12 performs various operations by reading predetermined programs stored in a flash memory 13, which is a storage medium, into a RAM 14 and executing the programs.

[0044] The display 11 is configured with, for example, an LCD or an OLED, and displays various screens under the control of the control unit 12.

[0045] The communication I / F 15 communicates with other devices via a communication function such as USB, HDMI (registered trademark), wireless LAN, infrared, Bluetooth (registered trademark), etc. For example, the communication I / F 15 receives a control command for the image capturing device 2 from the controller unit 101.

[0046] The touch panel 16 is stacked on the display 11. The touch panel 16 accepts various operations from the user. The display 11 and the touch panel 16 constitute a graphical user interface (hereinafter referred to as GUI).

[0047] Fig. 3 is a block diagram showing the configuration of the image capture device 2. Fig. 4 is a schematic diagram showing the exterior of the image capture device 2. The image capture device 2 is an aircraft that performs autonomous flight, such as a multi-rotor helicopter. Such an image capture device 2 is called an unmanned aerial vehicle (UAV), an aircraft, a drone, a multicopter, an RPAS (Remote Piloted Aircraft Systems), or a UAS (Unmanned Aircraft Systems), etc.

[0048] The photographing device 2 includes a camera 21 , a control unit 22 , a photographing control unit, a flash memory 23 , a RAM 24 , a communication I / F 25 , a sensor 26 , a camera attitude driving unit, and a driving unit 27 .

[0049] The control unit 22 has one or more processors, such as a CPU, MPU, or DSP, and controls the operation of the image capture device 2. The control unit 22 performs various operations by loading predetermined programs stored in a flash memory 23, which is a storage medium, into a RAM 24 and executing the programs. For example, the control unit 22 controls the drive unit 27 to control the position, attitude, speed, etc. of the image capture device 2. Furthermore, if the image capture device 2 includes a camera attitude drive unit that can change the relative attitude between the aircraft and the camera 21, the control unit 22 controls the camera attitude drive unit to control the relative angle between the camera 21 and the image capture device 2 to an arbitrary angle. The image capture control unit also controls image capture parameters such as the zoom amount and brightness of the camera 21. The control unit 22 also transmits image information captured by the camera 21 to the information processing device 1 via the communication I / F 25.

[0050] As an example, the drive unit 27 has four motors and four propellers. The control unit 22 controls the four motors of the drive unit 27 to control the rotation speed of the propellers, thereby moving the image capture device 2 with six degrees of freedom (translational motion x, y, and z, and rotational motion θx, θy, and θz). In addition, as an example, the camera attitude drive unit controls a motor mounted on a holder that holds the camera to the image capture device, thereby changing the relative angle between the camera and the image capture device to any angle around three axes (θx, θy, and θz).

[0051] The flash memory 23 stores map information including two-dimensional or three-dimensional map data. The sensor 26 includes an acceleration sensor, a gyro sensor, a GPS sensor, etc. The control unit 22 reads the map information from the flash memory 23 and acquires from the sensor 26 information on the current position, attitude, speed, and acceleration of the image capture device 2 in two-dimensional or three-dimensional space. The control unit 22 may also acquire image capture control parameters, such as the zoom amount of the camera, controlled by the image capture control unit. Furthermore, the control unit 22 may acquire image capture data captured by the camera for temporary storage before transmitting it to the information processing device 1. The control unit 22 moves the image capture device 2 to a predetermined aerial position (a predetermined latitude, longitude, and altitude) and autonomously flies the image capture device 2 so that it hovers at that position.

[0052] The sensor 26 may include a proximity sensor (e.g., a LiDar lidar, a time-of-flight sensor, a laser radar, or a dual camera) for detecting obstacles. The proximity sensor may detect obstacles not only in one direction but also in all six directions, i.e., forward, backward, up, down, left, and right. When the sensor 26 detects an obstacle, the control unit 22 may control the drive unit 27 to avoid the obstacle and fly the imaging device 2 autonomously.

[0053] A user controls the imaging device 2 via the controller unit 101. The controller unit 101 is connected to the information processing device 1. The controller unit 101 has controls such as a control stick that can be tilted left and right, a pair of left and right push buttons (including the left and right buttons of a directional pad), or a rotary dial, and accepts control operations from the user. When the control is a control stick, the user can input control operations by tilting the control stick from a neutral, upright position.

[0054] The controller unit 101 may be realized by a software configuration using the GUI of the information processing device 1.

[0055] The control unit 12 of the information processing device 1 receives control commands corresponding to operations for operating the image capture device 2 from the user via the controller unit 101 connected by the communication I / F 15. The control commands include ascending, descending, moving forward, moving backward, moving left and right, turning left, turning right, turning up and down, turning forward and backward, etc. The control commands also include commands for panning and tilting the camera 21, changing the camera attitude, and changing the zoom amount. Alternatively, the information processing device 1 can display a map of potential image capture positions for the image capture device 2 on the display 11 and receive control commands for operating the image capture device 2 from the user by accepting input of a selected image capture position on the display.

[0056] Fig. 5 is a block diagram showing the configuration of the sensor 3. Fig. 6 is a schematic diagram showing how the sensor 3 is worn.

[0057] The sensor 3 includes a sensor unit 31 , a control unit 32 , a flash memory 33 , a RAM 34 , and a communication I / F 35 .

[0058] The sensor unit 31 has a compass sensor or a position sensor that uses satellite signals such as GPS. For example, the sensor unit 31 may consist of a pair of compass sensors attached to both shoulders of the player as shown in FIG. 6 . Alternatively, the sensor unit 31 may consist of a pair of compass sensors attached to the front and back of the player's torso. Alternatively, the sensor unit 31 may consist of a pair of compass sensors attached to each shoe of the player. The sensor unit 31 may also be a combination of these pairs of compass sensors. The sensor unit 31 outputs the direction information of each of these pair of compass sensors as sensing information.

[0059] However, in the present invention, the sensors worn by the player are not limited to a pair of compass sensors. For example, the sensor worn by the player may be a single compass sensor attached to the player's torso (for example, at the waist). In this case, the reference direction of the compass sensor corresponds to the direction in which the player is facing. Furthermore, the sensor worn by the player may be any sensor capable of sensing the orientation of the body. For example, the sensor may be a yaw rate sensor attached to the player's waist and detecting the rotation angle of the player's waist. Furthermore, the orientation of the player's torso may be detected by attaching a geomagnetic sensor that detects orientation using geomagnetism to the player's torso.

[0060] The control unit 32 has one or more processors such as a CPU, an MPU, or a DSP, and controls the operation of the sensor 3. The control unit 32 performs various operations by loading predetermined programs stored in a flash memory 33, which is a storage medium, into a RAM 34 and executing the programs. For example, the control unit 32 acquires sensing information from the sensor unit 31. The control unit 32 also transmits the sensing information to the information processing device 1 via the communication I / F 35.

[0061] The control unit 12 of the information processing device 1 determines the movement of the player, including the orientation of the player's body, based on the video information received from the image capturing device 2 and the sensing information received from the sensor 3 .

[0062] Fig. 7 is a functional block diagram of the control unit 12. Fig. 8 is a flowchart showing the operation of the control unit 12.

[0063] The control unit 12 functionally includes a video information acquisition unit 121, a sensing information acquisition unit 122, an operation determination unit 123, and an output unit 124.

[0064] The video information acquisition unit 121 acquires video information from the filming device 2 (s11). The video information includes video captured from above or obliquely above the stadium. The sensing information acquisition unit 122 acquires sensing information from the sensor 3 (s12).

[0065] The movement determination unit 123 determines the movement of the player, including the direction of the player's body, based on the acquired video information and sensing information (s13). The movement of the player may be determined based on the speed or acceleration of the player, in addition to the direction of the player's body.

[0066] 9 is a flowchart showing the processing of player movement determination by the movement determination unit 123. The movement determination unit 123 recognizes images of players from the acquired video information (s131). The movement determination unit 123 uses, for example, a deep-learned neural network model as a general object detection algorithm to recognize images of the field and players from the acquired video information and determine the position of the player on the field.

[0067] Furthermore, the motion determination unit 123 may recognize images of the player's uniform using a trained neural network model that has previously undergone deep learning of images of the player's uniform. FIG. 10(A) is a diagram showing an example of the front of a uniform, and FIG. 10(B) is a diagram showing an example of the back of the uniform. As shown in FIG. 10(A), as an example, the front of the uniform has the player's number and team emblem. As shown in FIG. 10(B), as an example, the back of the uniform has the player's number and name. Generally, the player's number is placed large in the center of the back of the uniform, and the player's number is placed small on the right side of the front.

[0068] The motion determination unit 123 constructs a trained neural network model by deep learning the arrangement of the player numbers, team emblems, and player names in a predetermined neural network model in advance. The motion determination unit 123 inputs the acquired video information into the trained neural network model and causes it to output the positions of the player images on the field.

[0069] The movement determination unit 123 stores the position information of the player's image on the field in the flash memory 13 (s132). The movement determination unit 123 reads the past position information stored in the flash memory 13, and calculates the player's speed or acceleration based on changes in the position information (s133).

[0070] The motion determination unit 123 then determines the player's motion, including the player's body orientation (s134). The motion determination unit 123 determines the player's orientation using a trained neural network model that has previously deep-learned the relationship between the image of the player's uniform and the player's orientation. Furthermore, the motion determination unit 123 determines the player's orientation from the sensing information. The sensor unit 31 includes, for example, a pair of compass sensors (first and second compass sensors) attached to the player's shoulders, as shown in FIG. 6 . The pair of compass sensors may also be attached to other positions where the player's body orientation can be determined, such as the player's hips, the front or back of the torso, or the player's shoes. The motion determination unit 123 calculates the average of the orientations indicated by the first and second compass sensors included in the sensing information, and determines the average as the player's facing orientation. The motion determination unit 123 then determines the player's body orientation based on the reference orientation of the field and the player's facing orientation. The reference orientation of the field can be obtained from map information, for example, or can be input in advance by the user.

[0071] The movement determination unit 123 determines, as a determination result, the average value of the player's body orientation information obtained based on both the video information and the sensing information. Alternatively, the movement determination unit 123 may determine a weighted average of the player's body orientation information obtained based on both the video information and the sensing information. For example, in sports with fast movements, the accuracy of the player's body orientation information obtained based on the video information is high, so the weighting of the player's body orientation information obtained based on the video information may be increased. On the other hand, in sports with fast movements, the accuracy of the player's body orientation information obtained based on the video information is low, so the weighting of the player's body orientation information obtained based on the sensing information may be increased.

[0072] In this way, the movement determination unit 123 determines the movement, including the orientation of the player's body, based on the acquired video information and sensing information. The output unit 124 outputs the result of the determination by the movement determination unit 123 (s14). For example, the output unit 124 transmits the determination result to another information processing device (an information processing device used by the team's manager or coach). The other information processing device displays the determination result. Alternatively, the output unit 124 may display the determination result on the display 11 of its own device. In this case, the control unit 12 functionally includes a display control unit.

[0073] 11 is a user interface screen showing an example of a display of the determination result. The determination result is displayed, for example, as a user interface screen (GUI) of an application program. The GUI includes a field image that resembles a field, player images that resemble players on the field, and motion images that correspond to the movements of the players. The determination result is displayed, for example, as an arrow as shown in FIG. 11.

[0074] Furthermore, if the determination result includes the speed or acceleration of the player, the GUI may include a vector display according to the speed or acceleration of the player.

[0075] This allows the team manager or coach to obtain more accurate information regarding the movements of the players, including the direction of their bodies, in order to coach the players more effectively. Also, in sports where the players move quickly, for example, it is difficult to accurately determine the movements of the players using video alone, but the movement determination system of this embodiment determines the direction of the players' bodies based on sensing information obtained by sensors worn by the players in addition to video information, so it is possible to determine the direction of the players' bodies with high accuracy even in sports where the movements are fast.

[0076] Next, Fig. 12 is a functional block diagram of the control unit 12 according to Modification 1. Components common to those in Fig. 7 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 13 is a flowchart showing the operation of the control unit 12 according to Modification 1. Processes common to those in Fig. 8 are given the same reference numerals, and descriptions thereof will be omitted.

[0077] The control unit 12 further includes a player identification unit 125. The player identification unit 125 identifies the players appearing in the captured video based on the video information or based on both the video information and the sensing information (s30).

[0078] 14 is a flowchart showing the operation of player identification. The player identification unit 125 acquires video information of each player (s301). The player identification unit 125 recognizes images of players from the acquired video information using, for example, a deep-learned neural network model as a general object detection algorithm.

[0079] Next, the player identification unit 125 determines the uniforms of the players using, for example, a trained neural network model that has previously undergone deep learning on images of the players' uniforms (s302). Figure 15 is a conceptual diagram showing the results of uniform determination. Using the trained neural network model, the player identification unit 125 tags the images of each player with the team they belong to, as shown in Figure 15, based on the clothing color or team emblem information in the video information.

[0080] The player identification unit 125 then determines the player's number using a trained neural network model that determines the player's number from the video information (s303). Figure 16 is a conceptual diagram showing the result of number determination. Using the trained neural network model, the player identification unit 125 tags the image of each player with a number, as shown in Figure 16.

[0081] The player identification unit 125 determines whether the uniforms and numbers of all players have been determined (s304). If the uniforms and numbers of all players have not been determined, the player identification unit 125 repeats the process from s301.

[0082] When the uniforms and numbers of all players have been determined, the player identification unit 125 identifies the players in the video based on the team to which the players belong obtained in s302 and the player numbers obtained in s303 (s305).

[0083] 17 is a conceptual diagram showing the results of player identification. The player identification unit 125 associates the team and number tags of images of the same player, among the team to which the player belongs obtained in Ss02 and the player number obtained in S303, and tags all players with identification information.

[0084] This allows the information processing device 1 to determine the orientation of each of multiple players with high accuracy even in a sport such as soccer, which is played by multiple players.

[0085] In the above embodiment, an example in which the sensor 3 is attached to one player has been described. However, in team sports such as soccer, rugby, American football, hockey, basketball, or yacht racing, a sensor 3 may be attached to each of multiple players. FIG. 18 is a flowchart showing the operation of another example of player identification. The player identification unit 125 in this example identifies players appearing in captured video based on both video information and sensing information. In this example, a sensor 3 is attached to each of multiple players.

[0086] The player identification unit 125 acquires sensor identification information and sensor position measurement information (s311). The sensor identification information is unique identification information (such as a serial number or MAC address) of the sensor 3 attached to each of the multiple players. The sensor position measurement information is acquired by the sensor unit 31 of the sensor 3 attached to each of the multiple players. In this case, the sensor unit 31 has a position sensor such as a GPS. Alternatively, the sensor position measurement information can be acquired by a beacon that transmits and receives radio waves, such as Bluetooth (registered trademark). At least three beacons are installed on the field in advance. Each beacon transmits radio waves. The sensor unit 31 of the sensor 3 receives the radio waves from the beacon and transmits a reply signal in response to the radio waves. Each beacon receives the reply signal from the sensor 3. Each beacon measures the distance to the sensor 3 based on the time difference between transmitting the radio waves and receiving the reply signal. The player identification unit 125 acquires distance information from the multiple beacons to the sensor 3. This allows the player identification unit 125 to uniquely determine the position of the sensor 3 within the field, and obtain sensor position measurement information.

[0087] The player identification unit 125 acquires correspondence information between the players and the worn sensors (s312). Fig. 19 is a conceptual diagram showing the correspondence information between the players and the worn sensors. The correspondence information indicates the correspondence between the identification information of the sensor 3 and the team and number information of each player. The identification information of the sensor 3 and the team and number information of each player are input in advance by the user when attaching the sensor 3 to each player.

[0088] Next, the player identification unit 125 acquires the sensor position information for each player and adds it to the correspondence information (s313). The sensor position information corresponds to the position information within the field as described above. This allows the identification information of the sensor 3 and the sensor position information to be associated with each player.

[0089] Next, the player identification unit 125 recognizes images of the field and players from the acquired video information using, for example, a deep-learned neural network model as a general object detection algorithm, and determines the position of each recognized player on the field (s314). Figure 20 is a conceptual diagram showing the results of determining positions from video. The player identification unit 125 attaches in-image player tags (a to j in the figure) to the images of each player recognized using the trained neural network model, and associates position information with each in-image player tag.

[0090] The player identification unit 125 determines whether the player's position information estimated in the video can be associated with the sensor position information (s315). The player identification unit 125 compares the sensor position information calculated in s313 with the position of each player on the field recognized from the video in s314, and associates the two if the distance between the two is within a predetermined value. The player identification unit 125 repeats the process from s311 until it determines that the player's position information estimated in the video can be associated with the sensor position information for all players. If the player identification unit 125 determines that the player's position information estimated in the video can be associated with the sensor position information for all players, it identifies the players in the video (s316).

[0091] In this way, the information processing device 1 can accurately determine the orientation of each of multiple players even in a sport such as soccer, which is played by multiple players. In particular, the information processing device 1 of this embodiment can accurately determine the orientation of each of multiple players even when multiple players overlap in an aerial video.

[0092] 21 is a conceptual diagram showing the results of player identification. The player identification unit 125 associates the sensor position information obtained in s313 with the position on the field of each player recognized from the video in s314, and associates the position, team, and number information for each player.

[0093] FIG. 22 is a flowchart showing the operation of another example of player identification. Processes common to those in FIG. 18 are assigned the same reference numerals and will not be described again. In this example, the player identification unit 125 performs uniform determination (s317) if it determines in s315 that the player's estimated position information in the video cannot be associated with the sensor position information. Uniform determination is similar to the process in s302, and is a process of determining the player's uniform using, for example, a trained neural network model that has previously undergone deep learning on images of the player's uniform. Using the trained neural network model, the player identification unit 125 tags each player's image with the team they belong to based on the clothing color or team emblem information in the video information.

[0094] The player identification unit 125 then identifies the players based on their team and uniform determination (s318). For example, when multiple players are crowded together, the sensing information from multiple sensors 3 may overlap, making it difficult to accurately identify players using only the sensor information from s311 to s314. However, as shown in FIG. 22 , the player identification unit 125 can accurately determine sensor identification information corresponding to player images in the video by combining the processing from s311 to s316 with the uniform determination processing based on video information from s317 and s318.

[0095] The player identification unit 125 may identify personal attributes including those of the referee and goalkeeper. Alternatively, the player identification unit 125 may identify substitute players on the bench, coaches, and other related parties. This allows the information processing device 1 of this embodiment to provide information identifying the teams to which players belong, the referees, the goalkeepers, and the like in a competition, and to provide more useful information for more effective coaching of players or competition tactics.

[0096] Next, Fig. 23 is a functional block diagram of the control unit 12 according to Modification 2. Components common to those in Fig. 12 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 24 is a flowchart showing the operation of the control unit 12 according to Modification 2. Processes common to those in Fig. 13 are given the same reference numerals, and descriptions thereof will be omitted.

[0097] The control unit 12 further includes a display control unit 126 in addition to the configuration of Modification Example 1. The display control unit 126 displays the player's identification information and body orientation information on the video (s90). FIG. 25 shows an example of an image displayed on the display 11. The display control unit 127 displays, as an example, an arrow image corresponding to the body orientation of each player on the video. The display control unit 127 also displays, as an example, identification information near each player on the video. The display control unit 127 also displays, as an example, an arrow image corresponding to the speed or acceleration of each player on the video (however, this image has a different form from the arrow indicating the body orientation).

[0098] In this way, the information processing device 1 of the second modification can visually provide information on the body orientation determined for each player.

[0099] Next, Fig. 26 is a functional block diagram of the control unit 12 according to Modification 3. Components common to those in Fig. 12 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 27 is a flowchart showing the operation of the control unit 12 according to Modification 3. Processes common to those in Fig. 13 are given the same reference numerals, and descriptions thereof will be omitted.

[0100] The control unit 12 further includes an individual performance determination unit 128. The individual performance determination unit 128 determines the performance of each identified player based on information including the determined body orientation. The individual performance includes, for example, the cumulative distance traveled in the competition, top speed, number of sprints, offensive evaluation score, or defensive evaluation score. The offensive evaluation score is added, for example, for sprinting in the opposite direction to the defensive player's body orientation, staying a predetermined distance or more away from the defensive player for a long time, running into space, creating a passing route, having a high top speed, high acceleration, a large number of sprints at a predetermined acceleration or more, a long cumulative running distance, or having the body facing forward toward the offensive team when receiving the ball.

[0101] Defensive evaluation points are awarded for, for example, turning the body halfway in relation to the vector direction of the offensive player's speed or acceleration (for example, the angle difference with the offensive player is 120 degrees or less), keeping the distance between other defensive players to a specified distance or less, keeping the distance between the offensive player to a specified distance or less, and being closer to the goal than the offensive player.

[0102] Alternatively, individual performance may be evaluated not only based on the instantaneous position and body orientation of a player, but also based on time-series information from when the player receives the ball to when the player takes a shot. For example, the individual performance determination unit 128 may generate a probability model for scoring opportunities based on statistical information on the positions of each player and the ball and the body orientation of each player in the time-series from when the player receives the ball to when the player takes a shot, and evaluate players based on the probability calculated by the probability model. For example, when the individual performance determination unit 128 determines that the probability of a scoring opportunity is high based on the probability model, it may lower the evaluation score of the player if the scoring opportunity does not occur. When the individual performance determination unit 128 determines that the probability of a scoring opportunity is low, it may increase the evaluation score of the player if the scoring opportunity occurs.

[0103] As a result, the movement determination system of this embodiment can provide more useful information for coaching athletes or for implementing competition tactics more effectively.

[0104] Fig. 28 is a functional block diagram of the control unit 12 according to Modification 4. Components common to those in Fig. 26 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 29 is a flowchart showing the operation of the control unit 12 according to Modification 4. Processes common to those in Fig. 27 are given the same reference numerals, and descriptions thereof will be omitted.

[0105] The control unit 12 further includes a display control unit 129 in addition to the configuration of Modification Example 4. The display control unit 129 displays the identification information of the player and the performance evaluation judgment information on the image (s91). Fig. 30 is an example of an image displayed on the display 11. The display control unit 129 displays, as an example, the identification information of each individual player on the image, and the performance evaluation judgment information, such as the accumulated movement distance, top speed, number of sprints, offensive evaluation score, defensive evaluation score, and trajectory image.

[0106] As a result, the information processing device 1 of the fourth modification can visually provide more useful information for coaching players or for implementing competition tactics more effectively.

[0107] Next, Fig. 31 is a functional block diagram of the control unit 12 according to Modification 5. Components common to those in Fig. 12 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 32 is a flowchart showing the operation of the control unit 12 according to Modification 5. Processes common to those in Fig. 13 are given the same reference numerals, and descriptions thereof will be omitted.

[0108] The control unit 12 includes an offensive / defensive state determination unit 1280 and an organizational performance determination unit 1281. The offensive / defensive state determination unit 1280 determines which team is in an offensive state and which team is in a defensive state (s45). The offensive / defensive state determination unit 1280 recognizes an image of the ball, for example, from video information. The offensive / defensive state determination unit 1280 determines that the team to which the player closest to the ball belongs is in an offensive state. Alternatively, the offensive / defensive state determination unit 1280 may determine the offensive / defensive state based on the vectors of the players of each team. For example, if more than half of the players have vectors pointing in the direction of the opposing team, the offensive / defensive state determination unit 1280 determines that the team is in an offensive state.

[0109] The organizational performance determination unit 1281 determines the performance of the organization of multiple offensive players or the performance of the organization of multiple defensive players (s46).

[0110] For example, the offensive team's organizational performance can be determined by determining whether each offensive player is positioned a predetermined distance away from the defensive player (1), whether the difference vector of the velocity or acceleration vector between the offensive player and the defensive player is greater than a predetermined value (2), whether the difference vector differs from the defensive player's body orientation by more than a predetermined angle (3), whether the value of the velocity or acceleration vector is greater than a predetermined value (4), etc.

[0111] The defensive team's organizational performance is determined by determining whether each defensive player is positioned within a predetermined distance from the offensive player (5), whether the difference vector of the velocity or acceleration vector between the offensive player and the defensive player is smaller than a predetermined value (6), whether the angle relative to the offensive player's body direction is less than a predetermined angle (7), and whether the value of the velocity or acceleration vector is greater than a predetermined value (8).

[0112] These predetermined values ​​may be changed depending on the state of the game. For example, these predetermined values ​​may be set high immediately after the start of the game and then set lower as the game progresses. Furthermore, the determinations (1) to (8) may be made using arithmetic expressions such as AND and OR.

[0113] 33 is a diagram showing a display example when the organizational performance determination unit 1281 determines whether an offensive player is unmarked. The organizational performance determination unit 1281 determines whether an offensive player is unmarked based on the identification information of the team to which the identified player belongs, the position of the offensive player, and the determination results of the position, speed, and body orientation of the defensive player.

[0114] For example, the organizational performance determination unit 1281 determines that an offensive player is unmarked when the determination values ​​(or values ​​of arithmetic expressions such as AND, OR) of (1) to (4) above exceed a predetermined value.

[0115] 33, the display control unit 129 may display on the video the offensive players and defensive players who are to be unmarked as determined by the organizational performance determination unit 1281. Furthermore, the display control unit 129 may highlight the offensive players who are to be unmarked as shown in FIG.

[0116] For example, in sports such as soccer, it is very important to determine whether an offensive player has become unmarked. The information processing device 1 of this embodiment can determine whether an offensive player has become unmarked and provide more useful information.

[0117] 34 is a diagram showing a display example when the organizational performance determination unit 1281 determines that the defensive formation has collapsed. The organizational performance determination unit 1281 determines that the organized defensive formation has collapsed, for example, when the determination values ​​(or values ​​of the arithmetic expressions such as AND and OR) of (5) to (8) above exceed a predetermined value.

[0118] The display control unit 129 may display on the video the players whose defensive formation has been disrupted as determined by the organizational performance determination unit 1281, as shown in Fig. 34. The display control unit 129 may also highlight the disruption in the defensive formation as shown in Fig. 34. For example, in the example of Fig. 34, the defensive formation is displayed with dashed lines, and offensive players who deviate from the dashed lines are highlighted.

[0119] For example, in sports such as soccer, it is very important to determine whether an organized defensive formation has collapsed. The information processing device 1 of this embodiment can determine whether the defensive formation has collapsed and provide more useful information.

[0120] 35 is a diagram showing a display example when the organizational performance determination unit 1281 determines the offensive pass route. The organizational performance determination unit 1281 determines the offensive pass route based on the team identification determination of each player, the positions and body orientations of the offensive players, and the positions of the defensive players.

[0121] 35, the organizational performance determination unit 1281 detects, as a passing route, a line that does not include a defensive player among the lines connecting offensive players. Note that the organizational performance determination unit 1281 may exclude a line that points towards the back of a player who is holding the ball, based on the direction in which the body of the player is facing, from the passing route.

[0122] The display control unit 129 may highlight the path course as shown in FIG.

[0123] For example, in sports such as soccer, determining the offensive pass route is very important. The information processing device 1 of this embodiment can determine the offensive pass route and provide more useful information.

[0124] 36 is a diagram showing a display example in which the organizational performance determination unit 1281 detects spaces into which the offensive players should run. The organizational performance determination unit 1281 detects spaces into which the offensive players should run based on the team identification determination of each player and the positions and body orientations of multiple defensive players.

[0125] The organizational performance determination unit 1281 determines, as a space into which a player should run, an area that is more than a predetermined distance from the defensive player, as shown in Fig. 36 . Note that the organizational performance determination unit 1281 may preferentially determine, as a space into which a player should run, a space behind the defensive player, based on the direction in which the player's body is facing. Furthermore, the display control unit 129 may highlight a player or position related to the space, as shown in Fig. 36 .

[0126] For example, in sports such as soccer, it is very important to detect spaces into which the offensive team should run. The information processing device 1 of this embodiment can detect spaces into which the offensive team should run and provide more useful information.

[0127] In addition, the evaluation of organizational performance may be based not only on the momentary position and body orientation of the players, but also on the time-series information from when the ball is received to when the shot is taken.

[0128] Furthermore, organizational performance may be evaluated based on the duration of time during which a missed mark or a space is determined to have occurred. For example, the evaluation may be different depending on whether the duration of time during which a missed mark or a space is determined to have occurred is 1 second or 5 seconds.

[0129] Furthermore, individual performance or organized offense may be evaluated based on the timing of when a player becomes unmarked or when space is created. For example, an offensive player who becomes unmarked when a crossing pass event occurs or when a final pass event occurs in front of the goal may be evaluated higher, while an offensive player who becomes unmarked at a time other than when these events occur may be evaluated relatively lower.

[0130] Next, Fig. 37 is a functional block diagram of the control unit 12 according to Modification 6. Components common to those in Fig. 31 are given the same reference numerals, and descriptions thereof will be omitted. Fig. 38 is a flowchart showing the operation of the control unit 12 according to Modification 6. Processes common to those in Fig. 32 are given the same reference numerals, and descriptions thereof will be omitted.

[0131] The control unit 12 according to the sixth modification includes a tag processing unit 220. The tag processing unit 220 performs tagging processing to accept a tagging input from a user for video information corresponding to a noteworthy play among the video information (S61). That is, the information processing device 1 according to the sixth modification accepts a manual tagging input for a video when a user such as a coach determines that the video is a noteworthy play. The tag may include, for example, time information from the start of the game. Alternatively, the tag may include additional information such as player information, area information, and whether the player is near or far from the ball. The tag is recorded for each player and stored in the flash memory 23.

[0132] This allows players, managers, coaches, and managers to review noteworthy plays later, and also reduces the effort required for managers to keep score.

[0133] Alternatively, the tag processing unit 220 may automatically tag the video of a play that has been lost from its marker or broken down in formation, a pass route, or space determined in S61 by the organizational performance determination unit 1281. In this case, tagging is automatically performed on the video of a play that has been lost from its marker or broken down in formation, a pass route, or space determined, making it easier for players, coaches, managers, or the like to analyze notable plays later when reviewing them.

[0134] FIG. 39 is a diagram showing a display example for making other judgments. In the example of FIG. 39 , for example, the offside line is displayed as a dashed line. The offside line is made up of the second defensive player from the defensive team's own goal, including the goalkeeper. Furthermore, in the example of FIG. 39 , the defensive players that make up the offside line are highlighted. Furthermore, in the example of FIG. 39 , lines connecting all players are displayed. For example, all offensive players are connected by solid lines. Furthermore, all defensive players are connected by dashed lines. This makes it easier for coaches and other personnel to recognize distance relationships and the presence or absence of passing routes. In particular, when acquiring video information captured from diagonally above, players often do not overlap. Therefore, by displaying the teammate's line surrounding a specific player as a triangle or larger, coaches and other personnel can easily recognize the support distance of a specific player.

[0135] In this embodiment, the imaging device 2 is shown as a multi-rotor helicopter that performs autonomous flight, but the imaging device of the present invention does not necessarily have to be a multi-rotor helicopter. For example, FIG. 40(A) is a schematic diagram of an imaging device 2A that is supported by multiple physical lines (wires, etc.) and is movable to any position in two-dimensional (planar) or three-dimensional space. The movement determination system of this embodiment can also be realized with this type of imaging device 2A. FIG. 40(B) is a schematic diagram of an imaging device 2B that is supported by rails and is movable to any position on the rails. The movement determination system of this embodiment can also be realized with this type of imaging device 2B. Furthermore, the imaging device 2 may be a fixed camera rather than a movable camera.

[0136] In the above embodiment, an example was shown in which various processes of the motion determination method (program) of the present invention are performed by an information processing device 1 used by a user. However, the motion determination method (program) of the present invention does not require all processes to be performed by a single device. Figure 41 is a block diagram showing the overall configuration of a motion determination system according to an application example. Components common to those in Figure 1 are assigned the same reference numerals, and descriptions thereof will be omitted.

[0137] The movement determination system in FIG. 41 includes a server 5. The movement determination system also includes multiple information processing devices 1. The server 5 performs various operations in place of the information processing devices 1 described in FIGS. 1 to 40. The first information processing device 1 connected to the controller unit 101 is used by a user who operates the image capture device 2 (for example, a user who provides the movement determination system of this embodiment as a business). The other second information processing device 1 is used by related parties such as coaches. The second information processing device 1 includes the tag processing unit 220 shown in Modification 6 and accepts tagging input from related parties such as coaches. The server 5, the multiple information processing devices 1, and the sensor 3 are connected via a network such as the Internet or a LAN.

[0138] The server 5 includes a video information acquisition unit 121, a sensing information acquisition unit 122, a movement determination unit 123, and an output unit 124. The server 5 acquires sensing information from the sensor 3, and acquires video information captured by the image capture device 2 from the information processing device 1 of the user who operates the image capture device 2. The server 5 determines the movement of the player, including the orientation of the player's body, and outputs the determination results to the information processing device 1 used by the coach or other related person. The information processing device 1 used by the coach or other related person executes an application program of the movement determination system of this embodiment, and displays the determination results using the application program.

[0139] In this case, the server 5 has the main function of implementing the movement determination system of this embodiment. Therefore, the information processing device 1 used by the coach or other related parties only needs to perform the transmission and reception of information and display processing. Therefore, the information processing device 1 does not need to have advanced processing functions or storage capacity.

[0140] The description of the present embodiment is illustrative in all respects and is not restrictive. The scope of the present invention is defined not by the above-described embodiments but by the claims. Furthermore, the scope of the present invention is intended to include all modifications that are equivalent to the claims and fall within the scope thereof.

[0141] For example, although soccer is shown as an example of a sport in the above embodiment, the motion determination system of this embodiment can also be applied to other sports such as rugby, American football, hockey, basketball, yacht racing, windsurfing, etc.

[0142] 42 and 43 are diagrams showing display examples when the movement determination system of this embodiment is applied to yacht racing or windsurfing. Fig. 42 shows an example in which the direction of the sail of a yacht or windsurfing, the vector of the yacht or windsurfing board, the wind vector, or performance evaluation information is displayed on the video. Fig. 43 shows an example in which identification information of the yacht or windsurfing athlete, the direction of the sail of the yacht or windsurfing, the vector of the yacht or windsurfing, the wind vector, or other information such as performance evaluation information is displayed on an image resembling a yacht or windsurfing.

[0143] In a yacht or windsurfing race, the competition venue is the sea, and video information is acquired by photographing the sea from the sky or from an oblique angle above. The sensors may be attached to the athletes or to the sails of the yacht or windsurfing boats. The orientation of the athletes' bodies corresponds to the orientation of the sails. The sensing information may also include information on wind direction and wind strength.

[0144] The wind direction may be determined from the wave shape of the water surface reflected in the image, or may be measured by attaching a sensor to the imaging device 2. For example, the performance of a yacht or windsurfing race may be evaluated by determining whether the sails are oriented in a way that generates maximum thrust based on the wind direction relative to the direction of travel to the finish line, and then the evaluation score may be calculated.

[0145] The movement determination system of this embodiment is not limited to team sports, but can also be applied to individual sports such as snowboarding or skiing.

[0146] Fig. 44(A) is a schematic plan view of a snowboard. The sensors 3 may be attached to the athlete, or may be attached to the front and back of the snowboard as shown in Fig. 44(A). The orientation of the athlete's body corresponds to the orientation of the snowboard. Fig. 44(B) is a schematic plan view of skis. The sensors 3 may be attached to the athlete, or may be attached to the front and back of the skis as shown in Fig. 44(B). The orientation of the athlete's body corresponds to the orientation of the skis.

[0147] The sensor 3 may include a barometric pressure sensor. Alternatively, the sensor 3 may be attached to the boot and sense the pressure of the athlete's feet against the ski. Performance may be determined by the jump height based on the sensor 3. For example, the motion judging system may consider the period when the pressure attached to the ski is zero as the jump time and estimate the jump height based on the barometric pressure. Alternatively, the motion judging system may determine the horizontal direction based on video information and estimate the jump height as the altitude of the camera device 2 at the time of capturing the highest point of the image. The motion judging system may estimate the rotation angle of the ski based on the video information and the sensing information. The motion judging system may evaluate performance in snowboard or ski jumping events (such as half-pipe, slopestyle, or big air) based on the jump height and rotation angle. The motion judging system may also superimpose the jump height, rotation angle, and evaluation score on the captured video.

[0148] 1: Information processing device, 2, 2A, 2B: Imaging device, 3: Sensor, 5: Server, 11: Display, 12: Control unit, 13: Flash memory, 14: RAM, 15: Communication I / F, 16: Touch panel, 21: Camera, 22: Control unit, 23: Flash memory, 24: RAM, 25: Communication I / F, 26: Sensor, 27: Drive unit, 31: Sensor unit, 32: Control unit, 33: Flash memory, 34: RAM, 35: Communication I / F, 101: Controller unit, 121: Video information acquisition unit, 122: Sensing information acquisition unit, 123: Action determination unit, 124: Output unit, 125: Player identification unit, 126: Display control unit, 127: Display control unit, 128: Individual performance determination unit, 129: Display control unit, 220: Tag processing unit, 1280: Offensive / defensive state determination unit, 1281: Organizational performance determination unit

Claims

1. A movement judgment system comprising: an image acquisition unit that acquires image information captured by a camera from above or diagonally above a stadium; a sensor information acquisition unit that acquires sensing information by a sensor worn by a player; a movement judgment unit that judges the movement, including the orientation of the player's body, based on the image information and the sensing information; and an output unit that outputs the judgment results.

2. A movement determination system according to claim 1, wherein the sensor information acquisition unit acquires the sensing information from a pair of compass sensors attached to the player's shoulders, hips, front and back of the torso, or shoes on both feet.

3. A motion determination system as claimed in claim 1, further comprising a player identification unit which identifies a player appearing in the captured image based on the video information or based on both the video information and the sensing information.

4. A motion determination system as described in claim 3, wherein the sensing information acquired by the sensor information acquisition unit includes sensor position information from a sensor attached to a player and sensor identification information of the sensor, and the player identification unit determines the position of the player based on the sensor position information and the sensor identification information, and identifies the player appearing in the captured video based on the determined player position information of the player.

5. A motion determination system as claimed in claim 3, wherein the player identification unit at least either identifies the team and number of each player, or identifies personal attributes including those of the referee and goalkeeper, based on at least one of information such as the colour of clothing or uniform number of the person appearing in the acquired video.

6. The movement determination system according to claim 3, further comprising a display control unit that displays player identification information and body orientation information on the video.

7. A movement judging system as claimed in claim 3, further comprising an individual performance judging section for judging the performance of each identified player based on information including the judged body orientation.

8. A motion judging system according to claim 7, further comprising a display control unit that displays, on an image, identification information of the player and judgment information of the performance judgment.

9. A motion determination system as claimed in claim 3, comprising: an offensive / defensive state determination unit which determines offensive players and defensive players based on the offensive / defensive state; and an organizational performance determination unit which determines the organizational performance of multiple offensive players or the organizational performance of multiple defensive players.

10. A motion determination system as claimed in claim 9, wherein the organizational performance determination unit determines whether or not an offensive player has become unmarked based on the identification information of the team to which the identified player belongs, the position of the offensive player, and the determination results of the position, speed and body orientation of the defensive player.

11. A movement determination system as described in claim 9, wherein the organizational performance determination unit determines whether or not the defensive formation has collapsed based on identification information of the teams to which the identified players belong, the positions of the offensive players, and the positions and body orientations of the defensive players.

12. A motion determination system as claimed in claim 9, wherein the organizational performance determination unit determines the offensive team's passing route based on the team identification determination of each player, the positions and body orientations of the offensive players, and the positions of the defensive players.

13. A motion determination system as claimed in claim 9, wherein the organizational performance determination unit detects spaces into which the offensive team should run based on the team identification determination of each player and the positions and body orientations of multiple defensive players.

14. A motion judgment system as claimed in any one of claims 10 to 13, further comprising a display control unit which highlights on the video the players or positions related to a player who has lost his / her mark or is out of formation, a passing route, or space, as judged by the organizational performance judgment unit.

15. A motion determination system according to any one of claims 1 to 13, further comprising a tag processing unit that receives a tagging input for video information corresponding to a play of interest among said video information.

16. A motion determination system as claimed in any one of claims 10 to 13, further comprising a tag processing unit which performs tagging on video when the organizational performance determination unit determines that a player has lost his / her mark or broken formation, a passing route, or space.

17. A movement determination method comprising: acquiring video information from a camera that acquires video information taken from above or diagonally above a stadium; acquiring sensing information by a sensor worn by a player; and determining the orientation of the player's body based on the video information and the sensing information.

18. A program that causes an information processing device to execute the following processes: acquiring video information from a camera that acquires video information taken from above or diagonally above a stadium, acquiring sensing information using a sensor worn by the player, and determining the orientation of the player's body based on the video information and the sensing information.

Images