Computer system and method for analyzing physical movements of a person performing exercise

Abstract

To identify synchronism between sections of a person during exercise.SOLUTION: A computing system is configured to: acquire a video obtained by imaging an exercising person; calculate, for each of multiple images constituting the video, coordinates of multiple element points representing movable sections in exercise; generate, based on the coordinates of the element points in the images, time-series data on moving amount for each of the element points; calculate, using the time-series data on the moving amount of the element points, for each of element point pairs, a synchronism index for evaluating synchronism between the element points during exercise; and generate display information indicating synchronism between the element points during exercise, based on the synchronism indices of the element point pairs, for output.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a system and method for analyzing a person's physical movements during exercise. [Background technology]

[0002] In sports coaching and research, physical movements such as form have been analyzed by motion capture using markers and cameras. In recent years, with the development of image recognition technology, technologies have emerged that recognize people and detect their skeletons and analyze physical movements using only images.

[0003] For example, Patent Document 1 discloses a system comprising "a camera 21 that photographs the user while exercising, a display 22 installed at the location where the exercise is performed, an explanation output unit that outputs explanations about the exercise to the display, a body part identification unit that analyzes the image captured by the camera and identifies the positions of the user's body parts, and an output unit that superimposes the positions of the body parts on the image and outputs it to the display." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-174910 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 only identifies the parts of the user's body during exercise, and does not analyze the relationship between the parts. In analyzing human body movements during exercise, it is important to quantitatively evaluate the synchrony (relationship or causal relationship) between parts.

[0006] The present invention aims to realize a system and method for quantitatively evaluating and presenting synchrony between human parts during exercise. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a computer system for analyzing the physical movements of a person performing exercise includes at least one computer having a processor, a memory connected to the processor, and an interface connected to the processor, the at least one computer acquires a video of a person performing exercise, calculates the coordinates of a plurality of element points representing body parts of the person in each of a plurality of images constituting the video, stores the image and the coordinates of the plurality of element points in the memory in correspondence with each other, generates time-series data of the amount of movement of each of the plurality of element points based on the coordinates of the plurality of element points in each of the plurality of images, calculates a synchrony index for evaluating the synchrony between the element points during exercise for each of a plurality of element point pairs using the time-series data of the amount of movement of the plurality of element points, stores the synchrony index of the plurality of element point pairs in the memory, generates display information indicating the synchrony between the element points of the person during exercise based on the synchrony index of the plurality of element point pairs, and outputs The synchrony index of the element point pair is a movement entropy from one element point constituting the element point pair to the other element point constituting the element point pair, and when calculating the synchrony index, the at least one computer discretizes time series data of the movement amount of each of the plurality of element points, normalizes the time series data of the movement amount of each of the plurality of element points to a value that can be treated as a random variable, and calculates the synchrony index of each of the plurality of element point pairs using the time series data of the movement amount of each of the plurality of element points that has been discretized and normalized. . Effect of the Invention

[0008] According to the present invention, it is possible to grasp the synchronicity between parts of a person during exercise. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a system configuration according to a first embodiment. [Diagram 2] 4 is a diagram illustrating an example of a data structure of a video management DB according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram illustrating an example of a data structure of a skeleton / feature point management DB according to the first embodiment. [Figure 4] 13 is a diagram illustrating an example of a data structure of a synchronicity index management DB according to the first embodiment; FIG. [Diagram 5]11 is a flowchart illustrating an overview of a process executed by a server according to the first embodiment. [Figure 6A] 11 is a diagram showing a specific example of element points calculated by the server of the first embodiment. FIG. [Figure 6B] 11 is a diagram showing a specific example of element points calculated by the server of the first embodiment. FIG. [Figure 7] 13 is a diagram illustrating an example of calculation of a movement amount of an element point by the server of the first embodiment. FIG. [Figure 8] 11 is a flowchart illustrating an example of a synchronicity index calculation process executed by the server according to the first embodiment. [Figure 9] 11 is an image diagram of specific processing contents of a synchronicity index calculation process executed by the server of the first embodiment. FIG. [Figure 10] FIG. 11 is a diagram illustrating an example of a specific method for calculating a synchronicity index by the server according to the first embodiment. [Figure 11] 11 is a flowchart illustrating an example of a display process executed by the server according to the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of information presented by the server according to the first embodiment. [Figure 13] 11 is a flowchart illustrating an example of a display process executed by the server according to the first embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of information presented by the server according to the first embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of information presented by the server according to the first embodiment. [Figure 16] 11 is a flowchart illustrating an example of a display process executed by the server according to the first embodiment. [Figure 17] FIG. 11 is a diagram illustrating an example of information presented by the server according to the first embodiment. [Figure 18] 11 is a flowchart illustrating an example of a display process executed by the server according to the first embodiment. [Figure 19] FIG. 11 is a diagram illustrating an example of information presented by the server according to the first embodiment. [Figure 20] FIG. 11 is a diagram illustrating an example of a data structure of a synchronicity index management DB according to the second embodiment. [Figure 21]13 is a flowchart illustrating an outline of a process executed by a server according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not to be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or purpose of the present invention.

[0011] In the configurations of the invention described below, the same or similar configurations or functions are given the same reference symbols, and duplicated descriptions are omitted.

[0012] In this specification, the terms "first," "second," "third," and the like are used to identify components and do not necessarily limit the number or order.

[0013] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc. EXAMPLES

[0014] FIG. 1 is a diagram illustrating an example of a system configuration according to a first embodiment.

[0015] The system is composed of a server 100 and a user terminal 101. The server 100 and the user terminal 101 are connected via a network 105. The network 105 is a LAN (Local Area Network) or a WAN (Wide Area Network), and the connection method may be either wired or wireless.

[0016] The user terminal 101 is a terminal operated by a user. The user terminal 101 transmits to the server 100 a video of a person performing an exercise such as batting or pitching. The user terminal 101 also displays various information transmitted from the server 100. The user terminal 101 is, for example, a personal computer having a processor, memory, network interface, input device, output device, etc. (not shown), or a smartphone having a processor, memory, network interface, imaging device, input device, output device, etc. (not shown). When the user terminal 101 is a personal computer, in this embodiment, the user terminal 101 is assumed to be a smartphone.

[0017] The server 100 uses a video to analyze the synchronicity between parts of a person during exercise and presents the analysis result to the user. The server 100 has a processor 110, a memory 111, and a network interface 112. The server 100 may also have storage devices such as a hard disk drive (HDD) and a solid state drive (SSD), input devices such as a keyboard, a mouse, and a touch panel, and an output device such as a display.

[0018] Processor 110 is a computing device that controls the entire server 100, and executes programs stored in memory 111. Processor 110 executes processes according to the programs, thereby operating as a functional unit (module) that realizes a specific function. In the following explanation, when a process is explained with a program as the subject, it indicates that processor 110 is executing the program.

[0019] The memory 111 is a storage device that stores the programs executed by the processor 110 and information executed by the programs, and is, for example, a volatile or non-volatile memory. The memory 111 is also used as a work area. The programs and information stored in the memory 111 will be described later.

[0020] The network interface 112 is an interface for connecting to an external device such as the user terminal 101 via the network 105 .

[0021] Here, a description will be given of the programs and information stored in the memory 111. The memory 111 stores, as programs, a skeleton recognition program 120, a synchrony index calculation program 121, and a display program 122. The memory 111 also stores, as information, a moving image DB 130, a moving image management DB 131, a skeleton / feature point management DB 132, a synchrony index management DB 133, and a display setting DB 134.

[0022] The moving image DB 130 is a database for managing moving images. This embodiment is not limited to the format and frame rate of moving images.

[0023] The video management DB 131 is a database for managing metadata of videos. The data structure of the video management DB 131 will be described with reference to FIG.

[0024] The skeleton / feature point management DB 132 is a database for managing the coordinates of human body parts and feature points in one frame of an image that constitutes a video. Here, a feature point is, for example, the center of gravity of a person. For exercises that use a tool, the center of gravity of the tool is also treated as a feature point.

[0025] In the following description, human body parts such as elbows, knees, shoulders, and ankles, and feature points, that is, points indicating movable parts in motion, are referred to as element points. Details of the data structure of the skeleton / feature point management DB 132 will be described with reference to FIG. 3.

[0026] The synchrony index management DB 133 is a database for managing synchrony indexes for evaluating causal relationships between element points during exercise, that is, synchrony between element points. The details of the data structure of the synchrony index management DB 133 will be described with reference to FIG. 4.

[0027] The display setting DB 134 is a database for managing settings related to the display of analysis results, and stores setting information related to the contents to be presented, the layout of the screen to be presented, and the like.

[0028] FIG. 2 is a diagram illustrating an example of a data structure of the video management DB 131 according to the first embodiment.

[0029] The video management DB 131 stores entries including a video ID 201, a shooting date and time 202, a data name 203, a player name 204, a category 205, a location 206, a weather 207, and a comment 208. One entry corresponds to one video. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

[0030] Video ID 201 is a field that stores identification information of a video. Shooting date and time 202 is a field that stores the date and time when the video was shot. Data name 203 is a field that stores the name of the video. In the case of a video in a file format, the data name 203 stores the file name.

[0031] Athlete name 204 is a field that stores the name of the person performing the exercise in the video. Category 205 is a field that stores the type of exercise. Location 206 is a field that stores information about the location where the exercise was performed. Weather 207 is a field that stores information about the weather at the time of the exercise. Comment 208 is a field that stores a comment about the exercise.

[0032] FIG. 3 is a diagram illustrating an example of a data structure of the skeleton / feature point management DB 132 according to the first embodiment.

[0033] The skeleton / feature point management DB 132 stores entries including a video ID 301, time 302, and coordinates 303. One entry corresponds to one video. One entry includes lines each consisting of time and coordinates for the number of frames. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

[0034] The moving image ID 301 is the same field as the moving image ID 201. The time 302 is a field that stores the time in the moving image of one frame image (still image) that constitutes the moving image. The coordinates 303 are a group of fields that store the coordinates of element points in the still image. The coordinates 303 include a field for each element point.

[0035] FIG. 4 is a diagram illustrating an example of a data structure of the synchronicity index management DB 133 according to the first embodiment.

[0036] The synchrony index management DB 133 stores entries including a video ID 401, an analysis time range 402, and a synchrony index 403. One entry corresponds to one video. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

[0037] The moving image ID 401 is the same field as the moving image ID 201. The analysis time range 402 is a field that stores the time range of the moving image used to calculate the synchrony index. The synchrony index 403 is a field group that stores the synchrony index between element points. The synchrony index 403 includes a field for each element point pair.

[0038] Fig. 5 is a flowchart for explaining an outline of a process executed by the server 100 of the first embodiment. Fig. 6A and Fig. 6B are diagrams showing specific examples of element points calculated by the server 100 of the first embodiment. Fig. 7 is a diagram showing an example of calculation of the movement amount of an element point by the server 100 of the first embodiment.

[0039] When the server 100 receives an analysis request including information specifying a video or a video to be analyzed and an analysis time range from the user terminal 101, the server 100 executes the process described below. The information specifying the video to be analyzed is, for example, the name of the video, the type of exercise, and a person's name. The analysis time range is information for specifying the range of images to be analyzed in the video.

[0040] The skeleton recognition program 120 executes a skeleton / feature point calculation process using a moving image (step S101). Specifically, the following process is executed.

[0041] (S101-1) The skeleton recognition program 120 acquires one frame of an image from a video, identifies element points from the image, and calculates the coordinates of the identified element points. For example, element points such as those in FIG. 6A and FIG. 6B are identified. Circle marks represent body parts, and star and triangular marks represent feature points. The skeleton recognition program 120 can recognize a human skeleton by combining body parts. Note that a method for identifying body parts and feature points can be achieved by using publicly known technology such as that described in Patent Document 1, and therefore detailed explanations will be omitted.

[0042] (S101-2) The skeleton recognition program 120 refers to the skeleton / feature point management DB 132 and searches for an entry in which the identification information of the moving image to be analyzed is stored in the moving image ID 301.

[0043] If no entry exists, the skeleton recognition program 120 adds an entry and sets the identification information of the moving image in the moving image ID 301. The skeleton recognition program 120 adds one row including the time 302 and the coordinates 303 to the entry, and sets the time of the image in the time 302 of the added row. The skeleton recognition program 120 also sets the coordinates of each element point in each field of the coordinates 303 of the added row.

[0044] If an entry exists, the skeleton recognition program 120 adds one row including a time 302 and coordinates 303 to the entry, and sets the time of the image to the time 302 of the added row. The skeleton recognition program 120 also sets the coordinates of each element point to each field of the coordinates 303 of the added row.

[0045] If the processing has not been completed for all images, the skeleton recognition program 120 returns to S101-1 and executes the same processing. If the processing has been completed for all images, the skeleton recognition program 120 ends the processing of step S101. This concludes the description of the processing of step S101.

[0046] Next, the skeleton recognition program 120 performs downsampling on the video (step S102). Note that downsampling may be omitted.

[0047] Next, the skeleton recognition program 120 executes a movement amount calculation process (step S103). Specifically, the following process is executed.

[0048] (S103-1) The skeleton recognition program 120 selects a target element point from among the element points.

[0049] (S103-2) The skeleton recognition program 120 refers to the entry corresponding to the moving image in the skeleton / feature point management DB 132, and acquires the coordinates of the element point selected from the row corresponding to the image included in the downsampled moving image.

[0050] (S103-3) The skeleton recognition program 120 generates time-series data on the amount of movement of the element points by using the coordinates of the element points at each time.

[0051] For example, as shown in FIG. 7, the coordinates of the element point at time t1 are (X t1 ,Y t1 ), and the coordinates of the element point at time t2 are (X t2 ,Y t2 ), the amount of movement V of the element point between time t1 and time t2 is t2can be calculated using equation (1).

[0052]

number

[0053] By repeating the same process for successive images, time-series data on the amount of movement of the element points is generated.

[0054] If the processing has not been completed for all element points, the skeleton recognition program 120 returns to S103-1 and executes the same processing. If the processing has been completed for all element points, the skeleton recognition program 120 ends the processing of step S103. This concludes the description of the processing of step S103.

[0055] Next, the synchrony index calculation program 121 executes a synchrony index calculation process using the time-series data of the movement amount of the element point (step S104). The details of the synchrony index calculation process will be described with reference to FIGS.

[0056] Next, the display program 122 executes the display process using the synchronicity index (step S105). After that, the server 100 ends the process. The details of the display process will be described with reference to Figs. 11 to 19.

[0057] Fig. 8 is a flowchart for explaining an example of a synchrony index calculation process executed by the server 100 of the first embodiment. Fig. 9 is an image diagram of specific processing contents of the synchrony index calculation process executed by the server 100 of the first embodiment. Fig. 10 is a diagram showing an example of a specific method of calculating the synchrony index by the server 100 of the first embodiment.

[0058] The synchrony index calculation program 121 starts loop processing of element points (step S201). Specifically, the synchrony index calculation program 121 selects a target element point from among the element points.

[0059] Next, the synchrony index calculation program 121 generates a frequency histogram of the movement amount based on the time-series data of the movement amount of the selected element point, as shown in FIG. 9 (step S202).

[0060] Next, the synchrony index calculation program 121 calculates the number of divisions of the frequency histogram of the movement amount using Sturges's rule, and divides the histogram based on the number of divisions, as shown in FIG. 9 (step S203).

[0061] Next, the synchrony index calculation program 121 normalizes the movement amount so that it can be treated as a random variable, and discretizes the time series data of the movement amount based on the division result of the histogram (step S204). In the following description, the time series data generated by the processing of step S204 is referred to as random variable time series data.

[0062] By performing normalization, it is possible to eliminate differences in the magnitude of the movement of element points due to physical characteristics and video characteristics (angle of view, shooting distance, etc.). In addition, by performing discretization, it is possible to calculate the movement entropy.

[0063] Next, the synchrony index calculation program 121 judges whether or not the processing has been completed for all element points (step S205).

[0064] If the processing has not been completed for all element points, the synchrony index calculation program 121 returns to step S201 and executes the same processing.

[0065] When the processing for all element points is completed, the synchrony index calculation program 121 calculates the transfer entropy between element points using the random variable time series data of the element points (step S206). After that, the synchrony index calculation program 121 ends the synchrony index calculation processing.

[0066] For example, as shown in Fig. 10, using m random variable time series data of element point I and m random variable time series data of element point j, the transfer entropy from element point J to element point I at time step n+1 can be calculated by formula (2). The function P represents probability.

[0067]

number

[0068] The synchrony index calculation program 121 adds an entry to the synchrony index management DB 133, sets the identification information of the video in the video ID 401 of the added entry, and sets the analysis time range in the analysis time range 402. If no particular analysis time range is specified, the analysis time range is set to be from the start time to the end time of the video. The synchrony index calculation program 121 also sets the calculated transfer entropy in each field of the synchrony index 403 of the added entry.

[0069] Incidentally, the "right shoulder-right elbow" in the synchrony index 403 is a field that stores the transfer entropy from the right shoulder to the right elbow. Therefore, the synchrony index 403 also has a "right elbow-right shoulder" field in addition to the "right shoulder-right elbow".

[0070] Next, the display process will be described with reference to Fig. 11 to Fig. 19. The display process differs in content depending on the display content and display method requested by the user.

[0071] Fig. 11 is a flowchart illustrating an example of a display process executed by the server 100 according to the first embodiment. Fig. 12 is a diagram illustrating an example of information presented by the server 100 according to the first embodiment.

[0072] The display program 122 acquires the synchronicity index of the dynamic image to be analyzed from the synchronicity index management DB 133 (step S301).

[0073] The display program 122 extracts element point pairs whose synchrony index is greater than a threshold value (step S302).

[0074] The display program 122 generates display information for displaying an image of the body model depicting an arrow connecting the two element points constituting the extracted element point pair, and transmits it to the user terminal 101 (step S303). The body model is a model of the body in which element points corresponding to the appearance and body parts are defined.

[0075] For example, a screen as shown in Fig. 12 is displayed on the user terminal 101. The screen shown in Fig. 12 displays the shooting date and time of the video, the name of the person performing the action, the analysis time range, and an image of a body model. An arrow connecting the two element points that make up the extracted element point pair is depicted on the image of the body model. Note that the thickness or color of the arrow may be changed depending on the size of the synchrony index.

[0076] Fig. 13 is a flowchart illustrating an example of a display process executed by the server 100 according to the first embodiment. Figs. 14 and 15 are diagrams illustrating an example of information presented by the server 100 according to the first embodiment.

[0077] The display program 122 acquires the synchronicity index of the moving image to be analyzed from the synchronicity index management DB 133 (step S401).

[0078] The display program 122 acquires the moving image to be analyzed from the moving image DB 130, and also acquires the coordinates of the element points from the skeleton / feature point management DB 132 (step S402).

[0079] The display program 122 extracts element point pairs whose synchrony index is greater than a threshold value (step S403).

[0080] The display program 122 generates display information for displaying an animation depicting the element points and arrows connecting the two element points that make up the extracted element point pair based on the coordinates of the element points, the synchronicity index, and the animation, and transmits the display information to the user terminal 101 (step S404).

[0081] For example, a screen as shown in Fig. 14 is displayed on the user terminal 101. The screen shown in Fig. 14 displays the shooting date and time of the video, the name of the person performing the exercise, the analysis time range, the playback time, and the video. The video depicts element points and arrows connecting the two element points that make up the extracted element point pair. The thickness or color of the arrows may be changed depending on the size of the synchrony index.

[0082] As shown in FIG. 15, the server 100 may receive input of two moving images and display a screen that enables comparison of the synchronicity of element points of the two moving images.

[0083] Fig. 16 is a flowchart illustrating an example of a display process executed by the server 100 according to the first embodiment. Fig. 17 is a diagram illustrating an example of information presented by the server 100 according to the first embodiment.

[0084] In the following description, it is assumed that the server 100 has received information on the target element point and the maximum number of layers N from the user.

[0085] The display program 122 acquires the synchronicity index of the moving image to be analyzed from the synchronicity index management DB 133 (step S501).

[0086] The display program 122 acquires the moving image to be analyzed from the moving image DB 130, and also acquires the coordinates of the element points from the skeleton / feature point management DB 132 (step S502).

[0087] The display program 122 extracts element point pairs that include the target element point and have a synchrony index equal to or greater than a threshold (step S503).

[0088] The display program 122 generates a first-level element point list based on the extracted element point pairs (step S504). Specifically, the display program 122 obtains an element point that is different from the target element point for each element point pair, and registers it in the first-level element point list.

[0089] The display program 122 sets the variable n, which indicates the layer, to 1 (step S505).

[0090] The display program 122 acquires the element point list of the nth hierarchy (step S506) and starts loop processing of the element points (step S507). Specifically, the display program 122 selects one element point from the element point list. In the following description, the selected element point is referred to as the starting element point.

[0091] The display program 122 extracts element point pairs that include the starting element point and have a synchronicity index equal to or greater than a threshold (step S508).

[0092] The display program 122 generates an element point list of the (n+1)th hierarchy based on the extracted element point pairs (step S509). Specifically, the display program 122 obtains an element point different from the starting element point for each element point pair, and registers it in the element point list of the (n+1)th hierarchy.

[0093] The display program 122 determines whether or not the processing has been completed for all element points in the element point list of the nth layer (step S510).

[0094] If processing has not been completed for all element points in the element point list of the nth layer, the display program 122 returns to step S507 and executes the same processing.

[0095] When the processing has been completed for all element points in the element point list of the nth layer, the display program 122 determines whether or not n+1 matches the maximum layer number N (step S511).

[0096] If n+1 does not match the maximum number of layers N, the display program 122 adds 1 to the variable n and sets the value of the variable n (step S512), and then returns to step S506.

[0097] If n+1 matches the maximum number of hierarchical levels N, the display program 122 generates display information for displaying an animation depicting the element points and arrows connecting the two element points that make up the extracted element point pair based on the element point list, the coordinates of the element points, the synchronicity index, and the animation, and transmits it to the user terminal 101 (step S513).

[0098] For example, a screen as shown in Fig. 17 is displayed on the user terminal 101. The screen shown in Fig. 17 displays the shooting date and time of the video, the name of the person performing the exercise, the analysis time range, the playback time, and the video. In addition to the element points, the video also depicts arrows indicating connections between element points with high synchronicity, starting from a target element point. The thickness or color of the arrows may be changed depending on the size of the synchronicity index.

[0099] Fig. 18 is a flowchart illustrating an example of a display process executed by the server 100 according to the first embodiment. Fig. 19 is a diagram illustrating an example of information presented by the server 100 according to the first embodiment.

[0100] The display program 122 acquires the synchronicity index of the moving image to be analyzed from the synchronicity index management DB 133 (step S601).

[0101] The display program 122 acquires the moving image to be analyzed from the moving image DB 130, and also acquires the coordinates of the element points from the skeleton / feature point management DB 132 (step S602).

[0102] The display program 122 extracts element point pairs whose synchrony index is greater than a threshold value (step S603).

[0103] The display program 122 calculates a network index that indicates the importance of element points in a network formed by the connections of the extracted element point pairs (step S604). For example, a centrality index such as PageRank is calculated as the network index.

[0104] The display program 122 generates display information for displaying a video in which element points whose network indices are greater than a threshold are highlighted, based on the coordinates of the element points, the video, and the network indices, and transmits the display information to the user terminal 101 (step S605).

[0105] For example, a screen as shown in Fig. 19 is displayed on the user terminal 101. The screen shown in Fig. 19 displays the shooting date and time of the video, the name of the person performing the exercise, the analysis time range, the playback time, and the video. Element points are depicted in the video. Element points whose network index is greater than a threshold value are highlighted by changing the size or color, etc.

[0106] As described above, according to the first embodiment, the server 100 can quantitatively evaluate the synchrony between the body parts of a person and between the body parts and the tool, and present the evaluation result. This allows the user to understand the synchrony between the body parts of a person and between the body parts and the tool during exercise.

[0107] The same functions may be realized by using an analysis system made up of a plurality of servers 100. In this case, the functions may be distributed among the servers 100.

[0108] The programs held by the server 100 may be a single program in which multiple programs are integrated, or a single program may be divided into multiple programs for each function. EXAMPLES

[0109] The difference in the second embodiment is that a synchrony index is calculated for each unit time range for one video. This makes it possible to grasp the time change in synchrony of body parts during exercise. The following describes the second embodiment, focusing on the differences from the first embodiment.

[0110] The system configuration of the second embodiment is the same as that of the first embodiment. The hardware configuration and the software configuration of the server 100 of the second embodiment are the same as those of the first embodiment. The data structures of the moving image DB 130, the moving image management DB 131, the skeleton / feature point management DB 132, and the display setting DB 134 of the second embodiment are the same as those of the first embodiment.

[0111] The second embodiment differs from the first embodiment in the data structure of the synchronicity index management DB 133. Fig. 20 is a diagram showing an example of the data structure of the synchronicity index management DB 133 according to the second embodiment.

[0112] The structure of the fields constituting the entries stored in the synchrony index management DB 133 of the second embodiment is the same as that of the first embodiment. However, one entry includes a row storing the synchrony index for each unit time range. The unit time range may be a value common to the entire system, or may be a different value for each video.

[0113] FIG. 21 is a flowchart illustrating an outline of the process executed by the server 100 according to the second embodiment.

[0114] After the processes of steps S101 to S103 are executed, the synchrony index calculation program 121 sets the analysis start time to a variable t representing time (step S111).

[0115] Next, the synchrony index calculation program 121 acquires time series data of the movement amount of element points in the unit time range [t, t+l] (step S112). The unit time range is a time range extracted by sliding a window of an arbitrary window width l along the time series.

[0116] Next, the synchrony index calculation program 121 executes a synchrony index calculation process using the time series data of the movement amount of the element point (step S104). The synchrony index calculation process is the same as that in the first embodiment, but the range of time series data used in the process is different. In the first embodiment, the time series data of the entire analysis time range is targeted, but in the second embodiment, the time series data of the unit time range [t, t+l] is targeted. In addition, the synchrony index calculation program 121 adds a row of the unit time range [t, t+l] to the entry, and sets the calculated synchrony index to the synchrony index 403 of the row.

[0117] Next, the synchrony index calculation program 121 judges whether the time (t+l+Δt) is after the analysis end time (step S113). Here, Δt represents the update time interval. The update time interval is set in advance. It is to be noted that it may be a different value for each moving image.

[0118] If the time (t+l+Δt) coincides with or is earlier than the analysis end time, the synchrony index calculation program 121 sets the time obtained by adding the update time width Δt to the value of the variable t (step S114), and then returns to step S112.

[0119] If the time (t+l+Δt) is after the analysis end time, the synchrony index calculation program 121 executes a display process using the synchrony index for each unit time range (step S105).

[0120] In the second embodiment, the synchrony index changes over time. Therefore, in the display process, the synchrony index is dynamically changed over time. The contents of the display process are the same as those in the first embodiment, so a description thereof will be omitted.

[0121] According to the second embodiment, it is possible to grasp the change over time in synchrony between body parts and between a body part and a tool during exercise.

[0122] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment describes the configuration in detail to easily explain the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0123] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0124] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

[0125] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0126] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]

[0127] 100 Servers 101 User terminal 105 Network 110 Processor 111 Memory 112 Network Interface 120 Skeleton Recognition Program 121 Synchrony Index Calculation Program 122 Display Program 130 Video DB 131 Video Management DB 132 Skeleton / feature point management DB 133 Synchrony index management DB 134 Display settings DB

Claims

1. A computer system for analyzing physical movements of a person performing exercise, comprising: at least one computer having a processor, a memory coupled to the processor, and an interface coupled to the processor; The at least one computer Obtain a video of a person exercising, Calculating the coordinates of a plurality of element points representing movable parts in a movement for each of a plurality of images constituting the video, and storing the image and the coordinates of the plurality of element points in the memory in association with each other; generating time series data of the amount of movement of each of the element points based on coordinates of the element points of each of the images; Using time series data of the amount of movement of the plurality of element points, a synchrony index is calculated for each of a plurality of element point pairs to evaluate synchrony between the element points during exercise, and the synchrony index of the plurality of element point pairs is stored in the memory; generating and outputting display information indicating synchrony between the element points during exercise based on the synchrony indexes of the plurality of element point pairs; the synchrony index of the element point pair is a transfer entropy from one element point constituting the element point pair to the other element point constituting the element point pair; The at least one computer, when calculating the synchrony index, discretizing time series data of the amount of movement of each of the plurality of element points; Normalizing the time series data of the amount of movement of each of the plurality of element points into a value that can be treated as a random variable; A computer system comprising: a computer that calculates the synchrony index for each of the plurality of element point pairs using time series data of the amount of movement of each of the plurality of element point pairs that has been discretized and normalized.

2. The computer system according to claim 1, The at least one computer Set the analysis time range of the video, Calculating the synchrony index of the element point pairs based on time series data of the movement amounts of the element points included in the analysis time range; A computer system comprising: a memory that stores the analysis time range and the synchrony index of the plurality of element point pairs in association with each other.

3. The computer system of claim 1, The at least one computer Set the analysis time range of the video, calculating the synchrony index of the element point pairs based on time series data of the movement amounts of the element points included in each unit time range smaller than the analysis time range; The computer system is characterized in that the unit time range and the synchrony indexes of the plurality of element point pairs are stored in the memory in association with each other.

4. A computer system as claimed in claim 1, The at least one computer is characterized in that the display information is generated based on the synchronicity index of the multiple element point pairs, in such a way that the element points that constitute the element point pairs with high synchronicity are displayed more emphatically than the other element points.

5. A computer system as claimed in claim 1, The at least one computer searching for the element point pair having high synchronicity, starting from a target element point, based on the synchronicity indexes of the plurality of element point pairs; a display information generating unit that generates the display information so as to emphasize the element points constituting the searched element point pair more than the other element points;

6. A computer system as claimed in claim 1, The at least one computer Identifying the element point pairs having high synchrony based on the synchrony indexes of the plurality of element point pairs; Calculating a network index of a network composed of the identified element point pairs; a display information generating unit that generates, based on the network index, the display information emphasizing the element points that are important in the network more than the other element points;

7. A computer system according to any one of claims 4 to 6, The at least one computer generates the display information for displaying the element points by superimposing the display information on the moving image.

8. A method for analyzing physical movements of a person performing exercise, executed by a computer system, comprising: The computer system includes at least one computer having a processor, a memory coupled to the processor, and an interface coupled to the processor; The method for analyzing the physical movement of a person performing exercise includes: a first step of acquiring, by the at least one computer, a video of a person performing an exercise; a second step in which the at least one computer calculates coordinates of a plurality of element points representing movable parts in a movement for each of a plurality of images constituting the moving image, and stores the image and the coordinates of the plurality of element points in the memory in correspondence with each other; a third step of generating, by the at least one computer, time series data of the amount of movement of each of the plurality of element points based on coordinates of the plurality of element points of each of the plurality of images; a fourth step in which the at least one computer calculates a synchrony index for evaluating synchrony between the element points during exercise for each of a plurality of element point pairs using time series data of the amount of movement of the plurality of element points, and stores the synchrony index for the plurality of element point pairs in the memory; a fifth step of generating and outputting display information indicating synchrony between the element points during exercise based on the synchrony indexes of the plurality of element point pairs by the at least one computer; the synchrony index of the element point pair is a transfer entropy from one element point constituting the element point pair to the other element point constituting the element point pair; The fourth step includes: The at least one computer discretizes time series data of the movement amount of each of the plurality of element points; normalizing, by the at least one computer, time series data of the amount of movement of each of the plurality of element points into a value that can be treated as a random variable; and calculating, by the at least one computer, the synchrony index for each of the plurality of element point pairs using time series data of the amount of movement of each of the plurality of element points that have been discretized and normalized.

9. A method for analyzing physical movements of a person performing the exercise according to claim 8, comprising: The fourth step includes: said at least one computer setting an analysis time range for said video; The at least one computer calculates the synchrony index of the plurality of element point pairs based on time series data of the movement amounts of the plurality of element points included in the analysis time range; and storing, by the at least one computer, in the memory, the analysis time range and the synchrony index of the plurality of element point pairs in association with each other.

10. A method for analyzing physical movements of a person performing the exercise according to claim 8, comprising: The fourth step includes: said at least one computer setting an analysis time range for said video; The at least one computer calculates the synchrony index of the plurality of element point pairs based on time series data of the movement amounts of the plurality of element points included in each unit time range smaller than the analysis time range; and storing, by the at least one computer, in the memory, the unit time range and the synchrony index of the plurality of element point pairs in association with each other.

11. A method for analyzing physical movements of a person performing the exercise according to claim 8, comprising: The fifth step is a method for analyzing the physical movements of a person performing exercise, characterized in that the at least one computer generates the display information in such a way that, based on the synchronicity index of the plurality of element point pairs, the element points that constitute the element point pairs having a high degree of synchronicity are displayed more emphatically than the other element points.

12. A method for analyzing physical movements of a person performing the exercise according to claim 8, comprising: The fifth step includes: The at least one computer searches for the element point pair having high synchronicity, starting from a target element point, based on the synchronicity index of the plurality of element point pairs; and generating, by the at least one computer, the display information for displaying the element points constituting the searched element point pair in a more emphasized manner than the other element points.

13. A method for analyzing physical movements of a person performing the exercise according to claim 8, comprising: The fifth step includes: The at least one computer identifies the element point pairs having high synchronicity based on the synchronicity indexes of the plurality of element point pairs; A step of calculating a network index of a network composed of the identified element point pairs by the at least one computer; and generating, by the at least one computer, the display information based on the network index, which displays the element points that are important in the network in a more emphasized manner than the other element points.

Images