Information processing device, information processing method, and program

Abstract

Provided are an information processing device, an information processing method, and an information processing program for performing musculoskeletal analysis using a desired musculoskeletal model. A motion analysis device 13 is provided with an acquisition unit 31 and a pseudo-marker data acquisition unit 21. The acquisition unit 31 acquires a captured image obtained by imaging an analysis target as the subject. The pseudo-marker data acquisition unit 21 identifies a pseudo-marker on a body image, which is a subject image in the captured image, and acquires the coordinates of the pseudo-marker.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] A known form of musculoskeletal analysis for analyzing human movement is analysis using a musculoskeletal model. Analysis using a musculoskeletal model involves acquiring data indicating body movement using motion capture or the like, and then using the acquired data to have a musculoskeletal model reproduce human movement, thereby obtaining analysis results such as the amount of change in body parts such as joints during movement and the forces acting on each part of the body. A musculoskeletal model has muscles and bones, and multiple markers are set for analysis, with the skeleton serving as an indicator for the placement of the markers. There are various musculoskeletal models that differ in skeletal structure, number and position of markers, and names of markers, and a musculoskeletal model is selected and used appropriately depending on the target analysis, etc.

[0003] In musculoskeletal analysis, a widely known method is to estimate three-dimensional posture (hereinafter referred to as three-dimensional posture) by using a human body model in which the surface of the human body is represented by multiple polygons. A representative example of a human body model is the SMPL (Skinned Multi-Person Linear) model. In a human body model, the shape of a person is determined by the vertices of each polygon, and time-series data of the coordinates of these vertices is obtained to indicate changes in three-dimensional posture. Analysis is then performed based on the time-series data of the coordinates of the vertices.

[0004] For example, Patent Literature 1 discloses a movement feature acquisition device including a memory unit, a shape representative value calculation unit, and a movement feature value calculation unit. The memory unit stores time-series data of skin polygons that specify the shape of a target during movement. Each vertex of the skin polygon has a vertex ID and coordinates that depend on the target's posture. The target's shape is represented by one or more representative regions selected from the skin polygon, and each representative region is a group of vertices specified by the vertex IDs and coordinates of multiple vertices. The shape representative value calculation unit calculates a shape representative value that represents the posture-dependent shape of the target using one or more representative regions. The movement feature value calculation unit uses the time-series data of shape representative values ​​acquired over multiple frames to calculate, as a movement feature, a value that represents the temporal change in one or more representative regions accompanying the target's movement.

[0005] Japanese Patent Application Laid-Open No. 2005-316807

[0006] However, similar to the musculoskeletal model, there are various types of human body models that are used for estimating three-dimensional posture and for which the vertices are set. Furthermore, the skeletal structure of the human body model used for estimating three-dimensional posture may differ from the skeletal structure of the musculoskeletal model that is desired to be used.

[0007] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and an information processing program for performing musculoskeletal analysis using a desired musculoskeletal model.

[0008] The information processing device of the present invention includes an acquisition unit and a pseudo marker data acquisition unit. The acquisition unit acquires a captured image of a subject of analysis. The pseudo marker data acquisition unit identifies pseudo markers on a body image, which is an image of the subject, in the captured image and acquires coordinates of the pseudo markers.

[0009] The information processing device preferably further includes an ID storage unit that stores the names of a plurality of standard markers associated with a preselected general skeletal model and identification information of selected points selected from a plurality of points set in a human body model applied to the body image, the number of which is greater than the number of standard markers. In this case, the pseudo marker data acquisition unit identifies, as pseudo markers, points associated with the same identification information as the identification information from among the plurality of points in the human body model applied to the body image.

[0010] It is preferable that the human body model is represented by a collection of skin polygons, and each of the plurality of points set on the human body model is a vertex of a skin polygon.

[0011] The human body model is preferably a Skinned Multi-Person Linear (SMPL) model.

[0012] The selected points are preferably selected based on a video image showing the movement of the human body model.

[0013] It is preferable that the system further comprises an ID registration unit that stores the identification information of the selected point in the ID storage unit based on a registration request for the identification information of the selected point to be associated with the name of any standard marker.

[0014] The ID registration unit preferably determines whether or not the name and the identification information indicated in the registration request are appropriate, based on the degree of difference between the three-dimensional coordinates of the determination reference marker obtained by motion capture in which the determination reference marker is set at a position corresponding to the standard marker of the subject who is the subject of the captured image, and the three-dimensional coordinates of the selection point indicated in the registration request. If the ID registration unit makes a positive determination, it preferably stores the identification information indicated in the registration request in the ID storage unit, and if the ID registration unit makes a negative determination, it preferably issues a negative determination notification indicating a negative determination.

[0015] It is preferable to further include an inverse kinematics processing unit that performs inverse kinematics processing using a general skeleton model based on the three-dimensional coordinates of the pseudo markers and the three-dimensional coordinates of the standard markers.

[0016] The system may further include a subject marker data calculation unit that calculates the three-dimensional coordinates of the reference model markers for a subject skeletal model in which the height of the subject, who is the subject of the captured image, is applied to a general skeletal model. In this case, it is preferable to further include an inverse kinematics processing unit that performs inverse kinematics processing on the subject skeletal model based on the three-dimensional coordinates of the pseudo markers and the three-dimensional coordinates of the standard markers.

[0017] The pseudo marker data acquisition unit may have a trained model generated by machine learning using, as input information, one selected from a captured image of a human subject, two-dimensional coordinates of joint positions, and three-dimensional coordinates of joint positions, and output information of three-dimensional coordinates of pseudo markers. The pseudo marker data acquisition unit acquires the pseudo marker data by inputting, to the trained model, one selected from a captured image of the human subject, two-dimensional coordinates of joint positions obtained based on the captured image of the human subject, and two-dimensional coordinates obtained based on the captured image of the human subject, and outputting the three-dimensional coordinates of the pseudo markers.

[0018] The pseudo marker data acquisition unit preferably acquires three-dimensional coordinates of each of the pseudo markers for each of a plurality of frames constituting a moving image as the captured images of the subject captured from a fixed point.

[0019] The information processing method of the present invention includes an acquisition step and a pseudo marker data acquisition step. In the ID storage step, an information processing device including an acquisition unit and a pseudo marker data acquisition unit acquires a captured image of a subject of analysis by the acquisition unit. In the pseudo marker data acquisition step, the pseudo marker data acquisition unit identifies pseudo markers on a body image that is the subject image in the captured image and acquires coordinates of the pseudo markers.

[0020] The information processing program of the present invention causes a computer to execute an acquisition step and a pseudo marker data acquisition step. The acquisition step acquires a captured image of a subject of analysis as a subject. The pseudo marker data acquisition step identifies pseudo markers on a body image, which is an image of the subject, in the captured image and acquires coordinates of the pseudo markers.

[0021] According to the present invention, a musculoskeletal analysis can be performed using a desired musculoskeletal model.

[0022] 1 is a schematic diagram of a motion analysis system. FIG. 1 is an explanatory diagram of an imaging method performed by an imaging device. FIG. 2 is a configuration diagram of a motion analysis device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of a storage mode of a general skeletal model storage unit. FIG. 4 is an explanatory diagram of a storage mode of a marker set storage unit. FIG. 5 is an explanatory diagram of a storage mode of a marker correspondence storage unit. FIG. 6 is an explanatory diagram of a storage mode of an ID storage unit. FIG. 7 is an explanatory diagram of an example of a standard marker of a marker set. FIG. 8 is an explanatory diagram of a standard marker associated with a general skeletal model. FIG. 9 is an explanatory diagram of skin polygons and points of a human body model. FIG. 10 is an explanatory diagram of a method of requesting registration of a selected point. FIG. 11 is an explanatory diagram of extraction of pseudo marker data. FIG. 12 is a schematic diagram of an example of an image displayed on a client terminal. FIG. 13 is a configuration diagram of a motion analysis device according to another embodiment. FIG. 14 is a configuration diagram of a motion analysis device according to another embodiment. FIG. 15 is an explanatory diagram of joint positions and markers, which are training data used in machine learning, where (A) shows key points and (B) shows markers.

[0023] The motion analysis system 10 shown in FIG. 1 is an information processing system that captures video of a subject whose body movements are to be analyzed, and analyzes the subject's body movements based on the captured images. The motion analysis system 10 includes a user terminal 11 used by the subject, a motion analysis device 13, and a client terminal 15 that acquires analysis results and displays images showing the analysis results. The user terminal 11, the motion analysis device 13, and the client terminal 15 communicate with each other via a communication network CN, i.e., transmit and receive various data. For example, a registration request for identification information of a selection point (described below) is sent from the client terminal 15 to the motion analysis device 13, and the motion analysis device 13 stores a marker set including the identification information of the selection point indicated in the received registration request. The user terminal 11 captures an image of the subject using an imaging device (camera) 17 (see FIG. 2) and transmits a video as an example of the captured image to the motion analysis device 13. The motion analysis device 13 analyzes the subject's movements based on the received video and transmits the analysis results to the client terminal 15. As a result, analysis results and the like are displayed on the display (display unit) of the client terminal 15, allowing the client to verify the subject's movements. The motion analysis system 10 may also include a reflective marker data acquisition device 19, and this is also the case in this example. The reflective marker data acquisition device 19 acquires time-series data of the three-dimensional coordinates of the reflective markers by optical motion capture using reflective markers as judgment reference markers, which will be described later. The reflective marker data acquisition device 19 communicates with the motion analysis device 13 via the communication network CN and sends the time-series data of the three-dimensional coordinates of the reflective markers to the motion analysis device 13. However, the reflective marker data acquisition device 19 may also communicate with the client terminal 15 via the communication network CN, thereby sending the time-series data of the three-dimensional coordinates of the reflective markers to the motion analysis device 13 via the client terminal 15.

[0024] The user terminal 11 includes an imaging device 17 that captures moving images of the subject as a subject, and a transmission unit (not shown) that sends the captured moving images to the motion analysis device 13. The imaging device 17 in this example is capable of capturing still images in addition to moving images, and may also send the still images to the motion analysis device 13. In this example, a smartphone is used as the user terminal 11, but it is not limited to a smartphone as long as it is equipped with the imaging device 17 and transmission unit. One imaging device 17 is sufficient for one user. Although only one user terminal 11 is depicted in FIG. 1, the number of user terminals 11 may be multiple so that multiple users can use each terminal.

[0025] The client terminal 15 is, for example, a terminal used by a client to understand and verify the analysis results of the subject's movement, i.e., the analysis results obtained by the movement analysis device 13. The movement analysis system 10 may not include a client terminal 15. For example, if the user terminal 11 includes a display for displaying images and a receiving unit for acquiring (receiving) image data, and the subject understands the analysis results of their own movement, the user terminal 11 may be used as the client terminal 15. Although only one client terminal 15 is depicted in FIG. 1, there may be multiple client terminals 15, each used by multiple clients. The client terminal 15 may also be used as a management terminal for managing various settings and / or data of the movement analysis device 13, and the various settings may include the above-mentioned registration request. The management terminal may be provided separately from the client terminal 15.

[0026] The motion analysis device 13 is for analyzing the motion of the subject based on the acquired moving images. Details of the motion analysis device 13 will be described later with reference to another drawing.

[0027] The user terminal 11, the client terminal 15, and the motion analysis device 13 are each composed of a computer. The user terminal 11 and the client terminal 15 may operate on a browser by receiving a program that runs on a browser from the motion analysis device 13, or may be embedded with predetermined application software and operate by executing the program of this application software.

[0028] A predetermined program is installed in the motion analysis device 13, and execution of this program causes the motion analysis device 13 to function as each unit described below and perform predetermined processing. The motion analysis device 13 performs predetermined processing in response to input of information from the user terminal 11, the client terminal 15, and the reflective marker data acquisition device 19. The program installed in the motion analysis device 13 and the motion analysis devices 50, 70, and 80 described below causes a computer to execute an acquisition step and a pseudo marker data acquisition step. The acquisition step acquires a captured image of a person to be analyzed as a subject. The pseudo marker data acquisition step identifies pseudo markers (described below) on a body image, which is the image of the subject, in the captured image and acquires the coordinates of the pseudo markers. Note that the reflective marker data acquisition device 19 may be installed in the motion analysis device 13 as part of the motion analysis device 13, i.e., as a reflective marker data acquisition unit.

[0029] In FIG. 2 , the user terminal 11 is installed so that the mounted imaging device 17 is fixed in position and orientation. As a result, moving images are obtained as fixed-point captured moving images captured from a fixed point. The imaging device 17 is positioned so that the entire body of the moving subject is captured during the time period to be analyzed for the movement, i.e., from the start to the end of the period to be analyzed. In this example, as shown in FIG. 2 , a linear walking line (walkway) WL is set on the floor as the area in which the subject P moves, and the subject P's walking on the walking line WL from one end of the walking line WL to the other end is considered as an example of the subject P's movement. The user terminal 11 is positioned so that the imaging optical axis L of the imaging device 17 intersects with the walking line WL when viewed from above. The height of the imaging device 17 from the floor is set to be approximately 0.6 m to 1.2 m, e.g., 1 m, so that the entire body of the subject P is captured. In this way, the subject P walking along the walking line WL is imaged from the feet to the head during the analysis time. In this example, the imaging device 17 is positioned at the center of the walking line WL so that the walking line WL and the imaging optical axis L are perpendicular to each other. However, as long as the subject P's entire body is imaged, the imaging device 17 may be positioned so that the imaging optical axis L intersects the walking line WL at an angle other than 90° at a position offset from the center of the walking line WL, as shown by the two-dot chain line in FIG. 2 . Note that the exercise is not limited to walking and may also be exercise performed at a fixed position on the floor. Examples of such exercise include baseball and golf swings and tennis practice swings. Furthermore, while in this example, fixed-point imaging video is used as the captured image, the subject P may instead be imaged while the imaging device 17 is moving, and the moving-point imaging video captured from the moving point may be used as the captured image.

[0030] 3 , the motion analysis device 13 includes an ID storage unit 20, a pseudo marker data acquisition unit 21, and an acquisition unit 31. The motion analysis device 13 preferably further includes a general skeletal model storage unit 22, a marker set storage unit 23, a marker correspondence unit 24, a marker correspondence storage unit 25, a human body model storage unit 26, an ID registration unit 27, a three-dimensional posture estimation unit 32, an inverse kinematics processing unit 33, an external force estimation unit 36, a display control unit 37, and a controller 38, and this example also includes these. The ID storage unit 20, the general skeletal model storage unit 22, the marker set storage unit 23, the marker correspondence storage unit 25, and the human body model storage unit 26 constitute a database (DB).

[0031] The controller 38 responds to receiving various data from the user terminal 11, the client terminal 15, and the reflective marker data acquisition device 19, and controls all the components of the motion analysis device 13 other than the controller 38 in an integrated manner.

[0032] The general skeleton model storage unit 22 stores general skeleton models. The general skeleton model storage unit 22 may store multiple general skeleton models that differ from one another in at least one of the following: the number of bones, the number of joints, and the weights set for parts of the entire skeleton. In this example, eight general skeleton models M1a to M1h are stored (see FIG. 4A). The general skeleton models M1a to M1h may be stored in the general skeleton model storage unit 22 using the client terminal 15 or the management terminal, or may be incorporated into a program in advance and stored in the general skeleton model storage unit 22. In the following description, when there is no need to distinguish between the general skeleton models M1a to M1h, they will be referred to as the general skeleton model M1.

[0033] The general skeletal model M1 is a human rigid-body link model (segment-link model, skeletal model) having a skeleton composed of rigid bodies (segments) and connecting portions (links), and may be a known model. The general skeletal model M1 may be any model having a skeleton, and may be a skeletal portion of a musculoskeletal model having bones and muscles. The general skeletal model M1 has the same skeletal structure as the musculoskeletal model used in the analysis. These multiple general skeletal models M1a to M1h are pre-selected as having the same skeletal structure as each of the candidate musculoskeletal models used in the analysis. When performing an analysis, one musculoskeletal model is selected from these candidates according to the desired analysis, and the general skeletal model M1 having the same skeletal structure as the selected musculoskeletal model is selected and used by the client. Therefore, the general skeletal models M1a to M1h may be selected in advance by the client or an administrator managing the motion analysis device 13 according to the desired analysis. In the following explanation, a case where the musculoskeletal model used in the analysis has the same skeletal structure as the general skeletal model M1a will be described as an example.

[0034] The marker set storage unit 23 stores a marker set having a plurality of markers (hereinafter referred to as standard markers). The marker set storage unit 23 may store a plurality of marker sets, and in this example, three marker sets Sa to Sc are stored (see FIG. 4B). Of these, the marker set Sa has 39 standard markers SM1 to SM39 (see FIG. 4B). The marker sets Sa to Sc may be stored in the marker set storage unit 23 using the client terminal 15 or the management terminal, or may be incorporated into a program in advance and stored in the marker set storage unit 23. In the following description, when the marker sets Sa to Sc are not distinguished, they will be referred to as the marker set S, and when the standard markers SM1 to SM39 are not distinguished, they will be referred to as the standard markers SM.

[0035] The marker set S can be any known marker set that has multiple standard markers SM set on the skeleton of the musculoskeletal model used for analysis. Known marker sets include the plug-in-gait (PiG) model, the Helen Hayes model, and the Data Interface File Format (DIFF) model. The number of set standard markers SM and the position of each standard marker SM vary from marker set to marker set. Therefore, the marker set S may be selected according to the intended analysis content, etc. Similarly to the number and position, the names of the standard markers SM also vary from marker set to marker set. The number of standard markers SM is not particularly limited; the fewer the number, the faster the analysis process, and the more the number, the more reliable the analysis results. In order to balance the speed of the analysis process and the reliability of the analysis results, the number of standard markers SM is preferably in the range of 20 to 80, and more preferably in the range of 30 to 60. Furthermore, the standard markers SM may be set unevenly over the whole body depending on the target analysis. For example, the number of standard markers SM may be set to be greater on the legs than on other parts, or the number of standard markers SM may be set to be greater on the shoulders and arms than on other parts.

[0036] In this example, a PiG model is used as the marker set Sa, and this marker set Sa has 39 standard markers SM1 to SM39 (see FIG. 5A). The names and setting positions of the standard markers SM1 to SM39 are shown in Table 1. The names include official names and abbreviations that are abbreviated versions of the official names. The names of the standard markers SM stored in the marker set storage unit 23 may be either official names or abbreviations, and in this example, abbreviations are used. Similarly, in the other marker sets Sb and Sc in this example, multiple standard markers SM are set, and each standard marker SM is given a name.

[0037]

[0038] The marker correspondence unit 24 associates one of the marker sets Sa to Sc with each of the general skeleton models M1a to M1h, associates the names of the multiple standard markers SM included in the associated marker set S, and generates marker correspondence information in the marker correspondence storage unit 25. For example, if the marker set Sa is associated with the general skeleton model M1a, the standard markers SM1 to SM39 are associated with the general skeleton model M1a, and the marker correspondence information is generated. Each rigid body and each connecting portion of the general skeleton model M1a has coordinates that define them, and the standard markers SM1 to SM39 are associated with specific parts of the general skeleton model M1a by associating them with these coordinates (see FIG. 5B ). The names of the standard markers SM in the marker correspondence information may be either full names or abbreviations; in this example, abbreviations are used. The association between the general skeleton model M1 and the marker set S does not need to be a one-to-one correspondence and may be determined appropriately by the client or the administrator. For example, the marker association unit 24 may generate, for the general skeleton model M1a, first marker association information associated with the marker set Sa, second marker association information associated with the marker set Sb, and third marker association information associated with the marker set Sc, and store the three pieces of marker association information associated with the general skeleton model M1a in the marker association storage unit 25. The association of the marker set S and the association of the standard marker with the general skeleton model M1 may be performed in response to an input operation of an execution request from the client terminal 15 or the management terminal, or may be incorporated into a program in advance.

[0039] The marker correspondence storage unit 25 stores marker correspondence information. That is, the marker correspondence storage unit 25 stores marker correspondence information in which general skeleton models M1a to M1h, marker sets Sa to Sc, and the names (abbreviations in this example) of the multiple standard markers SM that each of the marker sets Sa to Sc has are associated with each other (see FIG. 4C ). When there is only one general skeleton model M1 stored in the general skeleton model storage unit 22 and only one marker set S stored in the marker set storage unit 23, the marker correspondence storage unit 25 stores one piece of marker correspondence information.

[0040] The human body model storage unit 26 stores identification information (hereinafter referred to as ID, where ID stands for Identification) of multiple points V (see FIG. 6) set in the human body model M2 (see FIG. 6). The human body model M2 is a model used by the three-dimensional posture estimation unit 32 to estimate a three-dimensional posture (hereinafter referred to as three-dimensional posture), and is applied to a body image that is a subject image in a captured image. In this example, the human body model M2 is an SMPL (Skinned Multi-Person Linear) model that represents the surface of a human body as a collection of multiple skin polygons (polygons) PG (see FIG. 6), and the number of points V (see FIG. 6) is 6,890. The points V are vertices of the skin polygons PG. Note that in FIG. 6, only a portion of the skin polygons PG and points V are depicted to avoid cluttering the illustration. The human body model M2 is not limited to SMPL, and may be any model that defines a human shape using a plurality of points V. Since the plurality of points V defines a human shape, the number of points V is sufficient to define the human shape and is greater than the number of standard markers SM in each of the marker sets Sa to Sc. The number of points V is preferably at least 500, more preferably 1,000 or more, and preferably 3,000 to 10,000. By setting a large number of points V in this manner, pseudo marker data, described below, can be extracted with greater accuracy, resulting in more reliable analysis results. When using a human body model M2 in which the surface of a human body is represented by a collection of a plurality of skin polygons PG, the skin polygons are not particularly limited to triangles, rectangles, or the like.

[0041] The IDs of the points V are not particularly limited and may be numbers, letters, or combinations thereof, as long as they can identify each point V. In this example, numbers from 1 to 6890 are used as the IDs of the 6890 points V.

[0042] The ID registration unit 27 stores the ID of the selection point SV (see FIG. 7 ) in the ID storage unit 20 based on a request to register the ID of the selection point SV to be associated with the name of an arbitrary standard marker SM. The selection point SV is selected from a plurality of points V set on the human body model M2, and the ID of the selection point SV is the ID of the point V that is the subject of selection. The selection of the selection point SV from the point V is performed by the client at the client terminal 15, and a request to register the ID of the selection point SV is sent from the client terminal 15. Furthermore, when the ID of the selection point SV and the name of the standard marker SM have already been stored in association with each other in the ID storage unit 20, if a new request to register the ID of the selection point SV to be associated with an arbitrary standard marker SM is sent from the client terminal 15, the ID of the selection point SV in the ID storage unit 20 may be updated and stored. In this way, storing the ID of the selection point SV includes updating. Alternatively, the selection point SV may be selected by an administrator using the management terminal, and a registration request may be sent from the management terminal. The method of sending a registration request will be described later with reference to another drawing.

[0043] The ID registration unit 27 may perform a suitability determination process to determine whether the association between the ID of the selection point SV indicated in the registration request and the name of the standard marker SM is appropriate. If the ID registration unit 27 determines that the association is appropriate, it associates the ID of the selection point SV with the name of the standard marker SM and stores the association in the ID storage unit 20. If it determines that the association is inappropriate, it sends a notification indicating the negative determination (hereinafter referred to as a negative determination notification) to the client terminal 15 or management terminal that sent the registration request. When the client terminal 15 receives the negative determination notification, the client may use the client terminal 15 to select a selection point SV from multiple points V that is different from the selection point SV with the ID indicated in the previous registration request and send a new registration request to the ID registration unit 27, or it may simply confirm the ID of the selection point SV that was the subject of the negative determination and send a confirmation notification to the ID registration unit 27. The same applies when the management terminal receives the negative determination notification. When a confirmation notification is received, the ID registration unit 27 associates the ID of the selection point SV that was the subject of the negative judgment with the name of the standard marker SM and stores the association in the ID storage unit 20. When a registration request is received, the ID registration unit 27 again determines whether the association is appropriate, and repeats the appropriateness determination process until a positive judgment is made or a confirmation notification is received. In this way, by determining whether the association between the ID of the selection point SV and the name of the standard marker SM is appropriate, pseudo marker data (described below) can be extracted with greater accuracy, and more reliable analysis results can be obtained.

[0044] The suitability determination can be performed in the following manner. First, the three-dimensional coordinates of a reference marker obtained by motion capture for the subject of the analysis, i.e., the subject of the moving image, are obtained. The reference marker is a marker used as a criterion for determining whether the ID of the selection point SV indicated in the registration request is suitable for association with the standard marker SM indicated in the registration request. The reference marker is set at each body part when determining the three-dimensional coordinates of that part using various motion capture methods. The reflective marker data acquisition device 19 is an example of a motion capture device, and any known motion capture method may be used. In this example, the aforementioned optical motion capture is used. Alternatively, instead of optical motion capture, motion capture may be used in which the three-dimensional coordinates of each body part are determined using AI (artificial intelligence). When AI is used, the reference marker may be set at any position for determining the three-dimensional coordinates. The reflective marker data acquisition device 19 includes a plurality of reference markers (reflective markers) attached to the subject's body that reflect light, and a plurality of detection units, and acquires time-series data of the three-dimensional coordinates of the reference markers using optical motion capture. The detection unit includes a camera, a light emission unit disposed, for example, near the camera, that emits light toward the subject, and a calculation unit configured as a computer. The plurality of detection units are disposed around the subject so that the camera and the light emission unit capture the subject at different angles. Each camera receives light emitted from the light emission unit and reflected by the reflective markers. The reflected light is received by the plurality of cameras at different angles, and the calculation unit calculates the three-dimensional coordinates of each of the plurality of reflective markers based on the light received by each of the plurality of cameras. When a request to register the ID of the selected point SV is sent from the client terminal 15 to the ID registration unit 27, the ID registration unit 27 requests the reflective marker data acquisition device 19 to send the time series data of the three-dimensional coordinates of the reflective marker thus acquired, and the data is then sent from the reflective marker data acquisition device 19 to the ID registration unit 27 via the controller 38.However, instead of this embodiment, for example, time series data of the three-dimensional coordinates of the reflective markers may be stored in advance in the client terminal 15, and the time series data of the three-dimensional coordinates of the reflective markers may be sent from the client terminal 15 to the ID registration unit 27.

[0045] Furthermore, when performing the optical motion capture, the subject wearing the reflective markers is captured as a video using, for example, the imaging device 17, and the video (hereinafter referred to as the "suitability determination video") is sent from the user terminal 11 to the acquisition unit 31. Next, the three-dimensional coordinates of the selection point SV, whose ID is indicated in the registration request, are calculated. The three-dimensional coordinates of the selection point SV can be calculated for the suitability determination video acquired by the acquisition unit 31 using a method similar to the method used to calculate the three-dimensional coordinates of point V using the three-dimensional posture estimation unit 32 (described below) and the method used to acquire pseudo marker data using the pseudo marker data acquisition unit 21. The ID registration unit 27 then performs suitability determination based on the degree of difference between the three-dimensional coordinates of the judgment reference marker acquired by the reflective marker data acquisition device 19 and the three-dimensional coordinates of the selection point SV sent from the pseudo marker data acquisition unit 21 as the three-dimensional coordinates of the pseudo marker. The degree of difference can be set based on the distance between the three-dimensional coordinates of the judgment reference marker and the three-dimensional coordinates of the selection point SV. For example, when the distance between the two is 0 (zero), the dissimilarity is set to 0, and the greater the distance, the greater the dissimilarity value, and the maximum value at which a positive judgment is made is set as the threshold value. The distance value itself may also be used as the dissimilarity and threshold value.

[0046] The ID storage unit 20 stores marker ID information that associates the names of multiple standard markers SM with the IDs of multiple selection points SV. In this example, as described above, multiple marker sets Sa-Sc are stored in the marker correspondence storage unit 25. Therefore, the ID storage unit 20 similarly stores, for each of the marker sets Sa-Sc, marker ID information that associates the names of multiple standard markers SM with the IDs of multiple selection points SV (see FIG. 4D ). In this example, since the general skeleton models M1a-M1h and the marker sets Sa-Sc are associated and stored in the marker correspondence storage unit 25 as described above, the ID storage unit 20 similarly stores the general skeleton models M1a-M1h in association with the marker sets Sa-Sc. For example, as shown in FIG. 4D , the ID storage unit 20 stores the general skeleton model M1a in association with the marker set Sa.

[0047] When the acquisition unit 31 acquires a captured image, i.e., a moving image in this example, from the imaging device 17, it sends the acquired moving image to the three-dimensional posture estimation unit 32. The same applies when a still image is acquired as the captured image and when the above-mentioned moving image for suitability determination is acquired. The acquisition unit 31 may have a memory unit (not shown) for storing the captured image and / or the moving image for suitability determination, and this is also the case in this example. When the acquisition unit 31 has such a memory unit, it sends the captured image previously stored in the memory unit to the three-dimensional posture estimation unit 32 when a processing request for, for example, estimating a three-dimensional posture is sent from the client terminal 15 to the motion analysis device 13. In this example, the processing request for estimating a three-dimensional posture is sent to the motion analysis device 13 after one of the general skeletal models M1a to M1h is selected in the client terminal 15. In the following explanation, a case in which the general skeletal model M1a is selected will be used as an example. Furthermore, when acquisition unit 31 includes the storage unit, for example, when a processing request for estimating a three-dimensional posture is sent from ID registration unit 27, acquisition unit 31 sends the suitability determination moving image stored in advance in the storage unit to three-dimensional posture estimation unit 32. At this time, acquisition unit 31 sends suitability determination information indicating that the information is for performing suitability determination to three-dimensional posture estimation unit 32 together with the suitability determination moving image.

[0048] When a moving image is sent from the acquisition unit 31, the three-dimensional posture estimation unit 32 reads from the human body model storage unit 26 a human body model M2 (see FIG. 6 ) in which multiple points V (see FIG. 6 ) are set. For each frame image constituting the moving image, the three-dimensional posture estimation unit 32 applies the human body model M2 to a subject, which is a subject image in the frame image, identifies two-dimensional joint position coordinates (two-dimensional coordinates of joint positions) in the subject image corresponding to each of the three-dimensional joint position coordinates (three-dimensional coordinates of joint positions) of the human body model M2, and calculates the three-dimensional coordinate of point V based on the identified two-dimensional joint position coordinates. The three-dimensional posture estimation unit 32 then sends the calculated three-dimensional coordinates of each of the multiple points V to the pseudo marker data acquisition unit 21 along with selection information indicating that the general skeletal model M1a has been selected. The same process is performed when a moving image for suitability assessment is sent from the acquisition unit 31. In addition, when the three-dimensional posture estimation unit 32 performs the above processing on the moving image for suitability determination to obtain the three-dimensional coordinates of point V, it sends the suitability determination information together with the three-dimensional coordinates of point V to the pseudo marker data acquisition unit 21.

[0049] The two-dimensional coordinates of each joint position are calculated for each frame. The two-dimensional coordinates of each joint position are expressed in the image coordinate system. In the image coordinate system, the vertex located at the leftmost and topmost corner of the image area defined on the xy plane is set as the origin (0,0), and the direction to the right is treated as the positive direction of the x-axis (positive values ​​increase gradually), and the direction downward is treated as the positive direction of the y-axis (positive values ​​increase gradually).

[0050] The method for calculating the two-dimensional coordinates of each joint position is not particularly limited, and any known calculation method (estimation method) can be used. For example, in DeepPose (described in the following Non-Patent Document 1: A. Toshev and C. Szegedy, "DeepPose: Human Pose Estimation via Deep Neural Networks," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, USA, 2014, pp. 1653-1660.), time-series data of frame images constituting a previously acquired video is input to a convolutional neural network, and each joint point on the image coordinate system is estimated by regression. The two-dimensional coordinates of the joint positions can also be obtained by representing the probability distribution of joint positions as a heat map, such as in OpenPose (described in the following non-patent document 2: Z. Cao, G. Hidalgo, T Simon, S. Wei, and Y. Sheikh. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Trans. Pattern Anal. Mach. Intell. 43, 1 (Jan. 2021), 172-186.), or by using an improved method that improves estimation accuracy by grouping candidate joint points.

[0051] Furthermore, the two-dimensional coordinates of the joint positions and the three-dimensional coordinates of point V are associated in advance by, for example, an arithmetic expression, etc., and therefore the three-dimensional coordinates of point V can be found based on the two-dimensional coordinates of the joint positions. When finding the three-dimensional coordinates of point V from the two-dimensional coordinates of each joint position, a silhouette image of the subject, who is the subject, may be generated for each frame image. The silhouette image may be an image in which the foreground is extracted by classifying the area inside the outline that indicates the subject's outline as the foreground and the area outside the outline as the background, an image in which the foreground is shown in a color different from the background, or an image in which the foreground is shown in a pattern different from the background; in this example, an image in which the foreground is extracted is used.

[0052] In this example, AI is incorporated into the three-dimensional posture estimation unit 32. When the two-dimensional coordinates of each joint position and time-series data of a silhouette image are input, the AI ​​of the three-dimensional posture estimation unit 32 estimates the three-dimensional coordinates and three-dimensional posture of point V in response to these inputs. In this example, the AI ​​generates data using a conditional variation autoencoder (CVAE), which can impose conditions on generated data. This AI estimates the three-dimensional posture at time t+1 from the three-dimensional posture at time t using a trained model (posture estimation model) trained on time-series data of three-dimensional posture calculated from optical motion capture data, for example, using a skeletal model such as a general skeletal model and inverse kinematics. The trained model uses a dataset of paired information in which the three-dimensional posture at a certain time t on the time axis is input information and the three-dimensional posture at time t+1 is output information. At this time, for example, the internal parameters obtained by the calibration process for the image capture device 17 are used to project the three-dimensional joint positions onto a two-dimensional plane in the image coordinate system, and an optimal three-dimensional posture is estimated so as to minimize the error with the two-dimensional coordinates of point V obtained by the estimation of the two-dimensional coordinates. If the two-dimensional coordinates obtained by the estimation of the two-dimensional coordinates fall outside the outline of the silhouette shown in the silhouette image, the silhouette image can be used as a basis for determining that the estimation is an incorrect estimation and ignoring the estimated data. However, the method for acquiring the three-dimensional posture data when creating the aforementioned data set is not critical. Furthermore, the three-dimensional posture estimation unit 32 in this example estimates and obtains the three-dimensional posture in addition to the three-dimensional coordinates of point V as described above. However, from the perspective of identifying the pseudo marker PM (see FIG. 8 ), it is sufficient that the three-dimensional coordinates of point V are obtained. Note that in FIG. 8 , only a portion of point V and pseudo marker PM are depicted to avoid cluttering the illustration.

[0053] When the pseudo marker data acquisition unit 21 receives time-series data of the three-dimensional coordinates of multiple points V determined in each frame image and selection information indicating that the general skeleton model M1a has been selected, the pseudo marker data acquisition unit 21 identifies the marker set S associated with the general skeleton model M1a in the ID storage unit 20. In this example, the marker set S associated with the general skeleton model M1a is marker set Sa, so marker set Sa is identified. The pseudo marker data acquisition unit 21 compares the IDs of the multiple points V with the IDs of selected points SV (see FIG. 7 ) included in the identified marker set Sa, and identifies points V having the same ID as the selected points SV as pseudo markers (also referred to as virtual markers) PM (see FIG. 8 ). Along with this identification, the pseudo marker data acquisition unit 21 associates the names of standard markers SM with each of the identified pseudo markers PM. The name of the associated standard marker SM is the name associated with the ID of the selection point SV, which has the same ID as the pseudo marker PM, when the pseudo marker PM was identified. The pseudo marker PM is associated with the standard marker SM in the inverse kinematics processing described below, thereby causing the general skeletal model M1 to be displayed on the image with a movement corresponding to the body movement of the subject. The pseudo marker data acquisition unit 21 then sets the three-dimensional coordinates of the point V identified as the pseudo marker PM as the three-dimensional coordinates of the pseudo marker PM. In this manner, the pseudo marker PM is extracted from the multiple points V, and the three-dimensional coordinates of the pseudo marker PM are obtained. In this example, the selection points SV are identified in one-to-one correspondence with the standard markers SM1 to SM39 set in the general skeletal model M1a, so that 39 pseudo markers PM are identified in each frame image GF (see FIG. 8), and pseudo marker data including the three-dimensional coordinates of the 39 pseudo markers is generated for each frame image GF. As a result, time-series data consisting of the same number of pseudo markers PM as the frame images GF is generated for each of the 39 pseudo markers PM. The pseudo marker data acquisition unit 21 sends the time-series data of the three-dimensional coordinates of the pseudo markers PM to the inverse kinematics processing unit 33. When the suitability determination information is sent from the three-dimensional posture estimation unit 32 along with the three-dimensional coordinates of point V, the pseudo marker data acquisition unit 21 sends the obtained time-series data of the three-dimensional coordinates of the pseudo markers PM to the ID registration unit 27.

[0054] As described above, the ID storage unit 20 stores marker ID information that associates the ID of the selected point SV selected from point V on the human body model M2 with the standard markers SM (BM1 to BM39 in this example) on the general skeletal model M1 (M1a in this example). Using this marker ID information, the pseudo marker data acquisition unit 21 extracts the ID of the selected point SV from the time-series data of the three-dimensional coordinates of point V, thereby obtaining pseudo markers PM that can be considered to be positioned in the same way as the markers used in the analysis, and thus obtaining pseudo marker data. Furthermore, in order to perform analysis using a musculoskeletal model having a skeletal structure different from that of the human body model, the pseudo marker data can be used to perform inverse kinematics processing and determine three-dimensional joint angles, three-dimensional joint positions, and the like in the musculoskeletal model.

[0055] The inverse kinematics processor 33 performs inverse kinematics processing on the general skeletal model M1a based on the three-dimensional coordinates of the pseudo markers PM and the three-dimensional coordinates of the standard markers SM. This inverse kinematics processing is an inverse kinematics processing in musculoskeletal analysis, in which each standard marker SM set in the general skeletal model M1a shown in the image is brought close to the corresponding pseudo marker PM, i.e., the pseudo marker PM associated with the same name, to determine three-dimensional joint angles and three-dimensional joint positions. Since inverse kinematics processing in musculoskeletal analysis is well known, details will be omitted. The inverse kinematics processor 33 sends the obtained time-series data of the three-dimensional joint angles and three-dimensional joint positions to the external force estimator 36. Inverse kinematics processing in musculoskeletal analysis can also be performed on still images. Joint angles may be expressed using any expression, such as a rotation vector, Euler angles, a rotation matrix, or a quaternion. Three-dimensional information such as three-dimensional joint angles and three-dimensional joint positions has the origin at (x, y, z) = (0, 0, 0) in three-dimensional space.

[0056] In this example, three-dimensional joint angles and three-dimensional joint positions are expressed in a generalized coordinate system. A generalized coordinate system is a coordinate system that can express the three-dimensional posture of a modeled subject using variables corresponding to the degrees of freedom of the model. In this embodiment, the generalized coordinate system defines segments SG, each representing the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet, and joints JT(i) connecting the segments (where i = 1 to N). Of the segments SG, segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six-degree-of-freedom displacement of segments SG other than the base segment SG(B) relative to the origin of the absolute coordinate system is not a variable in the generalized coordinate system; only the base segment SG(B) is treated as displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, and pitch). In this example, the three translational directions are time-series data in which the three-dimensional coordinates of the pseudo markers PM associated with the pelvis are the three-dimensional joint positions, and the three rotational directions are joint rotation vectors as the joint angles of the pseudo markers PM associated with the pelvis. In this example, the pseudo markers PM of the pelvis are identified as corresponding to the selected points SV of the IDs associated with the names "LASI," "RASI," "LPSI," and "RPSI," respectively.

[0057] The external force estimator 36 is an example of an analyzer. In this example, the external force estimator 36 obtains analysis results based on time-series data of three-dimensional joint angles and three-dimensional joint positions, such as time-series data of external forces (e.g., floor reaction forces), the foot pressure center position, and the center of gravity. External forces (e.g., floor reaction forces), the foot pressure center position, and the like can be obtained by a known calculation method using a musculoskeletal model having the same skeletal structure as the general skeletal model M1a. The musculoskeletal model has named markers set using the skeleton as indexes, and analysis is performed based on the coordinates, rotation vectors, movement velocities, and the like of these markers. In this example, the pseudo markers PM are identified based on the general skeletal model M1a, which has standard markers SM1 to SM39 with the same names as the markers. Therefore, the pseudo markers PM can be regarded as the markers, thereby enabling analysis using a known calculation method. The external force estimator 36 sends the obtained analysis results to the display controller 37 along with the time-series data of three-dimensional joint angles and three-dimensional joint positions.

[0058] In addition to, or instead of, the external force estimation unit 36, a functional unit that obtains a floor reaction force measured using a floor reaction force meter as an analysis result, or a functional unit that obtains a floor reaction force obtained from an insole-type pressure sensor as an analysis result, can be used.

[0059] The display control unit 37 generates an image displaying the analysis results sent from the external force estimation unit 36, sends it to the client terminal 15, and displays it on the display 15a of the client terminal 15. In this example, based on the three-dimensional joint angles, three-dimensional joint positions, and foot pressure center position obtained by the inverse kinematics processing, a moving image of an animation model M3 showing the subject's movement is generated, and graphs showing changes over time in the three-dimensional joint angles, floor reaction forces, etc. are generated. An image G2 having a first image portion G2a showing the moving image and a second image portion G2b showing the graphs, etc., is displayed on the display 15a (see FIG. 9 ). The display control unit 37 may further generate a file showing the analysis results and transmit the generated file to the client terminal 15. Examples of file formats include the well-known C3D format and TRC format. A C3D format file can represent time-series data such as the three-dimensional coordinates of markers and analog data acquired from a floor reaction force meter and / or an electromyograph. A TRC format file can represent time-series data of the three-dimensional coordinates of markers. If the client terminal 15 is equipped with a program for visualizing the information shown in these files, the display control unit 37 can transmit at least one of these files to the client terminal 15, thereby displaying the analysis results on the client terminal 15. This method further reduces the processing load involved in sending and receiving data to and from the client terminal 15.

[0060] The method for requesting registration of the ID of the selected point SV will be described with reference to FIG. 7 . The selected point SV is preferably a point on the human body model M2 that matches the standard marker SM in terms of anatomical or physical characteristics. Therefore, in this example, the selected point SV is selected as follows to request registration of the ID. When a display request for displaying an image G1 for requesting registration of the ID of the selected point SV is sent from the client terminal 15, the ID registration unit 27 of the motion analysis device 13 causes the display control unit 37 to generate the image G1, and the display control unit 37 causes the image G1 to be displayed on the display 15a of the client terminal 15. In the image G1, the human body model M2 is displayed in a manner that allows the shape to be grasped by the point V.

[0061] Image G1 has a selection image section G1a for selecting a selection point SV from points V (see FIG. 6). The selection image section G1a includes a first selection section B1 for selecting a standard marker SM to be associated with, and a second selection section B2 for selecting a point V to be associated with any selected standard marker SM. The first selection section B1 and the second selection section B2 are operation image sections that allow input operations. For example, when selecting a point V to be associated with each of standard markers SM1 to SM39 of marker set Sa (see FIG. 4B), the first selection section B1 displays the names of the standard markers SM1 to SM39 in a pull-down list, and any one of the markers is selected from the displayed list to become the standard marker SM to be associated. In the example shown in FIG. 7, the "RSHO" standard marker SM11 (see Table 1) is selected. The second selection section B2 displays a list of IDs of points V, and in the example shown in Fig. 7, the IDs of 6,890 points V are displayed in a scroll format. When an arbitrary point V of the human body model M2 is selected, the selected point V is displayed distinguishably from unselected points V. In the example shown in Fig. 7, unselected points V are displayed as black circles (●), and selected points V are displayed as white circles (◯) larger than the black circles. The ID of the selected point V is also displayed distinguishably from other IDs in the second selection section B2. In the example shown in Fig. 7, the ID "5282" of the selected point V is displayed with a colored periphery. In this example, the display 15a is a touch panel display. By touching a checkbox provided next to the ID of the selected point V and then touching the "Register" button B3, a registration request for associating the ID "5282" with the name "RSHO" of the standard marker SM11 is sent from the client terminal 15 to the motion analysis device 13.

[0062] The image G1 preferably includes an operation button B4 for changing the orientation of the displayed human body model M2. The up, down, left, right, front, and back of the body of the human body model M2 are defined by three mutually perpendicular axes: the X-axis, the Y-axis, and the Z-axis. Moving the operation button B4 displaces the three axes, thereby changing the orientation of the human body model M2. In this example, the orientation of the human body model M2 can be changed by a swipe operation, in which the client touches and moves their finger on the operation button B4. For example, in FIG. 7 , by operating the operation button B4, the upper part of the right shoulder of the human body model M2 is oriented downward, so that the human body model M2 is displayed as if viewed from the front. In this example, a point V in the depth direction of the page in FIG. 7 is also displayed to allow the three-dimensional shape of the human body model M2 to be grasped. However, in FIG. 7 , only the point on the front side of the page is depicted to avoid cluttering the illustration. In this way, by displaying the human body model M2 in three dimensions and making it possible to change the orientation, the operation of selecting point V becomes easier and faster, and point V can be selected more accurately.

[0063] When selecting the selection point SV from the point V, it is preferable to display a moving image showing the movement of the human body model M2 in a predetermined area of ​​the image G1, or to switch between the images displayed in addition to the image G1. This allows the person making the registration request to select the selection point SV based on the moving image. This allows the selection point SV to be selected more easily, quickly, and accurately.

[0064] When the ID registration unit 27 performs the aforementioned suitability determination, if a positive determination is made, the ID "5282" indicated in the received registration request is associated with the name "RSHO" and stored in the ID storage unit 20. If a negative determination is made, a negative determination notice is sent to the client terminal 15 and displayed, for example, in image G1. In the example shown in FIG. 7 , the text "Please select another point" is displayed as a notification display B5 indicating the negative determination notice. After displaying this notification display B5, the client terminal 15 selects another point, for example, another point V1 located near the point previously requested to be registered as the selected point SV, and then makes a registration request. In addition to the negative determination notice, a "Confirm" button B6 for confirming the ID indicated in the registration request may be displayed in image G1. Touching this button B6 may cause the ID indicated in the registration request for which a negative determination notice has been received to be sent from the client terminal 15 to the ID registration unit 27 as a registration confirmation request. In response to receiving the registration confirmation request, the ID registration unit 27 associates the negatively determined ID "5282" with the name "RSHO" and stores it in the ID storage unit 20. The suitability determination may be performed for each selection point SV, or may be performed collectively for all selection points SV (in this example, 39 selection points SV corresponding to the number of standard markers SM1 to SM39). When performing the suitability determination collectively, the notification display B5 may display the ID of the selection point SV that was the subject of the negative determination in addition to the above-mentioned "Please select another point."

[0065] The ID storage unit 20 stores the ID of the point V selected as the selection point SV as described above, associating it with the name of the standard marker SM, and stores marker ID information that associates the names of all standard markers SM1 to SM39 with the selection point SV, as shown in Figure 4D. The names of the standard markers SM in the other marker sets Sb and Sc are also similarly stored in association with the selection point SV. In this way, the ID storage unit 20 stores marker ID information for each marker set S.

[0066] In the above example, the pseudo marker data acquisition unit 21 sends the acquired pseudo marker data to the inverse kinematics processing unit 33. However, this is not limiting. For example, the pseudo marker data acquisition unit 21 may include a storage unit (not shown) and store the acquired pseudo marker data in the storage unit. Then, using a trained model that has trained a dataset in which at least one of a captured image and two-dimensional coordinates of joint positions is used as input information and the stored pseudo marker data is used as output information, the pseudo marker data may be estimated by AI from the captured image and the two-dimensional coordinates of joint positions. The estimated pseudo marker data may then be sent to the inverse kinematics processing unit 33 for inverse kinematics processing. In this way, the acquired pseudo marker data may be used to generate a trained model, and inverse kinematics processing may be performed using the estimated pseudo marker data during analysis.

[0067] According to the above example, the ID storage unit 20 stores the names of the standard markers SM1 to SM39 associated with the general skeletal model M1a, which has the same skeletal structure as the musculoskeletal model to be analyzed, and the IDs of the selected points SV selected from the multiple points V set on the human body model M2, in association with each other. The pseudo marker data acquisition unit 21 extracts pseudo markers PM from the multiple points V on the human body model M2 applied to the body image, which is the subject image in the captured image, and acquires the coordinates of the pseudo markers PM. Therefore, musculoskeletal analysis can be performed using the human body model M2 and a musculoskeletal model having the same skeletal structure as the general skeletal model M1a. By replacing the general skeletal model M1a with one of the general skeletal models M1b to M1h, musculoskeletal analysis can be performed using a musculoskeletal model having the same skeletal structure as any one of the general skeletal models M1b to M1h. Thus, according to this example, musculoskeletal analysis can be performed using a desired musculoskeletal model. In addition, any one of various human body models M2 whose shape is determined by a plurality of points V, which is greater than the standard markers SM, can be used.

[0068] In the above example, the process of determining whether or not the ID of the selection point SV is associated with the name of the standard marker SM is performed using time-series data of the three-dimensional coordinates of the reference marker and a moving image for determining whether or not the reference marker is appropriate. However, instead of this, the process may be performed using the three-dimensional coordinates of the reference marker at a single point in time and a still image of the subject wearing the reference marker (hereinafter referred to as a still image for determining whether or not the reference marker is appropriate). Using the time-series data of the three-dimensional coordinates of the reference marker and a moving image for determining whether or not the reference marker is appropriate is preferable because it allows for more accurate extraction of pseudo marker data and more reliable analysis results.

[0069] In the above example, inverse kinematics processing is performed using the general skeletal model M1, but inverse kinematics processing may also be performed using a subject skeletal model. The subject skeletal model is obtained by applying the height of the subject, who is the subject of the captured image, to the general skeletal model M1 and correcting for the height. The motion analysis device 50 shown in FIG. 10 differs from the motion analysis device 13 in that it further includes a subject skeletal model generation unit 51, a subject skeletal model storage unit 53, and a subject marker data calculation unit 55. In the following explanation, only the differences from the motion analysis device 13 will be described, and in FIG. 10, functional units that are the same as those in the motion analysis device 13 are assigned the same reference numerals as in FIG. 3 and will not be described again.

[0070] When one of the general skeletal models M1a to M1h is selected on the client terminal 15 and the height of the subject is sent from the client terminal 15, the subject skeletal model generation unit 51 reads out the selected general skeletal model M1 from the general skeletal model storage unit 22, and applies the subject's height to the general skeletal model M1 to generate a subject skeletal model, which is stored in the subject skeletal model storage unit 53. The subject skeletal model has the same standard markers SM as the general skeletal model M1, and the subject skeletal model storage unit 53 stores the names of the standard markers SM, similar to the general skeletal model storage unit 22.

[0071] The pseudo marker data acquisition unit 21 in this example sends the time-series data of the acquired pseudo marker data to the subject marker data calculation unit 55. The subject marker data calculation unit 55 corrects the three-dimensional coordinates of the pseudo markers PM (see FIG. 8 ) in each frame image GF (see FIG. 8 ) based on the subject's height, and obtains subject marker data. The obtained time-series data of the subject marker data is sent to the inverse kinematics processing unit 33. When the time-series data of the subject marker data is input, the inverse kinematics processing unit 33 reads out the subject skeletal model from the subject skeletal model storage unit 53 in response to this input, performs inverse kinematics processing, and sends the three-dimensional coordinates of the joint angles and joint positions to the external force estimator 36.

[0072] When making the suitability determination in this embodiment, the three-dimensional coordinates of the selection point SV may be found by executing processing by the subject marker data calculation unit 55 after pseudo marker data acquisition by the pseudo marker data acquisition unit 21. Specifically, when the pseudo marker data acquisition unit 21 receives the three-dimensional coordinates of multiple points V in the suitability determination moving image from the three-dimensional posture estimation unit 32 during suitability determination in this embodiment, the pseudo marker data acquisition unit 21 generates pseudo marker data including the three-dimensional coordinates of the pseudo markers PM, as in the case of the motion analysis device 13, and sends the pseudo marker data to the subject marker data calculation unit 55. The subject marker data calculation unit 55 corrects the three-dimensional coordinates of the pseudo markers PM based on the height of the subject, sets the pseudo marker data as subject marker data, and sends the corrected data to the ID registration unit 27. The ID registration unit 27 regards the coordinates indicated in the subject marker data as the three-dimensional coordinates of the selection point SV, and makes a judgment on suitability based on the degree of difference between this selection point SV and the three-dimensional coordinates of the judgment reference marker sent from the reflective marker data acquisition device 19.

[0073] According to this example, the three-dimensional coordinates of the pseudo markers PM according to the height of the subject can be obtained, so in addition to the effects of the example using the motion analysis device 13 described above, more accurate pseudo marker data can be obtained.

[0074] It is also possible to perform musculoskeletal analysis using various musculoskeletal models without using the human body model M2. A motion analysis device 70 shown in Fig. 11 includes a pseudo marker data acquisition unit 71 instead of the pseudo marker data acquisition unit 21 of the motion analysis device 13 (see Fig. 3). The motion analysis device 70 differs from the motion analysis device 13 in that it does not need to include the human body model storage unit 26, ID registration unit 27, and ID storage unit 20 of the motion analysis device 13. In the following explanation, only the differences from the motion analysis device 13 will be described, and in Fig. 11, functional units that are the same as those in the motion analysis device 13 are assigned the same reference numerals as in Fig. 3 and will not be described again.

[0075] The pseudo marker data acquisition unit 71 is configured with AI and has trained models LMa to LMc. In the following description, when there is no need to distinguish between the trained models LMa to LMc, they will be referred to as trained model LM. The trained model LMa is generated by machine learning using as training data a dataset in which captured images of a human subject are used as input information and the names and three-dimensional coordinates of the markers are used as output information.

[0076] The captured images of the training data used as input information for each of the trained models LMa to LMc need only be images of at least a part of a human body. That is, the captured images may be images of the entire human body or images of any part of a human body, and in this example, both of these are used for training.

[0077] The markers in the training data trained by the trained model LMa correspond to the standard markers SM1 to SM39 of the marker set Sa, and there are 39 of them. The names of the markers are the same as the names of the standard markers SM1 to SM39. Like the trained model LMa, each of the trained models LMb and LMc is generated by machine learning training data in which captured images of a human subject are used as input information and the names and three-dimensional coordinates of the markers are used as output information. The markers trained by the trained model LMb correspond to the standard markers of the marker set Sb as shown in FIG. 4C, and the markers trained by the trained model LMc correspond to the standard markers of the marker set Sc. In this way, the trained models LMa to LMc are trained by machine learning markers corresponding to the standard markers of the marker sets Sa to Sc. In this example, the number of learned models LM is three, but this may be increased or decreased depending on the number of marker sets S prepared to be associated with the general skeleton model M1 used in the inverse kinematics processing unit 33. In this example, three marker sets S, marker sets Sa to Sc, are prepared as in the example described above with reference to Figure 3, and therefore the number of learned models LM is also three.

[0078] The three-dimensional coordinates of the markers in the training data that are trained as output information for each of the trained models LMa to LMc can be obtained by, for example, the following various methods. In a method using optical motion capture, reflective markers are attached to the human body, and the three-dimensional coordinates of the markers can be measured and determined using multiple infrared cameras. In a method using markerless motion capture, a subject's skeletal model is created, a marker set identical to the pseudo markers to be trained is set on the subject's skeletal model, and the three-dimensional coordinates of the pseudo markers can be obtained using forward kinematics.

[0079] The acquisition unit 31 of the motion analysis device 70 acquires a captured image of a subject, and in response to this acquisition, sends the captured image to the pseudo marker data acquisition unit 71. In addition, a selection instruction indicating one selected from the multiple marker sets S is sent from the client terminal 15 or the user terminal 11 to the pseudo marker data acquisition unit 71 via the controller 38, and a selection instruction indicating one selected from the multiple general skeleton models M1 is sent to the inverse kinematics processing unit 33 via the controller 38. In this example, the description will be given assuming that the marker set Sa is selected, and the general skeleton model M1a is selected as in the embodiment of the motion analysis device 13.

[0080] When a selection instruction indicating one marker set S is input, the pseudo marker data acquisition unit 71 selects a learned model LM that has been trained using, as output information of teacher data, a marker corresponding to the standard marker SM of the marker set S indicated in the selection instruction. In this example, the pseudo marker data acquisition unit 71 receives a selection instruction indicating the marker set Sa, and therefore selects the learned model LMa. When a captured image of the subject P (see FIG. 2 ) is input from the acquisition unit 31, the pseudo marker data acquisition unit 71 inputs the captured image to the selected learned model LMa and outputs the three-dimensional coordinates of the pseudo markers PM. The pseudo marker data acquisition unit 71 sends the three-dimensional coordinates of the pseudo markers PM identified (estimated) and acquired in this way to the inverse kinematics processing unit 33. As described above, identifying the pseudo markers PM includes estimation by AI.

[0081] When the inverse kinematics processing unit 33 receives the three-dimensional coordinates of the pseudo marker PM from the pseudo marker data acquisition unit 71 and a selection instruction indicating the general skeleton model M1a from the controller 38, it identifies the above-mentioned first marker correspondence information including the general skeleton model M1a from the marker correspondence storage unit 25, and identifies the standard markers SM1 to SM39 of the marker set Sa included in this first marker correspondence information. The inverse kinematics processing unit 33 performs inverse kinematics processing with the general skeleton model M1a based on the three-dimensional coordinates of the identified standard markers SM1 to SM39 and the three-dimensional coordinates of the pseudo marker PM.

[0082] According to this example, a learned model LMa is used that has learned the three-dimensional coordinates of markers corresponding to standard markers SM1 to SM39 associated with a general skeletal model M1a. Furthermore, the general skeletal model M1a has the same skeletal structure as the musculoskeletal model of the target musculoskeletal analysis, and the standard markers SM1 to SM39 are associated with the same names as the markers of the musculoskeletal model. Therefore, as in the above-described embodiments, pseudo markers PM that can be regarded as markers of the musculoskeletal model can be identified from the captured image, thereby enabling analysis using a known calculation method. In this way, analysis can be performed using a desired musculoskeletal model without using the above-described human body model M2.

[0083] The trained model LM may be replaced with another data set that has been machine-learned as training data. For example, the training data may be a data set in which two-dimensional coordinates of joint positions are input information and names of markers and three-dimensional coordinates of the markers are output information. Alternatively, the training data may be a data set in which three-dimensional coordinates of joint positions are input information and names of markers and three-dimensional coordinates of the markers are output information. A method for performing analysis using a trained model trained on these data sets will be described with reference to FIGS. 12 and 13 .

[0084] A motion analysis device 80 shown in FIG. 12 includes a pseudo marker data acquisition unit 81 instead of the pseudo marker data acquisition unit 71 ( FIG. 11 ). The pseudo marker data acquisition unit 81 includes trained models LMa to LMc obtained by machine learning using a data set in which two-dimensional coordinates of joint positions are input information and names of markers KM and three-dimensional coordinates of the markers KM are output information as training data. The trained models LMa to LMc are the same as the trained models LMa to LMc of the pseudo marker data acquisition unit 71 in that they have learned markers KM corresponding to standard markers SM set in association with one selected from marker sets Sa to Sc. Furthermore, the number of trained models LM may be increased or decreased depending on the number of marker sets S prepared for the inverse kinematics processing unit 33. In this example, three marker sets S, namely marker sets Sa to Sc, are prepared, so the number of trained models LM is also three, which is also the same as the trained models LMa to LMc of the pseudo marker data acquisition unit. The motion analysis device 80 differs from the motion analysis device 70 in that it further includes a joint position coordinate estimation unit 82 .

[0085] As shown in FIG. 13A, the joint positions serving as input information of the training data used to generate the trained model LMa of the pseudo marker data acquisition unit 81 are defined by the two-dimensional coordinates of multiple key points KP set on the skeleton of a human body model M4, which is different from the human body model M2 (see FIG. 6). The human body model M4 may be one in which the above-mentioned key points KP are set, and the shape of the human body model M4 may not be determined by a skin polygon PG (see FIG. 6) and a point V (see FIG. 6). The three-dimensional coordinates of the markers KM serving as output information of the training data used to generate the trained model LMa are the three-dimensional coordinates of markers KM1 to KM39 corresponding to the standard markers SM1 to SM39, as with the trained model LMa of the pseudo marker data acquisition unit 71. In FIG. 13B, the markers KM1 to KM39 located on the front side are depicted. When this trained model M1a is used, the pseudo marker data acquisition unit 81 receives the two-dimensional coordinates of the joint position estimated based on the captured image of the subject P and defined by the two-dimensional coordinates of the key point KP, inputs the two-dimensional coordinates into the trained model M1a, and outputs the three-dimensional coordinates of the pseudo marker PM.

[0086] The two-dimensional coordinates of the key points KP used as training data to generate the learned model LM can be obtained by a known method, for example, as described in the above-mentioned Non-Patent Document 1 or Non-Patent Document 2.

[0087] When the acquisition unit 31 of the motion analysis device 80 acquires a captured image of the subject P (see FIG. 2 ), it sends the captured image to the joint position coordinate estimation unit 82. In addition, a selection instruction indicating one selected from the multiple marker sets S is sent from the client terminal 15 or the user terminal 11 via the controller 38 to the pseudo marker data acquisition unit 81, and a selection instruction indicating one selected from the multiple general skeleton models M1 is sent via the controller 38 to the inverse kinematics processing unit 33. In this example, the marker set Sa is selected, and the description will be given assuming that the general skeleton model M1a is selected as in the example of the motion analysis device 70.

[0088] The joint position coordinate estimation unit 82 estimates the two-dimensional coordinates of the joint positions based on the input captured image. The estimation of the two-dimensional coordinates of the joint positions can be performed by a known method such as that described in the above-mentioned Non-Patent Document 1 or Non-Patent Document 2.

[0089] When a selection instruction indicating one marker set S is input, the pseudo marker data acquisition unit 81 selects a learned model LM that has been trained using, as output information, a marker KM corresponding to a standard marker SM of the marker set S associated with the general skeletal model M1 indicated in the selection instruction. In this example, the pseudo marker data acquisition unit 81 receives a selection instruction indicating the marker set Sa, and therefore selects the learned model LMa. When time-series data of two-dimensional coordinates of joint positions is input from the joint position coordinate estimation unit 82, the pseudo marker data acquisition unit 81 inputs the time-series data of the two-dimensional coordinates to the selected learned model LMa and outputs time-series data of three-dimensional coordinates of the pseudo marker PM. The pseudo marker data acquisition unit 81 sends the time-series data of the three-dimensional coordinates of the pseudo marker PM identified (estimated) and acquired in this manner to the inverse kinematics processing unit 33. The subsequent processing is similar to that of the motion analysis device 70.

[0090] The trained models LMa to LMc of the pseudo marker data acquisition unit 81 may be replaced with models trained by machine learning using a data set in which the three-dimensional coordinates of joint positions are input information and the names and three-dimensional coordinates of the markers KM are output information. In this case, the joint positions serving as input information of the training data to be trained when generating the trained model LMa of the pseudo marker data acquisition unit 81 are defined by the three-dimensional coordinates of multiple key points KP set on the skeleton of the human body model M4.

[0091] The three-dimensional coordinates of the keypoints KP used as training data to generate the trained model LM can be obtained, for example, by the method described in Non-Patent Document 3 (D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 2019. “3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training.” 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019-June, 7745-7754.) or Non-Patent Document 4 (U. Solichah, M. H. Purnomo and E. M. Yuniarno. 2020 “Marker-less Motion Capture Based on Openpose Model Using Triangulation,” International Seminar on Intelligent Technology and Its Applications (ISITIA), Surabaya, Indonesia, 2020, pp. 217-222).

[0092] In this case, the joint position coordinate estimation unit 82 estimates the three-dimensional coordinates of the joint positions based on the input captured image of the subject P. The estimation of the three-dimensional coordinates of the joint positions can be performed by the method described in the above-mentioned Non-Patent Document 3 or Non-Patent Document 4. The joint position coordinate estimation unit 82 sends the estimated three-dimensional coordinates of the joint positions to the pseudo marker data acquisition unit 81.

[0093] When the pseudo marker data acquisition unit 81 receives time-series data of the three-dimensional coordinates of the joint positions from the joint position coordinate estimation unit 82, it inputs the time-series data of the three-dimensional coordinates to the selected trained model LMa and outputs the time-series data of the three-dimensional coordinates of the pseudo markers PM. The pseudo marker data acquisition unit 81 sends the time-series data of the three-dimensional coordinates of the pseudo markers PM thus identified (estimated) and acquired to the inverse kinematics processing unit 33.

[0094] According to the example of the motion analysis device 80, a learned model LMa is used that has learned the three-dimensional coordinates of markers KM1 to KM39 corresponding to standard markers SM1 to SM39. Furthermore, the general skeletal model M1a has the same skeletal structure as the musculoskeletal model of the target musculoskeletal analysis, and the standard markers SM1 to SM39 are associated with the same names as the markers of the musculoskeletal model. Therefore, as with the above-described embodiments, pseudo markers PM that can be regarded as markers of the musculoskeletal model can be identified from the captured image, and analysis can be performed using a known calculation method. In this way, as with the example of the motion analysis device 70, analysis can be performed using a desired musculoskeletal model without using the above-described human body model M2.

[0095] 13, 50, 70, 80 Motion analysis device 20 ID storage unit 21, 71, 81 Pseudo marker data acquisition unit 27 ID registration unit 31 Acquisition unit 33 Inverse kinematics processing unit 36 ​​External force estimation unit 55 Subject marker data calculation unit SM1 to SM39 Standard markers M1a to M1h General skeletal model M2 Human body model PG Skin polygon PM Pseudo marker SV Selection point V Point (human body model)

Claims

1. An information processing device comprising: an acquisition unit that acquires an image of a subject to be analyzed; and a pseudo marker data acquisition unit that identifies pseudo markers on a body image that is the subject image in the captured image and acquires the coordinates of the pseudo markers.

2. An information processing device as described in claim 1, further comprising an ID memory unit that stores the names of each of a plurality of standard markers associated with a pre-selected general skeletal model and identification information of a selected point selected from a plurality of points set in a human body model applied to the body image that is greater than the number of the standard markers, and wherein the pseudo marker data acquisition unit identifies, from among the plurality of points in the human body model applied to the body image, a point associated with the same identification information as the identification information as the pseudo marker.

3. The information processing device according to claim 2, wherein the human body model is represented by a collection of skin polygons, and each of the plurality of points set on the human body model is a vertex of the skin polygon.

4. The information processing device according to claim 2 or 3, wherein the human body model is an SMPL (Skinned Multi-Person Linear) model.

5. The information processing device according to claim 2 or 3, wherein the selected point is selected based on a moving image showing the movement of the human body model.

6. An information processing device according to claim 2 or 3, further comprising an ID registration unit that stores the identification information of the selected point in the ID storage unit based on a registration request for the identification information of the selected point to be associated with the name of any of the standard markers.

7. The information processing device described in claim 6, wherein the ID registration unit determines whether or not the name is associated with the identification information indicated in the registration request based on the degree of difference between the three-dimensional coordinates of the judgment reference marker obtained by motion capture in which a judgment reference marker is set at a position corresponding to the standard marker of the subject and the three-dimensional coordinates of the selection point indicated in the registration request, and if a positive judgment is made, stores the identification information indicated in the registration request in the ID storage unit, and if a negative judgment is made, issues a negative judgment notification indicating a negative judgment.

8. The information processing device according to claim 2 or 3, further comprising an inverse kinematics processing unit that performs inverse kinematics processing with the general skeleton model based on the three-dimensional coordinates of the pseudo markers and the three-dimensional coordinates of the standard markers.

9. The information processing device according to claim 2 or 3, further comprising a subject marker data calculation unit that calculates three-dimensional coordinates of the standard markers for a subject skeletal model in which the height of the subject is applied to the general skeletal model.

10. The information processing device according to claim 9, further comprising an inverse kinematics processing unit that performs inverse kinematics processing with the subject's skeletal model based on the three-dimensional coordinates of the pseudo markers and the three-dimensional coordinates of the standard markers.

11. The information processing device according to claim 1, wherein the pseudo marker data acquisition unit has a trained model generated by machine learning with input information selected from a captured image of a human subject, two-dimensional coordinates of joint positions, and three-dimensional coordinates of joint positions, and output information being the three-dimensional coordinates of the pseudo markers, and acquires the three-dimensional coordinates of the pseudo markers by inputting one selected from a captured image of the human subject, two-dimensional coordinates of joint positions obtained based on the captured image of the human subject, and two-dimensional coordinates obtained based on the captured image of the human subject into the trained model and outputting the three-dimensional coordinates of the pseudo markers.

12. The information processing device according to claim 2 or 8, wherein the pseudo marker data acquisition unit acquires three-dimensional coordinates of the pseudo markers for each of a plurality of frames constituting a moving image as the captured image of the subject captured from a fixed point.

13. An information processing method comprising: an acquisition step in which an acquisition unit and a pseudo marker data acquisition unit of an information processing device acquires a captured image of a subject of analysis as a subject, and a pseudo marker data acquisition step in which the pseudo marker data acquisition unit identifies pseudo markers on a body image that is the subject image in the captured image and acquires the coordinates of the pseudo markers.

14. An information processing program that causes a computer to execute the following steps: an acquisition step of acquiring a captured image of a person to be analyzed as a subject; and a pseudo marker data acquisition step of identifying pseudo markers on a body image that is the subject image in the captured image and acquiring the coordinates of the pseudo markers.

Images