Processing device, processing system, processed model construction method, and program

Abstract

The purpose of the present invention is to automatically analyze a turning operation state of the human body. A turning feature amount extraction unit extracts feature points to be trained of a subject from two frames selected from moving image data to be trained of the subject, and calculates, on the basis of the difference between turning operations of the body, turning feature amounts that indicate the turning operations to be trained of the body of the subject, the difference being calculated from the feature points extracted from the two frames. A training data construction unit imparts, to the turning feature amount F, a teaching level indicating a corresponding state to be trained of the subject and constructs training data. A training processing unit constructs a trained model by performing training with the training data.

Description

Processing device, processing system, method for constructing trained model, and program

[0001] The present disclosure relates to a processing device, a processing system, a method for constructing a trained model, and a program.

[0002] In the medical field, analyzing the state of human movement and planning treatment and rehabilitation based on the analysis results is a common practice. While such analyses have traditionally been carried out by specialists such as doctors and physical therapists, in recent years, there has been progress in the development of methods for analyzing the state of human movement.

[0003] As a method for analyzing such human body movements, a processing device that calculates the range of motion of a subject's movements has been proposed (Patent Document 1). This processing device takes an image of the subject, selects a movement to be measured from the image, and measures the range of motion of the selected movement.

[0004] Various other methods for analyzing human body movements have also been proposed (Patent Documents 2 to 5).

[0005] International Publication No. 2020 / 021873 JP 2022-131397 A JP 2022-65241 A JP 2018-94084 A JP 2015-61579 A

[0006] Although the above-mentioned analysis methods can measure the range of motion of movements, analysis of rotational movements, one of the main movements of the human body, is more difficult than linear movements, and the development of automatic recognition technology has not progressed. Therefore, in order to analyze and improve physical function problems related to rotational movements, the intervention of experts such as doctors and physical therapists is still required. Therefore, there is a growing need for posture state recognition technology that can also automatically analyze rotational movements.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to automatically analyze the state of rotational movement of the human body.

[0008] A processing device according to one aspect of the present disclosure includes a rotation feature extraction unit that extracts feature points of a training subject from two frames, one before and one after the body rotation movement, selected from video image data capturing the body rotation movement of the training subject, and calculates a rotation feature that indicates the body rotation movement of the training subject based on a difference in the positions of the feature points extracted from the two frames before and after the body rotation movement; a training data construction unit that constructs training data by assigning teacher labels that indicate the state of the training subject that correspond to the rotation feature; and a learning processing unit that constructs a trained model by learning the training data.

[0009] A processing device according to one aspect of the present disclosure includes a rotation feature amount extraction unit that extracts feature points of a subject to be estimated from two frames before and after a rotation movement of the body selected from video image data capturing the rotation movement of the body of the subject to be estimated, and calculates a rotation feature amount indicating the rotation movement of the body of the subject to be estimated based on a difference in the positions of the feature points extracted from the two frames before and after the rotation movement of the body; an estimation processing unit that extracts feature points of the subject to be trained, and inputs the rotation feature amounts of the subject to be trained calculated by the rotation feature amount extraction unit into a trained model constructed by learning a correspondence between rotation feature amounts indicating the rotational movement of the body of the subject to be trained, the rotation feature amounts being calculated based on a difference between the positions of the feature points extracted from the two frames before and after the rotational movement of the body, and the state of the subject to be trained corresponding to the rotation feature amounts.

[0010] A processing system according to one aspect of the present disclosure includes an imaging device that acquires video image data of a rotational movement of a body of a subject to be trained, and a processing device that learns rotational features indicative of the rotational movement of the body of the subject to be trained from the video image data of the subject to be trained, and constructs a trained model that estimates a state of the subject to be trained. The processing device includes: a rotational feature extraction unit that extracts feature points of the subject to be trained from two frames, one before and one after the body rotational movement, selected from the video image data of the body rotational movement of the subject to be trained, and calculates rotational features indicative of the body rotational movement of the subject to be trained based on a difference in the positions of the feature points extracted from the two frames before and after the body rotational movement; a training data construction unit that constructs training data by assigning teacher labels indicative of the state of the subject to be trained that correspond to the rotational features; and a learning processing unit that constructs a trained model by learning the training data.

[0011] A method for constructing a trained model that is one aspect of the present disclosure involves extracting feature points of a training subject from two frames, one before and one after the body rotation movement, selected from video image data capturing the body rotation movement of the training subject, calculating rotation feature amounts indicating the body rotation movement of the training subject based on the difference between the positions of the feature points extracted from the two frames before and after the body rotation movement, constructing training data by assigning teacher labels indicating the state of the training subject that correspond to the rotation feature amounts, and learning the training data to construct a trained model.

[0012] A program according to one aspect of the present disclosure causes a computer to execute the following processes: extracting feature points of a subject to be trained from two frames, one before and one after the body rotation movement, selected from video image data capturing the body rotation movement of the subject to be trained; calculating rotation feature amounts indicating the body rotation movement of the subject to be trained based on the difference between the positions of the feature points extracted from the two frames before and after the body rotation movement; constructing training data by assigning teacher labels indicating the state of the subject to be trained that correspond to the rotation feature amounts; and constructing a trained model by learning the training data.

[0013] According to the present disclosure, the state of rotational movement of the human body can be automatically analyzed.

[0014] 1 is a diagram schematically illustrating a configuration of a processing system according to a first embodiment. FIG. 2 is a diagram illustrating a modified example of the processing system according to the first embodiment. FIG. 3 is a diagram schematically illustrating a configuration of a processing device in a learning phase according to the first embodiment. FIG. 4 is a flowchart illustrating processing in the learning phase of the processing device according to the first embodiment. FIG. 4 is a diagram schematically illustrating a configuration of a rotation feature amount extraction unit according to the first embodiment. FIG. 5 is a diagram schematically illustrating skeletal points extracted by a feature point extraction unit. FIG. 6 is a diagram illustrating an outline of calculation of an amount of acromion rotation in each frame. FIG. 7 is a diagram illustrating an outline of calculation of left upper arm separation in each frame. FIG. 8 is a diagram illustrating an outline of calculation of right upper arm separation in each frame. FIG. 9 is a diagram illustrating an outline of calculation of left lower arm flexion in each frame. FIG. 10 is a diagram illustrating an outline of calculation of right lower arm flexion in each frame. FIG. 11 is a diagram illustrating an outline of calculation of acromion level in each frame. FIG. 12 is a diagram illustrating an outline of calculation of upper trunk anterior / posterior tilt in each frame. FIG. 13 is a diagram illustrating an outline of calculation of pelvic level in each frame. FIG. 14 is a diagram illustrating an outline of calculation of upper trunk lateral flexion in the previous frame. FIG. 15 is a diagram illustrating an outline of calculation of upper trunk lateral flexion in the subsequent frame. FIG. 16 is a diagram illustrating an outline of calculation of an amount of pelvic rotation in each frame. FIG. 17 is a diagram illustrating a list of teacher labels. FIG. 18 is a diagram schematically illustrating a configuration of a processing device in an estimation phase according to the first embodiment. 1 is a flowchart showing processing in an estimation phase of a processing device according to a first embodiment; FIG. 2 is a diagram showing an example of a configuration of a processing device 10 according to a first embodiment; FIG. 3 is a diagram showing an example of a configuration of a processing device that executes only processing in a learning phase; FIG. 4 is a diagram showing an example of a configuration of a processing device that executes only processing in an estimation phase; FIG. 5 is a diagram showing a configuration in a learning phase of a processing device according to a second embodiment; FIG. 6 is a flowchart showing processing in a learning phase of a processing device according to a second embodiment; FIG. 7 is a diagram showing a configuration of a convolution feature extraction unit according to a second embodiment; FIG. 8 is a diagram showing a list of teacher labels; FIG. 9 is a diagram showing a configuration in an estimation phase of a processing device according to a second embodiment;FIG. 1 is a diagram schematically illustrating a configuration of a computer, which is an example of a hardware configuration for realizing a processing device or a processing system.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are designated by the same reference numerals, and redundant explanations will be omitted as necessary.

[0016] A processing system 100 according to embodiment 1 will be described. The processing system 100 is configured to estimate the motion state of a body part to be analyzed when the subject OBJ performs a motion of twisting the body to the right or left, i.e., a rotational motion of rotating the body to the right or left, based on an image or video of the body of the subject OBJ.

[0017] The rotational movement of the body here refers to the movement of rotating the body to the right or left while keeping the ground position and direction of both feet fixed, and this means that each part of the upper body, such as the arms, shoulders, neck, waist, and legs, moves in unison.

[0018] 1 is a schematic diagram showing a configuration of a processing system 100 according to the first embodiment. The processing system 100 includes a processing device 10 and a camera 20.

[0019] The camera 20 captures an image or video of the subject OBJ, who is the imaging target, and outputs the captured image or video data to the processing device 10. In the following description, the camera 20 will be described as outputting video data MOV to the processing device 10. In addition, in the following description, video data and image data will be collectively referred to as moving image data.

[0020] 1, the moving image data MOV is output from the camera 20 to the processing device 10, but this is merely an example. For example, the moving image data MOV may be stored in another storage device, and the processing device 10 may read the moving image data MOV from the storage device as needed.

[0021] 2 shows a modified example of the processing system according to the first embodiment. The processing system 101 in FIG. 2 is different from the processing system 100 in FIG. 1 in that a moving image database 30 is added. The moving image database 30 is configured as one of various storage devices or is configured to be storable in one of various storage devices. The moving image database 30 appropriately stores moving image data MOV captured by the camera 20. The processing device 10 reads the moving image data MOV from the moving image database 30 as necessary.

[0022] The processing device 10 is configured to estimate, based on the received video data MOV, a feature amount indicating the movement state of the part to be analyzed when the imaged subject OBJ turns his / her body right or left.

[0023] A processing device according to embodiment 1 will be described. The processing device 10 constructs a trained model by learning the correspondence between rotation feature amounts extracted from a video of a subject OBJ to be trained and the state of the subject OBJ to be trained, and estimates the movement state of a body part to be analyzed when the subject to be trained rotates his / her body to the right or left by inputting the rotation feature amounts calculated for the subject to be estimated into the trained model.

[0024] The configuration and operation of the processing device 10 will be described below, separately for a learning phase and an estimation phase. Fig. 3 schematically shows the configuration of the processing device 10 according to the first embodiment in the learning phase. Fig. 4 shows the processing of the processing device 10 according to the first embodiment in the learning phase.

[0025] The processing device 10 includes a rotation feature extraction unit 11 , a learning data construction unit 12 , and a learning processing unit 13 .

[0026] Step S11: The rotation feature extraction unit 11 receives the video data MOV and extracts the rotation feature of the subject OBJ to be trained from the video data MOV. Fig. 5 shows a schematic configuration of the rotation feature extraction unit 11 according to the first embodiment. The rotation feature extraction unit 11 includes a data reading unit 1, a feature point extraction unit 2, and a feature calculation unit 3.

[0027] Step S111: The data reading unit 1 receives the moving image data MOV of the subject OBJ to be learned from, for example, a camera.

[0028] In step S112, the feature point extraction unit 2 detects feature points for detecting the movements of the subject OBJ, which is the learning target, based on the video data MOV received by the data reading unit 1. Here, the feature points to be detected are the skeletal points of the human body of the subject OBJ, which is the learning target. The method for detecting the skeletal points is not limited to a specific method, and various methods can be applied.

[0029] Fig. 6 shows a schematic diagram of skeleton points extracted by the feature point extraction unit 2. Fig. 6 is a front view of the subject OBJ to be learned, with the direction from the back of the subject OBJ to the front being the x-direction. However, in Fig. 6, the x-direction is tilted to make the drawing easier to see. Furthermore, the direction from right to left of the subject OBJ, i.e., from left to right on the drawing, is the y-direction, and the direction from bottom to top is the z-direction.

[0030] The feature point extraction unit 2 extracts 15 skeletal points from the subject OBJ. As shown in FIG. 6 , the feature points extracted from top to bottom on the midline of the subject OBJ are the nose C1, neck C2, and waist C3. For the right half of the body, the feature points extracted from top to bottom are the right shoulder R1, right elbow R2, and right wrist R3 for the right arm, and the feature points extracted from top to bottom are the right waist R4, right knee R5, and right ankle R6 for the right waist and right lower leg. For the left half of the body, the feature points extracted from top to bottom are the left shoulder L1, left elbow L2, and left wrist L3 for the left arm, and the feature points extracted from top to bottom are the left waist L4, left knee L5, and left ankle L6 for the left waist and left lower leg. Hereinafter, the feature points extracted by the feature point extraction unit 2 will also be referred to as a feature point group P.

[0031] In step S113, the feature calculation unit 3 extracts feature amounts indicating the motion state of the analysis target part of the subject OBJ to be studied before and after the body rotation motion, based on the skeletal points estimated by the feature point extraction unit 2. The rotation feature amounts extracted by the feature calculation unit 3 will be described below. In this embodiment, the feature calculation unit 3 calculates the following 10 types of rotation feature amounts.

[0032] Hereinafter, in calculating the rotation feature, two frames separated in time from a moving image are selected, and the rotation feature of the analysis target part of the subject OBJ to be learned between these two frames is calculated. Hereinafter, of the two frames separated in time, the earlier frame in time will be referred to as the previous frame, and the later frame in time will be referred to as the later frame.

[0033] In this embodiment, a frame in which the subject is standing facing forward is set as the front frame, and a frame in which the subject is rotating his / her body to the right or left is set as the back frame, so that the rotation feature amount can be suitably calculated. Hereinafter, the rotation feature amount is expressed as the displacement of the position of a feature point, i.e., a skeleton point, as the angular displacement of a vector connecting the feature points between two frames.

[0034] F0: Amount of acromion rotation A feature quantity indicating the amount of rotation of the line connecting the left shoulder L1 and the right shoulder R1 in the upper trunk relative to the midline of the subject OBJ is defined as the acromion rotation quantity F0. Calculation of the acromion rotation quantity F0 will be described below.

[0035] The following calculations are performed for each frame. Figure 7 shows an outline of the calculation of the amount of acromion rotation for each frame. First, a plane S0 perpendicular to vector a extending from the neck C2 to the waist C3 is fixed. Next, the angle θ formed by vector B obtained by projecting vector b extending from left shoulder L1 to right shoulder R1 onto plane S0, and vector C obtained by projecting vector c extending from left waist L4 to right waist R4 onto plane S0 is calculated. Note that in the following, the angle calculated from the subsequent frame is referred to as θ Ｌ , the angle calculated from the previous frame is θ Ｆ Let's say.

[0036] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting F from the acromion rotation amount F0 is calculated.

[0037] F1: Left Upper Arm Separation The feature quantity indicating how far the left upper arm is separated from the upper trunk when the body is rotated to the right or left, i.e., the compensatory movement of the left upper arm, is defined as left upper arm separation F1. The calculation of left upper arm separation F1 will be described below.

[0038] The following calculation is performed for each frame. Figure 8 shows an overview of the calculation of the left upper arm separation for each frame. First, a vector a extending from the neck C2 to the waist C3 and a vector b extending from the left shoulder L1 to the left elbow L2 are generated, and the angle θ between these vectors is calculated.

[0039] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting the above is calculated as the left upper arm separation F1.

[0040] F2: Right Upper Arm Separation The feature quantity indicating how far the right upper arm is separated from the upper trunk when the body is rotated to the right or left, i.e., the compensatory movement of the right upper arm, is defined as right upper arm separation F2. The calculation of right upper arm separation F2 will be described below.

[0041] The following calculation is performed for each frame. Figure 9 shows an overview of the calculation of right upper arm separation for each frame. First, a vector a extending from the neck C2 to the waist C3 and a vector b extending from the right shoulder R1 to the right elbow R2 are generated, and the angle θ between these vectors is calculated.

[0042] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting this is calculated as the right upper arm separation F2.

[0043] F3: Left Lower Arm Flexion The feature quantity indicating the degree of bending of the arm from the left elbow down when the body is rotated to the right or left, i.e., the compensatory movement of the left lower arm, is defined as left lower arm flexion F3. The calculation of left lower arm flexion F3 will be described below.

[0044] The following calculations are performed for each frame. Figure 10 shows an overview of the calculation of left lower arm flexion for each frame. First, a vector a pointing from the left shoulder L1 to the left elbow L2 and a vector b pointing from the left elbow L2 to the left wrist L3 are generated, and the angle θ between these vectors is calculated.

[0045] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting the above is calculated as the left lower arm flexion F3.

[0046] F4: Right Lower Arm Flexion The feature quantity indicating the degree of bending of the arm from the right elbow down when the body is rotated to the right or left, i.e., the compensatory movement of the right lower arm, is defined as right lower arm flexion F4. The calculation of right lower arm flexion F4 will be described below.

[0047] The following calculations are performed for each frame. Figure 11 shows an overview of the calculation of right lower arm flexion for each frame. First, a vector a pointing from the right shoulder R1 to the right elbow R2 and a vector b pointing from the right elbow R2 to the right wrist R3 are generated, and the angle θ between these vectors is calculated.

[0048] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting the above is calculated as the right lower arm flexion F4.

[0049] F5: Acromion horizontal The feature value indicating the inclination of the line connecting the right shoulder R1 and the left shoulder L2 with respect to the upper trunk is defined as the acromion horizontal F5. The calculation of the acromion horizontal F5 will be described below.

[0050] The following calculations are performed for each frame. Figure 12 shows an overview of the calculation of the acromion horizontal for each frame. First, a vector a pointing from the left shoulder L1 to the right shoulder R1 and a vector b pointing from the neck C2 to the lower back C3 are generated, and the angle θ between these vectors is calculated.

[0051] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting this is calculated as the acromion horizontal F5.

[0052] F6: Upper Trunk Anterior / posterior Tilt A feature quantity indicating the forward or backward tilt of the upper trunk is defined as upper trunk anterior / posterior tilt F6. The calculation of upper trunk anterior / posterior tilt F6 will be described below.

[0053] The following calculation is performed for each frame. Figure 13 shows an overview of the calculation of the forward / backward tilt of the upper trunk for each frame. First, a plane S6 perpendicular to vector a extending from the left hip L4 to the right hip R4 is fixed. Then, the angle θ formed by vector B obtained by projecting vector b extending from the neck C2 to the middle waist C3 onto plane S6 and vector G obtained by projecting vertical vector g onto plane S6 is calculated.

[0054] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting the above is calculated as the forward / backward upper trunk tilt F6.

[0055] F7: Pelvic horizontal level A feature value indicating the tilt of the pelvis to the right or left is defined as the pelvic horizontal level F7. The calculation of the pelvic horizontal level F7 will be described below.

[0056] The following calculations are performed for each frame. An outline of the calculation of the pelvic horizontal position for each frame is shown in Figure 14. First, the angle θ formed by the vector a pointing from the left hip L4 to the right hip R4 and the vertical vector g is calculated.

[0057] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting this is calculated as the pelvis horizontal F7.

[0058] F8: Upper Trunk Lateral Flexion A feature quantity indicating the inclination of the upper trunk to the right or left is defined as upper trunk lateral flexion F8. The calculation of upper trunk lateral flexion F8 will be described below.

[0059] Figure 15 shows an outline of the calculation of upper trunk lateral bending in the previous frame. For the previous frame, the angle θ between the vector a pointing from the neck C2 to the middle waist C3 and the vertical vector g is Ｆ Calculate.

[0060] Figure 16 shows an outline of the calculation of upper trunk lateral bending in the rear frame. For the rear frame, vector b from the neck C2 to the middle waist C3 is rotated around vector c from the left waist L4 to the right waist R4 by a negative multiple of the upper trunk forward / backward tilt F6 to generate vector B. Then, the angle θ between vector B and vertical vector g is Ｌ Calculate.

[0061] Then, the angle θ calculated in the next frame Ｌ The angle θ calculated in the previous frame Ｆ The difference Δθ obtained by subtracting this is calculated as the upper trunk lateral flexion F8.

[0062] F9: Amount of Pelvic Rotation A feature quantity indicating the amount of pelvic rotation is referred to as the amount of pelvic rotation F9. The calculation of the amount of pelvic rotation F9 will be described below.

[0063] The following calculations are performed for each frame. Figure 17 shows an overview of the calculation of the amount of pelvic rotation for each frame. First, a plane S9 perpendicular to the vertical vector g is fixed. Next, a vector A is calculated by projecting a vector a directed from the left hip L4 to the right hip R4 onto the plane S9.

[0064] Then, the vector A calculated in the next frame Ｌ and the vector A calculated in the previous frame Ｆ The angle θ formed by these angles is calculated as the amount of pelvic rotation F9.

[0065] In step S12, the training data constructing unit 12 constructs training data DAT by associating the rotation features F0 to F9 calculated by the rotation feature extracting unit 11 with corresponding teacher labels. Here, the rotation features F0 to F9 are also referred to as a rotation feature group F. For example, if the subject OBJ in the video data MOV is leaning his / her body to the left, the state of the subject OBJ is quantitatively represented by the rotation features F0 to F9.

[0066] In contrast to this, by providing information indicating the posture of the subject OBJ as a teacher label, it is possible to generate data elements that make up the training data DAT. The training data DAT is configured to include a plurality of data elements generated in this way.

[0067] In other words, each data element of the training data DAT is expressed as the following vector d ｉ In the following formula, i is an index indicating the video data MOV, and is an integer between 1 and N, inclusive, where N is the number of video data MOVs. In equation [1], the rotation feature group vector F ｉ is a vector whose elements are the rotation feature amounts F0 to F9 calculated from the i-th video data MOV. ｉ is the rotation feature group vector F ｉFrom the above, the learning data DAT is a vector whose elements are the teacher labels assigned to the vector d １ ~d Ｎ It is structured as a dataset including:

[0068] The teacher labels may be generated automatically by applying various analytical methods to the video data MOV, or may be input by the user of the processing system 100 according to the video data MOV.

[0069] A specific example of the teacher labels will now be described. FIG. 18 shows a list of the teacher labels. Here, the teacher label items include the degree of upper trunk rotation, shoulder level, trunk forward / backward tilt, trunk lateral bending, pelvic rotation, and pelvic level. Furthermore, for the center of gravity position, a teacher label can be assigned, for example, indicating that the center of gravity is located at the center of both feet or that the center of gravity is located on one foot. Each label can also be represented by a numerical value, as shown in FIG. 18.

[0070] Step S13: The learning processing unit 13 learns the learning data DAT by supervised learning and constructs a learned model M. The supervised learning method is not limited to a specific method, and various supervised learning methods can be used.

[0071] Next, the estimation phase in the processing device 10 will be described. Fig. 19 schematically shows the configuration of the processing device 10 according to the first embodiment in the estimation phase. Fig. 20 shows the processing in the estimation phase in the processing device 10 according to the first embodiment.

[0072] The processing device 10 in FIG. 19 includes a rotation feature extraction unit 11 and an estimation processing unit 14 .

[0073] In step S14, the rotation feature extraction unit 11 calculates rotation features f0 to f9 to be input to the trained model from the video data MOV_IN of the subject to be estimated. The rotation features f0 to f9 in the estimation phase correspond to the rotation features F0 to F9 in the learning phase, respectively, and are calculated in the same manner as in the learning phase. Note that the rotation features f0 to f9 are also referred to as a rotation feature group f.

[0074] In step S15, the estimation processing unit 14 inputs the rotation feature amounts f0 to f9 in the estimation phase as explanatory variables into the stored trained model, and outputs an objective variable indicating the state of the rotational movement of the body of the subject to be estimated that appears in the video data MOV_IN, i.e., an estimation result OUT. The estimation result may be, for example, information indicating the posture of the subject to be estimated.

[0075] Although the configuration of the processing device 10 relating to the learning phase and the configuration relating to the estimation phase have been described above separately, this is merely an example. Fig. 21 shows an example of the configuration of the processing device 10 according to the first embodiment. As shown in Fig. 21, the processing device 10 may have a configuration relating to the learning phase and a configuration relating to the estimation phase.

[0076] Furthermore, the processing device may be configured to have only the components related to the learning phase and to execute only the processing of the learning phase. Fig. 22 schematically shows an example configuration of a processing device that executes only the processing of the learning phase. In the processing device 10, the learning processing unit 13 may output output data D_M including information such as the constructed trained model and weight parameters that define the trained model.

[0077] Furthermore, the processing device may be configured to have only the components related to the estimation phase and to execute only the processing of the estimation phase. FIG. 23 schematically shows an example configuration of a processing device that executes only the processing of the estimation phase. In the processing device 10, a trained model can be constructed in the estimation processing unit 14 by receiving output data D_M including information such as a trained model constructed by another processing device or parameters that define the trained model. For example, the estimation processing unit 14 can receive output data D_M output from another processing device and construct a trained model based on the output data D_M.

[0078] This configuration enables automatic analysis of the target part of the subject's body, such as the amount of rotation of a joint, captured in image data or video data. It also makes it possible to estimate the validity of the analysis results using a trained model.

[0079] Furthermore, the processing device according to this embodiment can be installed in a terminal, such as a smartphone, that has an imaging device such as a camera and a processing device. This allows even ordinary users, not just experts, to analyze their body rotational movements and know the validity of the analysis results. Using these analysis results, each user can engage in rehabilitation activities through online training or self-training.

[0080] Embodiment 2 In the first embodiment, the analysis of the subject's body rotation movement in one direction from the rotation feature amount was described. However, the human body can perform symmetrical movements on both the left and right sides. However, even if the subject intends to perform symmetrical movements, there are cases where the subject is unable to actually perform the movements symmetrically on both sides due to a malfunction of a joint or the like. Therefore, in this embodiment, a method for evaluating the left-right symmetry of the subject's movements will be described.

[0081] In this case, video data MOV_L when the subject turns to the left and video data MOV_R when the subject turns to the right are captured by the camera 20. These two video data recording symmetrical movements of the same subject are hereinafter referred to as a video pair MP.

[0082] The configuration and operation of the processing device 40 according to the second embodiment will be described below with reference to Fig. 24 to Fig. 26. Fig. 24 schematically shows the configuration of the processing device 40 according to the second embodiment in the learning phase. Fig. 25 shows the processing of the processing device 40 according to the second embodiment in the learning phase. Fig. 26 schematically shows the configuration of the rotation feature extraction unit 11 according to the second embodiment.

[0083] In step S21, the rotation feature extraction unit 11 receives the moving image pair MP and extracts rotation features from each of the moving image data MOV_L and MOV_R included in the moving image pair MP. Here, the rotation features F0 to F9 extracted from the moving image data MOV_L are referred to as a rotation feature group FL, and the rotation features F0 to F9 extracted from the moving image data MOV_R are referred to as a rotation feature group FR.

[0084] Step S211: The data reading unit 1 receives the moving image pair MP.

[0085] In step S212, the feature point extraction unit 2 detects skeleton points from each of the video data MOV_L and MOV_R. Here, the skeleton point group extracted from the video data MOV_L is designated PL, and the skeleton point group extracted from the video data MOV_R is designated PR.

[0086] In step S213, the feature calculation unit 3 calculates rotation feature values ​​F0 to F9, i.e., rotation feature value group FL, from the skeleton point group PL extracted from the video data MOV_L. Also, the feature calculation unit 3 calculates rotation feature values ​​F0 to F9, i.e., rotation feature value group FR, from the skeleton point group PR extracted from the video data MOV_R.

[0087] In step S22, the training data constructing unit 12 constructs training data DAT including a plurality of data elements each consisting of a rotation feature group FL and a rotation feature group FR for one moving image pair MP, and a teacher label L indicating the left-right symmetry of the movement.

[0088] In other words, each data element of the training data is expressed as the following vector e ｊ In the following equation, j is an index indicating a moving image pair, and is an integer between 1 and M, inclusive, where M is the number of moving image pairs. In equation [2], the rotation feature group vector FL ｊ is a vector whose elements are the rotation feature amounts F0 to F9 calculated from the video data MOV_L of the j-th video pair. ｊ is a vector whose elements are the rotation feature amounts F0 to F9 calculated from the video data MOV_R of the j-th video pair. ｊ is the rotation feature group vector FL ｊ and FR ｊ From the above, the learning data DAT is a vector whose elements are the teacher labels assigned to the vector e １ ~e Ｍ It is structured as a dataset including:

[0089] In this embodiment, the learning labels may include labels indicating the symmetry of the upper trunk, the symmetry of the pelvis, and the symmetry of the center of gravity position, as labels indicating the left-right symmetry of the human body movements. FIG. 27 shows a list of the training labels. Here, a label indicating whether each item is "symmetrical in left-right movements" or "asymmetrical in left-right movements" may be assigned. For example, "0" may be assigned to "symmetrical in left-right movements," and "1" may be assigned to "asymmetrical in left-right movements."

[0090] For example, when a subject performs a left rotation movement, twisting the body to the left, it is assumed that the center of gravity may shift to the left foot due to a physical malfunction. In this case, if the center of gravity shifts to the right foot during a right rotation movement, a teacher label for center of gravity position symmetry is assigned as "0," and if not, a teacher label for center of gravity position symmetry is assigned as "1." Furthermore, if the center of gravity is at the center of both feet during a left rotation movement, and if the center of gravity is at the center of both feet during a right rotation movement, a teacher label for center of gravity position symmetry is assigned as "0," and if not, a teacher label for center of gravity position symmetry is assigned as "1."

[0091] Step S23: The learning processing unit 13 learns the learning data DAT through supervised learning, and constructs a learned model M.

[0092] Next, the estimation phase in the processing device 40 will be described. Fig. 28 schematically shows the configuration of the processing device 40 in the estimation phase according to the second embodiment. Fig. 29 shows the processing in the estimation phase in the processing device 40 according to the second embodiment.

[0093] The processing device 40 in FIG. 28 includes a rotation feature extraction unit 11 and an estimation processing unit 14 .

[0094] In step S24, the rotation feature extraction unit 11 calculates rotation feature groups fL and fR to be input to the trained model from the video pair MP_IN of the subject to be estimated. The rotation feature groups fL and fR in the estimation phase correspond to the rotation feature groups FL and FR in the learning phase, respectively, and are calculated in the same way as in the learning phase.

[0095] In step S25, the estimation processing unit 14 inputs the rotation feature sets fL and fR in the estimation phase as explanatory variables into the stored trained model, and outputs an objective variable indicating the left-right symmetry of the motion of the estimated subject captured in the video pair MP_IN, i.e., the estimation result OUT. This makes it possible to determine whether the left and right symmetry of the upper trunk, pelvis, and center of gravity is balanced based on the estimation result OUT.

[0096] Although the processing device 40 has been described with the configuration relating to the learning phase and the configuration relating to the estimation phase separated, this is merely an example. As in the first embodiment, the processing device 40 may have a configuration relating to the learning phase and a configuration relating to the estimation phase.

[0097] The processing device may also be configured to have only the configuration related to the learning phase and to execute only the processing of the learning phase, as in embodiment 1. Furthermore, the processing device may also be configured to have only the configuration related to the estimation phase and to execute only the processing of the estimation phase, as in embodiment 1.

[0098] As described above, this configuration makes it possible not only to analyze the posture of the subject and the unidirectional rotation of the body shown in the video, but also to further analyze the bilateral symmetry of the subject's rotation, thereby enabling a more comprehensive analysis of the body's rotation.

[0099] Third Embodiment A processing system 300 according to a third embodiment will be described. Fig. 30 schematically shows the configuration of the processing system 300 according to the third embodiment. The processing system 300 has a configuration in which a display unit 50 is added to the processing system 100 according to the first embodiment.

[0100] The display unit 50 is configured to display the estimation result OUT of the learning processing unit 13. At this time, the display unit 50 may display various information useful to the user based on the estimation result OUT.

[0101] Fig. 31 shows a first display example of the display unit 50. Fig. 31 shows an example in which a touch panel of a smartphone is used as the display unit 50. In this example, an image or video of the person to be estimated is displayed in the upper part of the screen, and the estimation result of the body rotation movement is displayed in the lower part.

[0102] In this configuration, the processing system is installed in a smartphone, and the smartphone captures video of the person to be estimated, processes the video in the processing device, and displays the necessary information on the touch panel. This makes it possible to know the state of the person's body rotational movement using widely available smartphones without requiring special hardware.

[0103] In the first example, a display example for general users was described, but more specialized displays are also possible. FIG. 32 shows a second display example of the display unit 50. FIG. 32 shows an example in which a display panel of a personal computer or the like is used as the display unit 50. In this example, the upper part of the screen displays, from left to right, a video of the person to be estimated captured from the front, a video of the person to be estimated captured from the side, and the estimation result of the body rotation movement. The lower part of the screen displays frames of the video in chronological order, allowing a frame at any time to be selected using a selection means such as a slider. For operations using the slider and for playing, stopping, and pausing the video, general video operation methods can be applied as appropriate.

[0104] In this configuration, compared to the first example, a user can analyze specific frames of a video in more detail. This allows a person with specialized knowledge, such as an expert, to analyze the state of the rotational movement of the body of a person to be estimated using general hardware such as a personal computer. It goes without saying that the user of this configuration is not limited to experts, and can also be used by non-experts.

[0105] Embodiment 4 In the above-described embodiments, the calculation of the upper trunk forward / backward tilt F6, the pelvic horizontality F7, the upper trunk lateral flexion F8, and the pelvic rotation amount F9 requires the determination of the vertical vector g. The direction of the vertical vector g can be determined in advance, for example, by checking the horizontality when installing the camera 20. However, this method requires manual work, and it is therefore desirable to be able to automatically obtain the vertical vector g. Therefore, in this embodiment, a processing system capable of automatically obtaining the vertical vector g will be described.

[0106] 33 is a schematic diagram showing the configuration of a processing system 400 according to the fourth embodiment. The processing system 400 has a configuration in which an acceleration sensor 60 is added to the processing system 100 according to the first embodiment. In this example, the acceleration sensor 60 is physically fixed to the camera 20. When the video data MOV is captured, the acceleration sensor 60 outputs, to the processing device 10, gravity direction information GV indicating the direction of gravity relative to the posture of the camera 20.

[0107] As a result, the processing device 10 can set an optimal vertical vector g for each piece of video data MOV by using the direction of gravity indicated by the gravity direction information GV as the direction of the vertical vector g.

[0108] Here, the acceleration sensor 60 has been described as being fixed to the camera 20, but the acceleration sensor 60 can be installed in any position and in any manner as long as it can detect the direction of gravity in association with image or video data.

[0109] As described above, this configuration makes it possible to automatically obtain the vertical vector g. For example, when a processing system is installed on a device such as a smartphone that is capable of software processing and has an acceleration sensor and a camera, the vertical vector g can be easily obtained.

[0110] Other Embodiments The present invention is not limited to the above-described embodiments, and modifications can be made as appropriate without departing from the spirit of the present invention. For example, the processing system according to the third embodiment has been described as adding a display unit to the processing system according to the first embodiment, but this is merely an example. A display unit may be added to the processing system according to the second embodiment, and the estimation results may be displayed on the display unit, as in the third embodiment.

[0111] Although the processing system according to the fourth embodiment has been described as adding an acceleration sensor to the processing system according to the first embodiment, this is merely an example. An acceleration sensor may be added to the processing system according to the second or third embodiment, and a vertical vector may be automatically acquired, as in the fourth embodiment.

[0112] Furthermore, it goes without saying that the above-described processing device may be provided with both the display unit according to the third embodiment and the acceleration sensor according to the fourth embodiment.

[0113] In the above embodiment, the feature point extraction unit 2 has been described as using skeleton points as feature points extracted from the subject, but this is merely an example, and other feature points may be used, or multiple types of feature points extracted using different extraction methods may be used together. For example, the silhouette of the subject may be detected in each frame of the video, and points on the outline of the silhouette may be extracted as feature points. Furthermore, skeleton points and points on the outline of the silhouette may be used together as feature points.

[0114] The processes performed by the measuring device according to the above-described embodiments may be realized by causing a computer to execute a program. Specifically, one or more programs including instructions for causing a computer system to execute an algorithm related to the transmission signal processing or the reception signal processing may be created, and the programs may be supplied to the computer.

[0115] These programs can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-RWs, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The programs may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the programs to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0116] FIG. 34 schematically shows the configuration of a computer 1000, which is an example of a hardware configuration for realizing a processing device or a processing system. The computer 1000 can be configured as various types of computers, such as a dedicated computer or a personal computer (PC). However, the computer does not need to be physically a single computer; multiple computers may be used when performing distributed processing. As shown in FIG. 34, the computer 1000 has a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003, which are interconnected via a bus 1004. Note that although explanation of the OS software and other software required to operate the computer is omitted, it is assumed that the computer also has these software.

[0117] An input / output interface 1005 is connected to the bus 1004. To the input / output interface 1005, an input unit 1006, an output unit 1007, a communication unit 1008, and a storage unit 1009 are connected.

[0118] The input unit 1006 is composed of, for example, a keyboard, a mouse, a sensor, etc. The output unit 1007 is composed of, for example, a display device such as an LCD, and an audio output device such as headphones and speakers, etc. The communication unit 1008 is composed of, for example, a router, a terminal adapter, etc. The storage unit 1009 is composed of a storage device such as a hard disk, a flash memory, etc.

[0119] The CPU 1001 can perform various processes in accordance with various programs stored in the ROM 1002 or various programs loaded from the storage unit 1009 to the RAM 1003. In this embodiment, the CPU 1001 executes processes performed by, for example, a measurement device. A graphics processing unit (GPU) may be provided separately from the CPU 1001, and, like the CPU 1001, executes various processes (in this embodiment, processes performed by, for example, a measurement device) in accordance with various programs stored in the ROM 1002 or various programs loaded from the storage unit 1009 to the RAM 1003. Note that the GPU is suitable for performing routine processes in parallel, and by applying it to neural network processing, etc., described below, it is possible to improve processing speed compared to the CPU 1001. The RAM 1003 also stores data necessary for the CPU 1001 and the GPU to execute various processes, as appropriate.

[0120] The communication unit 1008 is capable of two-way communication with the server 1030 via the network 1020. The communication unit 1008 can transmit data provided by the CPU 1001 to the server 1030, and output data received from the server 1030 to the CPU 1001, RAM 1003, storage unit 1009, etc. The communication unit 1008 may communicate with other devices using analog signals or digital signals. The storage unit 1009 is capable of exchanging data with the CPU 1001, and stores and erases information.

[0121] A drive 1010 may be connected to the input / output interface 1005 as needed. Storage media such as a magnetic disk 1011, an optical disk 1012, a flexible disk 1013, or a semiconductor memory 1014 may be appropriately mounted in the drive 1010. Computer programs read from each storage medium may be installed in the storage unit 1009 as needed. Furthermore, data required for the CPU 1001 to execute various processes, data obtained as a result of the processes of the CPU 1001, and the like may be stored in each storage medium as needed.

[0122] In the above embodiment, the image data and video data are described as being acquired by a camera, but this is merely an example. The image data and video data can be acquired by any of a variety of imaging devices.

[0123] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0124] (Supplementary Note 1) A processing device comprising: a rotation feature extraction unit that extracts feature points of a subject to be trained from two frames before and after a body rotation movement selected from video image data capturing the body rotation movement of the subject to be trained; and calculates rotation feature amounts that indicate the body rotation movement of the subject to be trained based on the difference in positions of the feature points extracted from the two frames before and after the body rotation movement; a training data construction unit that constructs training data by assigning teacher labels that indicate a state of the subject to be trained that correspond to the rotation feature amounts; and a learning processing unit that constructs a trained model by learning the training data.

[0125] (Supplementary Note 2) The processing device according to Supplementary Note 1, further comprising an estimation processing unit that estimates a state of a rotational movement of the body of a subject to be estimated by inputting input data into the trained model, wherein the rotation feature extraction unit extracts feature points of the subject to be estimated from two frames before and after the body rotational movement selected from video image data capturing the body rotational movement of the subject to be estimated, calculates a rotation feature that indicates the body rotational movement of the subject to be estimated based on a difference between the positions of the feature points extracted from the two frames before and after the body rotational movement, and outputs the calculated rotation feature to the estimation processing unit as the input data.

[0126] (Supplementary Note 3) The processing device described in Supplementary Note 1, wherein the rotation feature extraction unit selects two frames relating to symmetrical body rotation movements performed by the subject to be trained from video image data of the subject to be trained, calculates rotation features for the two selected frames, and the training data construction unit constructs the training data by assigning teacher labels indicating the state of the symmetrical movement of the subject to be trained corresponding to the rotation features.

[0127] (Supplementary Note 4) The processing device according to Supplementary Note 2, wherein the rotation feature extraction unit selects two frames of symmetrical body rotation movements performed by the subject to be learned from video image data of the subject to be learned and calculates rotation features for the two selected frames, the training data construction unit constructs the training data by assigning teacher labels indicating a state of the symmetrical movement of the subject to be learned that corresponds to the rotation feature, the rotation feature extraction unit selects two frames of symmetrical body rotation movements performed by the subject to be estimated from video image data of the subject to be estimated and calculates rotation features for the two selected frames, and the estimation processing unit estimates a state of the symmetrical movement of the subject to be estimated that corresponds to the rotation feature.

[0128] (Supplementary Note 5) A rotation feature amount extraction unit extracts feature points of a subject to be estimated from two frames before and after a rotation movement of the body selected from video image data of the rotation movement of the body of the subject to be estimated, and calculates a rotation feature amount indicating the rotation movement of the body of the subject to be estimated based on a difference in the positions of the feature points extracted from the two frames before and after the rotation movement of the body; and a rotation feature amount extraction unit extracts feature points of a subject to be estimated from two frames before and after the rotation movement of the body selected from video image data of the rotation movement of the body of the subject to be estimated, and calculates a rotation feature amount indicating the rotation movement of the body of the subject to be estimated based on a difference in the positions of the feature points extracted from the two frames before and after the rotation movement of the body. and an estimation processing unit that extracts feature points of the subject, and inputs the rotation feature amounts of the subject to be trained calculated by the rotation feature extraction unit into a trained model constructed by learning a correspondence between rotation feature amounts indicating the rotational movement of the body of the subject to be trained, the rotation feature amounts being calculated based on a difference between positions of the feature points extracted from the two frames before and after the rotational movement of the body, and states of the subject to be trained corresponding to the rotation feature amounts.

[0129] (Supplementary Note 6) The processing device described in Supplementary Note 5, wherein the trained model is constructed by selecting, from video image data of a subject to be trained, two frames relating to symmetrical body rotational movements performed by the subject to be trained, calculating rotational features for the two selected frames, constructing training data by assigning teacher labels indicating a state of the symmetrical movement of the subject to be trained corresponding to the rotational features, and training the training data, wherein the rotational feature extraction unit selects, from video image data of the subject to be estimated, two frames relating to symmetrical body rotational movements performed by the subject to be estimated, and calculates rotational features for the two selected frames, and the estimation processing unit estimates a state of the symmetrical movement of the subject to be estimated corresponding to the rotational features.

[0130] (Appendix 7) A processing device described in any one of Appendices 2, 4 to 6, further comprising a display unit that displays the judgment result of the estimation processing unit on the state of the rotational movement of the body of the subject to be estimated and a moving image of the subject to be estimated.

[0131] (Supplementary Note 8) The processing device described in Supplementary Note 7, wherein the display unit displays two or more moving images of the subject to be estimated taken from different positions.

[0132] (Supplementary Note 9) The processing device described in Supplementary Note 8, wherein the display unit displays a selection means for selecting any frame of a video of the subject to be estimated.

[0133] (Supplementary Note 10) The processing device according to any one of Supplementary Notes 1 to 9, wherein the rotation feature extraction unit extracts skeletal points of the subject as the feature points.

[0134] (Supplementary Note 11) The processing device described in Supplementary Note 10, wherein the rotation feature extraction unit extracts the subject's neck, waist, left and right shoulders, left and right elbows, left and right wrists, and left and right waists as the skeletal points.

[0135] (Supplementary Note 12) The processing device described in Supplementary Note 11, wherein the rotation feature extraction unit determines a plane perpendicular to a line connecting the neck and the mid-waist, calculates an angle between a line connecting the left shoulder and the right shoulder projected onto the plane and a line connecting the left hip and the right hip projected onto the plane, for the two frames, and calculates the difference between the two calculated angles as a first rotation feature.

[0136] (Supplementary Note 13) The processing device described in Supplementary Note 11 or 12, wherein the rotation feature extraction unit calculates the angle between a line connecting the neck and mid-waist and a line connecting the left shoulder and left elbow for the two frames, and calculates the difference between the two calculated angles as the second rotation feature.

[0137] (Supplementary Note 14) The processing device described in any one of Supplementary Notes 11 to 13, wherein the rotation feature extraction unit calculates the angle between a line connecting the neck and mid-waist and a line connecting the right shoulder and right elbow for the two frames, and calculates the difference between the two calculated angles as a third rotation feature.

[0138] (Supplementary Note 15) The processing device described in any one of Supplementary Notes 11 to 14, wherein the rotation feature extraction unit calculates the angle between a line connecting the left shoulder and left elbow and a line connecting the left elbow and left wrist for the two frames, and calculates the difference between the two calculated angles as a fourth rotation feature.

[0139] (Supplementary Note 16) The processing device described in any one of Supplementary Notes 11 to 15, wherein the rotation feature extraction unit calculates the angle between a line connecting the right shoulder and the right elbow and a line connecting the right elbow and the right wrist for the two frames, and calculates the difference between the two calculated angles as a fifth rotation feature.

[0140] (Supplementary Note 17) The processing device described in any one of Supplementary Notes 11 to 16, wherein the rotation feature extraction unit calculates the angle between a line connecting the right shoulder and the left shoulder and a line connecting the neck and the middle waist for the two frames, and calculates the difference between the two calculated angles as a sixth rotation feature.

[0141] (Supplementary Note 18) The processing device described in any one of Supplementary Notes 11 to 17, wherein the rotation feature extraction unit determines a plane perpendicular to a line connecting the left hip and the right hip, calculates an angle between a line connecting the neck and the middle waist projected onto the plane and a line projecting the vertical direction onto the plane, for the two frames, and calculates the difference between the two calculated angles as a seventh rotation feature.

[0142] (Supplementary Note 19) The processing device described in Supplementary Note 18, which calculates, for one of the two frames, a first angle formed by a line connecting the neck and the mid-waist and the vertical direction, and for the other of the two frames, obtains a line by rotating the line connecting the neck and the mid-waist around a line connecting the left hip and the right hip by the seventh rotation feature in a direction opposite to the seventh rotation feature, calculates a second angle formed by the obtained line and the vertical direction, and calculates the difference between the first angle and the second angle as a ninth rotation feature.

[0143] (Supplementary Note 20) The processing device described in any one of Supplementary Notes 11 to 19, wherein the rotation feature extraction unit calculates the angle between a line connecting the left hip and the right hip and the vertical direction for the two frames, and calculates the difference between the two calculated angles as an eighth rotation feature.

[0144] (Supplementary Note 21) The processing device described in any one of Supplementary Notes 11 to 20, wherein the rotation feature extraction unit determines a plane perpendicular to the vertical direction, calculates a line connecting the left hip and the right hip projected onto the plane for the two frames, and calculates the angle between the two calculated lines as a tenth rotation feature.

[0145] (Supplementary Note 22) A processing device described in any one of Supplementary Notes 18 to 21, wherein the vertical vector is determined based on the direction of gravity acquired by an acceleration sensor that detects the posture of an imaging device that captures the moving image data.

[0146] (Supplementary Note 23) A processing system comprising: an imaging device that acquires video data capturing a rotational movement of a body of a subject to be trained; and a processing device that learns rotational features that indicate the rotational movement of the body of the subject to be trained from the video data of the subject to be trained, and constructs a trained model that estimates a state of the subject to be trained, wherein the processing device comprises: a rotational feature extraction unit that extracts feature points of the subject to be trained from two frames, one before and one after the body rotational movement, selected from the video data capturing the body rotational movement of the subject to be trained, and calculates rotational features that indicate the body rotational movement of the subject to be trained based on differences in positions of the feature points extracted from the two frames before and after the body rotational movement; a training data construction unit that constructs training data by assigning teacher labels that indicate the state of the subject to be trained that correspond to the rotational features; and a learning processing unit that constructs a trained model by learning the training data.

[0147] (Appendix 24) A method for constructing a trained model, comprising: extracting feature points of a training subject from two frames, one before and one after the body rotation movement, selected from video image data capturing the body rotation movement of the training subject; calculating rotation feature amounts indicating the body rotation movement of the training subject based on the difference between the positions of the feature points extracted from the two frames before and after the body rotation movement; constructing training data by assigning teacher labels indicating the state of the training subject corresponding to the rotation feature amounts; and constructing a trained model by learning the training data.

[0148] (Supplementary Note 25) A program that causes a computer to execute the following processes: extracting feature points of a subject to be trained from two frames before and after a rotational movement of the body selected from video image data capturing the rotational movement of the body of the subject to be trained; calculating rotational feature amounts that indicate the rotational movement of the body of the subject to be trained based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; constructing training data by assigning teacher labels that indicate the state of the subject to be trained that correspond to the rotational feature amounts; and constructing a trained model by learning the training data.

[0149] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0150] This application claims priority based on Japanese Patent Application No. 2022-188607, filed November 25, 2022, the disclosure of which is incorporated herein in its entirety.

[0151] REFERENCE SIGNS LIST 1 Data reading unit 2 Feature point extraction unit 3 Feature calculation unit 11 Rotation feature extraction unit 12 Learning data construction unit 13 Learning processing unit 14 Estimation processing unit 10, 40 Processing device 20 Camera 30 Video database 50 Display unit 60 Acceleration sensor 100, 101, 300, 400 Processing system 1000 Computer 1001 CPU 1002 ROM 1003 RAM 1004 Bus 1005 Input / output interface 1006 Input unit 1007 Output unit 1008 Communication unit 1009 Storage unit 1010 Drive 1011 Magnetic disk 1012 Optical disk 1013 Flexible disk 1014 Semiconductor memory 1020 Network 1030 Server C1 Nose C2 Neck C3 Waist D_M Output data DAT Learning data F0 Acromial rotation amount F1 Left upper arm separation F2 Right upper arm separation F3 Left lower arm flexion F4 Right lower arm flexion F5 Acromion horizontal F6 Upper trunk anteroposterior tilt F7 Pelvic horizontal F8 Upper trunk lateral flexion F9 Pelvic rotation amount L1 Left shoulder L2 Left elbow L3 Left wrist L4 Left hip L5 Left knee L6 Left ankle MOV, MOV_L, MOV_R Video data MP Video pair OBJ Subject OUT Estimation result P, PL, PR Skeletal point group R1 Right shoulder R2 Right elbow R3 Right wrist R4 Right hip R5 Right knee R6 Right ankle

Claims

1. A rotation feature amount extraction unit that extracts feature points of a subject to be learned from two frames before and after the rotational movement of the body of the subject to be learned selected from moving image data obtained by imaging the rotational movement of the body of the subject to be learned, and calculates a rotation feature amount indicating the rotational movement of the body of the subject to be learned based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; A learning data construction unit that constructs learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotation feature amount; A learning processing unit that constructs a learned model by learning the learning data, and A processing device.

2. Further comprising an estimation processing unit that estimates the state of the rotational movement of the body of the subject to be estimated by inputting input data into the learned model, The rotation feature amount extraction unit extracts feature points of the subject to be estimated from two frames before and after the rotational movement of the body of the subject to be estimated selected from moving image data obtained by imaging the rotational movement of the body of the subject to be estimated, and calculates a rotation feature amount indicating the rotational movement of the body of the subject to be estimated based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, and outputs the calculated rotation feature amount as the input data to the estimation processing unit. The processing device according to claim 1.

3. The rotation feature amount extraction unit selects two frames related to the symmetric rotational movement of the body performed by the subject to be learned from the moving image data of the subject to be learned, and calculates a rotation feature amount for the selected two frames, The learning data construction unit constructs the learning data by assigning a teacher label indicating the state of the symmetric movement of the subject to be learned corresponding to the rotation feature amount. The processing device according to claim 1.

4. The rotation feature amount extraction unit selects two frames related to the symmetric rotational movement of the body performed by the subject to be learned from the moving image data of the subject to be learned, and calculates a rotation feature amount for the selected two frames, The learning data construction unit constructs the learning data by assigning a teacher label indicating the state of the symmetric movement of the subject to be learned corresponding to the rotation feature amount. The rotational feature amount extraction unit selects two frames related to the rotational motion of the symmetric body performed by the subject to be estimated from the moving image data of the subject to be estimated, calculates the rotational feature amount for the selected two frames, The estimation processing unit estimates the state of the symmetric motion of the subject to be estimated corresponding to the rotational feature amount. The processing device according to claim 2.

5. Extract the feature points of the subject to be estimated from two frames before and after the rotational motion of the body selected from the moving image data of the rotational motion of the body of the subject to be estimated, and based on the difference in the positions of the feature points extracted from the two frames before and after the rotational motion of the body, a rotational feature amount extraction unit that calculates a rotational feature amount indicating the rotational motion of the body of the subject to be estimated, Extract the feature points of the subject to be learned from two frames before and after the rotational motion of the body selected from the moving image data of the rotational motion of the body of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the rotational motion of the body, a rotational feature amount indicating the rotational motion of the body of the subject to be learned, and a learned model constructed by learning the association between the state of the subject to be learned corresponding to the rotational feature amount, and inputting the rotational feature amount of the subject to be learned calculated by the rotational feature amount extraction unit, and an estimation processing unit that estimates the state of the rotational motion of the body of the subject to be estimated. Processing device.

6. The learned model is Select two frames related to the rotational motion of the symmetric body performed by the subject to be learned from the moving image data of the subject to be learned, and calculate the rotational feature amount for the selected two frames. Construct learning data by assigning a teacher label indicating the state of the symmetric motion of the subject to be learned corresponding to the rotational feature amount. Constructed by learning the learning data. The rotational feature amount extraction unit selects two frames related to the rotational motion of the symmetric body performed by the subject to be estimated from the moving image data of the subject to be estimated, and calculates the rotational feature amount for the selected two frames. The estimation processing unit estimates the state of the symmetric motion of the subject to be estimated corresponding to the rotational feature amount. The processing device according to claim 5.

7. Further comprising a display unit that displays the estimation result of the state of the rotational movement of the body of the subject to be estimated by the estimation processing unit and the moving image of the subject to be estimated. The processing device according to claim 2 or 4.

8. An imaging device that acquires moving image data obtained by imaging the rotational movement of the body of a subject to be learned, A processing device that constructs a learned model that learns a rotational feature amount indicating the rotational movement of the body of the subject to be learned from the moving image data of the subject to be learned and estimates the state of the subject to be learned, The processing device is A rotational feature amount extraction unit that extracts feature points of the subject to be learned from two frames before and after the rotational movement of the body selected from the moving image data obtained by imaging the rotational movement of the body of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, calculates a rotational feature amount indicating the rotational movement of the body of the subject to be learned, A learning data construction unit that constructs learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotational feature amount, A learning processing unit that constructs a learned model by learning the learning data, Processing system.

9. Extract the feature points of the subject to be learned from two frames before and after the rotational movement of the body selected from the moving image data obtained by imaging the rotational movement of the body of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, calculate a rotational feature amount indicating the rotational movement of the body of the subject to be learned, Construct learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotational feature amount, Construct a learned model by learning the learning data, Method for constructing a learned model.

10. A process of extracting feature points of the subject to be learned from two frames before and after the rotational movement selected from the moving image data obtained by imaging the rotational movement of the body of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, calculating a rotational feature amount indicating the rotational movement of the body of the subject to be learned, A process of constructing learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotational feature amount, A process of constructing a learned model by learning the learning data, and causing a computer to execute the process. Program.