Processing device, processing system, method for building a trained model, and program

Abstract

The purpose of the present invention is to automatically analyze a turning operation state of the human body. A turning feature amount extraction unit extracts feature points to be trained of a subject from two frames selected from moving image data to be trained of the subject, and calculates, on the basis of the difference between turning operations of the body, turning feature amounts that indicate the turning operations to be trained of the body of the subject, the difference being calculated from the feature points extracted from the two frames. A training data construction unit imparts, to the turning feature amount F, a teaching level indicating a corresponding state to be trained of the subject and constructs training data. A training processing unit constructs a trained model by performing training with the training data.

Description

[Technical Field]

[0001] This disclosure relates to a processing device, a processing system, a method for building a trained model, and a program. [Background technology]

[0002] In the medical field, it is common practice to analyze the state of human movement and plan treatment and rehabilitation based on the analysis results. While such analyses have traditionally been carried out with the involvement of specialists such as doctors and physical therapists, in recent years, there has been progress in the development of methods for analyzing the state of human movement.

[0003] As a method for analyzing human body movements, a processing device has been proposed that calculates the range of motion of a subject's movements (Patent Document 1). This processing device takes images of the subject, selects a movement to be measured from the images, and measures the range of motion of the selected movement.

[0004] In addition, various methods for analyzing human body movements have been proposed (Patent Documents 2-5). [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] International Publication No. 2020 / 021873 [Patent Document 2] Japanese Patent Publication No. 2022-131397 [Patent Document 3] Japanese Patent Publication No. 2022-65241 [Patent Document 4] Japanese Patent Publication No. 2018-94084 [Patent Document 5] Japanese Patent Publication No. 2015-61579 [Overview of the project] [Problems that the invention aims to solve]

[0006] While the aforementioned analysis methods can measure the range of motion, analyzing rotational movements, one of the main movements of the human body, is more difficult compared to straight-line movements, and the development of automated recognition technology for this has not progressed. Therefore, the involvement of specialists such as doctors and physical therapists is still required to analyze and improve physical dysfunctions related to rotational movements. Consequently, there is a growing need for posture recognition technology that can automatically analyze rotational movements as well.

[0007] This disclosure is made in view of the above circumstances and aims to automatically analyze the state of rotational movement of the human body. [Means for solving the problem]

[0008] A processing device according to one aspect of the present disclosure includes: a rotation feature extraction unit that extracts feature points of the subject to be trained from two frames selected from video image data of the rotational movement of the subject's body, selected from the rotational movement of the subject's body, and calculates a rotation feature quantity indicating the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; a learning data construction unit that constructs learning data by assigning teacher labels indicating the state of the subject to be trained corresponding to the rotation feature quantity; and a learning processing unit that constructs a trained model by learning the learning data.

[0009] A processing device according to one aspect of the present disclosure extracts feature points of a subject to be estimated from two frames before and after the body rotation motion of the subject to be estimated, selected from moving image data obtained by imaging the body rotation motion of the subject to be estimated, and based on the difference in the positions of the feature points extracted from the two frames before and after the body rotation motion, a rotation feature quantity extraction unit that calculates a rotation feature quantity indicating the body rotation motion of the subject to be estimated, extracts feature points of the subject to be learned from two frames before and after the body rotation motion of the subject to be learned, selected from moving image data obtained by imaging the body rotation motion of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the body rotation motion, a rotation feature quantity calculated indicating the body rotation motion of the subject to be learned, and a learned model constructed by learning the association between the rotation feature quantity and the state of the subject to be learned, inputs the rotation feature quantity of the subject to be learned calculated by the rotation feature quantity extraction unit, and an estimation processing unit that estimates the state of the body rotation motion of the subject to be estimated.

[0010] A processing system according to one aspect of the present disclosure includes an imaging device that acquires moving image data obtained by imaging the body rotation motion of a subject to be learned, and a processing device that constructs a learned model that learns a rotation feature quantity indicating the body rotation motion of the subject to be learned from the moving image data of the subject to be learned and estimates the state of the subject to be learned. The processing device extracts feature points of the subject to be learned from two frames before and after the body rotation motion of the subject to be learned, selected from moving image data obtained by imaging the body rotation motion of the subject to be learned, and based on the difference in the positions of the feature points extracted from the two frames before and after the body rotation motion, a rotation feature quantity extraction unit that calculates a rotation feature quantity indicating the body rotation motion of the subject to be learned, a learning data construction unit that constructs learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotation feature quantity, and a learning processing unit that constructs a learned model by learning the learning data.

[0011] A method for constructing a learned model according to one aspect of the present disclosure extracts feature points of a subject to be learned from two frames before and after the body rotation motion of the subject to be learned selected from moving image data obtained by imaging the body rotation motion of the subject to be learned, and calculates a rotation feature amount indicating the body rotation motion of the subject to be learned based on the difference in the positions of the feature points extracted from the two frames before and after the body rotation motion, constructs learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotation feature amount, and constructs a learned model by learning the learning data.

[0012] A program according to one aspect of the present disclosure causes a computer to execute a process of extracting feature points of a subject to be learned from two frames before and after the rotation motion selected from moving image data obtained by imaging the body rotation motion of the subject to be learned, a process of calculating a rotation feature amount indicating the body rotation motion of the subject to be learned based on the difference in the positions of the feature points extracted from the two frames before and after the body rotation motion, a process of constructing learning data by assigning a teacher label indicating the state of the subject to be learned corresponding to the rotation feature amount, and a process of constructing a learned model by learning the learning data.

Effect of the Invention

[0013] According to the present disclosure, the state of the body rotation motion of a human can be automatically analyzed.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram schematically showing the configuration of a processing system according to Embodiment 1. [Figure 2] It is a diagram showing a modification example of the processing system according to Embodiment 1. [Figure 3] It is a diagram schematically showing the configuration in the learning phase of a processing device according to Embodiment 1. [Figure 4] It is a flowchart showing the processing in the learning phase of a processing device according to Embodiment 1. [Figure 5]This figure schematically shows the configuration of the rotation feature extraction unit according to Embodiment 1. [Figure 6] This diagram schematically shows the skeletal points extracted by the feature point extraction unit. [Figure 7] This figure shows an overview of how the acromion rotation amount is calculated in each frame. [Figure 8] This figure shows an overview of the calculation of left upper arm separation in each frame. [Figure 9] This figure shows an overview of the calculation of the right upper arm separation in each frame. [Figure 10] This figure shows an overview of the calculation of left forearm flexion in each frame. [Figure 11] This figure shows an overview of the calculation of right forearm flexion in each frame. [Figure 12] This diagram shows an overview of the calculation of acromion horizontality in each frame. [Figure 13] This figure shows an overview of the calculation of upper trunk anterior-posterior tilt in each frame. [Figure 14] This diagram shows an overview of the calculation of pelvic horizontality in each frame. [Figure 15] This figure shows an overview of the calculation of upper trunk lateral flexion in the anterior frame. [Figure 16] This figure shows an overview of the calculation of upper trunk lateral flexion in the posterior frame. [Figure 17] This figure shows an overview of how pelvic rotation is calculated in each frame. [Figure 18] This is a diagram showing a list of teacher labels. [Figure 19] This diagram schematically shows the configuration of the processing device in the estimation phase according to Embodiment 1. [Figure 20] This is a flowchart showing the processing of the estimation phase of the processing device according to Embodiment 1. [Figure 21] This figure shows an example of the configuration of the processing device 10 according to Embodiment 1. [Figure 22] This diagram schematically shows an example configuration of a processing unit that performs only the learning phase processing. [Figure 23]This diagram schematically shows an example configuration of a processing unit that performs only the estimation phase. [Figure 24] This diagram schematically shows the configuration of the processing device in the learning phase according to Embodiment 2. [Figure 25] This is a flowchart of the learning phase processing of the processing device according to Embodiment 2. [Figure 26] This figure schematically shows the configuration of the rotation feature extraction unit according to Embodiment 2. [Figure 27] This is a diagram showing a list of teacher labels. [Figure 28] This diagram schematically shows the configuration of the processing device in the estimation phase according to Embodiment 2. [Figure 29] This is a flowchart showing the processing of the estimation phase of the processing apparatus according to Embodiment 2. [Figure 30] This diagram schematically shows the configuration of the processing system according to Embodiment 3. [Figure 31] This is a diagram showing the first display example of the display unit. [Figure 32] This figure shows a second example of the display unit. [Figure 33] This diagram schematically shows the configuration of the processing system according to Embodiment 4. [Figure 34] This diagram schematically shows the configuration of a computer, which is an example of a hardware configuration for realizing a processing unit or processing system. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will now be described with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted where necessary.

[0016] Embodiment 1 The processing system 100 according to Embodiment 1 will now be described. The processing system 100 is configured to estimate the motion state of the body part to be analyzed when the subject's OBJ performs a twisting motion to the right or left, that is, a rotational motion in which the body rotates to the right or left, based on images or videos of the subject's OBJ.

[0017] In this context, body rotation refers to the movement of rotating the body to the right or left while keeping the position and orientation of both feet fixed. This movement involves the coordinated movement of various parts of the body, including the arms, shoulders, neck, waist, and legs.

[0018] Figure 1 schematically shows the configuration of the processing system 100 according to Embodiment 1. The processing system 100 includes a processing unit 10 and a camera 20.

[0019] Camera 20 captures images or videos of the subject's oblique body (OBJ) and outputs the captured image or video data to the processing unit 10. In the following description, camera 20 will be described as outputting video data (MOV) to the processing unit 10. In the following description, video data and image data will be collectively referred to as moving image data.

[0020] Note that in Figure 1, the video data MOV is output from the camera 20 to the processing unit 10, but this is merely an example. For example, the video data MOV may be stored in another storage device, and the processing unit 10 may read the video data MOV from the storage device as needed.

[0021] Figure 2 shows a modified version of the processing system according to Embodiment 1. The processing system 101 in Figure 2 has a video database 30 added compared to the processing system 100 in Figure 1. The video database 30 is configured as various storage devices, or is configured to be storable by various storage devices. The video database 30 appropriately stores MOV video data captured by the camera 20. The processing device 10 reads the MOV video data from the video database 30 as needed.

[0022] The processing unit 10 is configured to estimate feature quantities that indicate the motion state of the target body part when the captured subject's oblique body (OBJ) rotates to the right or left, based on the received video data MOV.

[0023] The processing device according to Embodiment 1 will now be described. The processing device 10 learns the correspondence between rotational features extracted from a video of the subject's object-by-joint (OBJ) to be trained and the state of the subject's OBJ to be trained, constructs a trained model, and inputs the rotational features calculated for the subject to be estimated into the trained model to estimate the motion state of the body part to be analyzed when the subject rotates their body to the right or left.

[0024] The configuration and operation of the processing unit 10 will be described below, divided into a learning phase and an estimation phase. Figure 3 schematically shows the configuration of the processing unit 10 in the learning phase according to Embodiment 1. Figure 4 shows the processing of the processing unit 10 in the learning phase according to Embodiment 1.

[0025] The processing unit 10 includes a rotation feature extraction unit 11, a training data construction unit 12, and a training processing unit 13.

[0026] Step S11 The rotational feature extraction unit 11 receives video data MOV and extracts rotational features of the subject OBJ to be trained from the video data MOV. Figure 5 schematically shows the configuration of the rotational feature extraction unit 11 according to Embodiment 1. The rotational feature extraction unit 11 includes a data reading unit 1, a feature point extraction unit 2, and a feature calculation unit 3.

[0027] Step S111 The data loading unit 1 receives video data (MOV) of the subject's object john (OBJ) to be studied, for example, from a camera.

[0028] Step S112 The feature point extraction unit 2 detects feature points for detecting the movement of the subject object OBJ to be trained, based on the video data MOV received by the data loading unit 1. Here, the feature points to be detected are the skeletal points of the subject object OBJ to be trained. The method for detecting skeletal points is not limited to a specific method, and various methods can be applied.

[0029] Figure 6 schematically shows the skeletal points extracted by the feature point extraction unit 2. Figure 6 is a front view of the subject's oblique joint (OBJ) being studied, with the x-direction being the direction from the posterior to the front of the subject's OBJ. However, in Figure 6, the x-direction is tilted for easier viewing. The y-direction is the direction from right to left of the subject's OBJ, i.e., from left to right in the drawing, and the z-direction is the direction from bottom to top.

[0030] The feature point extraction unit 2 extracts 15 skeletal points from the subject's OBJ. As shown in Figure 6, the nose C1, neck C2, and lower back C3 are extracted as feature points from top to bottom along the midline of the subject's OBJ. For the right half of the body, the right shoulder R1, right elbow R2, and right wrist R3 are extracted from top to bottom on the right arm, and the right hip R4, right knee R5, and right ankle R6 are extracted from top to bottom on the right hip and right lower limb. For the left half of the body, symmetrically to the right half, the left shoulder L1, left elbow L2, and left wrist L3 are extracted from top to bottom on the left arm, and the left hip L4, left knee L5, and left ankle L6 are extracted from top to bottom on the left hip and left lower limb. Hereafter, the feature points extracted by the feature point extraction unit 2 will also be referred to as the feature point group P.

[0031] Step S113 The feature calculation unit 3 extracts features that indicate the motion state of the target part of the subject's OBJ (oblique jejunum) before and after a rotational movement of the body, based on the skeletal points estimated by the feature point extraction unit 2. The rotational features extracted by the feature calculation unit 3 are described below. In this embodiment, the feature calculation unit 3 calculates the following 10 types of rotational features.

[0032] In calculating rotational features, we will select two frames that are temporally separated from the video image and calculate the rotational features of the target region of the subject's oblique junction (OBJ) between these two frames. Hereafter, the frame that is earlier in time will be referred to as the "previous frame," and the frame that is later in time will be referred to as the "later frame."

[0033] In this embodiment, the rotational feature quantity can be suitably calculated by defining the frame in which the subject is standing facing forward as the front frame and the frame in which the subject is rotating their body to the right or left as the back frame. Hereinafter, the rotational feature quantity is expressed as the displacement of the position of feature points, i.e., skeletal points, as the angular displacement of the vector connecting the feature points between the two frames.

[0034] F0: Acromial rotation amount In the upper trunk, the acromion rotation F0 is defined as a feature quantity that indicates the amount of rotation of the subject's oblique joint (OBJ) relative to the midline of the line connecting the left shoulder L1 and the right shoulder R1. The calculation of the acromion rotation F0 is explained below.

[0035] For each frame, the following calculations are performed. Figure 7 shows an overview of the calculation of acromion rotation in each frame. First, a plane S0 is fixed that is perpendicular to the vector a that goes from the neck C2 to the middle of the waist C3. Next, the angle θ between vector B, which is obtained by projecting the vector b that goes from the left shoulder L1 to the right shoulder R1 onto plane S0, and vector C, which is obtained by projecting the vector c that goes from the left waist L4 to the right waist R4 onto plane S0, is calculated. In the following, the angle calculated from the later frame is used as θ. L θ is the angle calculated from the previous frame. F Let's assume that.

[0036] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the acromion rotation amount F0.

[0037] F1: Left upper arm separation When the body is rotated to the right or left, the distance the left upper arm moves from the upper trunk is defined as the left upper arm separation F1, which is a feature quantity indicating compensatory movement by the left upper arm. The calculation of left upper arm separation F1 is explained below.

[0038] For each frame, the following calculations are performed. Figure 8 shows an overview of the calculation of the left upper arm separation in each frame. First, a vector a is generated from the neck C2 to the middle of the waist C3, and a vector b is generated from the left shoulder L1 to the left elbow L2, and the angle θ between these vectors is calculated.

[0039] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the left upper arm separation F1.

[0040] F2: Right upper arm separation When the body is rotated to the right or left, the distance the right upper arm moves from the upper torso is defined as the right upper arm separation F2, which is a feature quantity indicating compensatory movement by the right upper arm. The calculation of the right upper arm separation F2 is explained below.

[0041] For each frame, the following calculations are performed. Figure 9 shows an overview of the calculation of the right upper arm separation in each frame. First, a vector a is generated from the neck C2 to the middle of the waist C3, and a vector b is generated from the right shoulder R1 to the right elbow R2, and the angle θ between these vectors is calculated.

[0042] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the right upper arm separation F2.

[0043] F3: Left forearm flexion When the body is rotated to the right or left, the degree of bending of the left arm from the elbow down, i.e., the characteristic quantity indicating compensatory movement by the left forearm, is defined as left forearm flexion F3. The calculation of left forearm flexion F3 is explained below.

[0044] For each frame, the following calculations are performed. Figure 10 shows an overview of the calculation of left lower arm flexion in each frame. First, vectors a from the left shoulder L1 to the left elbow L2 and b from the left elbow L2 to the left wrist L3 are generated, and the angle θ formed by these vectors is calculated.

[0045] Then, the angle θ calculated in the subsequent frame L is subtracted from the angle θ calculated in the previous frame F to calculate the difference Δθ as the left lower arm flexion F3.

[0046] F4: Right lower arm flexion The feature amount indicating the degree of bending of the arm from the right elbow when the body is rotated to the right or left, that is, the compensatory movement by the right lower arm, is defined as the right lower arm flexion F4. The calculation of the right lower arm flexion F4 will be described below.

[0047] For each frame, the following calculations are performed. Figure 11 shows an overview of the calculation of right lower arm flexion in each frame. First, vectors a from the right shoulder R1 to the right elbow R2 and b from the right elbow R2 to the right wrist R3 are generated, and the angle θ formed by these vectors is calculated.

[0048] Then, the angle θ calculated in the subsequent frame L is subtracted from the angle θ calculated in the previous frame F to calculate the difference Δθ as the right lower arm flexion F4.

[0049] F5: Acromion level The feature amount indicating the inclination of the line connecting the right shoulder R1 and the left shoulder L2 with respect to the upper trunk is defined as the acromion level F5. The calculation of the acromion level F5 will be described below.

[0050] For each frame, the following calculations are performed. Figure 12 shows an overview of the calculation of the acromion level in each frame. First, vectors a from the left shoulder L1 to the right shoulder R1 and b from the neck C2 to the middle of the waist C3 are generated, and the angle θ formed by these vectors is calculated.

[0051] Then, the angle θ calculated in the subsequent frame LThe angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the acromion horizontal F5.

[0052] F6: Upper trunk anterior-posterior tilt The feature quantity indicating the forward or backward tilt of the upper trunk is defined as upper trunk anterior-posterior tilt F6. The calculation of upper trunk anterior-posterior tilt F6 is explained below.

[0053] For each frame, the following calculations are performed. Figure 13 shows an overview of the calculation of the anterior-posterior tilt of the upper trunk in each frame. First, a plane S6 is fixed that is perpendicular to the vector a that goes from the left hip L4 to the right hip R4. Then, the angle θ is calculated between vector B, which is obtained by projecting the vector b that goes from the neck C2 to the middle of the waist C3 onto plane S6, and vector G, which is obtained by projecting the vertical vector g onto plane S6.

[0054] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the upper trunk anterior-posterior tilt F6.

[0055] F7: Pelvis horizontal The feature quantity indicating the tilt of the pelvis to the right or left is defined as pelvic horizontal F7. The calculation of pelvic horizontal F7 is explained below.

[0056] For each frame, the following calculations are performed. Figure 14 shows an overview of the calculation of pelvic horizontality in each frame. First, the angle θ between the vector a, which points from the left hip L4 to the right hip R4, and the vertical vector g is calculated.

[0057] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as the pelvic horizontal F7.

[0058] F8: Upper trunk lateral flexion The feature quantity indicating the tilt of the upper trunk to the right or left is defined as upper trunk lateral flexion F8. The calculation of upper trunk lateral flexion F8 is explained below.

[0059] Figure 15 shows an overview of the calculation of upper trunk lateral flexion in the anterior frame. For the anterior frame, the angle θ is formed by the vector a, which is directed from the neck C2 to the mid-lumbar region C3, and the vertical vector g. F Calculate.

[0060] Figure 16 shows an overview of the calculation of upper trunk lateral flexion in the posterior frame. For the posterior frame, vector b, which is directed from the neck C2 to the middle of the waist C3, is rotated by a negative multiple of the upper trunk anterior-posterior tilt F6 around vector c, which is directed from the left waist L4 to the right waist R4, to generate vector B. Then, the angle θ between vector B and the vertical vector g is calculated. L Calculate.

[0061] Then, the angle θ calculated in the later frame L The angle θ calculated in the previous frame F The difference Δθ obtained by subtracting is calculated as upper trunk lateral flexion F8.

[0062] F9: Pelvic rotation amount The feature variable indicating the amount of pelvic rotation is denoted as pelvic rotation F9. The calculation of pelvic rotation F9 is explained below.

[0063] The following calculations are performed for each frame. Figure 17 shows an overview of the calculation of pelvic rotation in each frame. First, a plane S9 perpendicular to the vertical vector g is fixed. Next, vector A is calculated by projecting the vector a from the left hip L4 to the right hip R4 onto plane S9.

[0064] Then, vector A calculated in the later frame L And, vector A calculated in the previous frame F The angle θ formed by these two angles is calculated as the pelvic rotation amount F9.

[0065] Step S12 The training data construction unit 12 constructs the training data DAT by associating the rotational features F0 to F9 calculated by the rotational feature extraction unit 11 with corresponding teacher labels. Here, the rotational features F0 to F9 are also referred to as the rotational feature group F. For example, if the subject's OBJ in the video data MOV is tilted to the left, the state of the subject's OBJ is quantitatively represented by the rotational features F0 to F9.

[0066] In response to this, by providing information indicating the posture of the subject's object-oriented body (OBJ) as a teacher label, data elements that constitute the training data (DAT) can be generated. The training data (DAT) is composed of multiple data elements generated in this way.

[0067] In other words, each data element of the training data DAT is represented by the following vector d i It is expressed as follows. In the following formula, i is an index representing the video data MOV, and is an integer between 1 and N, where N is the number of video data MOVs.

number

[0068] The training labels may be automatically generated, for example, by applying various analysis methods to the video data (MOV), or they may be input by a user of the processing system 100 according to the video data (MOV).

[0069] Specific examples of teacher labels will be explained. Figure 18 shows a list of teacher labels. Here, the teacher label items include the degree of upper trunk rotation, shoulder horizontality, trunk anterior / posterior tilt, trunk lateral flexion, pelvic rotation, and pelvic horizontality. In addition, regarding the center of gravity, teacher labels can also be assigned, for example, that the center of gravity is located in the middle of both feet, or that the center of gravity is on one foot. Each label can also be represented by a numerical value, as shown in Figure 18.

[0070] Step S13 The learning processing unit 13 learns the training data DAT using supervised learning and constructs a trained model M. The supervised learning method is not limited to a specific method, and various supervised learning methods can be used.

[0071] Next, the estimation phase in the processing unit 10 will be described. Figure 19 schematically shows the configuration of the processing unit 10 in the estimation phase according to Embodiment 1. Figure 20 shows the processing of the estimation phase in the processing unit 10 according to Embodiment 1.

[0072] The processing unit 10 shown in Figure 19 includes a rotation feature extraction unit 11 and an estimation processing unit 14.

[0073] Step S14 The rotation feature extraction unit 11 calculates rotation features f0 to f9 to be input into the trained model from the video data MOV_IN of the subject to be estimated. The rotation features f0 to f9 in the estimation phase correspond to the rotation features F0 to F9 in the training phase, and are calculated in the same way as in the training phase. Here, the rotation features f0 to f9 are also referred to as the rotation feature group f.

[0074] Step S15 The estimation processing unit 14 inputs the rotational features f0 to f9 from the estimation phase as explanatory variables into the trained model it holds, and outputs the target variable, i.e., the estimation result OUT, which indicates the state of the rotational movement of the subject's body as captured in the video data MOV_IN. The estimation result may also output information such as the posture of the subject being estimated.

[0075] Although the configurations for the learning phase and estimation phase of the processing unit 10 were described separately above, this is merely illustrative. Figure 21 shows an example of the configuration of the processing unit 10 according to Embodiment 1. As shown in Figure 21, the processing unit 10 may have a configuration for the learning phase and a configuration for the estimation phase.

[0076] Furthermore, the processing unit may have only the components necessary for the learning phase and may be configured to perform only the processing of the learning phase. Figure 22 schematically shows an example of the configuration of a processing unit that performs only the processing of the learning phase. In the processing unit 10, the learning processing unit 13 may output output data D_M which includes information such as the constructed trained model and the weight parameters that define the trained model.

[0077] Furthermore, the processing unit may have only the configuration for the estimation phase and be configured to perform only the estimation phase processing. Figure 23 schematically shows an example of the configuration of a processing unit that performs only the estimation phase processing. The processing unit 10 can construct a trained model in the estimation processing unit 14 by receiving output data D_M containing information such as a trained model constructed by another processing unit or parameters that define the trained model. For example, the estimation processing unit 14 can receive output data D_M output from another processing unit and construct a trained model based on it.

[0078] This configuration allows for the automatic analysis of the amount of rotation of target body parts, such as joints, of a subject as captured in image or video data. Furthermore, it enables the estimation of the validity of the analysis results using a pre-trained model.

[0079] Furthermore, the processing device according to this embodiment can be installed in a terminal having an imaging device such as a camera and a processing device, such as a smartphone. This allows even ordinary users, not just experts, to analyze their body's rotational movements and understand the validity of the analysis results. Using these analysis results, each user can engage in rehabilitation activities through online training or self-training.

[0080] Embodiment 2 In Embodiment 1, the analysis of the subject's unidirectional body rotational movement was described using rotational features. However, the human body can perform symmetrical movements to the left and right. Nevertheless, even if a subject intends to perform a symmetrical movement, they may not actually be able to do so due to joint malfunctions or other issues. Therefore, this embodiment describes a method for evaluating the left-right symmetry of a subject's movements.

[0081] In this case, camera 20 captures video data MOV_L when the body is rotated to the left, and video data MOV_R when the body is rotated to the right. Two video data sets recording symmetrical movements of the same subject, as described above, will be referred to as a video pair MP.

[0082] The configuration and operation of the processing unit 40 according to Embodiment 2 will be described below with reference to Figures 24 to 26. Figure 24 schematically shows the configuration of the processing unit 40 according to Embodiment 2 during the learning phase. Figure 25 shows the processing of the processing unit 40 during the learning phase according to Embodiment 2. Figure 26 schematically shows the configuration of the rotational feature extraction unit 11 according to Embodiment 2.

[0083] Step S21 The rotation feature extraction unit 11 receives a video pair MP and extracts rotation features from the video data MOV_L and video data MOV_R contained in the video pair MP. Here, the rotation features F0 to F9 extracted from the video data MOV_L are denoted as the rotation feature group FL, and the rotation features F0 to F9 extracted from the video data MOV_R are denoted as the rotation feature group FR.

[0084] Step S211 The data reading unit 1 receives the video pair MP.

[0085] Step S212 The feature point extraction unit 2 detects skeletal points from the video data MOV_L and the video data MOV_R, respectively. Here, the skeletal point group extracted from the video data MOV_L is referred to as PL, and the skeletal point group extracted from the video data MOV_R is referred to as PR.

[0086] Step S213 The feature calculation unit 3 calculates rotational features F0 to F9, i.e., the rotational feature group FL, from the skeletal point cloud PL extracted from the video data MOV_L. The feature calculation unit 3 also calculates rotational features F0 to F9, i.e., the rotational feature group FR, from the skeletal point cloud PR extracted from the video data MOV_R.

[0087] Step S22 The training data construction unit 12 constructs training data DAT which includes multiple data elements consisting of rotation feature group FL and rotation feature group FR for a pair of video MP and a teacher label L that indicates the left-right symmetry of the motion.

[0088] In other words, each data element of the training data is represented by the following vector e j It is expressed as follows. In the following formula, j is an index representing a video pair, and if M is the number of video pairs, then j is an integer between 1 and M.

number

[0089] In this embodiment, the learning labels may include labels indicating the symmetry of the upper trunk, the symmetry of the pelvis, and the symmetry of the center of gravity, as these indicate the left-right symmetry of human body movements. Figure 27 shows a list of teacher labels. Here, each item is given a label indicating whether it is "symmetrical in left-right movement" or "asymmetrical in left-right movement." For example, "0" may be assigned if it is "symmetrical in left-right movement," and "1" may be assigned if it is "asymmetrical in left-right movement."

[0090] For example, when a subject performs a leftward rotation, it is conceivable that a physical malfunction may cause their center of gravity to shift towards their left foot. In this case, if the center of gravity shifts towards the right foot during a rightward rotation, a "0" is assigned as the center of gravity position symmetry teacher label; otherwise, a "1" is assigned. Also, if the center of gravity is in the center of both feet during a leftward rotation, and is in the center of both feet during a rightward rotation, a "0" is assigned as the center of gravity position symmetry teacher label; otherwise, a "1" is assigned.

[0091] Step S23 The learning processing unit 13 learns the training data DAT using supervised learning and constructs a trained model M.

[0092] Next, the estimation phase in the processing unit 40 will be described. Figure 28 schematically shows the configuration of the processing unit 40 in the estimation phase according to Embodiment 2. Figure 29 shows the processing of the estimation phase in the processing unit 40 according to Embodiment 2.

[0093] The processing unit 40 shown in Figure 28 includes a rotational feature extraction unit 11 and an estimation processing unit 14.

[0094] Step S24 The rotation feature extraction unit 11 calculates rotation feature sets fL and fR to be input to the trained model from the video pair MP_IN of the subject to be estimated. The rotation feature sets fL and fR in the estimation phase correspond to the rotation feature sets FL and FR in the training phase, respectively, and are calculated in the same way as in the training phase.

[0095] Step S25 The estimation processing unit 14 inputs the rotation feature sets fL and fR from the estimation phase as explanatory variables into the stored trained model, thereby outputting an objective variable, i.e., the estimation result OUT, which indicates the left-right symmetry of the estimated symmetrical subject's movements as seen in the video pair MP_IN. This makes it possible to determine from the estimation result OUT whether there is balance between the left and right sides regarding the symmetry of the upper trunk, pelvis, and center of gravity.

[0096] Although the configuration for the learning phase and the configuration for the estimation phase of the processing unit 40 have been described separately, this is merely an example. Similar to Embodiment 1, the processing unit may also have a configuration for the learning phase and a configuration for the estimation phase.

[0097] Furthermore, the processing unit may be configured to have only the components for the learning phase, similar to Embodiment 1, and to execute only the processing for the learning phase. In addition, the processing unit may be configured to have only the components for the estimation phase, similar to Embodiment 1, and to execute only the processing for the estimation phase.

[0098] In summary, this configuration allows for not only analysis of the subject's posture and unidirectional rotational movement as seen in the video, but also further analysis of the left-right symmetry of the subject's rotational movement. This enables a more comprehensive analysis of the body's rotational movement.

[0099] Embodiment 3 The processing system 300 according to Embodiment 3 will now be described. Figure 30 schematically shows the configuration of the processing system 300 according to Embodiment 3. The processing system 300 has a configuration in which a display unit 50 is added to the processing system 100 according to Embodiment 1.

[0100] The display unit 50 is configured to display the estimation result OUT of the learning processing unit 13. In this case, the display unit 50 may display various information useful to the user based on the estimation result OUT.

[0101] Figure 31 shows a first display example of the display unit 50. In Figure 31, an example is shown in which a smartphone's touch panel is used as the display unit 50. In this example, an image or video of the person to be estimated is displayed at the top of the screen, and the estimated result of the body's rotational movement is displayed below it.

[0102] In this configuration, the processing system is installed in a smartphone, which captures video of the person to be estimated, processes the data in the processing unit, and displays the necessary information on the touch panel. This makes it possible to determine the rotational movement of the person to be estimated using a widely available smartphone, without requiring any special hardware.

[0103] The first example described a display example for general users, but more specialized displays are also possible. Figure 32 shows a second display example of the display unit 50. Figure 32 shows an example in which a display panel of a personal computer or the like is used as the display unit 50. In this example, the top of the screen displays, from left to right, a video of the person being estimated being filmed from the front, a video of the person being estimated being filmed from the side, and the estimated result of the body's rotational movement. The bottom of the screen displays the video frames in chronological order, and frames at any time can be selected using a slider or other selection means. General video operation methods can be applied as appropriate to operations using the slider, as well as playback, stopping, and pausing of the video.

[0104] In this configuration, compared to the first example, users can analyze specific frames within a video in more detail. This makes it possible for experts, for example, to analyze the rotational movement of a person's body using common hardware such as a personal computer. Needless to say, users of this configuration are not limited to experts; it can also be used by non-experts.

[0105] Embodiment 4 In the above-described embodiment, it is necessary to determine the vertical vector g in the calculation of upper trunk anterior-posterior tilt F6, pelvic horizontality F7, upper trunk lateral flexion F8, and pelvic rotation amount F9. The direction of the vertical vector g can be determined in advance, for example, by checking the horizontal position when setting up the camera 20. However, since this method requires manual work, it is desirable to be able to acquire the vertical vector g automatically. Therefore, in this embodiment, a processing system capable of automatically acquiring the vertical vector g will be described.

[0106] Figure 33 schematically shows the configuration of the processing system 400 according to Embodiment 4. The processing system 400 has a configuration that adds an acceleration sensor 60 to the processing system 100 according to Embodiment 1. In this example, the acceleration sensor 60 is physically fixed to the camera 20. When video data MOV is captured, the acceleration sensor 60 outputs gravity direction information GV to the processing device 10, indicating the direction of gravity relative to the orientation of the camera 20.

[0107] As a result, the processing unit 10 can use the gravity direction indicated by the gravity direction information GV as the direction of the vertical direction vector g, thereby enabling it to set the optimal vertical direction vector g for each video data MOV.

[0108] Although this explanation assumes that the acceleration sensor 60 is fixed to the camera 20, the acceleration sensor 60 can be installed at any position and in any way as long as it can detect the direction of gravity in conjunction with image or video data.

[0109] As described above, this configuration makes it possible to automatically acquire the vertical vector g. For example, when the processing system is installed in a device such as a smartphone that is capable of software processing and is equipped with an accelerometer and a camera, it is possible to easily acquire the vertical vector g.

[0110] Other embodiments It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, although the processing system according to Embodiment 3 has been described as adding a display unit to the processing system according to Embodiment 1, this is merely an example. A display unit may be added to the processing system according to Embodiment 2, and the estimation results may be displayed on the display unit in the same manner as in Embodiment 3.

[0111] Although the processing system according to Embodiment 4 has been described as adding an acceleration sensor to the processing system according to Embodiment 1, this is merely an example. An acceleration sensor may also be added to the processing system according to Embodiment 2 or 3, and the system may be configured to automatically acquire vertical vectors, similar to Embodiment 4.

[0112] Furthermore, it goes without saying that the above-described processing apparatus may also be provided with a display unit according to Embodiment 3 and an acceleration sensor according to Embodiment 4.

[0113] In the above-described embodiment, the feature point extraction unit 2 was described as using skeletal points as feature points extracted from the subject, but this is merely an example, and other feature points may be used, or multiple types of feature points with different extraction methods may be mixed. For example, the silhouette of the subject may be detected in each frame of the video, and points on the contour of that silhouette may be extracted as feature points. Alternatively, skeletal points and points on the contour of the silhouette may be mixed as feature points.

[0114] According to the above-described embodiment process The processing performed by the device may be realized by having a computer execute a program. Specifically, one or more programs containing a set of instructions for causing a computer system to perform algorithms related to these transmission signal processing or reception signal processing methods can be created and supplied to the computer.

[0115] These programs can be stored and supplied to a computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (random access memory)). Programs may also be supplied to a computer using various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0116] Figure 34 schematically shows the configuration of computer 1000, which is an example of a hardware configuration for realizing a processing unit or processing system. Computer 1000 can be configured as various types of computers, such as a dedicated computer or a personal computer (PC). However, the computer does not need to be physically single; there may be multiple computers when performing distributed processing. As shown in Figure 34, computer 1000 has a CPU (Central Processing Unit) 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003, which are interconnected via a bus 1004. Although the explanation of OS software and other things necessary to operate the computer is omitted, it is assumed that this computer also has them.

[0117] An input / output interface 1005 is connected to bus 1004. An input unit 1006, an output unit 1007, a communication unit 1008, and a storage unit 1009 are connected to the input / output interface 1005.

[0118] The input unit 1006 consists of, for example, a keyboard, mouse, and sensors. The output unit 1007 consists of, for example, a display device such as an LCD and audio output devices such as headphones and speakers. The communication unit 1008 consists of, for example, a router and terminal adapter. The storage unit 1009 consists of a storage device such as a hard disk and flash memory.

[0119] The CPU 1001 can perform various processes according to various programs stored in the ROM 1002 or various programs loaded from the storage unit 1009 into the RAM 1003. In this embodiment, the CPU 1001 executes, for example, the processing performed by a measuring device. A separate GPU (Graphics Processing Unit) may be provided in addition to the CPU 1001, and like the CPU 1001, it may perform various processes according to various programs stored in the ROM 1002 or various programs loaded from the storage unit 1009 into the RAM 1003, in this embodiment, for example, the processing performed by a measuring device. The GPU is suitable for performing routine processing in parallel, and by applying it to processing in neural networks, which will be described later, it is possible to improve the processing speed compared to the CPU 1001. The RAM 1003 also appropriately stores data necessary for the CPU 1001 and GPU to perform various processes.

[0120] The communication unit 1008 can communicate bidirectionally with the server 1030 via the network 1020. The communication unit 1008 can send data provided by the CPU 1001 to the server 1030, and output data received from the server 1030 to the CPU 1001, RAM 1003, and storage unit 1009, etc. The communication unit 1008 may also communicate with other devices using analog or digital signals. The storage unit 1009 can exchange data with the CPU 1001 and can store and erase information.

[0121] A drive 1010 may be connected to the input / output interface 1005 as needed. The drive 1010 can be appropriately fitted with storage media such as a magnetic disk 1011, an optical disk 1012, a flexible disk 1013, or a semiconductor memory 1014. Computer programs read from each storage medium may be installed in the storage unit 1009 as needed. Furthermore, data necessary for the CPU 1001 to perform various processes, or data obtained as a result of the CPU 1001's processing, may be stored in each storage medium as needed.

[0122] In the embodiments described above, image data and video data are acquired by a camera, but this is merely an example. Image data and video data can be acquired by any type of imaging device.

[0123] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0124] (Note 1) A processing device comprising: a rotation feature extraction unit that extracts feature points of the subject to be studied from two frames selected from video image data of the rotational movement of the subject's body, selected from the rotational movement of the subject's body, and calculates a rotation feature quantity indicating the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; a learning data construction unit that constructs learning data by assigning teacher labels indicating the state of the subject to be studied corresponding to the rotation feature quantity; and a learning processing unit that constructs a trained model by learning the learning data.

[0125] (Note 2) The apparatus according to Note 1, further comprising an estimation processing unit that estimates the state of rotational movement of the body of a subject to be estimated by inputting input data into the trained model, wherein the rotational feature extraction unit extracts feature points of the subject to be estimated from two frames selected from video image data of the rotational movement of the body of the subject to be estimated, selected from the frame before and after the rotational movement, calculates a rotational feature representing the rotational movement of the body of the subject to be estimated based on the difference in the position of the feature points extracted from the two frames before and after the rotational movement, and outputs the calculated rotational feature as input data to the estimation processing unit.

[0126] (Note 3) The processing apparatus according to Note 1, wherein the rotation feature extraction unit selects two frames from the motion image data of the subject to be trained that relate to a symmetrical rotational movement of the body performed by the subject to be trained, calculates rotation features for the two selected frames, and the training data construction unit constructs the training data by assigning teacher labels that indicate the state of the symmetrical movement of the subject to be trained that correspond to the rotation features.

[0127] (Note 4) The processing apparatus according to Note 2, wherein the rotation feature extraction unit selects two frames from the video image data of the subject to be studied that relate to a symmetrical body rotation movement performed by the subject to be studied, calculates rotation features for the two selected frames, the training data construction unit constructs the training data by assigning teacher labels indicating the state of the symmetrical movement of the subject to be studied that correspond to the rotation features, the rotation feature extraction unit selects two frames from the video image data of the subject to be estimated that relate to a symmetrical body rotation movement performed by the subject to be estimated, calculates rotation features for the two selected frames, and the estimation processing unit estimates the state of the symmetrical movement of the subject to be estimated that corresponds to the rotation features.

[0128] (Note 5) A rotation feature extraction unit extracts feature points of the subject to be estimated from two frames selected from video image data of the rotational movement of the subject to be estimated, taken before and after the rotational movement of the body, and calculates a rotation feature quantity indicating the rotational movement of the subject to be estimated based on the difference in the position of the feature points extracted from the two frames before and after the rotational movement of the body, and a rotation feature quantity that indicates the rotational movement of the subject to be learned from two frames selected from video image data of the rotational movement of the body of the subject to be learned A processing device comprising: extracts feature points of a subject, calculates rotational feature quantities indicating the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, learns the correspondence between the rotational feature quantities and the state of the subject being learned, and inputs the rotational feature quantities of the subject being learned calculated by the rotational feature quantity extraction unit into a trained model constructed by learning the correspondence between the rotational feature quantities and the state of the subject being learned, in order to estimate the state of the rotational movement of the subject being estimated.

[0129] (Note 6) The trained model is constructed by selecting two frames from video data of the subject to be trained that relate to a symmetrical body rotation movement performed by the subject to be trained, calculating rotational features for the two selected frames, assigning teacher labels indicating the state of the symmetrical movement of the subject to be trained that correspond to the rotational features, constructing training data, and training the training data; the rotational feature extraction unit selects two frames from video data of the subject to be estimated that relate to a symmetrical body rotation movement performed by the subject to be estimated, calculating rotational features for the two selected frames; and the estimation processing unit estimates the state of the symmetrical movement of the subject to be estimated that corresponds to the rotational features, as described in Note 5.

[0130] (Appendix 7) The state of rotational movement of the subject's body that is the subject of estimation in the estimation processing unit. Estimate The processing apparatus according to any one of appendices 2, 4 to 6, further comprising a display unit for displaying the results and a video image of the subject being estimated.

[0131] (Appendix 8) The processing apparatus according to Appendix 7, wherein the display unit displays two or more moving images of the subject being estimated, taken from different positions.

[0132] (Note 9) The processing apparatus according to Note 8, wherein the display unit displays a selection means for selecting any frame of the video of the subject to be estimated.

[0133] (Note 10) The processing apparatus according to any one of Notes 1 to 9, wherein the rotation feature extraction unit extracts the skeletal points of the subject as the feature points.

[0134] (Note 11) The processing apparatus according to Note 10, wherein the rotation feature extraction unit extracts the subject's neck, waist, left and right shoulders, left and right elbows, left and right wrists, and left and right hips as the skeletal points.

[0135] (Note 12) The processing apparatus described in Note 11, wherein the rotation feature extraction unit finds a plane perpendicular to the line connecting the neck and the middle of the waist, calculates the angle between the line projected onto the plane that connects the left shoulder and the right shoulder, and the line projected onto the plane that connects the left hip and the right hip, for the two frames, and calculates the difference between the two calculated angles as the first rotation feature.

[0136] (Note 13) The processing apparatus according to Note 11 or 12, wherein the rotation feature extraction unit calculates the angle formed by a line connecting the neck and the middle of the waist and a line connecting the left shoulder and the left elbow for the two frames, and calculates the difference between the two calculated angles as a second rotation feature.

[0137] (Note 14) The processing apparatus according to any one of Notes 11 to 13, wherein the rotation feature extraction unit calculates the angle formed by a line connecting the neck and the waist and a line connecting the right shoulder and the right elbow for the two frames, and calculates the difference between the two calculated angles as a third rotation feature.

[0138] (Note 15) The processing apparatus according to any one of Notes 11 to 14, wherein the rotation feature extraction unit calculates the angle between the line connecting the left shoulder and the left elbow and the line connecting the left elbow and the left wrist for the two frames, and calculates the difference between the two calculated angles as a fourth rotation feature.

[0139] (Note 16) The processing apparatus according to any one of Notes 11 to 15, wherein the rotation feature extraction unit calculates the angle between the line connecting the right shoulder and the right elbow and the line connecting the right elbow and the right wrist for the two frames, and calculates the difference between the two calculated angles as a fifth rotation feature.

[0140] (Note 17) The processing apparatus according to any one of Notes 11 to 16, wherein the rotation feature extraction unit calculates the angle between the line connecting the right shoulder and the left shoulder and the line connecting the neck and the waist for the two frames, and calculates the difference between the two calculated angles as a sixth rotation feature.

[0141] (Note 18) The processing apparatus according to any one of Notes 11 to 17, wherein the rotation feature extraction unit finds a plane perpendicular to the line connecting the left hip and the right hip, calculates the angle between the line projected onto the plane with the line connecting the neck and the middle of the waist and the line projected onto the plane with respect to the vertical direction for the two frames, and calculates the difference between the two calculated angles as the seventh rotation feature.

[0142] (Note 19) The processing apparatus according to Note 18, wherein for one of the two frames, a first angle is calculated between the line connecting the neck and the middle of the waist and the vertical direction; for the other of the two frames, a line is obtained by rotating the line connecting the neck and the middle of the waist around the line connecting the left waist and the right waist in the opposite direction to the seventh rotation feature by the amount of the seventh rotation feature; a second angle is calculated between the obtained line and the vertical direction; and the difference between the first angle and the second angle is calculated as the ninth rotation feature.

[0143] (Note 20) The processing apparatus according to any one of Notes 11 to 19, wherein the rotation feature extraction unit calculates the angle between the line connecting the left hip and the right hip and the vertical direction for the two frames, and calculates the difference between the two calculated angles as the eighth rotation feature.

[0144] (Note 21) The processing apparatus according to any one of Notes 11 to 20, wherein the rotation feature extraction unit determines a plane perpendicular to the vertical direction, calculates a line projected onto the plane by connecting the left hip and the right hip for the two frames, and calculates the angle between the two calculated lines as the tenth rotation feature.

[0145] (Note 22) The processing apparatus according to any one of Notes 18 to 21, wherein the vertical vector is determined based on the direction of gravity obtained by an acceleration sensor that detects the orientation of the imaging device that captures the motion image data.

[0146] (Note 23) A processing system comprising: an imaging device that acquires video data of the rotational movement of a subject to be studied; a processing device that learns rotational features indicating the rotational movement of the subject to be studied from the video data of the subject to be studied and constructs a trained model that estimates the state of the subject to be studied, wherein the processing device comprises: a rotational feature extraction unit that extracts feature points of the subject to be studied from two frames selected from video data of the rotational movement of the subject to be studied, one before and one after the rotational movement of the body, and calculates rotational features indicating the rotational movement of the subject to be studied based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; a training data construction unit that constructs training data by assigning teacher labels indicating the state of the subject to be studied corresponding to the rotational features; and a training processing unit that constructs a trained model by learning the training data.

[0147] (Note 24) A method for constructing a trained model, comprising: extracting feature points of the subject to be trained from two frames selected from video image data of the rotational movement of the subject's body, selecting frames before and after the rotational movement of the body; calculating rotational feature quantities that indicate the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; constructing training data by assigning teacher labels that indicate the state of the subject to be trained corresponding to the rotational feature quantities; and constructing a trained model by training the training data.

[0148] (Note 25) A program that causes a computer to perform the following steps: extract feature points of the subject to be trained from two frames selected from video image data of the rotational movement of the subject's body, one before and one after the rotational movement; calculate a rotational feature quantity indicating the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body; construct training data by assigning a teacher label indicating the state of the subject to be trained corresponding to the rotational feature quantity; and construct a trained model by training the training data.

[0149] Although the present invention has been described above with reference to embodiments, the present invention is not limited thereto. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the invention.

[0150] This application claims priority based on Japanese Patent Application No. 2022-188607, filed on 25 November 2022, and incorporates all of its disclosures herein. [Explanation of Symbols]

[0151] 1. Data reading unit 2. Feature point extraction unit 3. Feature Calculation Unit 11 Rotational Feature Extraction Unit 12. Training Data Construction Section 13 Learning Processing Unit 14 Estimation Processing Unit 10, 40 Processing Units 20 cameras 30 Video Databases 50 Display 60 Accelerometer 100, 101, 300, 400 processing systems 1000 computers 1001 CPU 1002 ROM 1003 RAM 1004 Bus 1005 Input / Output Interface 1006 Input section 1007 Output section 1008 Communications Department 1009 Storage section 1010 Drive 1011 Magnetic disk 1012 Optical Disc 1013 Flexible disk 1014 Semiconductor memory 1020 Network 1030 Server C1 nose C2 head C3 middle of the waist D_M output data DAT learning data F0 amount of acromion rotation F1 separation of the left upper arm F2 separation of the right upper arm F3 flexion of the left lower arm F4 flexion of the right lower arm F5 acromion level F6 anterior-posterior tilt of the upper trunk F7 pelvis level F8 lateral flexion of the upper trunk F9 amount of pelvic rotation L1 left shoulder L2 left elbow L3 left wrist L4 left waist L5 left knee L6 left ankle MOV, MOV_L, MOV_R video data MP video pair OBJ subject OUT estimation result P, PL, PR skeletal point group R1 right shoulder R2 right elbow R3 right wrist R4 right waist R5 right knee R6 right ankle

Claims

1. A rotation feature extraction unit extracts feature points of the subject from two frames selected from video data images of the rotational movement of the subject's body, one before and one after the rotational movement, and calculates a rotational feature quantity that represents the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement. A training data construction unit constructs training data by assigning teacher labels indicating the state of the subject being trained, corresponding to the rotational features, The system comprises a learning processing unit that constructs a trained model by learning the aforementioned training data, Processing device.

2. The system further includes an estimation processing unit that estimates the state of rotational movement of the subject's body by inputting input data into the aforementioned trained model. The rotational feature extraction unit is, From the video data capturing the rotational movement of the subject to be estimated, feature points of the subject to be estimated are extracted from two frames selected, one before and one after the rotational movement of the body. Based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body, a rotational feature quantity indicating the rotational movement of the subject to be estimated is calculated. The calculated rotational feature quantity is output to the estimation processing unit as input data. The apparatus according to claim 1.

3. The rotation feature extraction unit selects two frames from the video data of the subject being trained that relate to a symmetrical rotational movement of the body performed by the subject being trained, and calculates rotation features for the two selected frames. The training data construction unit constructs the training data by assigning teacher labels that indicate the state of symmetrical movement of the subject being trained, corresponding to the rotational features. The apparatus according to claim 1.

4. The rotation feature extraction unit selects two frames from the video data of the subject being trained that relate to a symmetrical rotational movement of the body performed by the subject being trained, and calculates rotation features for the two selected frames. The aforementioned training data construction unit constructs the training data by assigning teacher labels that indicate the state of symmetrical movement of the subject being trained, corresponding to the rotational features. The rotation feature extraction unit selects two frames from the motion image data of the subject to be estimated that relate to a symmetrical rotational movement of the body performed by the subject to be estimated, and calculates rotation features for the two selected frames. The estimation processing unit estimates the state of symmetrical movement of the subject being estimated, corresponding to the rotational feature quantity. The apparatus according to claim 2.

5. A rotation feature extraction unit extracts feature points of the subject to be estimated from two frames selected from video data capturing the rotational movement of the subject's body, one before and one after the rotational movement, and calculates a rotational feature quantity indicating the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement. The system includes: an estimation processing unit that extracts feature points of the subject to be studied from two frames selected from video data capturing the rotational movement of the subject's body, selected from the rotational movement of the subject to be studied, and calculates a rotational feature quantity representing the rotational movement of the subject to be studied, based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement of the body. This rotational feature quantity is then used to learn the correspondence between the rotational feature quantity and the state of the subject to be studied that corresponds to the rotational feature quantity. The estimation processing unit then inputs the rotational feature quantity of the subject to be studied, calculated by the rotational feature quantity extraction unit, into a trained model constructed by learning the correspondence between the rotational feature quantity and the state of the subject to be studied that corresponds to the rotational feature quantity, and estimates the state of the rotational movement of the subject to be estimated. Processing device.

6. The aforementioned trained model is From the video data of the subject being trained, two frames relating to a symmetrical body rotational movement performed by the subject are selected, and rotational features are calculated for the two selected frames. Training data is constructed by assigning teacher labels that indicate the state of symmetrical movement of the subject being trained, corresponding to the rotational features. It is constructed by training on the aforementioned training data, The rotation feature extraction unit selects two frames from the motion image data of the subject to be estimated that relate to a symmetrical rotational movement of the body performed by the subject to be estimated, and calculates rotation features for the two selected frames. The estimation processing unit estimates the state of symmetrical movement of the subject being estimated, corresponding to the rotational feature quantity. The apparatus according to claim 5.

7. The system further includes a display unit that displays the estimation result of the estimation processing unit's estimation of the rotational movement state of the subject's body, and a moving image of the subject. The apparatus according to claim 2 or 4.

8. An imaging device that acquires dynamic image data of the rotational movement of the body of a subject being studied, The processing device includes a device that learns rotational features indicating the rotational movement of the subject's body from the subject's video data and constructs a trained model for estimating the subject's state. The aforementioned processing apparatus is A rotation feature extraction unit extracts feature points of the subject from two frames selected from video data images of the rotational movement of the subject's body, one before and one after the rotational movement, and calculates a rotational feature quantity that indicates the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement. A training data construction unit constructs training data by assigning teacher labels indicating the state of the subject being trained, corresponding to the rotational features, The system comprises a learning processing unit that constructs a trained model by learning the aforementioned training data, Processing system.

9. A computer, From video data capturing the rotational movement of the subject being studied, feature points of the subject are extracted from two frames selected, one before and one after the rotational movement. Based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement, a rotational feature quantity representing the rotational movement of the subject being studied is calculated. Training data is constructed by assigning a teacher label indicating the state of the subject being trained, corresponding to the rotational feature. A trained model is constructed by training it with the aforementioned training data. How to build a pre-trained model.

10. The process involves extracting feature points of the subject from two frames selected from video data capturing the rotational movement of the subject's body, one before and one after the rotational movement, and calculating a rotational feature quantity representing the rotational movement of the subject's body based on the difference in the positions of the feature points extracted from the two frames before and after the rotational movement. A process to construct training data by assigning a teacher label indicating the state of the subject being trained, corresponding to the rotational feature, The process of constructing a trained model by training it with the aforementioned training data is performed on the computer. program.

Images