Posture evaluation device, posture evaluation system, posture evaluation method, and program

Abstract

Provided are a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program, with which it is possible to evaluate a posture with high accuracy. The posture evaluation device comprises: a skeleton extraction means that estimates, from a captured image obtained by imaging a subject, keypoint location information for keypoints that each comprise a joint or an anatomical feature point of a body and a reliability score for each of the keypoints; an image generation means that generates a display image obtained by superimposing a keypoint for which the reliability score is lower than a prescribed threshold value over the captured image in a display mode differing from that of another keypoint for which the reliability score is equal to or greater than the prescribed threshold value; a skeleton correction means that corrects the keypoint location information on the basis of keypoint correction information that is for correcting the keypoint location information and that has been received from a user; and a feature value calculation means that calculates a feature value relating to the joint or the anatomical feature point on the basis of the corrected keypoint location information.

Description

[Technical Field]

[0001] This disclosure relates to a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program. [Background technology]

[0002] In recent years, with the spread of online training and self-training, there has been a growing need for ordinary people without specialized knowledge to evaluate their own posture. For example, there is a demand for technology that evaluates a user's posture based on images taken of the user with a camera built into a smartphone or similar device.

[0003] Patent Document 1 describes a system that estimates the skeleton from images of a subject and classifies and searches for the subject's posture, behavior, and other conditions. In Patent Document 1, the skeleton is estimated by extracting characteristic points such as joints as key points.

[0004] Key points can be extracted using machine learning models such as deep learning. However, while general machine learning models for posture evaluation can extract key points relatively accurately in postures such as sitting or standing, they may not be able to accurately extract key points in postures in special states such as during exercise. If the accuracy of key point extraction is low, the accuracy of posture evaluation will also be low. Therefore, a technology has been developed that overlays the extracted key points onto an image of the user and allows the user to correct those key points on the display screen. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] International Publication No. 2021 / 250808 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, if the user is unaware that key points have been incorrectly extracted, the key points may not be corrected, potentially leading to lower accuracy in posture assessment.

[0007] The purpose of this disclosure is to provide a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program that can evaluate posture with high accuracy. [Means for solving the problem]

[0008] The posture evaluation device according to this disclosure includes: a skeleton extraction unit that estimates keypoint position information of keypoints consisting of joints or anatomical feature points of the body and the reliability of said keypoints from an image captured by imaging the body of a subject; an image generation unit that generates a display image in which the keypoints whose reliability is lower than a predetermined threshold are superimposed on the image in a different display manner from the other keypoints whose reliability is above the predetermined threshold; a skeleton correction unit that corrects the keypoint position information based on keypoint correction information for correcting the keypoint position information received from a user; and a feature quantity calculation unit that calculates feature quantities related to the joints or anatomical feature points based on the keypoint position information corrected by the skeleton correction unit.

[0009] The posture evaluation system according to this disclosure comprises a posture evaluation device and a subject terminal capable of communicating with the posture evaluation device, wherein the posture evaluation device comprises a skeleton extraction unit that estimates keypoint position information of keypoints consisting of joints or anatomical feature points of the subject's body and the reliability of said keypoints from an image of the subject's body acquired by the subject terminal, an image generation unit that generates a display image in which the keypoints whose reliability is lower than a predetermined threshold are superimposed on the image in a different display manner from the other keypoints whose reliability is above the predetermined threshold, a skeleton correction unit that corrects the keypoint position information based on keypoint correction information for correcting the keypoint position information received from a user, and a feature quantity calculation unit that calculates feature quantities related to the joints or anatomical feature points based on the keypoint position information corrected by the skeleton correction unit.

[0010] The posture evaluation method according to this disclosure is a method in which a posture evaluation device estimates keypoint position information of keypoints consisting of joints or anatomical feature points of the body and the reliability of the keypoints from an image obtained by imaging the body of a subject, generates a display image in which the keypoints whose reliability is lower than a predetermined threshold are superimposed on the image in a different display manner from the other keypoints whose reliability is above the predetermined threshold, modifies the keypoint position information based on keypoint modification information for modifying the keypoint position information received from the user, and calculates feature quantities related to the joints or anatomical feature points based on the keypoint position information modified by the skeletal modification unit.

[0011] The program according to the present disclosure causes a posture evaluation device to perform a process of estimating keypoint position information of keypoints composed of joints or anatomical feature points of the body and the reliability of the keypoints from a captured image obtained by imaging the body of a subject, a process of generating a display image in which the keypoints with a reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other keypoints with a reliability equal to or higher than the predetermined threshold, a process of correcting the keypoint position information based on keypoint correction information received from a user for correcting the keypoint position information, and a process of calculating a feature amount related to the joint or the anatomical feature point based on the keypoint position information corrected by the skeleton correction unit.

Effect of the Invention

[0012] It is possible to provide a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program that can evaluate a posture with high accuracy.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. [Figure 2] It is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. [Figure 3] It is a diagram showing an example of an image captured by an imaging unit according to the present disclosure. [Figure 4] It is a diagram showing an example of a display image according to the present disclosure. [Figure 5] It is a diagram showing an example of a display image in which a keypoint according to the present disclosure is corrected. [Figure 6] It is a diagram showing another example of a display image according to the present disclosure. [Figure 7] It is a flowchart showing a posture evaluation method according to the present disclosure. [Figure 8] It is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. [Figure 9] It is a diagram showing an example of a display image according to the present disclosure. [Figure 10] This figure shows an example of a display image in which the spinal column edge point cloud related to this disclosure has been corrected. [Figure 11] This flowchart shows the attitude evaluation method related to this disclosure. [Figure 12] This is a block diagram showing the configuration of the posture evaluation device related to this disclosure. [Figure 13] This figure shows an example of a tensor and a displayed image related to this disclosure. [Figure 14] This is a block diagram showing the configuration of the posture evaluation system related to this disclosure. [Figure 15] This block diagram shows the configuration of the target user's terminal related to this disclosure. [Figure 16] This figure shows an example of a display image related to this disclosure. [Figure 17] This block diagram shows an example of the computer configuration related to this disclosure. [Modes for carrying out the invention]

[0014] Embodiment 1 Embodiment 1 of the present invention will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the posture evaluation device 100 according to the present disclosure. The posture evaluation device 100 in this disclosure is a device that evaluates posture based on captured images obtained by imaging a subject's body with a camera such as a smartphone. The subject refers to the person whose posture is being evaluated by the posture evaluation device 100. Specifically, the posture evaluation device 100 estimates the subject's skeletal structure from the captured images and evaluates the subject's posture based on that skeletal structure. This allows for posture evaluation in situations such as online training and self-training.

[0015] As shown in Figure 1, the posture evaluation device 100 includes a skeleton extraction unit 101, an image generation unit 103, an input reception unit 104, a skeleton correction unit 105, and a feature quantity calculation unit 106.

[0016] The skeletal extraction unit 101 extracts joints or anatomical feature points of the body as key points from the captured images obtained by imaging the subject's body. In other words, the skeletal extraction unit 101 estimates the key point location information and the confidence level of the key points. That is, "extracting key points" means "estimating the key point location information and confidence level." Therefore, in this specification, it may also be expressed as "estimating key points." Here, anatomical feature points include, for example, the wrist, elbow, shoulder, hip, knee, and ankle. Here, the image obtained by imaging the subject's body is a two-dimensional image, and may be a two-dimensional RGB image. Furthermore, the image may be a side view of the subject's body. Furthermore, keypoint location information refers to the location information of joints or anatomical feature points on the image. Specifically, the location information of joints or anatomical feature points on the image refers to, for example, the location information of a pixel located at the center of the image region corresponding to a joint or anatomical feature point on the image. Furthermore, pixel position information refers to, for example, image coordinates. Here, image coordinates are coordinates used to indicate the position of a pixel on a two-dimensional image. For example, the origin is defined as the center of the leftmost and topmost pixel in the two-dimensional image, with the left-right or horizontal direction being the x-direction and the up-down or vertical direction being the y-direction. Furthermore, confidence is the probability that the pixel specified by the keypoint location information estimated by the skeleton extraction unit 101 is a joint or anatomical feature point. For example, when the skeleton extraction unit 101 extracts keypoints from an acquired image using a machine learning model, a score is calculated along with the keypoint location information indicating whether that keypoint is a joint or anatomical feature point. In this case, confidence is this score. The higher the score, the higher the confidence.

[0017] The image generation unit 103 generates a display image in which keypoints with a confidence level lower than a predetermined threshold are superimposed on the captured image in a different display manner from other keypoints with a confidence level above the predetermined threshold. In this display image, the keypoints are superimposed on the captured image so that they can be moved (dragged) by the user. Here, "user" refers to at least one of the following: the person whose posture is being evaluated by the posture evaluation device 100, and the evaluator who uses the posture evaluation device 100 to evaluate the posture of another person. Furthermore, when the person evaluates their own posture using the posture evaluation device 100 during self-training, etc., the person is also the evaluator. Furthermore, when the evaluator uses the posture evaluation device 100 to evaluate the posture of another person, the evaluator is, for example, a therapist or trainer. The display image may be displayed on the user's smartphone (the person's terminal) or on a display unit (not shown) provided by the posture evaluation device 100. Furthermore, a different display method means that the shape, color, or size of the key points may differ, or the display format (such as flashing) may differ. A different display method also includes displaying a message within the image indicating that extraction of any key points failed.

[0018] The input reception unit 104 receives keypoint correction information from the user to correct keypoint position information. Specifically, the input reception unit 104 receives keypoint correction information as keypoint correction information, which is the position information of keypoints that have been corrected to the correct position by the user moving (dragging) the keypoints on the displayed image. Note that moving keypoints on the displayed image is not limited to dragging. For example, the user may select the keypoint to be corrected and then select the destination position.

[0019] The frame correction unit 105 corrects the key point position information based on the key point correction information.

[0020] The feature calculation unit 106 calculates feature quantities related to joints or anatomical feature points based on the keypoint position information corrected by the skeletal correction unit 105.

[0021] According to this embodiment 1, a posture evaluation device 100 capable of evaluating posture with high accuracy can be provided. Specifically, the skeletal extraction unit 101 estimates the reliability along with keypoint position information, and the image generation unit 103 superimposes keypoints with low reliability onto the captured image in a different display manner from other keypoints. Therefore, the user can distinguish keypoints with low reliability and a high probability of incorrect estimation from other keypoints. This prevents situations where the user is unaware that keypoints have been incorrectly extracted and the keypoints are not corrected. This improves the accuracy of posture evaluation. Thus, a posture evaluation device 100 capable of evaluating posture with high accuracy can be provided.

[0022] Furthermore, since the feature calculation unit 106 calculates feature quantities related to joints or anatomical feature points, it becomes possible to evaluate posture based on these feature quantities. Therefore, posture can be evaluated with a high level of accuracy comparable to that of professionals such as therapists and trainers.

[0023] Embodiment 2 Embodiment 2 of the present invention will be described with reference to Figure 2. Figure 2 is a block diagram showing the configuration of the posture evaluation device 100A according to the present disclosure. The posture evaluation device 100A is, for example, a server that can communicate with a user terminal such as a smartphone, tablet terminal, or personal computer owned by the user. The posture evaluation device 100A may also be a user terminal (subject terminal) such as a smartphone, tablet terminal, or personal computer owned by the user.

[0024] As shown in Figure 2, the posture evaluation device 100A of this disclosure comprises a skeleton extraction unit 101, a failure judgment unit 102, an image generation unit 103, an input reception unit 104, a skeleton correction unit 105, a feature quantity calculation unit 106, a state estimation unit 107, an additional learning unit 108, an input unit 109, a storage unit 110, and a communication unit 111. The posture evaluation device 100A may also include a display unit (not shown). The input unit 109 and the display unit (not shown) may be configured as a single touch panel display, or they may be provided separately. The storage unit 110 stores a skeleton database (indicated as "Skeleton DB" in Figure 2) 112, a skeleton extraction model 113, etc.

[0025] The skeletal extraction unit 101 extracts joints or anatomical feature points of the body as key points from the captured images obtained by imaging the subject's body. In other words, the skeletal extraction unit 101 estimates the key point location information and the reliability of the key points. For example, from the captured images shown in Figure 3, the skeletal extraction unit 101 extracts the wrist, elbow, shoulder, hip, knee, and ankle as key points P1 to P6, and estimates the key point location information and reliability of key points P1 to P6. Details of the location information and reliability are as described in Embodiment 1, so a detailed explanation is omitted here. Specifically, the skeleton extraction unit 101 estimates keypoint position information from the captured image using the trained skeleton extraction model 113. The posture evaluation device 100A first performs machine learning using the machine learning model, which is the skeleton extraction model 113, and the training data, which is the skeleton database 112, to generate the trained skeleton extraction model 113. Furthermore, the keypoint position information may be represented by three-dimensional coordinates defined in the x-direction (left-right or horizontal direction), the y-direction (up-down or vertical direction) of the captured two-dimensional image, as well as the z-direction (depth direction). This is possible by using a skeletal extraction model 113 that estimates keypoint position information represented by three-dimensional coordinates from a two-dimensional image. Furthermore, the body parts from which the skeletal extraction unit 101 extracts key points may include, in addition to those mentioned above, other joints such as the cervical vertebrae, hip joints, and knee joints, as well as anatomical features such as the wrists, elbows, shoulders, hips, knees, and ankles.

[0026] The failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed if the number of keypoints with a confidence level below a predetermined threshold is greater than or equal to a predetermined standard. Here, the predetermined standard is, for example, the upper limit of the number of keypoints with a confidence level below a predetermined threshold. Alternatively, the predetermined standard may be, for example, the ratio of the number of keypoints with a confidence level below a predetermined threshold to the total number of keypoints.

[0027] The image generation unit 103 generates a display image in which keypoints with a reliability level lower than a predetermined threshold are superimposed on the captured image in a different display manner from other keypoints with a reliability level above the predetermined threshold. In this display image, the keypoints are superimposed on the captured image in a way that allows the user to move (drag) them. Figure 4 shows an example of a display image generated by the image generation unit 103. In the example shown in Figure 4, the confidence level of the wrist keypoint P1 is lower than a predetermined threshold, and it is superimposed on the captured image with a different shape and size from the other keypoints P2 to P6. This makes it immediately clear that the extraction of the wrist keypoint P1 was incorrect. Figure 5 also shows another example of a display image generated by the image generation unit 103. In the example shown in Figure 5, the message M "Left wrist detection failed" is displayed in the image. This makes it immediately clear that the extraction of the keypoint P1 of the left wrist was incorrect.

[0028] Here, the predetermined threshold is a value determined based on the importance of the key points in the estimation of the state estimation unit 107, which will be described later. Let's explain the predetermined threshold in more detail. First, the importance of keypoint k in the estimation of the state estimation unit 107 is k It is defined by the following equation (1).

number

Equation

[0029] The input receiving unit 104 receives keypoint correction information from the user to correct keypoint position information. Specifically, the input receiving unit 104 receives keypoint position information as keypoint correction information when the user corrects the keypoint to the correct position by dragging the keypoint on the displayed image. Figure 6 shows a displayed image showing the keypoint P1 corrected by the user. The input receiving unit 104 receives the corrected keypoint P1 position information shown in Figure 6 as keypoint correction information.

[0030] The frame correction unit 105 corrects the key point position information based on the key point correction information.

[0031] The feature calculation unit 106 calculates feature quantities related to joints or anatomical feature points based on the keypoint position information corrected by the skeletal correction unit 105. Specifically, the feature calculation unit 106 calculates feature quantities such as the angles between straight lines (also called "bones") connecting two or more of the keypoints P1 to P6, as well as the angles between vertical and horizontal lines in the image.

[0032] The state estimation unit 107 estimates the posture state of subject O based on the features calculated by the feature calculation unit 106. Specifically, for example, the state estimation unit 107 estimates the posture state of subject O based on the features and a list of reference values ​​(not shown) stored in the memory unit 110. The list of reference values ​​is, for example, data that associates the type of posture with the reference value of the features in that posture.

[0033] The additional learning unit 108 performs additional training on the skeleton extraction model 113 using the captured image and the keypoint position information corrected by the skeleton correction unit 105. This further improves the accuracy of keypoint extraction using the skeleton extraction model 113.

[0034] The input unit 109 may accept operation instructions from the user. The input unit 109 may be configured as a keyboard or as a touch panel display device. The input unit 109 may be configured as a keyboard or touch panel connected to the posture evaluation device 100A main unit.

[0035] The storage unit 110 stores the skeleton database 112, the skeleton extraction model 113, etc. The storage unit 110 may also include non-volatile memory (e.g., ROM (Read Only Memory)) in which various programs and data necessary for processing are fixedly stored. The storage unit 110 may also use an HDD or SSD. Furthermore, the storage unit 110 may include volatile memory (e.g., RAM (Random Access Memory)) used as a working area. The above programs may be read from a portable recording medium such as an optical disc or semiconductor memory, or downloaded from a server device on a network.

[0036] The skeletal database 112 is a database in which multiple images obtained by imaging the human body are associated with keypoint location information, which serves as the correct label.

[0037] The skeletal extraction model 113 is a machine learning model that estimates keypoint location information from captured images obtained by imaging a human body. In other words, the skeletal extraction model 113 is a machine learning model that estimates keypoint location information using captured images obtained by imaging a human body as input. In this specification, machine learning may be deep learning, but is not particularly limited.

[0038] The communication unit 111 communicates with the target terminal (not shown). The communication unit 111 may also communicate with external servers or other terminal devices. The communication unit 111 may be equipped with an antenna (not shown) for wireless communication, or it may be equipped with an interface such as a NIC (Network Interface Card) for wired communication.

[0039] Next, with reference to Figure 7, the attitude evaluation method related to this disclosure will be explained. First, the skeletal extraction unit 101 estimates key points from the captured images of the subject O's body (step S101). Specifically, the skeletal extraction unit 101 estimates the key point location information of key points consisting of joints or anatomical feature points of the body, as well as the confidence level of those key points, from the captured images.

[0040] Next, the failure determination unit 102 determines whether the number of keypoints with a confidence level lower than a predetermined threshold is greater than or equal to a predetermined standard (step S102). In step S102, if the number of keypoints with a confidence level lower than a predetermined threshold is greater than or equal to a predetermined standard (step S102; Yes), the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed and returns to the processing in step S101. As a result, a new image is acquired, and keypoint extraction is performed again based on the new image. On the other hand, in step S102, if the number of keypoints with a confidence level lower than a predetermined threshold is less than a predetermined standard (step S102; No), the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has succeeded and proceeds to the processing in step S103.

[0041] Next, the image generation unit 103 generates a display image in which keypoints with a reliability lower than a predetermined threshold are superimposed on the captured image in a different display manner from other keypoints with a reliability above a predetermined threshold (step S103).

[0042] Next, the input receiving unit 104 receives keypoint correction information from the user to correct the keypoint location information (step S104).

[0043] Next, the frame correction unit 105 corrects the key point position information based on the key point correction information (step S105).

[0044] Next, the feature calculation unit 106 calculates feature quantities related to joints or anatomical feature points based on the keypoint location information corrected in step S105 (step S106).

[0045] Next, the state estimation unit 107 estimates the posture state of the subject O based on the feature quantities calculated in step S106 (step S107).

[0046] According to this second embodiment, a posture evaluation device 100A capable of evaluating posture with high accuracy can be provided. Specifically, the skeletal extraction unit 101 estimates the reliability along with keypoint position information, and the image generation unit 103 superimposes keypoints with low reliability onto the captured image in a different display manner from other keypoints. Therefore, the user can distinguish keypoints with low reliability and a high probability of incorrect estimation from other keypoints. This prevents situations where the user is unaware that keypoints have been incorrectly extracted and therefore the keypoints are not corrected. This improves the accuracy of posture evaluation. Thus, a posture evaluation device 100A capable of evaluating posture with high accuracy can be provided.

[0047] Furthermore, since the feature calculation unit 106 calculates feature quantities related to joints or anatomical feature points, it becomes possible to evaluate posture based on these feature quantities. Therefore, posture can be evaluated with a high level of accuracy comparable to that of professionals such as therapists and trainers.

[0048] Furthermore, since a predetermined threshold is determined based on the importance of key points in the state estimation unit 107's estimation, it is possible to prevent wasted time caused by the user correcting the position of key points that are not so important in the posture evaluation.

[0049] Furthermore, since the additional learning unit 108 performs additional training on the skeleton extraction model 113 using the captured image and the corrected keypoint position information, the accuracy of keypoint extraction using the skeleton extraction model 113 can be further improved.

[0050] Furthermore, if the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed when the number of keypoints with a reliability lower than a predetermined threshold exceeds a predetermined standard, a new image is acquired, and keypoint extraction is performed again based on the new image. This prevents situations where the user has to correct a large number of keypoints.

[0051] Embodiment 3 Embodiment 3 of the present invention will be described with reference to Figure 8. Figure 8 is a block diagram showing the configuration of the posture evaluation device 100B according to the present disclosure. As shown in Figure 8, the posture evaluation device 100B according to the present disclosure newly includes a spinal column extraction unit 114 and a spinal column correction unit 117, and the processing in the image generation unit 115, input reception unit 116, and feature quantity calculation unit 118 differs from the posture evaluation device 100A according to the present disclosure. For this reason, the same reference numerals are used for components of the posture evaluation device 100B that are the same as those of the posture evaluation device 100A according to the present disclosure, and their descriptions are omitted.

[0052] The spinal column extraction unit 114 estimates spinal column edge position information of a spinal column edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image, based on the position information (image coordinates) of at least the cervical vertebrae, hip joints, and knee joints of the body on the captured image. For example, the spinal column extraction unit 114 may obtain spinal column edge position information of the spinal column edge point cloud by performing edge extraction processing after trimming the area around the trunk from the captured image based on the position information (image coordinates) of the cervical vertebrae, hip joints, and knee joints. Details of the processing in the spinal column extraction unit 114 are described, for example, in Japanese Patent Application No. 2022-058198, so their explanation is omitted here. Alternatively, the spinal column extraction unit 114 may extract the silhouette of the subject from the captured image using a machine learning model such as deep learning, and identify the spinal column region from the edges of the subject's silhouette based on the position information (image coordinates) of the cervical vertebrae, hip joints, and knee joints.

[0053] The image generation unit 115 superimposes keypoints with a confidence level lower than a predetermined threshold onto the captured image in a different display manner than other keypoints with a confidence level above the predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate a display image. The image generation unit 115 also superimposes the keypoints and spinal column edge point cloud onto the captured image in a way that allows the user to move (drag) them. Figure 9 shows an example of a display image generated by the image generation unit 115. In the example shown in Figure 9, the confidence levels of keypoints P1 to P6 are above the predetermined threshold, so they are all superimposed on the captured image in the same display manner. In the example shown in Figure 9, the spinal column edge point cloud P7,... is also superimposed on the captured image. Here, among the spinal column edge point cloud P1,..., the estimation of the fourth point P7, moving from the neck side to the waist side of subject O, is incorrect. Therefore, the user can move the fourth point P7 to correct its position.

[0054] Next, the input receiving unit 116 receives keypoint correction information from the user to correct keypoint position information, similar to the second embodiment. Furthermore, the input receiving unit 116 receives spinal edge correction information from the user to correct spinal edge position information. Specifically, the input receiving unit 116 receives the position information of the spinal edge point cloud P1,... corrected to the correct position by the user dragging a point among the spinal edge point cloud P1,... on the displayed image as spinal edge correction information. Figure 10 shows a displayed image showing the spinal edge point cloud P1,... corrected by the user. The input receiving unit 116 receives the position information of the corrected spinal edge point cloud P1,... shown in Figure 10 as spinal edge correction information.

[0055] The spinal column correction unit 117 corrects the spinal column edge position information based on the spinal column edge correction information.

[0056] The feature calculation unit 118 calculates features based on the keypoint position information corrected by the skeleton correction unit 105, similar to the second embodiment. Furthermore, the feature calculation unit 118 calculates features based on the spinal column edge position information corrected by the spinal column correction unit 117. Specifically, it calculates features related to the spine based on the position information (image coordinates) of the cervical vertebrae, hip joint, and knee joint estimated by the spinal column extraction unit 114 and the spinal column edge position information corrected by the spinal column correction unit 117. The details of the spinal column feature calculation process in the feature calculation unit 118 are described, for example, in Japanese Patent Application No. 2022-058198, so a detailed explanation is omitted here.

[0057] Next, the posture evaluation method related to this disclosure will be explained with reference to Figure 11. The processes of steps S201, S202, S206, and S209 shown in Figure 11 are the same as the processes of steps S101, S102, S105, and S107 shown in Figure 7, so their explanation will be omitted. The spinal column extraction unit 114 extracts a point cloud of spinal column edges from the captured images of the subject O's body, that is, it estimates the position information of the spinal column edges (step S203).

[0058] Next, the image generation unit 115 superimposes keypoints with a reliability lower than a predetermined threshold onto the captured image in a different display manner than other keypoints with a reliability above a predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate a display image (step S204).

[0059] Next, the input receiving unit 116 receives keypoint correction information from the user to correct keypoint position information, and also receives spinal edge correction information from the user to correct spinal edge position information. (Step S205).

[0060] Furthermore, after step S206, the spinal column correction unit 117 corrects the spinal column edge position information based on the spinal column edge correction information (step S207).

[0061] Next, the feature calculation unit 118 calculates feature quantities related to joints or anatomical feature points based on the keypoint position information corrected in step S105, and also calculates feature quantities related to the spine based on the spinal edge position information corrected in step S207 (step S208).

[0062] According to this embodiment 3, posture can be evaluated based on the spinal column shape in the image, thus enabling highly accurate posture evaluation. Furthermore, while estimating spinal column shape usually requires expensive specialized equipment, this embodiment 3 allows for posture evaluation based on spinal column shape without the need for expensive specialized equipment, thus enabling relatively inexpensive posture evaluation.

[0063] Embodiment 4 Embodiment 4 of the present invention will be described with reference to Figure 12. Figure 12 is a block diagram showing the configuration of the posture evaluation device 100C according to the present disclosure. As shown in Figure 12, the processing in the skeleton extraction unit 119 and the image generation unit 120 of the posture evaluation device 100C according to the present disclosure differs from that of the posture evaluation device 100A or 100B according to the present disclosure. Therefore, among the configuration of the posture evaluation device 100C according to the present disclosure, the same reference numerals are used for components that are the same as those of the posture evaluation device 100A or 100B according to the present disclosure, and their descriptions are omitted. In Figure 12 and the following description, the differences between the posture evaluation device 100C according to the present disclosure and the posture evaluation device 100B according to the present disclosure will be mainly described.

[0064] The skeletal extraction unit 119, similar to Embodiment 3, extracts joints or anatomical feature points of the body as keypoints from the captured images obtained by imaging the body of the subject O, that is, it estimates the keypoint location information of the keypoints and the reliability of the keypoints. Furthermore, the skeletal extraction unit 119 further estimates the probability that a pixel included in the captured image is a joint or anatomical feature point. For example, when the skeletal extraction unit 101 extracts keypoints from the captured image using a machine learning model, a score is calculated indicating that a pixel included in the captured image is a joint or anatomical feature point. In this case, the score corresponds to the probability. The higher the score, the higher the probability. The probability estimated by the skeletal extraction unit 119 for some or all pixels included in the captured image may be in matrix form.

[0065] Similar to Embodiment 3, the image generation unit 120 superimposes keypoints with a reliability lower than a predetermined threshold onto the captured image in a different display manner than other keypoints with a reliability above the predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate a display image. Furthermore, the image generation unit 120 generates a tensor that displays the probability by displaying pixels with pixel values ​​corresponding to the above-mentioned probability, and then superimposes the tensor onto the captured image to generate a display image. The left side of Figure 13 shows the tensor generated by the image generation unit 120. As shown in Figure 13, in this tensor, the pixel portion P8 with a high probability (large score value) is displayed with a larger pixel value than other pixel portions. The right side of Figure 13 shows the display image with the tensor superimposed on the captured image. The display image shown on the right side of Figure 13 is the display image when user U is dragging keypoint P1 on their wrist to correct its position. As shown on the right side of Figure 13, the image generation unit 120 generates a display image with the tensor superimposed on the captured image as the display image when user U corrects keypoint P1. As a result, candidate positions P8 for correcting keypoint P1 are displayed on the display image, making it easier for user U to correct keypoint P1.

[0066] According to this embodiment 4, the skeletal extraction unit 119 further estimates the probability that a pixel included in the captured image is a joint or anatomical feature point, and the image generation unit 120 further generates a tensor that displays the probability by displaying the pixels with pixel values ​​corresponding to the above probability. Then, as a display screen for correcting keypoints, the image generation unit 120 generates a display image in which the tensor is further superimposed on the captured image. As a result, candidate positions for the keypoint correction position are displayed on the display image, making it easier for user U to correct keypoints.

[0067] Embodiment 5 The posture evaluation system 200 according to the present disclosure will be described with reference to Figure 14. Figure 14 is a diagram showing the configuration of the posture evaluation system 200 according to the present disclosure. As shown in Figure 14, the posture evaluation system 200 comprises a posture evaluation device 100C and a subject terminal 300 that can communicate with the posture evaluation device 100C. The posture evaluation device 100C and the subject terminal 300 can communicate with each other via a network N. Also, as shown in Figure 14, one or more subject terminals 300, ... may be able to communicate with the posture evaluation device 100C. Furthermore, the target devices 300 include smartphones, tablet devices, personal computers, etc., owned by the target individuals.

[0068] The posture evaluation device 100C related to this disclosure acquires captured images of the subject's body from the subject's terminal 300. Furthermore, the display image created by the image generation unit 120 of the posture evaluation device 100C is transmitted to the subject terminal 300 and displayed on the display unit 302 (described later) of the subject terminal 300.

[0069] Figure 15 shows an example of the configuration of the subject terminal 300. As shown in Figure 15, the subject terminal 300 includes an imaging unit 301, a display unit 302, an input unit 303, a display control unit 304, and a communication unit 305.

[0070] The imaging unit 301 captures images of the subject's body and acquires the captured images. The images captured by the imaging unit 301 are two-dimensional images, and may be two-dimensional RGB images. The subject terminal 300 transmits the captured images to the posture evaluation device 100C. Furthermore, the imaging unit 301 may capture images by recording a video of the subject's body. In this case, the user may specify the time to perform the posture evaluation by operating the input unit 109 of the posture evaluation device 100C. Alternatively, the subject may specify the time to perform the posture evaluation by operating the input unit 303 of the subject terminal 300. The images at the time specified by the user or subject may then be input to the skeletal extraction unit 119 and the spinal column extraction unit 114.

[0071] The display unit 302 displays the display image received from the attitude evaluation device 100C. The display unit 302 is composed of various display means such as an LCD (Liquid Crystal Display) and an LED (Light Emitting Diode).

[0072] The input unit 303 receives operation instructions from the user. The input unit 303 may be configured as a keyboard or as a touch panel display device. The input unit 303 may be configured as a keyboard or touch panel connected to the user terminal 300 main unit.

[0073] The display control unit 304 displays the display image received from the posture evaluation device 100C on the display unit 302. Specifically, the display control unit 304 displays keypoints and spinal column edge point clouds on the display unit 302 in a draggable state. The display control unit 304 also changes the ease of dragging keypoints according to the probability in the tensor. More specifically, the display control unit 304 changes the ease of dragging keypoints on the display image based on the following equation (3).

number

number

[0074] Furthermore, the display control unit 304 may display the keypoint on the display unit 302 in a draggable manner, and may also enlarge and display a predetermined range including the pixel where the probability of the tensor takes a maximum value, based on the position of the pixel where the probability of the tensor takes a maximum value and the position of the keypoint being dragged by the user. For example, as shown in Figure 16, when user U corrects keypoint P1, if the tensor is superimposed on the captured image and candidate positions P8 for the correction position of keypoint P1 are displayed, the display control unit 304 will enlarge the area around the wrist on the display unit 302. Specifically, the display control unit 304 will enlarge the area around candidate position P8 of the correction position when both of the following two conditions are met. Condition 1: The distance between the pixel position where the probability in the tensor takes a maximum value (i.e., candidate position P8) and the position of the keypoint being dragged by the user is less than or equal to a predetermined distance. Condition 2: The probability that candidate position P8 is at or above a predetermined threshold is that the distance from the keypoint being dragged by the user is less than or equal to the predetermined distance.

[0075] Furthermore, the display control unit 304 may cause the display unit 302 to always display a predetermined range centered on the position of the key point being dragged by the user, with the display magnified.

[0076] The communication unit 305 communicates with the attitude evaluation device 100C. The communication unit 305 may also communicate with external servers or other terminal devices. The communication unit 305 may be equipped with an antenna (not shown) for wireless communication, or it may be equipped with an interface such as a NIC (Network Interface Card) for wired communication.

[0077] According to this embodiment 5, the display control unit 304 changes the ease of dragging keypoints according to the probability in the tensor. Specifically, the number of pixels moved in one drag operation in the high-probability part of the tensor is less than the number of pixels moved in one drag operation in other parts. As a result, the distance moved in one drag operation at candidate position P8 is reduced, making it easier to move the keypoint being modified to candidate position P8.

[0078] Furthermore, when user U is modifying a key point, the display control unit 304 magnifies the display around candidate position P8 on the display unit 302 when the moving key point approaches candidate position P8. This makes it easier to align the key point being modified with candidate position P8.

[0079] In the embodiments described above, the present invention was explained as a hardware configuration, but this disclosure is not limited thereto. The above-described functions (processes) for the posture evaluation device 100, 100A, 100B, 100C, or the subject terminal 300 may be implemented by a computer 400 having, for example, the following configuration.

[0080] Figure 17 is a block diagram showing the configuration of a computer 400 that performs processing for posture evaluation devices 100, 100A, 100B, 100C, or the subject terminal 300. As shown in Figure 17, the computer 400 includes memory 401 and a processor 402.

[0081] Memory 401 is composed of, for example, a combination of volatile memory and non-volatile memory. Memory 401 is used to store programs executed by the processor 402, and data used for various processes. The storage unit of the posture evaluation device 100 (not shown), the storage units 110 of posture evaluation devices 100A, 100B, and 100C, and the storage unit of the subject terminal 300 (not shown) may be implemented by memory 401. However, these may also be implemented by any other storage device.

[0082] The processor 402 performs processing for each device by reading and executing a program from the memory 401. The processor 402 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 402 may include multiple processors.

[0083] In the above example, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiment. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0084] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from its spirit. For example, each component of the posture evaluation device 100, 100A, 100B, and 100C may be provided in the subject terminal or user terminal. Furthermore, the subject terminal may be equipped with a skeletal extraction model 113 and an additional learning unit 108. In this case, only the parameters of the skeletal extraction model 113 after additional learning need to be transmitted from the subject terminal to the posture evaluation devices 100A, 100B, and 100C by a method such as associative learning. Therefore, the subject does not need to transmit the captured images of their own body to an external server such as the posture evaluation devices 100A, 100B, and 100C, which is preferable from the standpoint of protecting personal information.

[0085] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0086] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0087] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) A skeletal extraction unit estimates keypoint location information of keypoints consisting of joints or anatomical feature points of the subject's body, and the reliability of said keypoints, from the captured images obtained by imaging the subject's body. An image generation unit generates a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold, A frame correction unit that corrects the key point position information based on key point correction information received from the user for correcting the key point position information, A feature quantity calculation unit calculates feature quantities relating to the joint or the anatomical feature point based on the key point position information corrected by the skeletal correction unit, A posture evaluation device equipped with the following features. (Note 2) The system further includes a state estimation unit that estimates the posture state of the subject based on the feature quantities calculated by the feature quantity calculation unit, The predetermined threshold is determined based on the importance of the key points in the estimation of the state estimation unit. The posture evaluation device described in Appendix 1. (Note 3) The system further includes a spine extraction unit that estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image. The image generation unit superimposes the keypoints whose reliability is lower than the predetermined threshold onto the captured image in a different display manner than the other keypoints whose reliability is above the predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate the display image. The system further includes a spinal column correction unit that corrects the spinal column edge position information based on spinal column edge correction information received from the user for correcting the spinal column edge position information, The feature calculation unit calculates the feature based on the keypoint position information corrected by the skeleton correction unit and the spinal column edge position information corrected by the spinal column correction unit. A posture evaluation device as described in Appendix 1 or 2. (Note 4) The skeleton extraction unit estimates the keypoint location information and the confidence level using a machine learning-based skeleton extraction model. The system further includes an additional learning unit that performs additional training on the skeleton extraction model using the captured image and the keypoint position information corrected by the skeleton correction unit. A posture evaluation device as described in any one of the appendices 1 to 3. (Note 5) The skeletal extraction unit further estimates the probability that a pixel included in the captured image is a joint or an anatomical feature point. The image generation unit further generates a tensor that displays the probability by displaying the pixels with pixel values ​​corresponding to the probability, and further superimposes the tensor onto the captured image to generate the display image. A posture evaluation device as described in any one of the appendices 1 to 4. (Note 6) The system further includes a failure determination unit that determines that the processing in the skeleton extraction unit has failed if the number of keypoints whose reliability is lower than a predetermined threshold is greater than or equal to a predetermined standard. A posture evaluation device as described in any one of the appendices 1 to 5. (Note 7) The system comprises a posture evaluation device and a subject terminal capable of communicating with the posture evaluation device. The posture evaluation device is, A skeletal extraction unit estimates keypoint location information of keypoints consisting of joints or anatomical feature points of the subject's body, and the reliability of said keypoints, from the captured images of the subject's body acquired by the subject's terminal. An image generation unit generates a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold, A frame correction unit that corrects the key point position information based on key point correction information received from the user for correcting the key point position information, A feature quantity calculation unit calculates feature quantities relating to the joint or the anatomical feature point based on the key point position information corrected by the skeletal correction unit, A posture evaluation system equipped with the following features. (Note 8) The posture evaluation device further comprises a state estimation unit that estimates the posture state of the subject based on the feature quantities calculated by the feature quantity calculation unit, The predetermined threshold is determined based on the importance of the key points in the estimation of the state estimation unit. The posture evaluation system described in Appendix 7. (Note 9) The posture evaluation device further includes a spine extraction unit that estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image. The image generation unit superimposes the keypoints whose reliability is lower than the predetermined threshold onto the captured image in a different display manner than the other keypoints whose reliability is above the predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate the display image. The posture evaluation device further comprises a spinal column correction unit that corrects the spinal column edge position information based on spinal column edge correction information received from the user for correcting the spinal column edge position information, The feature calculation unit calculates the feature based on the keypoint position information corrected by the skeleton correction unit and the spinal column edge position information corrected by the spinal column correction unit. A posture evaluation system as described in Appendix 7 or 8. (Note 10) The skeleton extraction unit estimates the keypoint location information and the confidence level using a machine learning-based skeleton extraction model. The posture evaluation device further comprises an additional learning unit that performs additional learning of the skeleton extraction model using the captured image and the keypoint position information corrected by the skeleton correction unit. A posture evaluation system described in any one of the appendices 7-9. (Note 11) The skeletal extraction unit further estimates the probability that a pixel included in the captured image is a joint or an anatomical feature point. The image generation unit further generates a tensor that displays the probability by displaying the pixels with pixel values ​​corresponding to the probability, and further superimposes the tensor onto the captured image to generate the display image. A posture evaluation system described in any one of the appendices 7-10. (Note 12) The aforementioned user terminal includes a display control unit that causes the display image generated by the image generation unit to be displayed on the display unit, The display control unit makes the keypoints draggable on the display unit and changes the ease of dragging the keypoints according to the probability in the tensor. The posture evaluation system described in Appendix 11. (Note 13) The aforementioned user terminal includes a display control unit that causes the display image generated by the image generation unit to be displayed on the display unit, The display control unit displays the keypoint on the display unit in a draggable manner, and expands and displays a predetermined range including the pixel where the probability takes a maximum value, based on the position of the pixel where the probability in the tensor takes a maximum value and the position of the keypoint being dragged by the user. A posture evaluation system as described in Appendix 11 or 12. (Note 14) The posture evaluation device further includes a failure determination unit that determines that the processing in the skeleton extraction unit has failed if the number of key points whose reliability is lower than a predetermined threshold is greater than or equal to a predetermined standard. A posture evaluation system described in any one of the appendices 7-13. (Note 15) The posture evaluation device, From the images obtained by imaging the subject's body, the keypoint location information of keypoints consisting of joints or anatomical feature points of the body, and the reliability of said keypoints are estimated. A display image is generated in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold. Based on the keypoint correction information received from the user for correcting the keypoint location information, the keypoint location information is corrected. Based on the keypoint position information corrected by the skeletal correction unit, feature quantities relating to the joint or anatomical feature point are calculated. Posture assessment methods. (Note 16) The posture evaluation device, Based on the feature quantities calculated by the feature quantity calculation unit, the posture state of the subject is estimated. The predetermined threshold is determined based on the importance of the key points in estimating the state of posture. Posture evaluation method as described in Appendix 15. (Note 17) The posture evaluation device, The spinal column edge position information of a spinal column edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image is estimated. The keypoints whose reliability is lower than the predetermined threshold are superimposed on the captured image in a different display manner from the other keypoints whose reliability is above the predetermined threshold, and the spinal column edge point cloud is further superimposed on the captured image to generate the display image. Based on the spinal edge correction information received from the user for correcting the spinal edge position information, the spinal edge position information is corrected. Based on the corrected keypoint position information and the corrected spinal column edge position information, the feature quantities are calculated. Posture evaluation method as described in Appendix 15 or 16. (Note 18) The posture evaluation device, Using a machine learning-based skeleton extraction model, the keypoint location information and the confidence level are estimated. The captured image and the corrected keypoint position information are used to perform additional training on the skeleton extraction model. The posture evaluation method described in any one of the appendices 15-17. (Note 19) The posture evaluation device, Further estimate the probability that a pixel included in the captured image is the joint or the anatomical feature point. A tensor representing the probability is further generated by displaying the aforementioned pixels with pixel values ​​corresponding to the probability, and the display image is further generated by superimposing the tensor onto the captured image. The posture evaluation method described in any one of the appendices 15-18. (Note 20) The posture evaluation device, If the number of keypoints whose confidence level is lower than the predetermined threshold is greater than or equal to a predetermined standard, it is determined that the keypoint extraction has failed. The posture evaluation method described in any one of the appendices 15-19. (Note 21) In the posture evaluation device, A process for estimating keypoint location information of keypoints consisting of joints or anatomical feature points of the subject's body, and the reliability of said keypoints, from captured images obtained by imaging the subject's body, A process for generating a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is equal to or greater than the predetermined threshold; A process for correcting the keypoint location information based on keypoint correction information received from the user for correcting the keypoint location information, A process for calculating feature quantities relating to the joint or the anatomical feature point based on the key point position information corrected by the skeletal correction unit, A program that executes something. (Note 22) In the posture evaluation device, Based on the feature quantities calculated by the feature quantity calculation unit, the process is executed to estimate the posture state of the subject. The predetermined threshold is determined based on the importance of the key points in estimating the state of posture. The program described in Appendix 21. (Note 23) In the posture evaluation device, A process for estimating spinal edge position information of a spinal edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image, A process to generate a display image by superimposing the keypoints whose reliability is lower than the predetermined threshold onto the captured image in a different display manner from the other keypoints whose reliability is equal to or greater than the predetermined threshold, and further superimposing the spinal column edge point cloud onto the captured image, A process to correct the spinal edge position information based on spinal edge correction information for correcting the spinal edge position information received from the user, A process for calculating the feature quantities based on the corrected keypoint position information and the corrected spinal column edge position information, A program described in Appendix 21 or 22 that causes the execution of the program. (Note 24) In the posture evaluation device, A process for estimating the keypoint location information and the confidence level using a machine learning-prepared skeleton extraction model, A process for performing additional training on the skeleton extraction model using the captured image and the corrected keypoint position information, The program to execute is one of the programs described in one of the appendices 21-23. (Note 25) In the posture evaluation device, A process to further estimate the probability that a pixel included in the captured image is the joint or the anatomical feature point, The process involves generating a tensor that displays the probability by displaying the aforementioned pixels with pixel values ​​corresponding to the probability, and further superimposing the tensor onto the captured image to generate the display image, The program to execute is one of the programs described in one of the appendices 21-24. (Note 26) In the posture evaluation device, If the number of keypoints whose confidence level is lower than the predetermined threshold is greater than or equal to a predetermined standard, the process determines that the extraction of the keypoints has failed. The program to execute is one of the programs described in one of the appendices 21-25.

[0088] This application claims priority based on Japanese Patent Application No. 2022-188606, filed on 25 November 2022, and incorporates all of its disclosures herein. [Industrial applicability]

[0089] We can provide a posture evaluation device, posture evaluation system, posture evaluation method, and program that can evaluate posture with high accuracy. [Explanation of Symbols]

[0090] 100, 100A, 100B, 100C Posture Evaluation Device 101, 119 Skeleton extraction section 102 Failure Judgment Department 103, 115, 120 Image generation unit 104, 116 Input reception section 105 Skeletal correction section 106, 118 Feature calculation unit 107 State Estimation Unit 108 Additional Learning Section 114 Spinal column extraction part 117 Spinal Column Correction Department 109 Input section 110 Storage section 111 Communications Department 112 Skeleton Database 113 Skeleton Extraction Model 200 Posture Evaluation System 300 target user devices 301 Imaging Unit 302 Display section 303 Input section 304 Display Control Unit 305 Communications Department

Claims

1. A skeletal extraction means that estimates keypoint location information of keypoints consisting of joints or anatomical feature points of the body, and the reliability of said keypoints, from captured images obtained by imaging the subject's body, Image generation means for generating a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold, A frame correction means for correcting the key point position information based on key point correction information received from the user for correcting the key point position information, A feature quantity calculation means that calculates feature quantities relating to the joint or the anatomical feature point based on the key point position information corrected by the skeletal correction means, Equipped with, The skeletal extraction means further estimates the probability that a pixel included in the captured image is a joint or an anatomical feature point. The posture evaluation device further generates a tensor that displays the probability by displaying the pixels with pixel values ​​corresponding to the probability, and further superimposes the tensor onto the captured image to generate the display image.

2. The system further comprises a state estimation means for estimating the posture state of the subject based on the feature quantities calculated by the feature quantity calculation means, The predetermined threshold is determined based on the importance of the key points in the estimation of the state estimation means. The posture evaluation device according to claim 1.

3. The system further comprises a spine extraction means for estimating spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing the shape of the spine on the captured image, The image generation means superimposes the keypoints whose reliability is lower than the predetermined threshold onto the captured image in a different display manner from the other keypoints whose reliability is above the predetermined threshold, and further superimposes the spinal column edge point cloud onto the captured image to generate the display image. The system further comprises a spinal column correction means for correcting the spinal column edge position information based on spinal column edge correction information received from the user for correcting the spinal column edge position information, The feature calculation means calculates the feature based on the keypoint position information corrected by the skeleton correction means and the spinal column edge position information corrected by the spinal column correction means. The posture evaluation device according to claim 1.

4. The skeleton extraction means estimates the keypoint location information and the confidence level using a machine learning-based skeleton extraction model. The system further includes an additional learning means for performing additional training on the skeleton extraction model using the captured image and the keypoint position information corrected by the skeleton correction means. The posture evaluation device according to claim 1.

5. The system further includes a failure determination means that determines that the processing in the skeleton extraction means has failed if the number of keypoints whose reliability is lower than a predetermined threshold is greater than or equal to a predetermined standard. The posture evaluation device according to claim 1.

6. The system comprises a posture evaluation device and a subject terminal capable of communicating with the posture evaluation device. The posture evaluation device is, A skeletal extraction means for estimating keypoint location information of keypoints consisting of joints or anatomical feature points of the body, and the reliability of said keypoints, from the captured images of the subject's body acquired by the subject's terminal, Image generation means for generating a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold, A frame correction means for correcting the key point position information based on key point correction information received from the user for correcting the key point position information, A feature quantity calculation means that calculates feature quantities relating to the joint or the anatomical feature point based on the key point position information corrected by the skeletal correction means, Equipped with, The skeletal extraction means further estimates the probability that a pixel included in the captured image is a joint or an anatomical feature point. The posture evaluation system further generates a tensor that displays the probability by displaying the pixels with pixel values ​​corresponding to the probability, and further superimposes the tensor onto the captured image to generate the display image.

7. The aforementioned user terminal includes a display control means that causes the display image generated by the image generation means to be displayed on the display means, The display control means makes the keypoints draggable on the display means and changes the ease of dragging the keypoints according to the probability in the tensor. The posture evaluation system according to claim 6.

8. The aforementioned user terminal includes a display control means that causes the display image generated by the image generation means to be displayed on the display means, The display control means displays the keypoint in a draggable manner on the display means, and expands and displays a predetermined range including the pixel where the probability takes a maximum value, based on the position of the pixel where the probability in the tensor takes a maximum value and the position of the keypoint being dragged by the user. The posture evaluation system according to claim 6.

9. The posture evaluation device, From the images obtained by imaging the subject's body, the keypoint location information of keypoints consisting of joints or anatomical feature points of the body, and the reliability of said keypoints are estimated. A display image is generated in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is above the predetermined threshold. Based on the keypoint correction information received from the user for correcting the keypoint location information, the keypoint location information is corrected. Based on the corrected keypoint location information, feature quantities relating to the joint or anatomical feature point are calculated. Further estimate the probability that a pixel included in the captured image is the joint or the anatomical feature point. A tensor representing the probability is further generated by displaying the aforementioned pixels with pixel values ​​corresponding to the probability, and the display image is further generated by superimposing the tensor onto the captured image. Posture assessment methods.

10. In the posture evaluation device, A process for estimating keypoint location information of keypoints consisting of joints or anatomical feature points of the subject's body, and the reliability of said keypoints, from captured images obtained by imaging the subject's body, A process for generating a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a different display manner from the other key points whose reliability is equal to or greater than the predetermined threshold; A process for correcting the keypoint location information based on keypoint correction information received from the user for correcting the keypoint location information, A process for calculating feature quantities relating to the joint or the anatomical feature point based on the keypoint position information corrected by the process for correcting the keypoint position information, A process to further estimate the probability that a pixel included in the captured image is the joint or the anatomical feature point, The process involves generating a tensor that displays the probability by displaying the aforementioned pixels with pixel values ​​corresponding to the probability, and further superimposing the tensor onto the captured image to generate the display image, A program that executes something.

Images