Posture evaluation device, posture evaluation system, posture evaluation method, and program

Abstract

Provided are a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program, with which it is possible to evaluate a posture with high accuracy. The posture evaluation device comprises: a skeleton extraction means that estimates, from a captured image obtained by imaging a subject, keypoint location information for keypoints that each comprise a joint or an anatomical feature point of a body and a reliability score for each of the keypoints; an image generation means that generates a display image obtained by superimposing a keypoint for which the reliability score is lower than a prescribed threshold value over the captured image in a display mode differing from that of another keypoint for which the reliability score is equal to or greater than the prescribed threshold value; a skeleton correction means that corrects the keypoint location information on the basis of keypoint correction information that is for correcting the keypoint location information and that has been received from a user; and a feature value calculation means that calculates a feature value relating to the joint or the anatomical feature point on the basis of the corrected keypoint location information.

Description

Posture evaluation device, posture evaluation system, posture evaluation method, and program

[0001] The present disclosure relates to a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program.

[0002] In recent years, the spread of online training and self-training has led to a growing need for ordinary people without specialized knowledge to evaluate their own posture. For example, there is a demand for technology that can evaluate a user's posture based on images of the user taken with a camera installed in a smartphone or other device.

[0003] Patent Document 1 describes a system that estimates a skeleton from an image of a subject, and classifies and searches for the subject's posture, behavior, etc. In Patent Document 1, the skeleton is estimated by extracting characteristic points such as joints as key points.

[0004] Keypoints can be extracted using machine learning models such as deep learning. However, while general machine learning models for posture evaluation can extract keypoints relatively accurately for postures such as sitting and standing, they may not be able to accurately extract keypoints for special postures such as those during exercise. If the accuracy of keypoint extraction is low, the accuracy of posture evaluation will also be low. Therefore, a technology has been developed that superimposes extracted keypoints on a captured image of the user and allows the user to modify the keypoints on the display screen.

[0005] International Publication No. 2021 / 250808

[0006] However, if the user does not notice that the keypoints have been extracted incorrectly, the keypoints may not be corrected, resulting in a low accuracy of posture estimation.

[0007] An object of the present disclosure is to provide a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program that can evaluate posture with high accuracy.

[0008] The posture assessment device according to the present disclosure includes a skeleton extraction unit that estimates, from an image obtained by imaging the body of a subject, key point position information of key points consisting of joints or anatomical landmarks of the body and the reliability of the key points; an image generation unit that generates a display image in which key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display manner that is different from that of other key points whose reliability is equal to or higher than the predetermined threshold; a skeleton correction unit that corrects the key point position information based on key point correction information for correcting the key point position information received from a user; and a feature calculation unit that calculates feature amounts related to the joints or the anatomical landmarks based on the key point position information corrected by the skeleton correction unit.

[0009] A posture evaluation system according to the present disclosure includes a posture evaluation device and a subject terminal capable of communicating with the posture evaluation device. The posture evaluation device includes: a skeleton extraction unit that estimates key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from an image of the body of the subject acquired by the subject terminal; an image generation unit that generates a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold; a skeleton correction unit that corrects the key point position information based on key point correction information for correcting the key point position information received from a user; and a feature calculation unit that calculates feature amounts related to the joints or the anatomical landmarks based on the key point position information corrected by the skeleton correction unit.

[0010] A posture assessment method according to the present disclosure is a method in which a posture assessment device estimates key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from an image obtained by imaging the body of a subject, generates a display image in which key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold, corrects the key point position information based on key point correction information for correcting the key point position information received from a user, and calculates feature amounts related to the joints or anatomical landmarks based on the key point position information corrected by the skeletal correction unit.

[0011] The program according to the present disclosure causes a posture evaluation device to execute the following processes: a process of estimating, from an image obtained by imaging the body of a subject, key point position information of key points consisting of joints or anatomical landmarks of the body and the reliability of the key points; a process of generating a display image in which the key points having the reliability lower than a predetermined threshold are superimposed on the captured image in a display manner different from that of other key points having the reliability equal to or higher than the predetermined threshold; a process of correcting the key point position information based on key point correction information for correcting the key point position information received from a user; and a process of calculating feature amounts related to the joints or the anatomical landmarks based on the key point position information corrected by the skeletal correction unit.

[0012] It is possible to provide a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program that can evaluate posture with high accuracy.

[0013] FIG. 1 is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. FIG. 1 is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. FIG. 2 is a diagram showing an example of an image captured by an imaging unit according to the present disclosure. FIG. 3 is a diagram showing an example of a display image according to the present disclosure. FIG. 4 is a diagram showing an example of a display image in which key points have been corrected according to the present disclosure. FIG. 5 is a diagram showing another example of a display image according to the present disclosure. FIG. 6 is a flowchart showing a posture evaluation method according to the present disclosure. FIG. 7 is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. FIG. 8 is a diagram showing an example of a display image according to the present disclosure. FIG. 9 is a diagram showing an example of a display image in which a spinal column edge point cloud has been corrected according to the present disclosure. FIG. 10 is a flowchart showing a posture evaluation method according to the present disclosure. FIG. 11 is a block diagram showing the configuration of a posture evaluation device according to the present disclosure. FIG. 12 is a diagram showing an example of a tensor and a display image according to the present disclosure. FIG. 13 is a block diagram showing the configuration of a posture evaluation system according to the present disclosure. FIG. 14 is a block diagram showing the configuration of a subject terminal according to the present disclosure. FIG. 15 is a diagram showing an example of a display image according to the present disclosure. FIG. 16 is a block diagram showing an example of a computer configuration according to the present disclosure.

[0014] Embodiment 1 A first embodiment of the present invention will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of a posture evaluation device 100 according to the present disclosure. The posture evaluation device 100 according to the present disclosure is a device that evaluates posture based on a captured image obtained by capturing an image of a subject's body using a camera such as a smartphone. The subject refers to a person whose posture is evaluated by the posture evaluation device 100. Specifically, the posture evaluation device 100 estimates the subject's skeleton and the like from the captured image and evaluates the subject's posture based on the skeleton and the like. This makes it possible to evaluate posture in situations such as online training and self-training.

[0015] As shown in FIG. 1, the posture evaluation device 100 includes a skeleton extraction unit 101, an image generation unit 103, an input reception unit 104, a skeleton correction unit 105, and a feature calculation unit .

[0016] The skeleton extraction unit 101 extracts body joints or anatomical landmarks as key points from captured images of the subject's body. In other words, the skeleton extraction unit 101 estimates key point position information of the key points and the reliability of the key points. In other words, "extracting key points" means "estimating key point position information and reliability." Therefore, in this specification, "estimating key points" may also be expressed. Here, anatomical landmarks include, for example, wrists, elbows, shoulders, hips, knees, and ankles. Here, the captured image obtained by capturing the subject's body is a two-dimensional image, which may be a two-dimensional RGB image. The captured image may also be an image of the side of the subject's body. The key point position information is the position information of the joints or anatomical landmarks on the image. The position information of the joints or anatomical landmarks on the image is, for example, the position information of a pixel located at the center of an image region corresponding to the joint or anatomical landmark on the image. The pixel position information is, for example, image coordinates. Here, image coordinates are coordinates for indicating the position of a pixel on a two-dimensional image. For example, the coordinates are defined as having the origin at the center of the leftmost and topmost pixel of the two-dimensional image, with the left-right or horizontal direction defined as the x-direction and the up-down or vertical direction defined as the y-direction. Furthermore, reliability is the probability that a pixel specified by keypoint position information estimated by the skeleton extraction unit 101 is a joint or an anatomical landmark. For example, when the skeleton extraction unit 101 extracts keypoints from a captured image using a machine learning model, a score indicating that the keypoint is a joint or an anatomical landmark is calculated along with the keypoint position information. In this case, the reliability is the score. The higher the score, the higher the reliability.

[0017] The image generation unit 103 generates a display image in which key points with reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points with reliability equal to or higher than the predetermined threshold. In the display image, the key points are superimposed on the captured image in a manner that allows them to be moved (dragged) by a user. Here, the term "user" refers to at least one of a subject whose posture is evaluated by the posture evaluation device 100 and an evaluator who evaluates the posture of others using the posture evaluation device 100. When a subject evaluates their own posture using the posture evaluation device 100 in self-training or the like, the subject is also the evaluator. When an evaluator evaluates the posture of others using the posture evaluation device 100, the evaluator is, for example, a therapist or trainer. The display image may be displayed on a smartphone (subject terminal) owned by the user or on a display unit (not shown) included in the posture evaluation device 100. Different display modes refer to different shapes, colors, sizes, or display formats (e.g., flashing) of key points. Furthermore, the different display mode also includes displaying a message in the displayed image indicating that extraction of any key point has failed.

[0018] The input accepting unit 104 accepts key point correction information for correcting key point position information from the user. Specifically, the input accepting unit 104 accepts, as key point correction information, position information of a key point corrected to a correct position by the user moving (dragging) the key point on the display image. Note that the movement of the key point on the display image is not limited to dragging. For example, an operation of selecting a key point to be corrected and then selecting a position to move it to may also be used.

[0019] The skeleton correction unit 105 corrects the key point position information based on the key point correction information.

[0020] The feature amount calculation unit 106 calculates feature amounts relating to joints or anatomical landmarks based on the key point position information corrected by the skeleton correction unit 105 .

[0021] According to the first embodiment, it is possible to provide a posture evaluation device 100 capable of evaluating posture with high accuracy. Specifically, the skeleton extraction unit 101 estimates the reliability along with the keypoint position information, and the image generation unit 103 superimposes keypoints with low reliability on the captured image in a display mode different from that of other keypoints. This allows the user to distinguish keypoints with low reliability and a high probability of being estimated incorrectly from other keypoints. This prevents a situation in which the user does not notice that a keypoint has been erroneously extracted and the keypoint is not corrected. This improves the accuracy of posture evaluation. Therefore, it is possible to provide a posture evaluation device 100 capable of evaluating posture with high accuracy.

[0022] Furthermore, since the feature amount calculation unit 106 calculates feature amounts related to joints or anatomical features, it becomes possible to evaluate posture based on the feature amounts, thereby enabling posture evaluation with the same degree of accuracy as that achieved by experts such as therapists and trainers.

[0023] Embodiment 2 A second embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of a posture evaluation device 100A according to the present disclosure. The posture evaluation device 100A is, for example, a server or the like capable of communicating with a user terminal such as a smartphone, tablet terminal, or personal computer owned by a user. Note that the posture evaluation device 100A may also be a user terminal (subject terminal) such as a smartphone, tablet terminal, or personal computer owned by a user.

[0024] As shown in FIG. 2 , the posture evaluation device 100A of the present disclosure includes a skeleton extraction unit 101, a failure determination unit 102, an image generation unit 103, an input reception unit 104, a skeleton correction unit 105, a feature calculation unit 106, a state estimation unit 107, an additional learning unit 108, an input unit 109, a memory unit 110, and a communication unit 111. The posture evaluation device 100A may also include a display unit (not shown). The input unit 109 and the display unit (not shown) may be configured as a single touch panel display, or may be provided separately. The memory unit 110 also stores a skeleton database (shown as "skeleton DB" in FIG. 2 ) 112, a skeleton extraction model 113, etc.

[0025] The skeleton extraction unit 101 extracts body joints or anatomical landmarks as key points from captured images of the subject's body. In other words, the skeleton extraction unit 101 estimates key point position information of the key points and the reliability of the key points. For example, the skeleton extraction unit 101 extracts wrists, elbows, shoulders, hips, knees, and ankles as key points P1 to P6 from the captured image shown in FIG. 3 , i.e., estimates key point position information and reliability of the key points P1 to P6. Details of the position information and reliability are as described in the first embodiment, and therefore will not be described again. Specifically, the skeleton extraction unit 101 estimates key point position information from the captured image using a trained skeleton extraction model 113. The posture evaluation device 100A performs machine learning in advance using the skeleton extraction model 113, which is a machine learning model, and the skeleton database 112, which is training data, to generate the trained skeleton extraction model 113. The keypoint position information may be information expressed in three-dimensional coordinates defined by the x-direction, which is the left-right or horizontal direction, the y-direction, which is the up-down or vertical direction, and the z-direction, which is the depth direction, of the two-dimensional image that is the captured image. This is possible by using the skeleton extraction model 113 that estimates the keypoint position information expressed in three-dimensional coordinates from the two-dimensional image. Furthermore, the body parts from which the skeleton extraction unit 101 extracts keypoints may be other joints such as the cervical vertebrae, hip joints, and knee joints, or anatomical landmarks such as wrists, elbows, shoulders, waists, knees, and ankles, in addition to the above.

[0026] The failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed if the number of keypoints whose reliability is lower than a predetermined threshold is equal to or greater than a predetermined criterion. Here, the predetermined criterion may be, for example, the upper limit of the number of keypoints whose reliability is lower than a predetermined threshold. Alternatively, the predetermined criterion may be, for example, the proportion of the number of keypoints whose reliability is lower than a predetermined threshold to the total number of keypoints.

[0027] The image generation unit 103 generates a display image in which key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold. In the display image, the key points are superimposed on the captured image in a manner that allows the user to move (drag) them. FIG. 4 shows an example of a display image generated by the image generation unit 103. In the example shown in FIG. 4, the reliability of the wrist key point P1 is lower than a predetermined threshold, and the wrist key point P1 is superimposed on the captured image in a shape and size different from that of the other key points P2 to P6. This makes it clear at a glance that the extraction of the wrist key point P1 is incorrect. FIG. 5 shows another example of a display image generated by the image generation unit 103. In the example shown in FIG. 5, a message M reading "Left wrist detection failed" is displayed in the display image. This makes it clear at a glance that the extraction of the left wrist key point P1 is incorrect.

[0028] Here, the predetermined threshold is a value determined based on the importance of the key point in the estimation by the state estimation unit 107, which will be described later. The predetermined threshold will be described in more detail. First, the importance of the key point k in the estimation by the state estimation unit 107 is defined as importance ｋ is defined by the following equation (1). Here, n is the number of parts to be evaluated, m is the number of features, and l is the number of key points. ｉｊ is a state estimation model f for evaluating the part i (i=1, . . . , n) to be evaluated. ｉ is the importance of feature j (j = 1, ..., m) in ｉｊ ≧0. In addition, if the feature j is not used to evaluate the part i, a ｉｊ= 0. When using linear regression as a state estimation model using permutation importance or Gini function, a ｉｊ Alternatively, the absolute value of the coefficient corresponding to each feature amount may be used as b ｊｋ is 1 if keypoint k is used to calculate feature quantity j, and is 0 if not used. ｋ is a monotonically increasing function of the importance of the feature calculated using keypoint k. ｋ ≧0). ｋ When is defined as above, a predetermined threshold thresh ｋ is defined by the following equation (2). where α ｋ and β ｋ is a constant empirically determined based on the captured image and the extraction accuracy of the machine learning model (skeleton extraction model 113) used in the skeleton extraction unit 101, and α ｋ >0. Also, α １ = α ２ =...=α ｌ , β １ = β ２ =...=β ｌ or α ｋ and β ｋ may be different for each keypoint. ｋ is the importance ｋ It may be a monotonically increasing function (range [0, 1]) of

[0029] The input accepting unit 104 accepts key point correction information for correcting key point position information from the user. Specifically, the input accepting unit 104 accepts, as key point correction information, position information of a key point corrected to a correct position by the user dragging the key point on the display image. Fig. 6 shows a display image showing a key point P1 corrected by the user. The input accepting unit 104 accepts, as key point correction information, the position information of the corrected key point P1 shown in Fig. 6.

[0030] The skeleton correction unit 105 corrects the key point position information based on the key point correction information.

[0031] The feature amount calculation unit 106 calculates feature amounts related to joints or anatomical landmarks based on the key point position information corrected by the skeleton correction unit 105. Specifically, the feature amount calculation unit 106 calculates, as feature amounts, angles formed between lines (also called "bones") connecting two or more of the key points P1 to P6, and between vertical and horizontal lines in the image.

[0032] The state estimation unit 107 estimates the posture state of the subject O based on the feature amounts calculated by the feature amount calculation unit 106. Specifically, for example, the state estimation unit 107 estimates the posture state of the subject O based on the feature amounts and a reference value list (not shown) stored in the storage unit 110. The reference value list is data in which, for example, types of postures are associated with reference values ​​of the feature amounts in the postures.

[0033] The additional learning unit 108 performs additional learning of the skeleton extraction model 113 using the captured image and the keypoint position information corrected by the skeleton correction unit 105. This makes it possible to further improve the accuracy of keypoint extraction using the skeleton extraction model 113.

[0034] The input unit 109 may receive operation instructions from a user. The input unit 109 may be configured with a keyboard or a touch panel display device. The input unit 109 may be configured with a keyboard or a touch panel connected to the main body of the posture evaluation device 100A.

[0035] The storage unit 110 stores a skeleton database 112, a skeleton extraction model 113, etc. The storage unit 110 may also include a non-volatile memory (e.g., a ROM (Read Only Memory)) in which various programs and various data required for processing are fixedly stored. The storage unit 110 may also use an HDD or an SSD. The storage unit 110 may also include a volatile memory (e.g., a RAM (Random Access Memory)) used as a working area. The above programs may be read from a portable recording medium such as an optical disc or a semiconductor memory, or may be downloaded from a server device on a network.

[0036] The skeleton database 112 is a database in which a plurality of images obtained by capturing images of the body are associated with key point position information as correct labels.

[0037] The skeleton extraction model 113 is a machine learning model that estimates keypoint position information from captured images obtained by capturing an image of a body. In other words, the skeleton extraction model 113 is a machine learning model that estimates keypoint position information using captured images obtained by capturing an image of a body as input. Note that in this specification, machine learning may be, but is not limited to, deep learning.

[0038] The communication unit 111 communicates with a subject terminal (not shown). The communication unit 111 may also communicate with an external server, another terminal device, etc. The communication unit 111 may include an antenna (not shown) for wireless communication, or may include an interface such as a NIC (Network Interface Card) for wired communication.

[0039] Next, a posture evaluation method according to the present disclosure will be described with reference to Fig. 7. First, the skeleton extraction unit 101 estimates key points from a captured image of the body of the subject O (step S101). Specifically, the skeleton extraction unit 101 estimates key point position information of key points consisting of body joints or anatomical landmarks and the reliability of the key points from the captured image.

[0040] Next, the failure determination unit 102 determines whether the number of key points whose reliability is lower than a predetermined threshold is equal to or greater than a predetermined standard (step S102). If the number of key points whose reliability is lower than a predetermined threshold is equal to or greater than the predetermined standard (step S102; Yes), the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed, and the process returns to step S101. As a result, a new captured image is acquired, and key points are extracted again based on the new captured image. On the other hand, if the number of key points whose reliability is lower than a predetermined threshold is less than the predetermined standard (step S102; No), the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has been successful, and the process proceeds to step S103.

[0041] Next, the image generating unit 103 generates a display image in which key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points whose reliability is equal to or higher than the predetermined threshold (step S103).

[0042] Next, the input receiving unit 104 receives key point correction information for correcting the key point position information from the user (step S104).

[0043] Next, the skeleton correction unit 105 corrects the key point position information based on the key point correction information (step S105).

[0044] Next, the feature calculation unit 106 calculates feature amounts relating to joints or anatomical landmarks based on the key point position information corrected in step S105 (step S106).

[0045] Next, the state estimation unit 107 estimates the posture state of the subject O based on the feature amount calculated in step S106 (step S107).

[0046] According to the second embodiment, a posture evaluation device 100A capable of evaluating posture with high accuracy can be provided. Specifically, the skeleton extraction unit 101 estimates the reliability along with the key point position information, and the image generation unit 103 superimposes key points with low reliability on the captured image in a display mode different from that of other key points. This allows the user to distinguish key points with low reliability and a high probability of being estimated incorrectly from other key points. This prevents a situation in which the user does not notice that a key point has been erroneously extracted and the key point is not corrected. This increases the accuracy of posture evaluation. Therefore, a posture evaluation device 100A capable of evaluating posture with high accuracy can be provided.

[0047] Furthermore, since the feature amount calculation unit 106 calculates feature amounts related to joints or anatomical features, it becomes possible to evaluate posture based on the feature amounts, thereby enabling posture evaluation with the same degree of accuracy as that achieved by experts such as therapists and trainers.

[0048] Furthermore, since the predetermined threshold is determined based on the importance of key points in the estimation by the state estimation unit 107, it is possible to prevent the user from wasting time by correcting the positions of key points that are not particularly important in assessing posture.

[0049] In addition, the additional learning unit 108 performs additional learning of the skeleton extraction model 113 using the captured image and the corrected keypoint position information, thereby further improving the accuracy of keypoint extraction using the skeleton extraction model 113.

[0050] Furthermore, if the number of key points whose reliability is lower than a predetermined threshold is equal to or greater than a predetermined standard, the failure determination unit 102 determines that the processing in the skeleton extraction unit 101 has failed, and a new captured image is acquired, and key points are extracted again based on the new captured image. This prevents a situation in which the user has to correct a large number of key points.

[0051] Embodiment 3 A third embodiment of the present invention will be described with reference to Fig. 8 . Fig. 8 is a block diagram showing the configuration of a posture evaluation device 100B according to the present disclosure. As shown in Fig. 8 , the posture evaluation device 100B according to the present disclosure differs from the posture evaluation device 100A according to the present disclosure in that it newly includes a spine extraction unit 114 and a spine correction unit 117, and in the processing performed by the image generation unit 115, the input reception unit 116, and the feature calculation unit 118. Therefore, among the configuration of the posture evaluation device 100B according to the present disclosure, the same components as those of the posture evaluation device 100A according to the present disclosure are denoted by the same reference numerals, and their description will be omitted.

[0052] The spine extraction unit 114 estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing the spine shape on the captured image based on position information (image coordinates) of at least the cervical vertebrae, hip joints, and knee joints of the body on the captured image. For example, the spine extraction unit 114 may trim the area around the trunk from the captured image based on position information (image coordinates) of the cervical vertebrae, hip joints, and knee joints, and then perform edge extraction processing to obtain spine edge position information of the spine edge point cloud. Details of the processing by the spine extraction unit 114 are as described, for example, in Japanese Patent Application No. 2022-058198, and therefore will not be described here. Alternatively, the spine extraction unit 114 may extract a silhouette of the subject from the captured image using a machine learning model such as deep learning, and identify the spine region from the edges of the subject's silhouette based on position information (image coordinates) of the cervical vertebrae, hip joints, and knee joints.

[0053] The image generation unit 115 superimposes key points whose reliability is lower than a predetermined threshold on the captured image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposes a spine edge point cloud on the captured image to generate a display image. The image generation unit 115 also superimposes the key points and spine edge point cloud on the captured image in a manner that allows the user to move them (drag them). Figure 9 shows an example of a display image generated by the image generation unit 115. In the example shown in Figure 9, the reliability of key points P1 to P6 is equal to or higher than a predetermined threshold, so they are all superimposed on the captured image in the same display manner. In the example shown in Figure 9, spine edge point cloud P7, ... is also superimposed on the captured image. Here, among the spine edge point cloud P1, ..., the fourth point P7, which is located from the neck side to the waist side of the subject O, is incorrectly estimated. Therefore, the user can move the fourth point P7 to correct it to the correct position.

[0054] Next, the input accepting unit 116 accepts key point correction information for correcting the key point position information from the user, as in the second embodiment. Furthermore, the input accepting unit 116 further accepts spine edge correction information for correcting the spine edge position information from the user. Specifically, the input accepting unit 116 accepts, as spine edge correction information, the position information of the spine edge point group P1, ... that has been corrected to the correct position by the user dragging a point of the spine edge point group P1, ... on the displayed image. FIG. 10 shows a display image showing the spine edge point group P1, ... corrected by the user. The input accepting unit 116 accepts, as spine edge correction information, the position information of the corrected spine edge point group P1, ... shown in FIG. 10.

[0055] The spine correction unit 117 corrects the spine edge position information based on the spine edge correction information.

[0056] As in the second embodiment, the feature calculation unit 118 calculates feature amounts based on the keypoint position information corrected by the skeleton correction unit 105. Furthermore, the feature calculation unit 118 calculates feature amounts based on the spine edge position information corrected by the spine correction unit 117. Specifically, the feature calculation unit 118 calculates feature amounts related to the spine based on the position information (image coordinates) of the cervical vertebrae, hip joints, and knee joints estimated by the spine extraction unit 114 and the spine edge position information corrected by the spine correction unit 117. Details of the calculation process of feature amounts related to the spine by the feature calculation unit 118 are as described in, for example, Japanese Patent Application No. 2022-058198, and therefore description thereof will be omitted.

[0057] Next, the posture evaluation method according to the present disclosure will be described with reference to Fig. 11. The processes of steps S201, S202, S206, and S209 shown in Fig. 11 are similar to the processes of steps S101, S102, S105, and S107 shown in Fig. 7, and therefore description thereof will be omitted. The spine extraction unit 114 extracts a spine edge point cloud from a captured image of the body of the subject O, i.e., estimates spine edge position information (step S203).

[0058] Next, the image generation unit 115 superimposes key points whose reliability is lower than a predetermined threshold on the captured image in a display manner different from that of other key points whose reliability is higher than the predetermined threshold, and further superimposes the spinal column edge point cloud on the captured image to generate a display image (step S204).

[0059] Next, the input receiving unit 116 receives key point correction information for correcting the key point position information from the user, and also receives spine edge correction information for correcting the spine edge position information from the user (step S205).

[0060] After step S206, the spine correction unit 117 corrects the spine edge position information based on the spine edge correction information (step S207).

[0061] Next, the feature calculation unit 118 calculates feature amounts related to the joints or anatomical landmarks based on the keypoint position information corrected in step S105, and calculates feature amounts related to the spine based on the spine edge position information corrected in step S207 (step S208).

[0062] According to the third embodiment, posture can be evaluated based on the spinal column shape on an image, and therefore posture can be evaluated with high accuracy. Furthermore, while expensive specialized equipment is usually required to estimate the spinal column shape, according to the third embodiment, posture can be evaluated based on the spinal column shape without using expensive specialized equipment, and therefore posture can be evaluated relatively inexpensively.

[0063] Embodiment 4 A fourth embodiment of the present invention will be described with reference to FIG. 12 . FIG. 12 is a block diagram showing the configuration of a posture evaluation device 100C according to the present disclosure. As shown in FIG. 12 , the posture evaluation device 100C according to the present disclosure differs from the posture evaluation device 100A or 100B according to the present disclosure in the processing of the skeleton extraction unit 119 and the image generation unit 120. Therefore, among the configuration of the posture evaluation device 100C according to the present disclosure, the same components as those of the posture evaluation device 100A or 100B according to the present disclosure are denoted by the same reference numerals, and their description will be omitted. Note that FIG. 12 and the following description will mainly describe the differences between the posture evaluation device 100C according to the present disclosure and the posture evaluation device 100B according to the present disclosure.

[0064] As in the third embodiment, the skeleton extraction unit 119 extracts body joints or anatomical landmarks as key points from captured images obtained by capturing images of the body of the subject O, i.e., estimates key point position information of the key points and the reliability of the key points. The skeleton extraction unit 119 also estimates the probability that pixels included in the captured images are joints or anatomical landmarks. For example, when the skeleton extraction unit 101 extracts key points from the captured images using a machine learning model, a score is calculated indicating that the pixels included in the captured images are joints or anatomical landmarks. In this case, the score corresponds to the probability. The larger the score value, the higher the probability. The probability of some or all pixels included in the captured images estimated by the skeleton extraction unit 119 may be in the form of a matrix.

[0065] As in the third embodiment, the image generation unit 120 superimposes key points whose reliability is lower than a predetermined threshold on the captured image in a display mode different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposes a spinal column edge point cloud on the captured image to generate a display image. The image generation unit 120 also generates a tensor that displays the probability by displaying pixels with pixel values ​​corresponding to the above-mentioned probability, and further superimposes the tensor on the captured image to generate a display image. The left side of Figure 13 shows the tensor generated by the image generation unit 120. As shown in Figure 13, in the tensor, a pixel portion P8 with a high probability (large score value) is displayed with a larger pixel value than other pixel portions. The right side of Figure 13 shows a display image in which the tensor is superimposed on the captured image. The display image shown on the right side of Figure 13 is a display image when the user U is dragging the wrist key point P1 to correct the wrist key point P1 to its correct position. 13, the image generating unit 120 generates a display image in which the tensor is further superimposed on the captured image as a display image when the user U corrects the key point P1. As a result, a candidate position P8 for correcting the key point P1 is displayed on the display image, allowing the user U to more easily correct the key point P1.

[0066] According to the fourth embodiment, the skeleton extraction unit 119 further estimates the probability that a pixel included in the captured image is a joint or an anatomical landmark, and the image generation unit 120 further generates a tensor that displays the probability by displaying the pixel with a pixel value according to the above-mentioned probability. Then, the image generation unit 120 generates a display image in which the tensor is further superimposed on the captured image as a display screen for correcting the key point. Therefore, candidate positions for correcting the key point are displayed on the display image, allowing the user U to correct the key point more easily.

[0067] Fifth Embodiment A posture evaluation system 200 according to the present disclosure of the present invention will be described with reference to FIG. 14 . FIG. 14 is a diagram showing the configuration of the posture evaluation system 200 according to the present disclosure. As shown in FIG. 14 , the posture evaluation system 200 includes a posture evaluation device 100C and a subject terminal 300 capable of communicating with the posture evaluation device 100C. The posture evaluation device 100C and the subject terminal 300 are capable of communicating with each other via a network N. Also, as shown in FIG. 14 , one or more subject terminals 300, ... may be capable of communicating with the posture evaluation device 100C. Furthermore, the subject terminal 300 is a smartphone, tablet terminal, personal computer, etc. owned by the subject.

[0068] The posture evaluation device 100C according to the present disclosure acquires a captured image of the subject's body from the subject's terminal 300. Furthermore, the display image created by the image generation unit 120 of the posture evaluation device 100C is transmitted to the subject's terminal 300 and displayed on a display unit 302 (described later) of the subject's terminal 300.

[0069] 15 shows an example of the configuration of the subject terminal 300. As shown in FIG. 15, the subject terminal 300 includes an imaging unit 301, a display unit 302, an input unit 303, a display control unit 304, and a communication unit 305.

[0070] The imaging unit 301 captures an image of the subject's body to obtain the captured image. The image captured by the imaging unit 301 is a two-dimensional image, and may be a two-dimensional RGB image. The subject's terminal 300 transmits the captured image to the posture evaluation device 100C. The imaging unit 301 may also capture a video of the subject's body to obtain the captured image. In this case, the user may specify the time point at which posture evaluation is to be performed by operating the input unit 109 of the posture evaluation device 100C. Alternatively, the subject may specify the time point at which posture evaluation is to be performed by operating the input unit 303 of the subject's terminal 300. Then, the image at the time point specified by the user or the subject may be input to the skeleton extraction unit 119 and the spine extraction unit 114.

[0071] The display unit 302 displays the display image received from the posture evaluation device 100 C. The display unit 302 is configured with various display means such as an LCD (Liquid Crystal Display) or an LED (Light Emitting Diode).

[0072] The input unit 303 receives operation instructions from the subject. The input unit 303 may be configured with a keyboard or a touch panel display device. The input unit 303 may be configured with a keyboard or a touch panel connected to the subject terminal 300 main body.

[0073] The display control unit 304 displays the display image received from the posture evaluation device 100C on the display unit 302. Specifically, the display control unit 304 displays the key points and spine edge point cloud in a draggable manner on the display unit 302. The display control unit 304 also changes the ease of dragging the key points according to the probability in the tensor. More specifically, the display control unit 304 changes the ease of dragging the key points on the display image based on the following equation (3): Here, Δx ｐｉｘｅｌ is the number of pixels that the key point moves by one drag operation, and p is the probability in the tensor, i.e., the probability that a pixel included in the captured image is a joint or an anatomical landmark. ０ is an integer greater than 0, and v is a real number greater than 0. is (v ０ -vp). In other words, equation (3) expresses the number of pixels Δx that the key point moves by one drag operation. ｐｉｘｅｌ is a function of the probability p in the tensor. Specifically, the number of pixels moved by one drag operation in the part of the tensor with a high probability is smaller than the number of pixels moved by one drag operation in other parts. Also, the number of pixels Δx that the key point moves by one drag operation is ｐｉｘｅｌ is a strictly monotonically decreasing function of probability p, and Δx ｐｉｘｅｌ may be a step function where .times. ...

[0074] The display control unit 304 may also display the key point on the display unit 302 so that it can be dragged, and may enlarge and display a predetermined range including the pixel where the probability is at a local maximum based on the position of the pixel where the probability in the tensor is at a local maximum and the position of the key point being dragged by the user. For example, as shown in FIG. 16 , when a tensor is superimposed on a captured image and a candidate position P8 for the correction position of the key point P1 is displayed as a display image when the user U corrects the key point P1, the display control unit 304 may enlarge and display an area around the wrist on the display unit 302. Specifically, the display control unit 304 performs an enlarged display near the candidate position P8 for the correction position when both of the following two conditions are satisfied: Condition 1: The distance between the position of the pixel where the probability in the tensor is at a local maximum (i.e., candidate position P8) and the position of the key point being dragged by the user is equal to or less than a predetermined distance; and Condition 2: The probability of the candidate position P8, where the distance from the position of the key point being dragged by the user is equal to or less than the predetermined distance, is equal to or greater than a predetermined threshold.

[0075] Furthermore, the display control unit 304 may cause the display unit 302 to constantly enlarge and display a predetermined range centered on the position of the key point being dragged by the user.

[0076] The communication unit 305 communicates with the posture evaluation device 100C. The communication unit 305 may also communicate with an external server, another terminal device, etc. The communication unit 305 may include an antenna (not shown) for wireless communication, or may include an interface such as a network interface card (NIC) for wired communication.

[0077] According to the fifth embodiment, the display control unit 304 changes the ease of dragging the keypoint according to the probability of the tensor. Specifically, the number of pixels moved by one drag operation in a part of the tensor with a high probability is smaller than the number of pixels moved by one drag operation in other parts. This reduces the distance moved by one drag operation at candidate position P8, making it easier to move the keypoint being modified to candidate position P8.

[0078] Furthermore, when the user U is correcting a key point and the moving key point approaches candidate position P8, the display control unit 304 causes the display unit 302 to enlarge and display the area around candidate position P8, making it easier to align the position of the key point being corrected with candidate position P8.

[0079] In the above-described embodiment, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The above-described functions (processing) of the posture evaluation device 100, 100A, 100B, 100C, or the subject terminal 300 may be realized by a computer 400 having the following configuration, for example.

[0080] 17 is a block diagram showing the configuration of a computer 400 that realizes the processing of the posture evaluation device 100, 100A, 100B, or 100C, or the subject terminal 300. As shown in FIG. 17 , the computer 400 includes a memory 401 and a processor 402.

[0081] The memory 401 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The memory 401 is used to store programs executed by the processor 402, data used for various processes, and the like. The memory unit (not shown) of the posture evaluation device 100, the memory unit 110 of the posture evaluation devices 100A, 100B, and 100C, and the memory unit (not shown) of the subject terminal 300 may be realized by the memory 401. However, these may also be realized by any other storage device.

[0082] The processor 402 performs processing of each device by reading and executing programs from the memory 401. The processor 402 may be, for example, a microprocessor, a microprocessor unit (MPU), or a central processing unit (CPU). The processor 402 may include multiple processors.

[0083] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0084] The present disclosure is not limited to the above-described embodiments and may be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, each component of the posture evaluation devices 100, 100A, 100B, and 100C may be provided in a subject terminal or a user terminal. Furthermore, the subject terminal may be provided with the skeleton extraction model 113 and the additional learning unit 108. In this case, only the parameters of the skeleton extraction model 113 after additional learning may be transmitted from the subject terminal to the posture evaluation devices 100A, 100B, and 100C by a method such as associative learning. Therefore, the subject does not need to transmit captured images of his or her body to the posture evaluation devices 100A, 100B, and 100C, which are external servers, which is preferable from the perspective of protecting personal information.

[0085] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0086] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0087] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes: (Supplementary Note 1) A posture evaluation device comprising: a skeleton extraction unit that estimates key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from a captured image obtained by capturing an image of the body of a subject; an image generation unit that generates a display image in which the key points, the reliability of which is lower than a predetermined threshold, are superimposed on the captured image in a display manner different from that of other key points, the reliability of which is equal to or higher than the predetermined threshold; a skeleton correction unit that corrects the key point position information based on key point correction information for correcting the key point position information received from a user; and a feature calculation unit that calculates feature amounts related to the joints or the anatomical landmarks based on the key point position information corrected by the skeleton correction unit. (Supplementary Note 2) The posture evaluation device according to Supplementary Note 1, further comprising a state estimation unit that estimates a posture state of the subject based on the feature calculated by the feature calculation unit, wherein the predetermined threshold is determined based on the importance of the key point in the estimation by the state estimation unit. (Supplementary Note 3) The posture assessment device according to Supplementary Note 1 or 2, further comprising: a spine extraction unit that estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points that represent the spine shape on the captured image; the image generation unit superimposes on the captured image the key points whose reliability is lower than the predetermined threshold in a display manner that is different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposes the spine edge point cloud on the captured image to generate the display image; and a spine correction unit that corrects the spine edge position information based on spine edge correction information for correcting the spine edge position information received from the user; and the feature calculation unit calculates the feature based on the key point position information corrected by the skeleton correction unit and the spine edge position information corrected by the spine correction unit.(Supplementary Note 4) The posture evaluation device according to any one of Supplements 1 to 3, wherein the skeleton extraction unit estimates the keypoint position information and the reliability using a skeleton extraction model that has undergone machine learning, and further comprises an additional learning unit that performs additional learning of the skeleton extraction model using the captured image and the keypoint position information corrected by the skeleton correction unit. (Supplementary Note 5) The posture evaluation device according to any one of Supplements 1 to 4, wherein the skeleton extraction unit further estimates a probability that a pixel included in the captured image is the joint or the anatomical landmark, and the image generation unit further generates a tensor that indicates the probability by displaying the pixel with a pixel value according to the probability, and further superimposes the tensor on the captured image to generate the display image. (Supplementary Note 6) The posture evaluation device according to any one of Supplements 1 to 5, further comprises a failure determination unit that determines that processing in the skeleton extraction unit has failed if the number of keypoints whose reliability is lower than the predetermined threshold is equal to or greater than a predetermined standard. (Supplementary Note 7) A posture evaluation system comprising: a posture evaluation device; and a subject terminal capable of communicating with the posture evaluation device, wherein the posture evaluation device comprises: a skeleton extraction unit that estimates key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from an image of the body of the subject acquired by the subject terminal; an image generation unit that generates a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold; a skeleton correction unit that corrects the key point position information based on key point correction information for correcting the key point position information received from a user; and a feature calculation unit that calculates feature amounts related to the joints or the anatomical landmarks based on the key point position information corrected by the skeleton correction unit. (Supplementary Note 8) The posture evaluation system according to Supplementary Note 7, wherein the posture evaluation device further includes a state estimation unit that estimates a posture state of the subject based on the feature calculated by the feature calculation unit, and the predetermined threshold is determined based on the importance of the key point in the estimation by the state estimation unit.(Supplementary Note 9) The posture evaluation system according to Supplementary Note 7 or 8, wherein the posture evaluation device further comprises a spine extraction unit that estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points that represent a spine shape on the captured image, the image generation unit superimposing the key points whose reliability is lower than the predetermined threshold on the captured image in a display manner that is different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposing the spine edge point cloud on the captured image to generate the display image, the posture evaluation device further comprises a spine correction unit that corrects the spine edge position information based on spine edge correction information for correcting the spine edge position information received from the user, and the feature calculation unit calculates the feature based on the key point position information corrected by the skeletal correction unit and the spine edge position information corrected by the spine correction unit. (Supplementary Note 10) The posture assessment system according to any one of Supplementary Notes 7 to 9, wherein the skeleton extraction unit estimates the keypoint position information and the reliability using a skeleton extraction model that has undergone machine learning, and the posture assessment device further comprises an additional learning unit that performs additional learning of the skeleton extraction model using the captured image and the keypoint position information corrected by the skeleton correction unit. (Supplementary Note 11) The posture assessment system according to any one of Supplementary Notes 7 to 10, wherein the skeleton extraction unit further estimates a probability that a pixel included in the captured image is the joint or the anatomical landmark, and the image generation unit further generates a tensor that indicates the probability by displaying the pixel with a pixel value according to the probability, and further superimposes the tensor on the captured image to generate the display image. (Supplementary Note 12) The posture evaluation system according to Supplementary Note 11, wherein the subject terminal includes a display control unit that causes the display image generated by the image generation unit to be displayed on a display unit, and the display control unit causes the key points to be displayed on the display unit in a draggable manner and changes the ease of dragging the key points according to the probability in the tensor.(Supplementary Note 13) The posture evaluation system according to Supplementary Note 11 or 12, wherein the subject terminal comprises a display control unit that causes a display unit to display the display image generated by the image generation unit, and the display control unit causes the display unit to display the key points in a draggable manner, and also causes the display unit to enlarge and display a predetermined range including the pixel where the probability has a maximum value, based on the position of the pixel where the probability in the tensor has a maximum value and the position of the key point being dragged by the user. (Supplementary Note 14) The posture evaluation system according to any one of Supplementary Notes 7 to 13, wherein the posture evaluation device further comprises a failure determination unit that determines that processing in the skeleton extraction unit has failed when the number of the key points whose reliability is lower than the predetermined threshold is equal to or greater than a predetermined standard. (Supplementary Note 15) A posture evaluation method, in which a posture evaluation device estimates key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from an image obtained by imaging the body of the subject, generates a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold, corrects the key point position information based on key point correction information for correcting the key point position information received from a user, and calculates feature amounts related to the joints or anatomical landmarks based on the key point position information corrected by the skeleton correction unit. (Supplementary Note 16) A posture evaluation method according to Supplementary Note 15, in which a posture evaluation device estimates a posture state of the subject based on the feature amounts calculated by the feature calculation unit, and the predetermined threshold is decided based on the importance of the key points in estimating the posture state.(Supplementary Note 17) The posture evaluation method according to Supplementary Note 15 or 16, wherein the posture evaluation device estimates spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing a spine shape on the captured image, superimposes on the captured image the key points whose reliability is lower than the predetermined threshold in a display mode different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposes the spine edge point cloud on the captured image to generate the display image, corrects the spine edge position information based on spine edge correction information for correcting the spine edge position information received from the user, and calculates the feature amount based on the corrected key point position information and the corrected spine edge position information. (Supplementary Note 18) The posture evaluation method according to any one of Supplementary Notes 15 to 17, wherein the posture evaluation device estimates the key point position information and the reliability using a skeleton extraction model that has been trained on machine learning, and performs additional learning of the skeleton extraction model using the captured image and the corrected key point position information. (Supplementary Note 19) The posture evaluation method according to any one of Supplementary Notes 15 to 18, wherein the posture evaluation device further estimates a probability that a pixel included in the captured image is the joint or the anatomical landmark, further generates a tensor indicating the probability by displaying the pixel with a pixel value according to the probability, and further superimposes the tensor on the captured image to generate the display image. (Supplementary Note 20) The posture evaluation method according to any one of Supplementary Notes 15 to 19, wherein the posture evaluation device determines that extraction of the keypoint has failed if the number of the keypoints whose reliability is lower than the predetermined threshold is equal to or greater than a predetermined standard.(Supplementary Note 21) A program causing a posture evaluation device to execute the following processes: estimating key point position information of key points consisting of joints or anatomical landmarks of the body and reliability of the key points from an image obtained by imaging the body of the subject, generating a display image in which the key points whose reliability is lower than a predetermined threshold are superimposed on the captured image in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold, correcting the key point position information based on key point correction information for correcting the key point position information received from a user, and calculating feature amounts related to the joints or anatomical landmarks based on the key point position information corrected by the skeleton correction unit. (Supplementary Note 22) The program according to Supplementary Note 21, causing a posture evaluation device to execute a process of estimating a posture state of the subject based on the feature amounts calculated by the feature amount calculation unit, wherein the predetermined threshold is determined based on the importance of the key points in estimating the posture state. (Supplementary Note 23) The program according to Supplementary Note 21 or 22, causing a posture assessment device to execute the following processes: a process of estimating spine edge position information of a spine edge point cloud consisting of a predetermined number of points representing the spine shape on the captured image; a process of superimposing, on the captured image, the key points whose reliability is lower than the predetermined threshold in a display manner different from that of other key points whose reliability is equal to or higher than the predetermined threshold, and further superimposing the spine edge point cloud on the captured image to generate the display image; a process of correcting the spine edge position information based on spine edge correction information for correcting the spine edge position information received from the user; and a process of calculating the feature amount based on the corrected key point position information and the corrected spine edge position information. (Supplementary Note 24) The program described in any one of Supplementary Notes 21 to 23, which causes a posture evaluation device to execute: a process of estimating the keypoint position information and the reliability using a skeleton extraction model that has undergone machine learning; and a process of performing additional learning of the skeleton extraction model using the captured image and the corrected keypoint position information.(Supplementary Note 25) The program according to any one of Supplements 21 to 24, causing a posture evaluation device to execute the following processes: further estimating the probability that a pixel included in the captured image is the joint or the anatomical landmark; further generating a tensor that indicates the probability by displaying the pixel with a pixel value according to the probability, and further superimposing the tensor on the captured image to generate the display image. (Supplementary Note 26) The program according to any one of Supplements 21 to 25, causing a posture evaluation device to execute the following processes:

[0088] This application claims priority based on Japanese Patent Application No. 2022-188606, filed November 25, 2022, the disclosure of which is incorporated herein in its entirety.

[0089] It is possible to provide a posture evaluation device, a posture evaluation system, a posture evaluation method, and a program that can evaluate posture with high accuracy.

[0090] 100, 100A, 100B, 100C Posture evaluation device 101, 119 Skeleton extraction unit 102 Failure determination unit 103, 115, 120 Image generation unit 104, 116 Input reception unit 105 Skeleton correction unit 106, 118 Feature calculation unit 107 State estimation unit 108 Additional learning unit 114 Spine extraction unit 117 Spine correction unit 109 Input unit 110 Storage unit 111 Communication unit 112 Skeleton DB (skeleton database) 113 Skeleton extraction model 200 Posture evaluation system 300 Subject terminal 301 Imaging unit 302 Display unit 303 Input unit 304 Display control unit 305 Communication unit

Claims

1. Skeleton extraction means for estimating, from a captured image obtained by imaging the body of a subject, the key point position information of key points consisting of joints or anatomical feature points of the body and the reliability of the key points; Image generation means for generating a display image in which the key points with a reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points with a reliability equal to or higher than the predetermined threshold; Skeleton correction means for correcting the key point position information based on key point correction information received from a user for correcting the key point position information; Feature quantity calculation means for calculating a feature quantity related to the joint or the anatomical feature point based on the key point position information corrected by the skeleton correction means; A posture evaluation device comprising:

2. Further comprising state estimation means for estimating the state of the posture of the subject based on the feature quantity calculated by the feature quantity calculation means, The predetermined threshold is determined based on the importance of the key points in the estimation by the state estimation means, The posture evaluation device according to claim 1.

3. Further comprising spine extraction means for estimating spine edge position information of a spine edge point group consisting of a predetermined number of points representing the spine shape on the captured image, The image generation means superimposes the key points with a reliability lower than the predetermined threshold on the captured image in a display mode different from that of other key points with a reliability equal to or higher than the predetermined threshold, and further superimposes the spine edge point group on the captured image to generate the display image, Further comprising spine correction means for correcting the spine edge position information based on spine edge correction information received from the user for correcting the spine edge position information, The feature quantity calculation means calculates the feature quantity based on the key point position information corrected by the skeleton correction means and the spine edge position information corrected by the spine correction means, The posture evaluation device according to claim 1.

4. The skeleton extraction means estimates the key point position information and the reliability using a pre-trained skeleton extraction model, Further comprising additional learning means for performing additional learning of the skeleton extraction model using the captured image and the key point position information corrected by the skeleton correction means, The posture evaluation device according to claim 1.

5. The skeleton extraction means further estimates the probability that a pixel included in the captured image is the joint or the anatomical feature point, The image generation means further generates a tensor for displaying the probability by displaying the pixel with a pixel value corresponding to the probability, and further superimposes the tensor on the captured image to generate the display image. The posture evaluation device according to claim 1.

6. Further comprising a failure determination means for determining that the processing in the skeleton extraction means has failed when the number of the key points whose reliability is lower than the predetermined threshold is equal to or more than a predetermined standard. The posture evaluation device according to claim 1.

7. A posture evaluation system comprising a posture evaluation device and a subject terminal capable of communicating with the posture evaluation device, The posture evaluation device is a skeleton extraction means for estimating key point position information of a key point composed of a joint or an anatomical feature point of the body and the reliability of the key point from a captured image of the body of the subject acquired by the subject terminal; an image generation means for generating a display image in which the key points with a reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points with a reliability equal to or higher than the predetermined threshold; a skeleton correction means for correcting the key point position information based on key point correction information for correcting the key point position information received from a user; a feature amount calculation means for calculating a feature amount related to the joint or the anatomical feature point based on the key point position information corrected by the skeleton correction means; A posture evaluation system comprising:

8. The skeleton extraction means further estimates the probability that a pixel included in the captured image is the joint or the anatomical feature point, The image generation means further generates a tensor for displaying the probability by displaying the pixel with a pixel value corresponding to the probability, and further superimposes the tensor on the captured image to generate the display image. The posture evaluation system according to claim 7.

9. The subject terminal includes a display control means for causing a display means to display the display image generated by the image generation means, The display control means causes the display means to display the key points so that they can be dragged, and changes the ease of dragging the key points according to the probability in the tensor. The posture evaluation system according to claim 8.

10. The target terminal includes a display control means for causing the display means to display the display image generated by the image generation means. The display control means causes the display means to display the key points in a draggable manner, and based on the position of the pixel at which the probability in the tensor takes a maximum value and the position of the key point being dragged by the user, expands and displays a predetermined range including the pixel at which the probability takes a maximum value. The posture evaluation system according to claim 8.

11. A posture evaluation device estimates key point position information of key points composed of joints or anatomical feature points of the body and the reliability of the key points from a captured image obtained by capturing an image of the body of a subject, generates a display image in which the key points with a reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points with a reliability equal to or higher than the predetermined threshold, corrects the key point position information based on key point correction information received from the user for correcting the key point position information, calculates a feature amount related to the joint or the anatomical feature point based on the corrected key point position information. A posture evaluation method.

12. In a posture evaluation device a process of estimating key point position information of key points composed of joints or anatomical feature points of the body and the reliability of the key points from a captured image obtained by capturing an image of the body of a subject, a process of generating a display image in which the key points with a reliability lower than a predetermined threshold are superimposed on the captured image in a display mode different from that of other key points with a reliability equal to or higher than the predetermined threshold, a process of correcting the key point position information based on key point correction information received from the user for correcting the key point position information, a process of calculating a feature amount related to the joint or the anatomical feature point based on the key point position information corrected by the process of correcting the key point position information, A program for causing the above to be executed.

Images