Determination method, determination device, and determination system

Abstract

A determination method according to this embodiment includes: an estimation step (S101) for estimating, on the basis of an image containing a target who performs a specified activity as a subject, a skeleton model of the target in the image; a setting step (S102) for setting a plurality of three-dimensional areas around the skeleton model on the basis of positions of a plurality of skeleton points in the skeleton model; and a specification step (S103) for specifying, among the plurality of three-dimensional areas, the three-dimensional area in which a skeleton point of a wrist among the plurality of skeleton points is positioned in the specified activity; and a determination step (S104) for determining a level of activities of daily living that the target can perform on the basis of the three-dimensional area specified in the specification step.

Description

[Technical field]

[0001] The present invention relates to a determination method, a determination device, and a determination system. [Background technology]

[0002] Traditionally, nursing care facilities provide services that train elderly people (so-called rehabilitation) so that they can live independently. Staff at nursing care facilities who are qualified to create training plans visit the homes of elderly people, assess the elderly people's physical functions and the state of their Activities of Daily Living (ADL), and create training plans according to the state of their ADL. Rehabilitation is carried out according to the created training plans.

[0003] For example, Patent Document 1 discloses a motion information processing device that acquires motion information of a subject performing a specified motion in a rehabilitation evaluation, analyzes the acquired motion information, and displays information based on analytical values ​​regarding the movement of a specified part of the body. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-061579 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to create an effective rehabilitation training plan for a patient, it is necessary to accurately assess the state of the patient's activities of daily living. It is also desirable to be able to assess the state of the patient's activities of daily living easily.

[0006] The present invention provides a method, device, and system for easily and accurately determining the state of a subject's activities of daily living. [Means for solving the problem]

[0007] A determination method according to one aspect of the present invention is a determination method executed by a computer, and includes the following steps: an estimation step of estimating a skeletal model of a subject in an image based on the image containing the subject performing a specific movement; a setting step of setting a plurality of three-dimensional regions around the skeletal model based on positions of a plurality of skeletal points in the skeletal model; an identification step of identifying a three-dimensional region among the plurality of three-dimensional regions in which a wrist skeletal point is located during the specific movement; and a determination step of determining a level of daily living activities that the subject can perform based on the three-dimensional region identified in the identification step.

[0008] In addition, a determination device according to one embodiment of the present invention includes an estimation unit that estimates a skeletal model of a subject in an image based on the image containing the subject performing a specific movement; a setting unit that sets multiple three-dimensional regions around the skeletal model based on positions of multiple skeletal points in the skeletal model; an identification unit that identifies, among the multiple three-dimensional regions, a three-dimensional region in which a wrist skeletal point is located during the specific movement; and a determination unit that determines a level of daily living activities that the subject can perform based on the three-dimensional region identified by the identification unit.

[0009] Moreover, a judgment system according to one embodiment of the present invention includes the judgment device described above and an information terminal, wherein the judgment device further includes a first communication unit that communicates with the information terminal, an acquisition unit that acquires the image from the information terminal via the first communication unit, and an output unit that outputs a judgment result by the judgment unit to the information terminal via the first communication unit, and the information terminal includes a second communication unit that communicates with the judgment device, an instruction unit that instructs the subject to perform the specific action, a camera that generates the image by photographing the subject performing the specific action, a control unit that outputs the image to the judgment device via the second communication unit and acquires the judgment result from the judgment device via the second communication unit, and a presentation unit that presents the judgment result. Effect of the Invention

[0010] According to the present invention, a method, device, and system for determining the state of a subject's activities of daily living can be realized simply and accurately. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of a determination system according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a skeletal model of a subject estimated by an estimation unit according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining a three-dimensional region set by the setting unit according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining a three-dimensional region identified by the identification unit according to the embodiment. [Diagram 5] FIG. 5 is a diagram showing a specific example of the determination criteria of the determination unit according to the embodiment. [Figure 6] FIG. 6 is a diagram showing a specific example of a determination result by a determination unit presented by a presentation unit according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating a processing procedure of the determination device according to the embodiment. [Figure 8]FIG. 8 is a flowchart illustrating a processing procedure of the determination system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0013] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In addition, in each drawing, the same reference numerals are used for substantially the same configuration, and duplicated explanations may be omitted or simplified.

[0014] (Embodiment) [composition] First, the configuration of a determination system according to an embodiment will be described.

[0015] FIG. 1 is a block diagram showing a functional configuration of a determination system 10 according to an embodiment.

[0016] The determination system 10 is a system that determines the level of activities of daily living (ADL) that a subject can perform, based on an image that includes the subject performing a specific movement as a subject (that is, an image in which the subject appears).

[0017] The determination system 10 includes an information terminal 30 and a determination device 40.

[0018] The user photographs the subject by, for example, operating the information terminal 30. The image (more specifically, a moving image) generated thereby is transmitted to the determination device 40. The determination device 40 determines the level of the activities of daily living that the subject included as a subject in the image can perform based on the received image. This determination result is transmitted to the information terminal 30 and is presented to the user by the information terminal 30. The determination device 40 evaluates the level of the activities of daily living that the subject can perform in multiple stages, for example, using a five-stage evaluation from 1 to 5, numerical values ​​such as 0%, 50%, 75%, or 100%, or symbols such as A, B, or C.

[0019] Here, the subject is a person whose degree of executable daily living activities is being determined, for example, a person whose physical function, i.e., the ability to move the body, has been reduced due to illness, injury, aging, or disability.

[0020] Furthermore, the user may be, for example, a physical therapist, an occupational therapist, a nurse, or a rehabilitation specialist.

[0021] Activities of daily living are the minimum daily activities necessary to lead a normal life, such as getting up and down, transferring, moving, eating, changing clothes (e.g. putting on and taking off shoes and putting on clothes), excretion, bathing (e.g. washing hair), and grooming.

[0022] The specific movement is a movement related to the activities of daily living. For example, the specific movement is a movement that is common to or similar to at least a part of the movements included in the activities of daily living. Specific examples of the specific movement will be described later.

[0023] The information terminal 30 is a computer that instructs the subject to perform a specific action, acquires images (image data) including the subject as a subject, which are generated by photographing the subject with the camera 20, and transmits the acquired images to the determination device 40. In this embodiment, the information terminal 30 generates a moving image composed of a plurality of images by photographing the subject, and transmits the generated moving image to the determination device 40.

[0024] The information terminal 30 is, for example, a portable computer device such as a smartphone or a tablet terminal used by a user. The information terminal 30 may be a stationary computer device such as a personal computer.

[0025] The information terminal 30 includes a camera 20, a communication unit 31, a control unit 32, a storage unit 33, a reception unit , a presentation unit 35, and an instruction unit .

[0026] Camera 20 is a camera that captures an image of a subject performing a specific action, thereby generating an image including the subject performing the specific action as a subject. In this embodiment, camera 20 is a video camera that captures an image of a subject performing a specific action, thereby generating a moving image including the subject performing the specific action as a subject (i.e., a moving image composed of a plurality of images each including the subject as a subject). Camera 20 may be a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a camera using a CCD (Charge Coupled Device) image sensor.

[0027] The camera 20 may be an external camera attached to the information terminal 30. In this case, the information terminal 30 does not need to include the camera 20, and only needs to include a communication interface for connecting to the camera 20 so as to be able to communicate with it.

[0028] The communication unit 31 is a communication interface that communicates with the determination device 40. Specifically, the communication unit 31 allows the information terminal 30 to communicate with the determination device 40 via a network 5 such as the Internet. The communication unit 31 is an example of a second communication unit. The communication unit 31 is realized by, for example, a wireless communication circuit for wirelessly communicating with the determination device 40.

[0029] The communication standard for communication performed by the communication unit 31 is not particularly limited.

[0030] Furthermore, the communication unit 31 may be connected to the determination device 40 so as to be capable of wireless communication, or may be connected to the determination device 40 so as to be capable of wired communication. For example, when the communication unit 31 is connected to the determination device 40 so as to be capable of wired communication, the communication unit 31 is realized by a connector connected to a communication line or the like.

[0031] The control unit 32 is a processing unit that performs various information processing in the information terminal 30. For example, the control unit 32 outputs an image generated by the camera 20 to the determination device 40 via the communication unit 31. For example, when outputting a moving image, the control unit 32 outputs a plurality of images that correct the moving image, together with time information at which each image was generated, in a linked manner. Also, for example, the control unit 32 acquires a determination result (determination result information) by the determination device 40 (more specifically, the determination unit 42e) from the determination device 40 via the communication unit 31. For example, the control unit 32 causes the presentation unit 35 to present information indicating the acquired determination result. Also, the control unit 32 performs various processing based on, for example, an operation input accepted by the acceptance unit 34. The control unit 32 is realized, for example, by a microcomputer. Alternatively, the control unit 32 may be realized by a processor. The function of the control unit 32 is realized, for example, by the microcomputer or processor constituting the control unit 32 executing a dedicated application program stored in the storage unit 33.

[0032] The storage unit 33 is a storage device that stores dedicated application programs and the like to be executed by the control unit 32. The storage unit 33 is realized by, for example, a semiconductor memory or a hard disk drive (HDD).

[0033] The reception unit 34 is an input interface that receives operation input by a user (e.g., a rehabilitation specialist) of the information terminal 30. For example, the reception unit 34 receives an input operation from the user, such as an instruction to start a process of determining the level of daily living activities that the subject can perform. The reception unit 34 is realized, for example, by a touch panel display. For example, when the reception unit 34 is realized by a touch panel display, the touch panel display functions as the presentation unit 35 and the reception unit 34.

[0034] The reception unit 34 is not limited to a touch panel display, and may be, for example, a keyboard, a pointing device such as a touch pen or a mouse, or a hardware button. The reception unit 34 may be a microphone when receiving voice input. The reception unit 34 may be a camera when receiving gesture input. In this case, the reception unit 34 may be realized by the camera 20, or may be realized by a camera different from the camera 20.

[0035] The presentation unit 35 is a presentation device that presents the determination result by the determination device 40. Specifically, the presentation unit 35 presents information indicating the degree of the state of the subject's activities of daily living determined by the determination device 40.

[0036] The manner in which the presentation unit 35 presents information to the user is not particularly limited. The presentation unit 35 may present information to the user, for example, by video (image or moving image), by audio, or by video and audio. The presentation unit 35 may be realized by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel, by an audio device such as a speaker or earphones, or by a display panel and an audio device.

[0037] The instruction unit 36 ​​is an instruction device that instructs the subject to perform a specific action. The instruction unit 36 ​​instructs the subject to perform a specific action, for example, by video and audio. That is, the instruction unit 36 ​​may instruct the user by video or audio.

[0038] The instruction unit 36 ​​issues instructions by video and / or audio, such as "Please raise your arms in the air", "Touch your back and maintain that posture", "Touch the back of your head and maintain that posture", "Touch your toes and maintain that posture", etc. The instruction unit 36 ​​may be realized by a display panel such as a liquid crystal panel or an organic EL panel, by an audio device such as a speaker or earphones, or by a display panel and an audio device.

[0039] The instruction unit 36 ​​and the presentation unit 35 may be realized by the same device such as a display panel and / or an audio device.

[0040] Furthermore, the video information and / or audio information used by the instruction unit 36 ​​to instruct the subject to perform a specific action may be stored in the storage unit 33 in advance.

[0041] The determination device 40 is a computer that acquires an image transmitted from the information terminal 30, estimates a skeletal model of the subject in the acquired image, and determines the state of the subject's activities of daily living based on the estimated skeletal model.

[0042] The determination device 40 includes a communication unit 41 , an information processing unit 42 , and a storage unit 43 .

[0043] The communication unit 41 is a communication interface that communicates with the information terminal 30. Specifically, the communication unit 41 allows the determination device 40 to communicate with the information terminal 30 via a network 5 such as the Internet. The communication unit 41 is an example of a first communication unit. The communication unit 41 is realized by, for example, a wireless communication circuit for wirelessly communicating with the information terminal 30.

[0044] The communication standard for communication performed by communication unit 41 is not particularly limited.

[0045] Furthermore, the communication unit 41 may be connected to the information terminal 30 so as to be capable of wireless communication, or may be connected to the information terminal 30 so as to be capable of wired communication. For example, when the communication unit 41 is connected to the information terminal 30 so as to be capable of wired communication, the communication unit 41 is realized by a connector connected to a communication line or the like.

[0046] The information processing unit 42 is a processing unit that performs various information processing in the determination device 40. The information processing unit 42 is realized, for example, by a microcomputer. Alternatively, the information processing unit 42 may be realized by a processor. The functions of the information processing unit 42 are realized, for example, by the microcomputer or processor constituting the information processing unit 42 executing a computer program stored in the storage unit 43.

[0047] The information processing unit 42 includes an acquisition unit 42a, an estimation unit 42b, a setting unit 42c, a specification unit 42d, a determination unit 42e, and an output unit 42f.

[0048] The acquisition unit 42a is a processing unit that acquires images from the information terminal 30 via the communication unit 41. Specifically, the acquisition unit 42a acquires images (e.g., a moving image composed of a plurality of images) output (transmitted) from the information terminal 30 via the communication unit 41.

[0049] The estimation unit 42b is a processing unit that estimates (calculates) a skeletal model of a subject in an image that includes a subject performing a specific action as a subject. Specifically, the estimation unit 42b estimates a skeletal model of the subject in the image based on an image acquired by the acquisition unit 42a. More specifically, the estimation unit 42b estimates a skeletal model in each of a plurality of images that constitute a moving image based on a moving image that is composed of a plurality of images.

[0050] Fig. 2 is a diagram for explaining a skeletal model of the subject 1 estimated by the estimation unit 42b according to the embodiment. Specifically, Fig. 2 is a diagram that diagrammatically illustrates an image including the subject 1 as a subject, in which the skeletal model of the subject 1 estimated by the estimation unit 42b is superimposed on the subject 1.

[0051] The skeletal model is a model generated by connecting a plurality of skeletal points, which are specific positions such as joints of the subject 1 in an image, with links (lines). Specifically, the skeletal model is coordinate data of a plurality of skeletal points. For example, the estimation unit 42b estimates the positions (more specifically, coordinates) of a plurality of predetermined skeletal points of the subject 1 in an image, including a neck skeletal point, an elbow skeletal point, and a wrist skeletal point, by performing image analysis or the like. Furthermore, the estimation unit 42b connects, for example, predetermined skeletal points such as an elbow skeletal point and a wrist skeletal point among the estimated plurality of skeletal points with lines. In this way, the estimation unit 42b estimates the skeletal model of the subject 1.

[0052] In addition, the skeletal model may be estimated by any method, as long as an existing posture and skeleton estimation algorithm is used, for example.

[0053] The estimation unit 42b may estimate a two-dimensional skeletal model of the subject, or may estimate a three-dimensional skeletal model of the subject. That is, the estimation unit 42b may estimate two-dimensional coordinates of skeletal points of the subject in the image, or may estimate three-dimensional coordinates of skeletal points of the subject. For example, the estimation unit 42b estimates a two-dimensional skeletal model of the subject (i.e., coordinates of each skeletal point in a two-dimensional orthogonal coordinate system) based on the image acquired by the acquisition unit 42a, and estimates a three-dimensional skeletal model of the subject (i.e., coordinates of each skeletal point in a three-dimensional orthogonal coordinate system) using the trained model 44, which is a trained machine learning model, based on the estimated two-dimensional skeletal model.

[0054] The trained model 44 is a classifier constructed in advance by machine learning using a two-dimensional skeletal model in which the three-dimensional coordinate data of each joint is known as training data and the three-dimensional coordinate data as teacher data. The trained model 44 receives the two-dimensional skeletal model as input and outputs three-dimensional coordinate data corresponding to the two-dimensional skeletal model, that is, a three-dimensional skeletal model. The trained model 44 is stored in advance in the storage unit 43, for example.

[0055] In this manner, the estimation unit 42b may estimate a three-dimensional skeletal model of the subject in the image acquired by the acquisition unit 42a.

[0056] The setting unit 42c is a processing unit that sets (calculates) a plurality of three-dimensional regions around the skeletal model based on the positions of a plurality of skeleton points in the skeletal model estimated by the estimation unit 42b.

[0057] FIG. 3 is a diagram for explaining a three-dimensional area set by the setting unit 42c according to the embodiment. Specifically, FIG. 3 is a diagram that shows an image including a subject, in which the three-dimensional area set by the setting unit 42c and the skeleton points of the subject estimated by the estimation unit 42b are superimposed. Note that (b), (d), and (f) of FIG. 3 are diagrams showing a case where the side of the subject is photographed, (a) and (c) of FIG. 3 are diagrams showing a case where the front of the subject is photographed, and (e) of FIG. 3 is a diagram showing a case where the back of the subject is photographed. Also, (a) and (b) of FIG. 3 are diagrams that show a front area A1 among the multiple three-dimensional areas set by the setting unit 42c. Also, (c) and (d) of FIG. 3 are diagrams that show a front area A2 among the multiple three-dimensional areas set by the setting unit 42c. Moreover, (e) and (f) of FIG. 3 are diagrams that typically show a rear surface area A3 out of the multiple three-dimensional areas set by the setting unit 42c.

[0058] As shown in (b), (d) and (f) of Figure 3, for example, the setting unit 42c sets, as three-dimensional areas, a back area A3 on the back side of the subject and a front area A2 on the front side of the subject, which are arranged on either side of a first reference axis Z1 that is an axis in the direction from the subject's head to the legs (also called the vertical direction) and passes through a base point (first base point), and a front area A1 adjacent to the front area and arranged on the front side of the subject.

[0059] The first base point may be arbitrarily determined in advance and is not particularly limited. The number of first base points, that is, the number of skeleton points that become the first base points, may be one or more and is not particularly limited. The first base points are, for example, the subject's neck skeleton point and waist skeleton point. That is, for example, the first reference axis Z1 is set based on the positions of the subject's neck skeleton point and waist skeleton point. Specifically, for example, the first reference axis Z1 is set so as to pass through the subject's neck skeleton point and waist skeleton point in a side view of the subject.

[0060] 3(a), (c) and (e), the setting unit 42c sets the back area A3, the front area A2 and the front area A1 so as to include (for example, overlap with the left area B2 and the right area B1) the left area B2 and the right area B1 adjacently arranged across the second reference axis Z2, which is a vertical axis and passes through the base point (second base point), in the front view of the subject. In other words, the setting unit 42c sets the left area B2 and the right area B1 adjacently arranged across the second reference axis Z2, which is a vertical axis and passes through the base point (second base point), in the front view of the subject, so as to include, for example, the back area A3, the front area A2 and the front area A1, respectively.

[0061] The second base point may be arbitrarily determined in advance and is not particularly limited. The number of second base points, that is, the number of skeleton points that are the second base points, may be one or more and is not particularly limited. The second base points are, for example, the skeleton points of the neck and elbows of the subject. That is, for example, the second reference axis Z2 is set based on the positions of the skeleton points of the neck and elbows of the subject. Specifically, for example, the second reference axis Z2 is set so as to pass through the middle between the skeleton points of the neck and the skeleton points of both elbows of the subject in a front view of the subject.

[0062] Also, as shown in (a) and (b) of Figures 3, in this example, the setting unit 42c divides each of the left side region B2 and the right side region B1 in the forward area A1 in a horizontal direction perpendicular to the vertical direction, thereby setting three three-dimensional regions in each of the left side region B2 and the right side region B1.

[0063] Also, as shown in (c) and (d) of Figure 3, in this example, the setting unit 42c divides each of the left side area B2 and the right side area B1 in the front area A2 in a horizontal direction perpendicular to the vertical direction, thereby setting five three-dimensional areas in each of the left side area B2 and the right side area B1.

[0064] Also, as shown in (e) and (f) of Figure 3, in this example, the setting unit 42c divides each of the left side region B2 and the right side region B1 in the back region A3 in a horizontal direction perpendicular to the vertical direction, thereby setting four three-dimensional regions in each of the left side region B2 and the right side region B1.

[0065] In this manner, for example, the setting unit 42c sets multiple three-dimensional regions D1, D21, D22, D3, E1, E21, E22, E31, E32, F1, F2, F3, G1, G21, G22, G3, H1, H21, H22, H31, H32, I1, I2, I3 around the skeletal model using at least one of the multiple skeletal points in the skeletal model as a base point.

[0066] In addition, the one or more skeleton points that serve as first base points and the one or more skeleton points that serve as second base points may all be the same skeleton points, may all be different skeleton points, or some may be the same skeleton points and others may be different skeleton points.

[0067] Furthermore, the positions, sizes, and numbers of the three-dimensional regions set by the setting unit 42c may be determined arbitrarily and are not particularly limited.

[0068] For example, the setting unit 42c sets a first distance L1, which is a distance from the skeleton point of the subject's elbow to the tip of the hand, as the width W1 of each of the back region A3, the front region A2, and the front region A1 in a side view of the subject. Also, for example, the setting unit 42c sets a distance twice the second distance L2, which is a distance from the skeleton point of the subject's neck to the skeleton point of the shoulder, as the width W2 of each of the left region B2 and the right region B1 in a front view of the subject.

[0069] Furthermore, the sizes and shapes of the multiple three-dimensional regions set by the setting unit 42c may be the same as or different from each other.

[0070] The identification unit 42d is a processing unit that identifies a three-dimensional area (target three-dimensional area) in which the skeletal point of the subject's wrist is located during a specific movement (i.e., while performing a specific movement) among the multiple three-dimensional areas set by the setting unit 42c. Specifically, the identification unit 42d identifies in which area of ​​the multiple three-dimensional areas the coordinates of the skeletal point of the subject's wrist are located (in other words, included in) based on three-dimensional coordinate data (i.e., three-dimensional skeletal model) of the subject in the image. Also, for example, the identification unit 42d identifies one or more three-dimensional areas in which the skeletal point of the subject's wrist is located during the execution of a specific movement among the multiple three-dimensional areas based on a three-dimensional skeletal model (i.e., three-dimensional coordinate data) of the subject in a moving image including the subject who performed a specific movement as a subject.

[0071] Fig. 4 is a diagram for explaining a three-dimensional region identified by the identification unit 42d according to the embodiment. In Fig. 4, the target three-dimensional region is shown with hatching. Also, the numerical values ​​shown in Fig. 4 are numerical values ​​showing, by way of example, coordinates according to each axis in a three-dimensional orthogonal coordinate system.

[0072] For example, the estimation unit 42b estimates a skeletal model of the subject (i.e., coordinates of skeletal points of the subject). Furthermore, the setting unit 42c sets a plurality of three-dimensional regions. Furthermore, the identification unit 42d identifies a target three-dimensional region, which is a three-dimensional region in which the wrist skeletal points estimated by the estimation unit 42b are located, from among the plurality of three-dimensional regions set by the setting unit 42c.

[0073] Furthermore, for example, the specifying unit 42d specifies a three-dimensional area through which a skeleton point of the wrist of the subject passes during a specific movement, from among the multiple three-dimensional areas set by the setting unit 42c.

[0074] The three-dimensional area through which the subject's wrist skeleton point has passed is, for example, three-dimensional area E22 when the wrist skeleton point located in three-dimensional area F2 moves to three-dimensional area E22 and then reaches three-dimensional area E21.

[0075] The determination unit 42e is a processing unit that determines (calculates) the level of the daily living activities that the subject can perform based on the three-dimensional area specified by the determination unit 42d. Specifically, the determination unit 42e determines (calculates) the level of the daily living activities that the subject can perform corresponding to the specific actions based on the three-dimensional area specified by the determination unit 42d. For example, the determination unit 42e determines the level of the daily living activities that the subject can perform based on the database 45 stored in the storage unit 43.

[0076] FIG. 5 is a diagram showing a specific example of the judgment criteria of the judgment unit 42e according to the embodiment. More specifically, FIG. 5 is a diagram showing a database 45. In the example shown in FIG. 5, the higher the achievement level, which is the judgment result when the subject performs a specific movement, the more correctly the subject can perform the daily living movement corresponding to the specific movement. For example, when the achievement level is 100%, it indicates that the subject can correctly perform the daily living movement corresponding to the specific movement. Also, for example, when the achievement level is 50%, it indicates that the subject can slightly perform the daily living movement corresponding to the specific movement, that is, the subject is not very good at the daily living movement. Also, for example, when the achievement level is 0%, it indicates that the subject cannot perform the daily living movement corresponding to the specific movement.

[0077] The database 45 is data that stores a specific movement, a three-dimensional area (reference three-dimensional area) in which the wrist is located during the specific movement, activities of daily living (ADL) corresponding to the specific movement, and a calculation method for the degree of correspondence to the specific movement (assessment criteria and achievement level), all of which are linked together.

[0078] The judgment criteria is information that indicates what judgment the judgment unit 42e will make regarding each of the change in the position of the target three-dimensional area, the time during which the position of the target three-dimensional area does not change, the speed at which the position of the target three-dimensional area changes, and the auxiliary operation information described below.

[0079] The achievement level is information indicating a calculation method by which the determination unit 42e calculates the degree of the daily living activities that the subject can perform based on each piece of information determined based on the determination criteria. In other words, the achievement level is information indicating, for example, a calculation method of the determination result by the determination unit 42e.

[0080] For example, when the specific action is "banzai (action of raising both hands from a lowered position)", if the skeletal point of the subject's wrist moves from the subject's body surface (front of the torso) to around the face during the specific action (i.e., while the subject is performing the specific action), the determination unit 42e determines that the subject is 100% capable of performing the action of "eating" as a daily living activity. Specifically, for example, when the subject performs "banzai", if the position of the skeletal point of the wrist passes through three-dimensional areas E22 and E21 in this order from the initial position (e.g., three-dimensional area F2) and reaches three-dimensional area D22 ("up to D22: 100%" shown in FIG. 5), the determination unit 42e determines that the subject is 100% capable of performing the action of "eating".

[0081] In addition, for example, when the subject performs a “banzai” pose, if the position of the wrist skeleton point passes through three-dimensional area E22 from the initial position and reaches three-dimensional area E21 but does not reach three-dimensional area D22 (”Up to E21: 75%” shown in FIG. 5), the judgment unit 42e judges that the subject is 75% capable of performing the “eating” action.

[0082] In addition, for example, when the subject performs a “banzai” pose and the position of the wrist skeleton point reaches from the initial position to three-dimensional area E22 but does not reach three-dimensional area E21 (“Up to E22: 50%” shown in FIG. 5), the judgment unit 42e judges that the subject is 50% capable of performing the “eating” action.

[0083] In this manner, when the subject is capable of performing activities of daily living, the determination unit 42e determines to what extent the subject is capable of performing the activities of daily living.

[0084] The determination unit 42e may also determine whether the subject is capable of performing daily living activities. For example, when the subject is instructed to perform a "banzai" motion but the position of the wrist skeleton point does not change from the initial position, that is, when the target three-dimensional area does not change from the three-dimensional area F2 ("Other: 0%" shown in FIG. 5), the determination unit 42e determines that the subject is 0% capable of performing (i.e., unable to perform) the "eating" motion.

[0085] In addition, the judgment unit 42e may judge the degree of daily living activities that the subject can perform based on the auxiliary movement information indicating whether or not the subject is able to perform auxiliary movements related to daily living activities corresponding to a specific movement and the three-dimensional area identified by the identification unit 42d.

[0086] Here, the assistive movement is a so-called compensatory movement, which is a movement that helps to perform an activity of daily living corresponding to a specific movement. For example, the assistive movement is a movement different from the specific movement corresponding to the activity of daily living.

[0087] For example, if the subject can perform "banzai" which is an example of a specific action, it is estimated that the subject can perform the action of raising both hands, and therefore it is considered that the subject can perform the action of "eating" which is an example of an activity of daily living corresponding to "banzai", that is, the action of moving the hands to bring food to the mouth. Here, for example, even if the hand can only be raised to about the chest level, that is, even if the "banzai" cannot be performed correctly, if the head can be lowered, it is considered that the action of "eating" can be performed more correctly than when the head cannot be lowered. Therefore, for example, when determining the level of the activity of daily living, the determination unit 42e makes a determination taking into consideration whether or not not only the specific action but also the auxiliary action can be performed. For example, the determination unit 42e determines the degree to which the subject can perform the specific action based on the three-dimensional area specified by the determination unit 42d, and weights the determination result according to whether or not the subject can perform the auxiliary action, thereby determining the level of the activity of daily living that the subject can perform.

[0088] For example, when the position of the wrist skeleton point passes through the three-dimensional area E22 from the initial position when the subject performs a banzai gesture and reaches the three-dimensional area E21 but does not reach the three-dimensional area D22, and the subject is able to perform an auxiliary movement ("auxiliary movement possible: 1.2" shown in FIG. 5), the determination unit 42e determines that the subject is able to perform the movement of "eating" 75%×1.2 ("(reached three-dimensional area)×(auxiliary movement possible or not)" shown in FIG. 5)=90%. That is, in this case, the determination result by the determination unit 42e is 90%. On the other hand, for example, when the position of the wrist skeleton point passes through the three-dimensional area E22 from the initial position and reaches the three-dimensional area E21 but does not reach the three-dimensional area D22, and the subject is unable to perform an auxiliary movement ("auxiliary movement not possible: 1.0" shown in FIG. 5), the determination unit 42e determines that the subject is able to perform the movement of "eating" 75%×1.0=75%. That is, in this case, the result of the determination by the determining unit 42e is 75%.

[0089] The auxiliary operation information may be stored in advance in the storage unit 43, or may be acquired from a user via the reception unit 34 or the like. Alternatively, the determination device 40 may determine whether or not the subject is capable of performing the auxiliary operation based on an image (e.g., a moving image) including the subject performing the auxiliary operation as a subject, and acquire the determination result as the auxiliary operation information. For example, the estimation unit 42b estimates a skeletal model of the subject in an image including the subject performing the auxiliary operation as a subject. Furthermore, for example, the setting unit 42c sets a plurality of three-dimensional regions around the skeletal model based on the positions of a plurality of skeletal points in the skeletal model estimated by the estimation unit 42b. Furthermore, for example, the specification unit 42d specifies, among the plurality of three-dimensional regions, a three-dimensional region (auxiliary three-dimensional region) in which a specific skeletal point (e.g., a skeletal point of the neck) among the multiple skeletal points in the auxiliary operation is located. Furthermore, for example, the determination unit 42e determines whether or not the subject is capable of performing the auxiliary operation based on the three-dimensional region specified by the specification unit 42d. The specific skeletal point may be arbitrarily determined.

[0090] Alternatively, for example, the determination device 40 determines whether the subject can perform an auxiliary movement by using an image used to determine the degree of the daily living movement that the subject can perform. For example, the identification unit 42d further identifies an auxiliary three-dimensional area in which one or more specific skeletal points other than the wrist skeletal point among the multiple skeletal points in a specific movement are located among the multiple three-dimensional areas set by the setting unit 42c. Furthermore, for example, the determination unit 42e determines whether the subject can perform an auxiliary movement based on the auxiliary three-dimensional area. That is, for example, the determination unit 42e determines the degree of the daily living movement that the subject can perform based on the target three-dimensional area and the auxiliary three-dimensional area. For example, if the skeletal point of the neck of the subject performing "banzai" is in any of the three-dimensional areas E21, E22, H21, and H22, the determination unit 42e determines that the subject can perform an auxiliary movement, and if it is located in any other three-dimensional area, the determination unit 42e determines that the subject cannot perform an auxiliary movement. Information indicating which three-dimensional area the auxiliary three-dimensional area must match to determine that the auxiliary movement is performable may be stored in advance in the storage unit 43.

[0091] Also, for example, the criterion may be the time during which the position of the target three-dimensional region does not change and / or the speed of change of the position of the target three-dimensional region, or may be a combination of the above-mentioned criteria and weightings.

[0092] For example, when the specific action is "back touch (action of touching the back with both hands down)", the determination unit 42e determines to what extent the subject can perform "dressing" based on the moving speed of the wrist skeleton point of the subject on the back of the subject and the time the position is maintained on the back in the specific action. Specifically, when the subject performs "back touch", the determination unit 42e determines to what extent the subject can perform "dressing" based on the speed at which the wrist skeleton point passes through the three-dimensional area I3 and the time it continues to be located in the three-dimensional area H32 when the position of the wrist skeleton point passes through the three-dimensional area I3 from the initial position and reaches the three-dimensional area H32.

[0093] The speed of passing through the three-dimensional area I3 is calculated, for example, based on the size of the three-dimensional area I3 and the time it takes for the position of the wrist skeleton point to move from being located in the three-dimensional area I3 to being located in the three-dimensional area H32.

[0094] The time during which the skeleton point of the wrist continues to be located in the three-dimensional area H32 is, for example, the time from when the skeleton point of the wrist is located in the three-dimensional area I3 to when the skeleton point is no longer located in the three-dimensional area I3.

[0095] For example, when the speed at which the wrist skeleton point passes through the three-dimensional region I3 is less than V1 m / s and greater than V2 m / s, and the time that the wrist skeleton point continues to be maintained in the three-dimensional region H32 is less than T2 seconds, the judgment unit 42e judges 75% x 50% = 37.5% to be the degree of ``dressing'' movement that the subject can perform.

[0096] In this manner, for example, the determination unit 42e determines the degree of activities of daily living that the subject can perform, based on the speed at which the wrist skeleton point passes through a first three-dimensional area among the multiple three-dimensional areas.

[0097] Alternatively, for example, the determination unit 42e determines the degree of activities of daily living that the subject can perform based on the time during which the wrist skeleton point continues to be located in a second three-dimensional area among the multiple three-dimensional areas.

[0098] The position and number of the first three-dimensional regions may be arbitrarily determined according to a specific movement. For example, the determination unit 42e may determine the level of the movement of daily living that the subject can perform based on the speed of passing through a plurality of three-dimensional regions (for example, the average speed of passing through each three-dimensional region, or the speed of the region with the slowest passing speed among the three-dimensional regions).

[0099] Furthermore, the position of the second three-dimensional area may be arbitrarily determined depending on a specific operation.

[0100] Furthermore, for example, when the specific action is "touching the back of the head (the action of touching the back of the head with both hands lowered)", the determination unit 42e determines to what extent the subject can perform "washing hair" based on the time during which the skeleton point of the wrist of the subject continues to be located at the back of the subject's head during the specific action. Specifically, when the subject performs "touching the back of the head", the determination unit 42e determines to what extent the subject can perform "washing hair" based on the time during which the skeleton point of the wrist continues to be located in the three-dimensional areas G3 and D3. For example, when the time during which the skeleton point of the wrist continues to be located in the three-dimensional areas G3 and D3 is less than the time T4 seconds, the determination unit 42e determines 50% as the degree of the "washing hair" action that the subject can perform.

[0101] Furthermore, for example, when the specific motion is "toe touch (motion of touching toes with both hands down)", the determination unit 42e determines to what extent the subject can perform "putting on and taking off shoes" based on the time during which the skeletal point of the wrist of the subject continues to be located on the lower body of the subject during the specific motion. Specifically, when the subject performs "toe touch", the determination unit 42e determines to what extent the subject can perform "putting on and taking off shoes" based on the time during which the skeletal point of the wrist continues to be located in the three-dimensional areas F2 and I2. For example, when the time during which the skeletal point of the wrist continues to be located in the three-dimensional areas F2 and I2 is longer than time T5 seconds, the determination unit 42e determines 100% as the degree of the motion of "washing hair" that the subject can perform.

[0102] The specific movements, daily living movements, three-dimensional regions, judgment criteria, and achievement levels shown in FIG. 5 are merely examples, and are not particularly limited, and may be arbitrarily determined. For example, the speeds V1 and V2 and the times T1, T2, T3, T4, T5, and T6 shown in FIG. 5 may be arbitrarily determined. Also, for example, the weighting value may be changed depending on the degree to which the subject can perform the assisting movement. Also, for example, the weighting value for the case of "assisting movement possible" may be 1.0, and the weighting value for the case of "assisting movement impossible" may be less than 1.0.

[0103] Furthermore, the wrist skeleton points used in the determination by the determining unit 42e may be the right wrist skeleton points, the left wrist skeleton points, or both the left and right wrist skeleton points.

[0104] The output unit 42f is a processing unit that outputs the determination result by the determination unit 42e to the information terminal 30 via the communication unit 41. The control unit 32 acquires the output determination result and causes the presentation unit 35 to present the determination result. As a result, the presentation unit 35 presents the determination result by the determination unit 42e to the user.

[0105] Fig. 6 is a diagram showing a specific example of a determination result by the determination unit 42e, presented by the presentation unit 35 according to the embodiment. The example shown in Fig. 6 is a specific example of an image that is one example of a determination result presented by the presentation unit 35 when the determination unit 42e has determined that the degree of the "eating" action that the subject can perform is 90%.

[0106] As shown in FIG. 6, the presentation unit 35 presents the assessment result such as "ADL: Eating" and "Achievement level is 90%," thereby allowing the user to easily and accurately know the degree of daily living activities that the subject can perform.

[0107] The output unit 42f may output a three-dimensional skeletal model in the moving image of the subject, feature values ​​(e.g., data on physical functions such as range of motion of joints) used in the determination result of the state of the daily living activities, the determination result of the subject's physical functions, a rehabilitation training plan, etc. The control unit 32 may cause the presentation unit 35 to present these pieces of information.

[0108] The storage unit 43 is a storage device that stores information such as the image (image data) acquired by the acquisition unit 42a, the control program executed by the information processing unit 42, the trained model 44, the database 45, and the above-mentioned various thresholds. The storage unit 43 is realized by, for example, a semiconductor memory or a HDD.

[0109] [Processing Procedure] Next, the processing procedure of the determination system 10 will be described.

[0110] FIG. 7 is a flowchart showing a processing procedure of the determination device 40 according to the embodiment.

[0111] First, the estimation unit 42b estimates a skeletal model of a subject in an image that includes a subject performing a specific action as a subject (S101). For example, the estimation unit 42b estimates a two-dimensional skeletal model of the subject based on the image, and estimates a three-dimensional skeletal model of the subject (i.e., three-dimensional coordinate data indicating the three-dimensional coordinates of each of a plurality of skeletal points) using the trained model 44, which is a trained machine learning model, based on the estimated two-dimensional skeletal model.

[0112] Next, the setting unit 42c sets a plurality of three-dimensional regions around the skeletal model based on the positions of a plurality of skeleton points in the skeletal model estimated by the estimation unit 42b (S102).

[0113] Next, the specifying unit 42d specifies, from among the multiple three-dimensional regions set by the setting unit 42c, a three-dimensional region in which the wrist skeleton point is located among the multiple skeleton points in the specific motion (S103).

[0114] Next, the determination unit 42e determines the level of the activities of daily living that the subject can perform based on the three-dimensional area identified by the identification unit 42d (S104). For example, the determination unit 42e determines the level of the activities of daily living that the subject can perform based on the three-dimensional area identified by the identification unit 42d and the database 45.

[0115] The determination device 40 may perform the processes of steps S101 to S104 as one loop process each time the subject performs each of the multiple specific movements.

[0116] FIG. 8 is a flowchart illustrating a processing procedure of the determination system according to the embodiment.

[0117] First, the instruction unit 36 ​​instructs the subject to perform a specific movement (S201). For example, when the reception unit 34 receives an instruction from the user to perform a process of determining the degree of the daily living activities that the subject can perform (i.e., an instruction to have the subject start a specific movement), the instruction unit 36 ​​gives an instruction such as "Please raise your arms in the air."

[0118] When the receiving unit 34 receives an instruction, the control unit 32 may acquire an image captured by the camera 20 and identify the subject in the acquired image. For example, a known image analysis technique is used to identify the subject in the image.

[0119] Next, the camera 20 captures an image of the subject performing the specific action as a subject, thereby generating an image (more specifically, a moving image) including the subject performing the specific action as a subject (S202).

[0120] Next, the control unit 32 outputs the image generated by the camera 20 to the determination device 40 via the communication unit 31 (S203). At this time, the control unit 32 may anonymize the image and transmit it to the determination device 40. This protects the privacy data of the subject.

[0121] Next, the acquisition unit 42a acquires, via the communication unit 41, the image generated by the camera 20 and output by the control unit 32 via the communication unit 31 (S100).

[0122] Next, the estimation unit 42b estimates a skeletal model of the subject in the image acquired by the acquisition unit 42a, that is, based on an image including the subject performing a specific action as a subject (S101).

[0123] When the acquisition unit 42a acquires a moving image made up of a plurality of images, the estimation unit 42b may estimate a skeletal model for each of the plurality of images constituting the moving image based on the acquired moving image.

[0124] Next, the setting unit 42c sets a plurality of three-dimensional regions around the skeletal model based on the positions of a plurality of skeleton points in the skeletal model estimated by the estimation unit 42b (S102).

[0125] Next, the specifying unit 42d specifies, from among the multiple three-dimensional regions set by the setting unit 42c, a three-dimensional region in which the wrist skeleton point is located among the multiple skeleton points in the specific motion (S103).

[0126] Next, the determination unit 42e determines the level of activities of daily living that the subject can perform, based on the three-dimensional area identified by the identification unit 42d (S104).

[0127] Next, the output unit 42f outputs the result of the determination made by the determination unit 42e to the information terminal 30 via the communication unit 41 (S105).

[0128] Next, the control unit 32 acquires, via the communication unit 31, the determination result output by the output unit 42f via the communication unit 41 (S204).

[0129] Next, the presentation unit 35 presents the determination result acquired by the control unit 32 (S205). Specifically, the control unit 32 causes the presentation unit 35 to present the acquired determination result.

[0130] The information terminal 30 may perform the processes of steps S201 to S203 as one loop process, executing them each time the subject performs each of the multiple specific movements, or may perform the processes of steps S201 and S202 for each of the multiple specific movements, and execute step S203 after the subject has completed all of the specific movements.

[0131] In addition, when the subject performs a plurality of specific actions, the judgment results associated with each of the specific actions may be presented, or only the actions with the poorest judgment results may be presented. These judgment results may be presented in order of poorest results.

[0132] Furthermore, the information terminal 30 may select a specific motion to be performed by the subject according to the subject's physical function before instructing the subject to perform the specific motion. For example, before step S201, the information terminal 30 may instruct the subject to perform a motion to stand up from a sitting position. At this time, the information terminal 30 may determine whether the subject can perform the motion to stand up based on an image of the subject captured by the camera 20. Alternatively, the specific motion to be performed by the subject may be selected based on a user's instruction received by the receiving unit 34.

[0133] This allows specific movements to be selected according to the subject's physical functions, making it possible to efficiently and accurately assess the state of the subject's daily living movements.

[0134] [Effects, etc.] As described above, the determination method of the embodiment is a determination method executed by a computer, and includes an estimation step (S101) of estimating a skeletal model of a subject in an image based on the image including the subject performing a specific movement, a setting step (S102) of setting multiple three-dimensional regions around the skeletal model based on positions of multiple skeletal points in the skeletal model, an identification step (S103) of identifying a three-dimensional region among the multiple three-dimensional regions in which a wrist skeletal point is located during the specific movement, and a determination step (S104) of determining the level of daily living activities that the subject can perform based on the three-dimensional region identified in the identification step.

[0135] The wrist is likely to be located at a specific position relative to the subject's body (e.g., torso) during daily living activities such as eating, dressing, excretion, bathing, and grooming. In the determination method according to the embodiment, a skeletal model of the subject is estimated, multiple three-dimensional regions are set around the estimated skeletal model, and in which of the multiple set three-dimensional regions the skeletal point of the wrist is located is identified. This makes it possible to easily and accurately identify where the wrist is located relative to the subject's body while the subject is performing a specific movement. Therefore, the determination method according to one aspect of the present invention makes it possible to easily and accurately determine the state of the subject's daily living activities.

[0136] Also, for example, in the identifying step, a three-dimensional area through which a skeleton point of the wrist passes during a specific motion is identified from among the multiple three-dimensional areas.

[0137] The wrist is likely to pass through a specific position relative to the subject's body (e.g., torso) during the execution of the daily living activity. Therefore, the state of the subject's daily living activity can be more accurately determined based on the three-dimensional area through which the wrist skeletal point passes during the specific movement, among the multiple three-dimensional areas.

[0138] Also, for example, in the determining step, the degree of the activities of daily living that the subject can perform is determined based on the speed at which the wrist skeleton point passes through a first three-dimensional area among the multiple three-dimensional areas.

[0139] For example, even if a subject can perform a specific movement, if the time it takes from starting to finishing the specific movement is too long, it is difficult to say that the subject can perform the daily living activities corresponding to the specific movement to the same degree as, for example, a healthy person. Therefore, for example, in a specific movement in which the wrist skeleton point passes through a specific three-dimensional area, the degree of daily living activities that the subject can perform is determined based on the speed at which the skeleton point passes through the specific three-dimensional area. This makes it possible to more accurately determine the state of the subject's daily living activities.

[0140] Also, for example, in the determining step, the degree of daily living activities that the subject can perform is determined based on the time during which the wrist skeleton point continues to be located in a second three-dimensional area among the multiple three-dimensional areas.

[0141] For example, daily living activities such as washing hair require the wrist to be kept in a specific position. Therefore, for a specific activity in which the wrist's skeleton point is kept in a specific three-dimensional area, the degree of daily living activities that the subject can perform is determined based on the time that the wrist's skeleton point is kept in the specific three-dimensional area. This makes it possible to more accurately determine the state of the subject's daily living activities.

[0142] Also, for example, in the determination step, the degree of daily living activities that the subject can perform is determined based on auxiliary movement information indicating whether or not the subject is able to perform auxiliary movements related to daily living activities corresponding to a specific movement and the three-dimensional area identified in the identification step.

[0143] For example, daily living activities such as eating require the ability to bring hands to the mouth. Therefore, if the subject can perform "banzai" which is an example of a specific action, it is estimated that the subject can perform the action of raising both hands, and therefore it is considered that the subject can perform the action of "eating", which is an example of daily living activities corresponding to "banzai", that is, the action of moving hands to bring food to the mouth. Here, for example, even if the hand can only be raised to about the chest level, that is, even if the "banzai" cannot be performed correctly, if the head can be lowered, it is considered that the "eating" action can be performed more correctly compared to when the head cannot be lowered. Therefore, for example, when determining the level of daily living activities, the determination unit 42e makes a determination taking into consideration whether or not not only the specific action but also the auxiliary action can be performed. This makes it possible to more accurately determine the state of the daily living activities of the subject.

[0144] Also, for example, the determination method according to the embodiment includes an output step of outputting a result of the determination step.

[0145] This allows the user to easily know the state of the subject's activities of daily living.

[0146] In addition, the determination device 40 of the embodiment includes an estimation unit 42b that estimates a skeletal model of a subject in an image based on the image including the subject performing a specific movement, a setting unit 42c that sets multiple three-dimensional regions around the skeletal model based on positions of multiple skeletal points in the skeletal model, an identification unit 42d that identifies, among the multiple three-dimensional regions, a three-dimensional region in which the wrist skeletal point is located during the specific movement, and a determination unit 42e that determines the level of daily living activities that the subject can perform based on the three-dimensional region identified by the identification unit 42d.

[0147] In addition, the judgment system 10 of the embodiment includes a judgment device 40 and an information terminal 30. In addition to the estimation unit 42b, the setting unit 42c, the identification unit 42d, and the judgment unit 42e, the judgment device 40 further includes a first communication unit (communication unit 41) that communicates with the information terminal 30, an acquisition unit 42a that acquires an image from the information terminal 30 via the first communication unit, and an output unit 42f that outputs the judgment result by the judgment unit 42e to the information terminal 30 via the first communication unit. The information terminal 30 includes a second communication unit (communication unit 31) that communicates with the judgment device 40, an instruction unit 36 ​​that instructs the subject to perform a specific action, a camera 20 that generates an image by photographing the subject performing the specific action, a control unit 32 that outputs an image to the judgment device 40 via the second communication unit and acquires the judgment result from the judgment device 40 via the second communication unit, and a presentation unit 35 that presents the judgment result.

[0148] According to these, the same effects as those of the determination method according to the above-mentioned embodiment can be achieved.

[0149] (Other embodiments) Although the embodiment has been described above, the present invention is not limited to the above embodiment.

[0150] For example, in the above embodiment, the determination device 40 determines the degree of ADL that the subject can perform based on the position of the wrist skeleton point in the three-dimensional skeleton model, but is not limited thereto. For example, the determination device may determine the degree of ADL that the subject can perform based on the position of the wrist skeleton point in the two-dimensional skeleton model. For example, in the above embodiment, the determination device 40 determines the degree of ADL that the subject can perform based on the three-dimensional area in which the wrist skeleton point is located among the multiple three-dimensional areas in the three-dimensional orthogonal coordinate system, but may determine the degree of ADL that the subject can perform based on the area in which the wrist skeleton point is located among the multiple areas in the two-dimensional orthogonal coordinate system.

[0151] Also, for example, an instruction to cause a subject to perform a specific action may be given by a user, in which case the information terminal does not need to have an instruction unit.

[0152] Furthermore, the information terminal 30 may transmit to the determination device 40 an image (moving image) of the subject when performing the specific movement together with the information indicating the specific movement.

[0153] Also, for example, the information terminal 30 may transmit information indicating an instruction to determine the degree of ADL that the subject can perform, to the determination device 40, based on an instruction from the user received by the reception unit 34. In this case, for example, the determination device 40 may transmit information indicating an instruction to have the subject perform a specific action, to the information terminal 30. Also, in this case, the information terminal 30 may cause the subject to perform a specific action by the instruction unit 36 ​​based on the received information, and may also photograph the subject by the camera 20.

[0154] The determination unit 42e may also calculate a feature amount indicating the feature of the movement of the subject in a specific movement based on the skeletal model estimated by the estimation unit 42b, and determine the subject's physical function, which is the ability to perform a physical movement, based on the calculated feature amount. For example, the determination unit 42e calculates the angle (joint angle) between two links connected to a specific skeletal point of the subject as a feature amount based on the skeletal model estimated by the estimation unit 42b. Alternatively, for example, the determination unit 42e calculates the distance between a specific skeletal point and an extremity part in a specific movement, and the fluctuation range of the position of the specific skeletal point in a specific movement, as a feature amount. For example, the determination unit 42e determines the subject's physical function based on whether each calculated value is equal to or greater than a predetermined threshold value, or whether each calculated value is within a predetermined range.

[0155] According to this, for example, for a subject who has no problem with daily living activities, a training plan necessary for maintaining or improving a physical function based on a physical function such as muscle strength can be provided. The predetermined skeletal points, the predetermined threshold, and the predetermined range may be arbitrarily determined. These pieces of information may be stored in the storage unit 43 in advance.

[0156] The determination unit 42e may further determine the degree of the daily living activities that the subject can perform based on the determination result of whether or not the subject can perform an action involving finger movement (for example, opening and closing the hand (opening the hand) or opposing the fingers (OK sign)). For example, when the reception unit 34 of the information terminal 30 receives an instruction to determine whether or not the subject can perform an action involving finger movement, the control unit 32 instructs the instruction unit 36 ​​to perform an action involving finger movement. When the information terminal 30 obtains an image including a subject performing an action involving finger movement captured by the camera 20, the information terminal 30 transmits the instruction received by the reception unit 34 and the image captured by the camera 20 to the determination device 40. The determination unit 42e of the determination device 40 determines whether or not the subject can perform an action involving opening and closing the hand, for example, by using another trained model (not shown) different from the trained model 44. The determination unit 42e may also use another trained model to determine whether the tip of the index finger and the tip of the thumb are touching in the image, and to identify the shape and size of the space between the index finger and the thumb to determine whether the finger opposition motion is possible. The other trained model may be stored in the storage unit 43 in advance.

[0157] This makes it possible to determine whether the subject can grasp an object, thereby making it possible to more accurately determine the degree of daily living activities that the subject can perform.

[0158] Furthermore, the information regarding the subject's physical functions may be stored in advance in the storage unit 43, or the information may be received from the user by the reception unit 34 and acquired by the acquisition unit 42a from the information terminal 30.

[0159] Furthermore, the determination unit 42e may generate a rehabilitation training plan based on the determination result. At this time, for example, the determination unit 42e may generate the rehabilitation training plan based on the subject's physical function in addition to the determination result.

[0160] In addition, for example, in the above embodiment, the process executed by a specific processing unit may be executed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0161] Also, for example, in the above embodiment, each component of a processing unit such as the information processing unit 42 may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0162] Furthermore, each component may be realized by hardware. Each component may be a circuit (or an integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit, or a dedicated circuit.

[0163] In addition, the general or specific aspects of the present invention may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0164] Also, for example, the present invention may be realized as a determination method, or as a program for causing a computer to execute the determination method, or as a non-transitory recording medium readable by a computer on which such a program is recorded.

[0165] In the above embodiment, the determination system 10 includes the information terminal 30 and the determination device 40, but the determination system according to the present invention may be realized as a single device such as an information terminal, or may be realized by multiple devices. For example, the determination system may be realized as a client-server system. When the determination system is realized by multiple devices, the components of the determination system described in the above embodiment may be distributed among the multiple devices in any manner.

[0166] In addition, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0167] 1. Target Audience 10 Judging System 20 Camera 30 Information terminal 31, 41 Communications Department 35 Presentation section 36 Instruction section 40 Judgment device 42b Estimation part 42c Setting section 42d Specific part 42e Judgment section 42f Output section D1, D21, D22, D3, E1, E21, E22, E31, E32, F1, F2, F3, G1, G21, G22, G3, H1, H21, H22, H31, H32, I1, I2, I3 Three-dimensional area

Claims

1. A computer-implemented method for determining whether a an estimation step of estimating a skeletal model of a subject in an image, the subject performing a specific action, based on the image; a setting step of setting a plurality of three-dimensional regions around the skeletal model based on positions of a plurality of skeleton points in the skeletal model; a step of identifying a three-dimensional region in which a wrist skeleton point of the plurality of skeleton points is located during the specific motion, among the plurality of three-dimensional regions; and a determining step of determining a level of the activities of daily living that the subject can perform based on the three-dimensional area identified in the identifying step. Judgment method.

2. In the identifying step, a three-dimensional area through which a skeleton point of the wrist passes during the specific motion is identified from among the plurality of three-dimensional areas. The method according to claim 1 .

3. In the determination step, a level of the activities of daily living that the subject can perform is determined based on a speed at which the skeleton point of the wrist passes through a first three-dimensional area among the plurality of three-dimensional areas. The method according to claim 2 .

4. In the determination step, a level of the activities of daily living that the subject can perform is determined based on a time during which the skeleton point of the wrist continues to be located in a second three-dimensional area among the plurality of three-dimensional areas. The method according to any one of claims 1 to 3.

5. In the determining step, a degree of the daily living activity that the subject can perform is determined based on assistive motion information indicating whether the subject can perform an assistive motion related to the daily living activity corresponding to the specific motion and the three-dimensional area identified in the identifying step. The method according to any one of claims 1 to 3.

6. Further, the method includes an output step of outputting a result of the determination step. The method according to any one of claims 1 to 3.

7. an estimation unit that estimates a skeletal model of a subject in an image, the subject performing a specific action, based on the image; a setting unit that sets a plurality of three-dimensional regions around the skeletal model based on positions of a plurality of skeleton points in the skeletal model; an identification unit that identifies, among the plurality of three-dimensional regions, a three-dimensional region in which a wrist skeleton point among the plurality of skeleton points is located during the specific motion; A determination unit that determines a level of an activity of daily living that the subject can perform based on the three-dimensional area identified by the identification unit. Judgment device.

8. The determination device according to claim 7 ; An information terminal, The determination device further comprises: A first communication unit that communicates with the information terminal; an acquisition unit that acquires the image from the information terminal via the first communication unit; an output unit that outputs a determination result by the determination unit to the information terminal via the first communication unit, The information terminal includes: A second communication unit that communicates with the determination device; An instruction unit that instructs the subject to perform the specific action; a camera that generates the image by photographing the subject performing the specific action; a control unit that outputs the image to the determination device via the second communication unit and acquires the determination result from the determination device via the second communication unit; A presentation unit that presents the determination result. Judging system.

Images