Evaluation system, and program

Abstract

To estimate walking motions of a subject when there are people other than the subject within an imaging range.SOLUTION: An evaluation system includes: an input reception part receiving walking image data; a subject identifying part for identifying a subject to be evaluated by using a plurality of images when two or more people are included in a plurality of images included in the walking image data; a walking phase determination part which divides each of the plurality of images for each of the plurality of walking phases included in one walking cycle of the subject; a joint feature amount extraction part extracting a feature amount about the subject's joint from the plurality of images; a part evaluation part which outputs a part evaluation result for each walking phase of the respective plurality of target parts obtained by evaluating movement for every walking phase to be a plurality of predetermined target parts of the subject body including at least a trunk by using a feature amount; and a total evaluation part which outputs a total evaluation result evaluating a walking motion at least in one walking cycle by using the part evaluation result for every walking phase of the target part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an evaluation system and a program. [Background technology]

[0002] Patent Document 1 describes a technique for analyzing the walking pattern of a trainee using images captured while the trainee is walking alone during walking training. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-62231 A Summary of the Invention [Problem to be solved by the invention]

[0004] In some cases, a trainee may be accompanied by an assistant during walking training. Therefore, there has been a demand for a technology that can evaluate the walking of a trainee even when a person other than the walking trainee is within the shooting range. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one embodiment of the present disclosure, an evaluation system is provided, which includes an input receiving unit that receives walking image data of at least one person performing a walking motion, the whole body of the person performing the walking motion being continuously photographed over at least one walking cycle, a subject identifying unit that identifies a subject to be evaluated using the images when two or more people are included in the images of the walking image data, a walking phase determining unit that divides each of the images into a plurality of walking phases included in one walking cycle of the subject, a joint feature extracting unit that extracts features related to the joints of the subject from the images, a part evaluation unit that outputs a part evaluation result for each walking phase of a plurality of predetermined target parts of the body of the subject, the target parts including at least the trunk, obtained by evaluating the movement of each of the walking phases of the plurality of target parts using the features, and a comprehensive evaluation unit that outputs a comprehensive evaluation result obtained by evaluating the walking motion in at least one walking cycle using the part evaluation result for each walking phase of the target parts. According to this aspect, even if a person other than the walking subject is within the shooting range, the walking of the subject can be evaluated.

[0007] (2) In the evaluation system of the above form, when two or more people are included in the multiple images, the subject identification unit may identify as the subject the person among the two or more people who occupies the largest proportion of an area in the images. According to this aspect, even if a person other than the walking subject is within the shooting range, the walking of the subject can be evaluated.

[0008] (3) In the evaluation system of the above form, when two or more people are included in the multiple images, the subject identification unit may identify as the subject the person among the two or more people who is furthest forward in a predetermined direction of travel. According to this aspect, even if a person other than the walking subject is within the shooting range, the walking of the subject can be evaluated.

[0009] (4) In the evaluation system of the above form, when two or more people are included in the multiple images, the subject identification unit may identify the subject from among the two or more people whose face and the front of their bodies are facing a predetermined direction of travel. The subject identification unit may determine whether two or more people have been detected in an image included in the plurality of image data, and if two or more people have been detected in an image included in the plurality of image data, may determine that among the two or more people, those whose faces and the front of their bodies are facing a predetermined traveling direction are candidates for the subject, and may determine that among the two or more people, those whose faces and the front of at least one of their bodies are facing a direction different from the traveling direction are not candidates for the subject. According to this embodiment, prior to the process of identifying the target person, a process of excluding people who are not candidates for the target person is performed in advance. Therefore, for example, even if multiple people are detected in the input image, the target person can be identified with high accuracy.

[0010] (5) In the evaluation system of the above form, the subject identification unit may determine whether two or more people are detected in one image included in the plurality of images and whether two or more corresponding people are detected in another image included in the plurality of images, and when two or more people are detected in the one image and two or more corresponding people are included in the other image, identify the subject from among those two or more people moving in a predetermined direction of travel. According to this embodiment, prior to the process of identifying the target person, a process of excluding people who are not candidates for the target person is performed in advance. Therefore, for example, even if multiple people are detected in the input image, the target person can be identified with high accuracy.

[0011] (6) In the evaluation system of the above form, the part evaluation results may include results of evaluating the movement of each of the target parts for each gait phase for at least one evaluation item set for each of the target parts. According to this embodiment, since evaluation items are set for each target part, the movement of each target part during each gait phase can be evaluated in detail.

[0012] (7) In the evaluation system of the above form, the part evaluation unit may evaluate the movement of each of the target parts for each gait phase using the evaluation items corresponding to attributes of the person, the attributes including at least age. According to the above aspect, evaluation items corresponding to human attributes including at least age are used, so that, for example, in walking training, the movement of the target part can be appropriately evaluated according to the attributes of the subject.

[0013] (8) In the evaluation system of the above form, the multiple target parts may include hip joints and knee joints, and the overall evaluation result may include an evaluation result of the stability of the person's walking, obtained using the part evaluation results for each gait phase of the trunk, hip joint, and knee joint. According to the above embodiment, since the stability of a person's walking can be evaluated, for example, in walking training, walking motion can be stably evaluated regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0014] (9) In the evaluation system of the above form, the multiple target parts may include hip joints and knee joints, and the overall evaluation result may include a result of evaluating the safety of the person's walking, obtained using the part evaluation results for each gait phase of the trunk, hip joint, and knee joint. According to the above embodiment, since the safety of a person's walking can be evaluated, for example, in walking training, walking motion can be stably evaluated regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0015] (10) In the evaluation system of the above form, the multiple target parts may include hip joints and knee joints, and the overall evaluation result may include a result of evaluating the walking endurance of the person, obtained using the part evaluation results for each gait phase of the trunk, the hip joint, and the knee joint. According to the above embodiment, since the walking endurance of a person can be evaluated, for example, in walking training, walking motion can be stably evaluated regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0016] (11) In the evaluation system of the above form, the multiple target parts may include hip joints and knee joints, and the overall evaluation result may include a result of evaluating the walking speed of the person obtained using the part evaluation results for each gait phase of the trunk, hip joint, and knee joint. According to the above embodiment, since the walking speed of a person can be evaluated, for example, in walking training, walking movement can be stably evaluated regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0017] (12) According to another aspect of the present disclosure, a program is provided for evaluating a person's walking motion, the program causing a computer to realize the following functions: receiving a plurality of image data obtained by continuously capturing the whole body of at least one person who is walking motion over at least one walking cycle; identifying a subject to be evaluated using the plurality of image data when the plurality of image data includes two or more people; dividing each of the plurality of image data into a plurality of walking phases included in one walking cycle; extracting feature amounts related to the joints of the subject from the plurality of image data; outputting a part evaluation result for each walking phase of a plurality of predetermined target parts of the body of the subject, the part evaluation result being obtained by evaluating the movement of each of the plurality of target parts including at least the trunk for each walking phase using the feature amounts; and outputting a comprehensive evaluation result obtained by evaluating the walking motion in at least one walking cycle using the part evaluation result for each walking phase of the each of the target parts. According to this aspect, even if a person other than the walking subject is within the shooting range, the walking of the subject can be evaluated. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an evaluation system according to an embodiment. [Diagram 2] FIG. 1 is a diagram for explaining a walking cycle and a walking phase. [Diagram 3] FIG. 1 is an explanatory diagram of key points. [Figure 4] FIG. 13 is a diagram showing a subject and a caregiver walking. [Diagram 5] 13 is a flowchart of an evaluation process. [Figure 6] FIG. 13 is a diagram showing evaluation items set for the trunk and pelvis. [Figure 7] FIG. 13 is a diagram showing evaluation items set for the hip joint and the knee joint. [Figure 8] FIG. 13 is a diagram showing evaluation items set for the ankle joints and toes. [Figure 9] 13 is an example of a screen showing an evaluation result transmitted to the imaging apparatus. [Figure 10] 13 is another example of a screen showing the evaluation result transmitted to the imaging apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] A. Embodiment: 1 is a diagram showing a schematic configuration of an evaluation system 10. The evaluation system 10 evaluates the walking movement of a person.

[0020] The evaluation system 10 includes an evaluation device 100 and a photographing device 200. The evaluation device 100 evaluates a person's walking motion using an image captured by the photographing device 200. In an embodiment, the functions of the evaluation device 100 are realized by a cloud server.

[0021] The evaluation device 100 includes a processor 101, a memory 102, a communication interface 103, and an input / output interface 104. The memory 102, the communication interface 103, and the input / output interface 104 are connected to the processor 101 via a bus 109. The processor 101 executes various programs stored in the memory 102 to realize various functions. The memory 102 includes a volatile memory and a non-volatile memory. The memory 102 stores various programs and data. For example, the memory 102 stores data representing a plurality of neural network models used in the evaluation process described below. The communication interface 103 enables wired or wireless communication between the evaluation device 100 and another device under the control of the processor 101. A display device such as a liquid crystal display or an organic EL (Electro Luminescence) display may be connected to the input / output interface 104. An input device such as a keyboard or a mouse may also be connected to the input / output interface 104. The evaluation device 100 and the photographing device 200 can communicate with each other via a network such as the Internet.

[0022] The evaluation device 100 functionally comprises an input receiving unit 110, a result output unit 120, a subject identifying unit 125, a walking cycle determining unit 130, a walking phase determining unit 140, a joint feature amount extracting unit 150, a first evaluating unit 160, and a second evaluating unit 170. The functions of each unit are realized by the processor 101 executing a program stored in the memory 102.

[0023] The input receiving unit 110 receives walking image data from the photographing device 200. The walking image data is data of a moving image captured by the photographing device 200 of a person's walking motion. The walking image data includes information on the shooting start time and shooting end time, and information on the frame rate. The input receiving unit 110 generates a plurality of still images from a plurality of frame images included in the walking image data. The generated still images are called input images. Furthermore, the input receiving unit 110 obtains the time when each frame image was photographed based on the information on the shooting start time and shooting end time, and the information on the frame rate, and adds information representing the obtained time as the shooting time of the input image corresponding to the frame image to the input image. A set of a plurality of input images generated from one walking image data is called an input image group. The input receiving unit 110 stores the input image group in the memory 102.

[0024] The result output unit 120 shown in FIG. 1 outputs the evaluation result obtained by the evaluation device 100 to the imaging device 200.

[0025] The subject identification unit 125 identifies a person to be evaluated that is included in each of the multiple input images. The multiple input images are also referred to as multiple images. Hereinafter, the person to be evaluated is referred to as a subject. Details of the process executed by the subject identification unit 125 will be described later.

[0026] The gait cycle determination unit 130 identifies the gait cycle of the subject using the multiple input images, and divides the multiple input images included in the input image group into groups of gait cycles. The process executed by the gait cycle determination unit 130 will be described in detail later.

[0027] The walking phase determination unit 140 identifies the walking phase of the subject using the multiple input images, and separates the multiple input images included in the input image group into input images for each walking phase. The details of the processing executed by the walking phase determination unit 140 will be described later.

[0028] FIG. 2 is a diagram for explaining a walking cycle and a walking phase. A section from when one foot touches the ground to when the same foot touches the ground again is called a walking cycle. A movement at the moment when the foot part of one side to be observed touches the floor or ground is called a touchdown. In the embodiment, touchdown refers to a movement and not a state. FIG. 2 shows one walking cycle from when the right foot touches the ground to when the right foot touches the ground again.

[0029] Walking is a cyclical continuous motion, but in the embodiment, continuous walking motion is divided into each walking cycle. Furthermore, in the embodiment, one walking cycle is divided into eight walking phases defined by the Rancho Los Amigos method. The eight walking phases are, in order from the start of the walking cycle, initial contact, loading response, mid stance, terminal stance, pre-swing, initial swing, mid swing, and terminal swing. For example, the proportion of each walking phase in one walking cycle C1 is defined as follows. One walking cycle C1 is divided into eight walking phases based on the defined proportions. The period of one walking cycle C1 is the period from the touchdown of one foot on one side to the touchdown of the same foot on the same side again. In the embodiment, the foot refers to the part beyond the ankle joint. (1) Initial grounding: 0~5% (2) Load response period: 5-15% (3) Midstance: 15-25% (4) End stance: 25-52% (5) Foreswing phase: 52~60% (6) Initial swing phase: 60-78% (7) Mid-swing phase: 78~88% (8) End swing phase: 88-100%

[0030] 1 detects features related to human joints in each input image. In the embodiment, key points used for skeleton detection are used as the features related to human joints.

[0031] Fig. 3 is an explanatory diagram of key points. Key points refer to feature points of the human body, including the positions of the joints of the human body. In Fig. 3, the feature points of person P1 are represented by black circles. Adjacent feature points are connected by links (straight lines).

[0032] 1 evaluates the movement of a predetermined part of a subject for each walking phase. The process executed by the first evaluation unit 160 will be described later in detail. The first evaluation unit 160 is also called a part evaluation unit.

[0033] The second evaluation unit 170 evaluates the walking movement of the subject using the results of evaluation by the first evaluation unit 160 regarding the movements of predetermined body parts in each walking phase. Details of the processing executed by the second evaluation unit 170 will be described later. The second evaluation unit 170 is also referred to as a comprehensive evaluation unit.

[0034] The photographing device 200 is a mobile terminal equipped with a camera, such as a tablet terminal, a smartphone, etc. The photographing device 200 supplies the photographed walking image data to the evaluation device 100.

[0035] Application software for walking training is pre-installed in the photographing device 200. In a walking training scene, a photographer photographs the walking movement of a subject of walking training using the photographing device 200. For example, the photographer is a physical therapist who instructs the subject on walking.

[0036] 4 is a diagram showing a state in which the subject T1 and the helper P11 are walking. In the embodiment, it is assumed that when the subject T1 undergoing walking training walks, the helper P11 who assists the subject T1 in walking walks together with the subject T1. The helper P11 has sufficient experience in assisting walking and can assist the subject T1 without interfering with the subject T1's walking movement.

[0037] For example, suppose that the subject T1 is right-handed. In this case, the caregiver P11 is on the opposite side of the subject T1's handedness, i.e., on the left side, and diagonally behind the subject T1, inserting his / her right arm into the left side of the subject T1 and holding the subject T1's left hand with his / her left hand, and assists the subject T1 in walking. The left side of the subject T1 means the left side as seen from the subject T1. The right side of the subject T1 means the right side as seen from the subject T1. The same applies to the following explanation.

[0038] The photographer P12 starts up the walking training application software on the photographing device 200 and selects a photographing menu. The selection of the photographing menu starts up the camera provided in the photographing device 200. When the photographing device 200 receives an instruction to start photographing by the operation of the photographer P12, it starts photographing the walking movement of the subject.

[0039] The photographer P12, holding the photographing device 200, photographs the subject T1 and the assistant P11 walking from the side of the subject T1 so that the entire body of the subject T1 is captured. For example, the subject T1 walks about 3 meters on a straight line drawn on the floor. The assistant P11 assists the subject T1 in walking as described above. Since the assistant P11 is close to the subject T1 on the left side, the photographer P12 photographs the subject T1 and the assistant P11 from the right side of the subject T1. Note that in this embodiment, it is assumed that only the subject T1 and the assistant P11 are within the photographing range of the photographing device 200.

[0040] When the photographing device 200 receives an instruction to stop photographing through an operation by the photographer P12, the photographing device 200 ends photographing.

[0041] After completing the photographing, the photographer P12 selects a data list menu of the walking training application software. In the data list menu, a list of walking image data acquired by the photographing device 200 is displayed. In the data list menu, the photographer P12 performs an operation to select, for example, the walking image data photographed immediately before from the list of photographed walking image data. Furthermore, the photographer P12 performs an operation to instruct transmission of the selected walking image data to the evaluation device 100. In response to this operation, the walking training application software transmits the selected walking image data to the evaluation device 100.

[0042] 5 is a flowchart of the evaluation process in the evaluation device 100. For example, the evaluation process is started when the input receiving unit 110 of the evaluation device 100 receives walking image data from the photographing device 200. As described above, the input receiving unit 110 generates an input image group consisting of a plurality of input images from the walking image data. At the start of the process shown in FIG. 5, the input image group is stored in the memory 102. The evaluation device 100 performs the process shown in FIG. 4 by executing a program for evaluating the walking motion of a person stored in the memory 102.

[0043] In step S5, the subject identification unit 125 identifies a subject to be evaluated using the input image group.

[0044] First, the subject identification unit 125 detects an area in each input image where a person exists using the machine learning model M1 for each of a plurality of input images included in the input image group. The machine learning model M1 is a neural network model for object detection. When an input image is input, the machine learning model M1 can output information on a partial area (bounding box) surrounding an area in the input image where a person exists. The subject identification unit 125 associates an input image number, a box ID, a coordinate value, and a label value, and stores them in the memory 102 as a detection result. The input image number is a number that identifies the input image. The box ID is a value for identifying the bounding box. By using the machine learning model M1, which is a neural network model for object detection, an area in the input image where a person exists is detected. The coordinate value represents the position and range in which the bounding box exists in the target input image. The coordinate value is represented by, for example, the coordinate value of the upper left corner and the coordinate value of the lower right corner of the bounding box. The label value represents the type of the detected object. The label value is "person". This is because in this embodiment, the target person identification unit 125 detects only people from the input image.

[0045] Next, the subject identification unit 125 determines whether two or more people are detected for each input image. If two or more bounding boxes are detected in an input image, then two or more people are detected in the input image.

[0046] When two or more people are detected in each input image, the subject identification unit 125 identifies the subject from the two or more detected people in order to evaluate only the walking movement of the subject.

[0047] Specifically, the subject identification unit 125 detects an area in each input image where a person exists using the machine learning model M2 for each of a plurality of input images included in the input image group. The machine learning model M2 is a neural network model for instance segmentation. The machine learning model M2 can detect a person in the input image in pixel units. The subject identification unit 125 associates a bounding box detected using the machine learning model M1 with an area of ​​each person in pixel units detected using the machine learning model M2. The subject identification unit 125 identifies, as the subject, a person having the largest area in pixel units in the input image among two or more detected people. The area in the bounding box for the person identified as the subject is called the subject area. The subject identification unit 125, for example, associates an input image number, a box ID representing the identified subject, coordinate values, and label values, and stores them in the memory 102 as subject information.

[0048] In step S10, the gait cycle determination unit 130 determines the gait cycle of the subject by using the target region.

[0049] Specifically, first, the gait cycle determination unit 130 selects all ground contact images from the multiple input images included in the input image group. The ground contact images are input images captured at the moment when one of the subject's lower legs touches the floor.

[0050] For example, the gait cycle determination unit 130 selects an image in which the moment one lower leg of the subject touches the floor using the machine learning model M3. The machine learning model M3 is a neural network model generated by machine learning. When an image is input, the machine learning model M3 outputs the probability that the input image corresponds to an image in which the moment one lower leg of the person touches the floor is captured. In an embodiment, a partial image of a target area in the input image is input to the machine learning model M3. The gait cycle determination unit 130 determines that the input image including the target area is a ground contact image when the probability that the partial image of the target area corresponds to an image in which the moment one lower leg of the person touches the floor is equal to or greater than a preset threshold. The gait cycle determination unit 130 selects all ground contact images of the right lower leg from the multiple input images included in the input image group, and then selects all ground contact images of the left lower leg from the multiple input images included in the input image group.

[0051] In addition, when a person can walk normally, the heel always steps on the ground, but there are cases where the heel cannot touch the ground due to reasons such as deterioration of physical functions due to aging or illness. In the embodiment, it is determined that the lower leg is in contact with the ground when at least a part of one foot is in contact with the floor.

[0052] Next, the gait cycle determination unit 130 determines the gait cycle based on the shooting time of the ground contact image. As described above, the shooting time of the input image is added to the input image when the input receiving unit 110 generates the input image group.

[0053] The walking cycle determination unit 130 selects two chronologically consecutive ground contact images from among the multiple ground contact images selected for the right lower limb. The walking cycle determination unit 130 specifies the ground contact image with the earlier photographing time as the ground contact image at the start of the walking cycle, and the ground contact image with the later photographing time as the ground contact image at the end of the walking cycle. The ground contact images being consecutive in chronological order means that there is no other ground contact image photographed between the photographing times of the two ground contact images. However, there is an input image other than the ground contact image between the photographing times of the two ground contact images. The walking cycle determination unit 130 specifies the photographing time of the ground contact image at the start of the walking cycle as the start time of the walking cycle. The walking cycle determination unit 130 specifies the photographing time of the ground contact image at the end of the walking cycle as the end time of the walking cycle. The start time of the walking cycle is also called the cycle start time.

[0054] For example, the walking cycle determination unit 130 specifies the landing image with the earliest photographed time among the multiple landing images selected for the right lower limb as the landing image at the start of the first walking cycle for the right side. The walking cycle determination unit 130 specifies the time when this landing image was photographed as the start time of the first walking cycle. The walking cycle determination unit 130 specifies the landing image showing that the right lower limb touches the ground next as the landing image at the end of the first walking cycle for the right side. The walking cycle determination unit 130 specifies the time when this landing image was photographed as the end time of the first walking cycle. In addition, the walking cycle determination unit 130 specifies this landing image as the landing image at the start of the second walking cycle for the right side. The walking cycle determination unit 130 specifies the time when this landing image was photographed as the start time of the second walking cycle.

[0055] The gait cycle determination unit 130 repeats the process of identifying the start time and end time of the gait cycle for the right lower limb from the multiple ground contact images selected for the right lower limb by the above-mentioned method. After identifying all gait cycles for the right lower limb, the gait cycle determination unit 130 repeats the process of identifying the start time and end time of the gait cycle for the left lower limb from the multiple ground contact images selected for the left lower limb.

[0056] The walking cycle determination unit 130 determines the walking cycle to which each of the multiple input images included in the input image group belongs, based on the start time and end time of the walking cycle for each of the identified right and left lower limbs and the photographing time of the input image. The walking cycle determination unit 130 stores in the memory 102, in association with information for identifying the input image and information for identifying the walking cycle for each of the multiple input images.

[0057] In step S20, the walking phase determination unit 140 identifies eight walking phases in each walking cycle of the subject. The walking phase determination unit 140 identifies the times that separate the eight walking phases based on the start time and end time of the walking cycle and the ratios defined for the durations of the above-mentioned eight walking phases. The walking phase determination unit 140 identifies the walking phase to which the input image belongs based on the times that separate the eight walking phases and the capture time of the input image that belongs to that walking cycle.

[0058] The walking phase determination unit 140 repeats the process of identifying input images belonging to eight walking phases for each walking cycle identified for the right lower limb. After that, the walking phase determination unit 140 repeats the process of identifying input images belonging to eight walking phases for each walking cycle identified for the left lower limb. The walking phase determination unit 140 stores in the memory 102, in association with information for identifying the input image, information for identifying the walking cycle, and information for identifying the walking phase for each of the multiple input images.

[0059] In step S30, the joint feature extraction unit 150 detects key points of the subject in each input image using the machine learning model M5. The machine learning model M5 is an existing neural network model for key point detection. For each input image, the joint feature extraction unit 150 stores information for identifying the input image and key point information about the key points detected in the target area of ​​the input image in association with each other in the memory 102. The key point information includes, for example, a value for identifying the key point, coordinates indicating the position of the key point in the input image, and a value for identifying one or more other adjacent key points.

[0060] In step S40, the first evaluation unit 160 calculates, for each walking phase, the amount of change in the angle of the target part of the subject in that walking phase as an index value of the target part, using key points detected for the subject using the input image belonging to that walking phase. The target part includes a plurality of predetermined parts of the human body. In the embodiment, the target part includes the trunk, the pelvis, the hip joints, the knee joints, the ankle joints, and the toes. In this specification, the trunk refers to the torso excluding the head and the limbs, and the part above the waist. The index value is used to evaluate the movement of the target part.

[0061] For example, in the walking movement of a subject shown in the input image group generated from one walking image data, the walking cycle for the right lower limb is three walking cycles, and the walking cycle for the left lower limb is three walking cycles. First, the first evaluation unit 160 calculates the change amount of each angle of the trunk, pelvis, hip joint, knee joint, ankle joint, and toes in each of the eight walking phases of the first walking cycle of the right lower limb. The first evaluation unit 160 performs the same processing for the second walking cycle and the third walking cycle of the right lower limb. The first evaluation unit 160 calculates the average value of the change amount of each angle of the trunk, pelvis, hip joint, knee joint, ankle joint, and toes in the same walking phase of each walking cycle, and sets the calculated average value as the change amount of the angle of the target part in that walking phase. The first evaluation unit 160 generates index data in which a value for identifying the walking phase is associated with the change amount of the angle of each target part. The first evaluation unit 160 performs the same processing for the left lower limb. The first evaluation unit 160 stores the generated index data in the memory 102.

[0062] In step S50, the first evaluation unit 160 evaluates the movement of the target part of the subject in each walking phase. In the embodiment, at least one evaluation item is set for each walking phase for each target part.

[0063] FIG. 6 is a diagram showing evaluation items set for the trunk and pelvis. FIG. 7 is a diagram showing evaluation items set for the hip joint and knee joint. FIG. 8 is a diagram showing evaluation items set for the ankle joint and toes. In FIGS. 6 to 8, the items shaded are not evaluated. As shown in FIGS. 6 to 8, even for the same target part, there are items that are evaluated and items that are not evaluated in different gait phases. For example, as shown in FIG. 7, the flexion restriction in the mid-stance phase and the end-stance phase of the hip joint is not evaluated. Also, for example, the hyperextension in the early swing phase and the mid-swing phase of the knee joint is not evaluated.

[0064] The first evaluation unit 160 evaluates the movement of each target part in each walking phase by comparing the index value of each target part in each walking phase with a reference value that is preset for each evaluation item in that walking phase.

[0065] For example, the first evaluation unit 160 compares the maximum value of the angle at which the trunk tilts forward during the load response period with a reference value set for the evaluation item of the forward tilt during the load response period. If the difference between the maximum value of the angle at which the trunk tilts forward during the load response period and the reference value is less than a predetermined first threshold, the first evaluation unit 160 determines that the forward tilt of the trunk during the load response period is within a normal range. If the difference between the maximum value of the angle at which the trunk tilts forward during the load response period and the reference value is equal to or greater than a predetermined second threshold, the first evaluation unit 160 determines that there is an abnormality in the forward tilt of the trunk during the load response period. If the difference between the maximum value of the angle at which the trunk tilts forward during the load response period and the reference value is equal to or greater than a predetermined first threshold and less than a predetermined second threshold, the first evaluation unit 160 determines that there is a minor abnormality in the forward tilt of the trunk during the load response period.

[0066] The first evaluation unit 160 evaluates the target part with the set evaluation items in each gait phase of the right lower limb. Furthermore, the first evaluation unit 160 evaluates the target part with the set evaluation items in each gait phase of the left lower limb. The first evaluation unit 160 stores the evaluation results in the memory 102. Furthermore, the first evaluation unit 160 outputs the evaluation results to the result output unit 120. The evaluation results output by the first evaluation unit 160 are also called part evaluation results. In this way, since evaluation items are set for each target part, the movement of each target part in each gait phase can be evaluated in detail.

[0067] In step S60 shown in FIG. 5, the second evaluation section 170 uses the evaluation result by the first evaluation section 160 to evaluate the walking movement of the subject based on the following indexes. (a) Walking stability (b) Pedestrian safety (c) Walking endurance Walking stability is an index that evaluates the degree of stability of walking. Walking safety is an index that evaluates whether walking is safe. For example, even if you can walk, if there is a high possibility of falling, the walking safety is low. Walking endurance is an index that evaluates whether you can walk continuously.

[0068] The second evaluation unit 170 evaluates the stability of walking using, for example, a machine learning model M71. The machine learning model M71 is a neural network model that has been trained using teacher data. As the teacher data, result data of evaluation of each target part in each of eight walking phases for the gait of a trainee undergoing walking training and a person who can walk normally, and result data of evaluation of the stability of walking by an experienced physical therapist are used. The result data of evaluation of each target part in each of the eight walking phases may be, for example, a result of evaluation by a computer based on an image of a person walking, or may be a result of evaluation by an experienced physical therapist.

[0069] The second evaluation unit 170 inputs the results of evaluating each of the target parts in each gait phase of the right lower limb and the results of evaluating each of the target parts in each gait phase of the left lower limb to the machine learning model M71, and obtains an evaluation value for the stability of walking output by the machine learning model M71. The evaluation value is, for example, a value indicating the probability that the stability of walking corresponds to each of a plurality of predefined levels. The second evaluation unit 170 outputs the evaluation result for the stability of walking obtained based on the evaluation value to the result output unit 120. For example, it is assumed that the plurality of levels are defined as "no problem", "slight abnormality", and "abnormality". When the evaluation value output by the machine learning model M71 indicates that the probability of corresponding to "no problem" is the highest, the second evaluation unit 170 outputs "no problem" to the result output unit 120 as the evaluation result for the stability of walking.

[0070] The second evaluation unit 170 evaluates the safety of walking using, for example, a machine learning model M72. The machine learning model M72 is a neural network model that has been trained using teacher data. As the teacher data, result data of evaluation of each target part in each of eight walking phases for the gait of a trainee undergoing walking training and a person who can walk normally, and result data of evaluation of the safety of walking by an experienced physical therapist are used. The result data of evaluation of each target part in each of the eight walking phases may be, for example, a result of evaluation by a computer based on an image of a person walking, or may be a result of evaluation by an experienced physical therapist.

[0071] The second evaluation unit 170 inputs the results of evaluating each target part in each gait phase of the right lower limb and the results of evaluating each target part in each gait phase of the left lower limb to the machine learning model M72, and obtains an evaluation value for the walking safety output by the machine learning model M72. The evaluation value is, for example, a value indicating the probability that the walking safety falls into each of a plurality of predefined levels. The second evaluation unit 170 outputs the evaluation result for the walking safety obtained based on the evaluation value to the result output unit 120.

[0072] The second evaluation unit 170 evaluates the walking durability using, for example, a machine learning model M73. The machine learning model M73 is a neural network model that has been trained using teacher data. As the teacher data, for each of the trainee undergoing walking training and the gait of a person who can walk normally, result data of evaluation of each target part in each of eight walking phases and result data of evaluation of walking durability by an experienced physical therapist are used. The result data of evaluation of each target part in each of the eight walking phases may be, for example, a result of evaluation by a computer based on an image of a person walking, or may be a result of evaluation by an experienced physical therapist.

[0073] The second evaluation unit 170 inputs the results of evaluating each of the target parts in each gait phase of the right lower limb and the results of evaluating each of the target parts in each gait phase of the left lower limb to the machine learning model M73, and obtains an evaluation value for walking durability output by the machine learning model M73. The evaluation value is, for example, a value indicating the probability that walking durability corresponds to each of a plurality of predefined levels. The second evaluation unit 170 outputs the evaluation result for walking durability obtained based on the evaluation value to the result output unit 120. The evaluation results for walking stability, walking safety, and walking durability output by the second evaluation unit 170 are also referred to as overall evaluation results.

[0074] In step S70, the result output unit 120 transmits the evaluation result for the subject to the photographing device 200. Fig. 9 is an example of a screen of the evaluation result transmitted to the photographing device 200. The result output unit 120 generates screen data of the evaluation result as shown in Fig. 9 based on the evaluation result of the target part of each walking phase evaluated by the first evaluation unit 160 and the evaluation results of walking stability, walking safety, and walking endurance evaluated by the second evaluation unit 170. Note that in Fig. 9, as in Figs. 6 to 8, the shaded items are items that are not evaluated.

[0075] Fig. 10 is another example of the evaluation result screen transmitted to the photographing device 200. The result output unit 120 further generates evaluation result screen data as shown in Fig. 10, which shows a graph indicating the change in angle in one walking cycle for each target part. The result output unit 120 transmits the generated evaluation result screen data to the photographing device 200. Thereafter, the evaluation process ends.

[0076] When the image capturing device 200 receives the screen data of the evaluation result, the image capturing device 200 displays the evaluation result via the walking training application software, for example, on a touch panel display. When any target part is tapped on the evaluation result screen shown in Fig. 9, the image capturing device 200 may display a graph showing the change in angle of the selected target part in one walking cycle, as shown in Fig. 10.

[0077] As described above, according to the configuration of the embodiment, even if a person other than the walking subject is within the shooting range, the walking of the subject can be evaluated.

[0078] In addition, since the evaluation device 100 evaluates the movement of the body part including the trunk, it is possible to evaluate the walking movement using not only the movement of the lower body but also the movement of the upper body. Conventionally, a physical therapist evaluates the walking of a subject by repeatedly checking a video of the walking of the subject. However, according to the configuration of the embodiment, the physical therapist can obtain the evaluation result of the walking without repeatedly checking the video. Therefore, it is possible to provide efficient walking instruction.

[0079] In addition, since the evaluation device 100 evaluates the stability, safety, and durability of a person's walking, for example, in walking training, a stable evaluation result of walking movement can be obtained regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0080] B. Other embodiments: B1. Alternative embodiment 1: In the embodiment, when two or more people are detected in the input image, the subject identification unit 125 identifies the person who occupies the largest area ratio in the input image among the two or more detected people as the subject. However, the method of identifying the subject is not limited to this.

[0081] When two or more people are detected in the input image, the subject identification unit 125 may identify the person who is the furthest forward in a predetermined traveling direction as the subject among the two or more people. When the photographing device 200 photographs the subject from the side, it is predetermined whether the traveling direction of the subject is rightward as seen from the photographing device 200 or leftward as seen from the photographing device 200. For example, when the traveling direction of the subject is rightward as seen from the photographing device 200, the subject identification unit 125 identifies the person who is on the right side of the other person among the two people detected in the input image as the subject. In the other embodiment 1, as in the embodiment, the walking of the subject can be evaluated even if a person other than the walking subject is within the photographing range.

[0082] B2. Alternative embodiment 2: In the embodiment, an example has been described in which only the subject and the caregiver are within the photographing range of the photographing device 200. However, there may be other people around the subject and the caregiver. For example, the subject and the caregiver are in a rehabilitation facility. In such a case, the subject identification unit 125 can perform the following process prior to identifying the subject.

[0083] When two or more people are detected in the input image, the subject identification unit 125 determines that, among the two or more detected people, a person whose face and body are facing the front of the body in a predetermined traveling direction is a candidate subject. The orientation of the face and body can be estimated, for example, by using an algorithm for pose estimation. In addition, the subject identification unit 125 determines that, among the two or more detected people, a person whose face and / or body is facing a direction different from the predetermined traveling direction is not a subject. Thereafter, the subject identification unit 125 identifies a subject from the candidate subjects by the method described in the above-mentioned other embodiment 1. Alternatively, the subject identification unit 125 identifies a subject from the candidate subjects by the method described in the embodiment.

[0084] In another embodiment 2, a process of excluding people who are not candidates for the subject is performed prior to the process of identifying the subject. Therefore, for example, even if multiple people are detected in the input image, the subject can be identified with high accuracy.

[0085] B3. Alternative embodiment 3: Alternatively, the subject identification unit 125 can perform the following process prior to identifying the subject in order to exclude in advance people who are not candidates for the subject.

[0086] When two or more people are detected in two or more input images that are consecutive in time series, the subject identification unit 125 determines, based on the two or more input images, that a person moving in a predetermined traveling direction is a candidate for the subject. The correspondence between the two or more people detected in the two or more input images can be performed, for example, using an algorithm for object tracking. In addition, the subject identification unit 125 determines, based on the two or more input images, that a person who is stationary is not a subject. Furthermore, the subject identification unit 125 determines, based on the two or more input images, that a person moving in a direction other than the traveling direction is not a subject.

[0087] In the third embodiment, a process of excluding people who are not candidates for the subject is also performed prior to the process of identifying the subject. Therefore, for example, even if multiple people are detected in the input image, the subject can be identified with high accuracy.

[0088] B4. Alternative embodiment 4: Also, the subject may be asked to wear a marker such as an armband in advance. The subject identification unit 125 may detect a person wearing the marker from the input image and identify the person wearing the marker as the subject. Alternatively, if the subject walks with a cane, the subject identification unit 125 may detect the person holding the cane and identify the person as the subject.

[0089] B5. Alternative embodiment 5: In the embodiment, the evaluation device 100 evaluates the walking movement based on the change in the angle of the target part for each walking phase. Alternatively, the evaluation device 100 may evaluate the walking movement by considering the attributes of the person walking in addition to the change in the angle of the target part. The attributes of the person are, for example, age, the cause of the need to undergo walking training, and the period of time for which the person has undergone walking training.

[0090] In the embodiment, in order to evaluate the movement of the target part in each walking phase, an evaluation item is set for each target part for each walking phase (see Figs. 6 to 8). In another embodiment 1, an evaluation item corresponding to a predetermined age group is set. For example, a reference value that is set in advance for each evaluation item in each walking phase can be made different depending on the age group. Also, for example, different reference values ​​may be used when the cause of the need for walking training is aging, injury, or illness. Since an evaluation item corresponding to a person's attributes is used, for example, in walking training, the movement of the target part can be appropriately evaluated depending on the attributes of the subject.

[0091] B6. Alternative Embodiment 6: In the embodiment, an example has been described in which three indices, namely, walking stability, walking safety, and walking endurance, are used when evaluating walking. However, indices other than these may be used to evaluate walking. Walking may be evaluated based on one or two of the above three indices.

[0092] Alternatively, in addition to the above three indexes, the walking movement may be evaluated based on the walking speed. For example, a reference value for determining whether the walking speed is normal is set in advance. The reference value may be set for indoor walking and outdoor walking, respectively. If the walking speed calculated using the walking image data is equal to or greater than the reference value, the evaluation device 100 determines that the walking speed is normal. In this case, since the walking speed of a person can be evaluated, for example, in walking training, walking movement can be stably evaluated regardless of the experience of the physical therapist. Therefore, it is possible to suppress the occurrence of variation in the instruction content in walking training.

[0093] B7. Alternative Embodiment 7: The evaluation device 100 may evaluate the walking movement by considering the change in the angle of the target part of the subject, as well as the environment around the subject. For example, poor physical condition may occur due to temperature differences. Also, for example, poor physical condition may occur due to changes in atmospheric pressure. If the physical condition is poor due to such causes, the results of rehabilitation may not be easily seen during walking training. Therefore, in the evaluation device 100, the results of evaluating the movement of the target part in each walking phase and weather information may be input into a neural network that has been trained in advance, and the walking movement may be evaluated. The weather information is, for example, information on atmospheric pressure and temperature for the past three days including the current day. The evaluation device 100 can acquire weather information via a weather information service, for example, based on the location information of the imaging device 200.

[0094] B8. Alternative Embodiment 8: Furthermore, during walking training, the subject may walk with a wearable sensor attached to, for example, the wrist. This wearable sensor measures the subject's blood pressure and heart rate and transmits the measured values ​​to the evaluation device 100. For example, when the amount of change in blood pressure or the amount of change in heart rate while walking exceeds a predetermined standard, the evaluation device 100 may transmit advice to the imaging device 200 that the walking training should be interrupted.

[0095] B9. Alternative Embodiment 9: In the embodiment, the evaluation device 100 evaluates the walking movement based on the amount of change in the angle of the target part for each walking phase. Alternatively, the evaluation device 100 may evaluate the walking movement in consideration of the moving speed of the target part for each walking phase in addition to the amount of change in the angle of the target part for each walking phase.

[0096] B10. Alternative Embodiment 10: In the embodiment, the movements of the trunk, pelvis, hip joint, knee joint, ankle joint, and toes are evaluated. However, the target parts are not limited to these. For example, the target parts may further include the shoulder joint, elbow joint, and wrist joint. In this case, the evaluation device 100 evaluates the movement of the arm using the change in the angle of each of the shoulder joint, elbow joint, and wrist joint. The evaluation device 100 evaluates whether the arm is swinging too much compared to normal walking, whether the arm movement is a compensatory movement for stabilizing walking, and the like. The evaluation device 100 can evaluate the walking movement over the entire body by evaluating the movement of the shoulder joint, elbow joint, and wrist joint in addition to the trunk.

[0097] B11. Other embodiment 11: In the embodiment, the first evaluation unit 160 of the evaluation device 100 evaluates the target parts with the set evaluation items in each gait phase of the right and left lower limbs. Alternatively, the first evaluation unit 160 may evaluate the movement of the target parts for each gait phase using a neural network that has been trained in advance. For example, when information for identifying a gait phase and a plurality of input images in that gait phase are input, this neural network outputs the probability that the movement of each target part in that gait phase corresponds to a normal movement and the probability that it corresponds to an abnormal movement. Even in this case, the movement of each target part for each gait phase can be evaluated in detail.

[0098] B12. Other embodiment 12: Furthermore, the evaluation device 100 may store past evaluation results in the memory 102. For example, the evaluation device 100 may generate screen data of the evaluation results displayed together with a graph showing the change in angle in one walking cycle of a certain target part in the current walking (see FIG. 10) and a graph showing the change in angle in one walking cycle of a certain target part in the previous walking, and transmit the screen data to the imaging device 200. This allows, for example, a physical therapist to visually recognize the past walking state and the current walking state.

[0099] Alternatively, the evaluation device 100 may generate screen data of the evaluation result displayed together with a graph showing the change in angle in one gait cycle of a certain target part in the current walking (see FIG. 10) and a graph showing the change in angle in one gait cycle of a certain target part in normal walking, and transmit the screen data to the imaging device 200. This allows, for example, a physical therapist to visually recognize the degree to which the current walking state deviates from normal walking.

[0100] B13. Other embodiment 13: In the embodiment, the joint feature extraction unit 150 detects keypoints using an existing neural network model for keypoint detection. Alternatively, the joint feature extraction unit 150 may extract keypoints using a trained machine learning model. Alternatively, the joint feature extraction unit 150 may detect keypoints by using a known skeletal information acquisition technology to obtain features related to human joints. An example of a known skeletal information acquisition technology is OpenPose (URL=https: / / github.com / CMU-Perceptual-Computing-Lab / openpose). The joint feature extraction unit 150 can detect the positions of keypoints and the connection relationships of keypoints by using OpenPose.

[0101] B14. Other embodiment 14: In the embodiment, an example has been described in which the photographing device 200 supplies moving image data to the evaluation device 100. Alternatively, the photographing device 200 may supply data of a plurality of still images photographed at a determined time interval to the evaluation device 100. The determined time interval is, for example, every 0.1 seconds.

[0102] The control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0103] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined in order to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0104] 10...Evaluation system, 100...Evaluation device, 101...Processor, 102...Memory, 103...Communication interface, 104...Input / output interface, 109...Bus, 110...Input reception unit, 120...Result output unit, 125...Object identification unit, 130...Gait cycle determination unit, 140...Gait phase determination unit, 150...Joint feature extraction unit, 160...First evaluation unit, 170...Second evaluation unit, 200...Photographing device, M1, M2, M3, M5...Neural network model, C1...Gait cycle, P11...Assistant, P12...Photographer, T1...Subject

Claims

1. An evaluation system for evaluating a person's walking motion, An input receiving unit that receives walking image data obtained by continuously capturing an image of at least one whole body of a person performing a walking motion over at least one walking cycle; A subject identification unit that identifies a subject to be evaluated using a plurality of images when two or more people are included in the plurality of images included in the walking image data; a gait phase determination unit that divides each of the plurality of images into a plurality of gait phases included in one gait cycle of the subject; a joint feature extraction unit that extracts features related to the joints of the subject from the plurality of images; a part evaluation unit that outputs part evaluation results for each of the gait phases of a plurality of predetermined target parts of the body of the subject, the part evaluation results being obtained by evaluating, using the feature amount, the movement of each of the plurality of target parts including at least a trunk for each of the gait phases of the target parts; a comprehensive evaluation unit that outputs a comprehensive evaluation result obtained by evaluating the walking movement in at least one walking cycle using the part evaluation result for each of the walking phases of the target parts; An evaluation system comprising:

2. The evaluation system according to claim 1 , The subject identification unit, If the plurality of images includes two or more people, Among the two or more people, the person occupying the largest proportion of an area in the image is identified as the subject. Rating system.

3. The evaluation system according to claim 1 , The subject identification unit, If the plurality of images includes two or more people, Among the two or more people, a person who is furthest forward in a predetermined traveling direction is identified as the target person. Rating system.

4. The evaluation system according to claim 1 , The subject identification unit, If the plurality of images includes two or more people, Identifying the target person from among the two or more people whose faces and bodies are facing a predetermined traveling direction; Rating system.

5. The evaluation system according to claim 2 or 3, The subject identification unit, determining whether two or more people are detected in one image included in the plurality of images and whether corresponding two or more people are detected in another image included in the plurality of images; If two or more people are detected in the one image and two or more corresponding people are included in the other image, Identifying the target person from among the two or more people who are moving in a predetermined traveling direction. Rating system.

6. The evaluation system according to claim 5, The site evaluation result includes a result of evaluating the movement of each of the target sites for each of the walking phases for at least one evaluation item set for each of the target sites. Rating system.

7. The evaluation system according to claim 6, The part evaluation unit evaluates the movement of each of the target parts for each gait phase by using the evaluation items corresponding to attributes of the subject, the attributes including at least age. Rating system.

8. The evaluation system according to claim 5, The plurality of target sites include a hip joint and a knee joint, The comprehensive evaluation result includes a result of evaluating the stability of the subject's walking, obtained using the part evaluation results for each gait phase of the trunk, the hip joint, and the knee joint, Rating system.

9. The evaluation system according to claim 5, The plurality of target sites include a hip joint and a knee joint, The comprehensive evaluation result includes a result of evaluating the safety of walking of the subject obtained by using the part evaluation results for each walking phase of the trunk, the hip joint, and the knee joint, Rating system.

10. The evaluation system according to claim 5, The plurality of target sites include a hip joint and a knee joint, The comprehensive evaluation result includes a result of evaluating the walking endurance of the subject, which is obtained using the part evaluation results for each walking phase of the trunk, the hip joint, and the knee joint. Rating system.

11. The evaluation system according to claim 5, The plurality of target sites include a hip joint and a knee joint, The comprehensive evaluation result includes a result of evaluating the walking speed of the subject obtained by using the part evaluation results for each gait phase of the trunk, the hip joint, and the knee joint, Rating system.

12. A program for evaluating a person's walking motion, On the computer, A function of receiving walking image data in which the whole body of at least one person performing a walking motion is continuously photographed over at least one walking cycle; A function of identifying a subject to be evaluated using a plurality of images when two or more people are included in the plurality of images included in the walking image data; A function of dividing each of the plurality of images into a plurality of gait phases included in one gait cycle; A function of extracting features related to the joints of the subject from the plurality of images; a function of outputting a part evaluation result for each of a plurality of predetermined target parts of the body of the subject, the part including at least a trunk, obtained by evaluating a movement of each of the plurality of target parts for each of the gait phases using the feature amount; and a function of outputting a comprehensive evaluation result obtained by evaluating the walking movement in at least one walking cycle using the part evaluation result for each of the walking phases of the target parts; A program to achieve this.

Images