Skeleton estimation system

Abstract

To provide a skeleton estimation system capable of estimating a position of a necessary portion even when the portion necessary for analysis of a behavior of a person is hidden by another portion or a physical difference between subjects is great.SOLUTION: A skeleton estimation system 10 comprises: an input unit 11 which inputs image data obtained by imaging or sensing a user moving on a tread mill; and a control unit 12 which recognizes a position of a skeleton of the user based on the image data and obtains user skeleton information indicating the position of the skeleton. When a position of a first prescribed portion cannot be recognized and a position of a second prescribed portion can be recognized among portions of the skeleton whose positions are indicated by the user skeleton information, the control unit 12 estimates the position of the first prescribed portion that cannot be recognized currently, based on information indicating the position of the second prescribed portion that can be recognized currently and information indicating a positional relation between the first prescribed portion and the second prescribed portion that was recognized before.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a skeletal estimation system. [Background technology]

[0002] Patent document 1 describes a behavior analysis device that acquires a captured image including a person at a specified angle of view, extracts a person image area in which the person exists from the captured image, detects a person's skeleton from the person image area, and analyzes the person's behavior from the person's skeleton.

[0003] The behavior analysis device described in Patent Document 1 uses either the person image area or the person skeleton to determine whether a person is in a state where they will go out of the specified angle of view or where they are predicted to go out of the frame, and if they have, notifies the person of the determination.

[0004] Here, the occurrence of out-of-frame is determined by any one of the following: a part of the person image area is present in the outermost boundary of the photographed image, a part of the person skeleton is present in the outermost boundary of the photographed image, or the depth is not within a predetermined range. Also, the occurrence of out-of-frame is determined by any one of the following: the center of gravity of the person image area is not present within a predetermined range from the center of the photographed image, or the center of gravity of the person skeleton is not present within a predetermined range from the center of the photographed image, or the person skeleton is not present in the outermost boundary of the photographed image and a part of the person skeleton is present within a predetermined range from the outermost boundary of the photographed image toward the inside, or a part of the person skeleton necessary for behavior analysis is not present in the photographed image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-084951 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology described in Patent Document 1, even if only the occurrence or prediction of frame-out for the human skeleton necessary for analyzing human behavior is taken into consideration, it is determined that frame-out has occurred or has been predicted and a notification is given even when the necessary human skeleton is hidden by other parts or when the difference in the subject's physique cannot be accommodated. Such notifications are bothersome to the user and cause unnecessary stress to the user.

[0007] In order to prevent such bothersome notifications, it is desirable to develop a method for estimating whether or not a necessary part is present, which can be used even when the necessary part is hidden by other parts in analysis, or when there is a large difference in the physique of the subjects.

[0008] The present disclosure has been made to solve such problems, and has as its purpose to provide a skeletal estimation system that is capable of estimating the position of necessary body parts for analyzing human behavior even when the body parts are hidden by other body parts or when there is a large difference in the physique of the subjects. [Means for solving the problem]

[0009] A skeletal estimation system according to one embodiment of the present disclosure includes an input unit that inputs image data obtained by photographing or sensing a user moving on a treadmill, and a control unit that recognizes the position of the user's skeleton based on the image data and obtains user skeletal information indicating the position of the skeletal position, wherein when the position of a first predetermined part of the skeletal parts whose positions are indicated by the user skeletal information cannot be recognized but the position of a second predetermined part can be recognized, the control unit estimates the position of the first predetermined part that cannot currently be recognized based on information indicating the position of the currently recognized second predetermined part and information indicating the positional relationship between the first predetermined part and the second predetermined part that was previously recognized.

[0010] In the above-mentioned skeletal estimation system, the position of a first specified part that cannot currently be recognized is estimated based on the information indicating the above-mentioned positional relationship, so that the position of the necessary part can be estimated even if the part necessary for analyzing human behavior is hidden by other parts or if there is a large difference in the physique of the subjects. Effect of the Invention

[0011] The present disclosure makes it possible to provide a skeletal estimation system that can estimate the position of necessary body parts even when the body parts necessary for analyzing human behavior are hidden by other body parts or when there is a large difference in the physiques of the subjects. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a configuration of a skeleton estimation system according to an embodiment. FIG. [Diagram 2] 2 is a schematic side view showing an example of the configuration of a walking training device in which the skeletal structure estimation system of FIG. 1 can be mounted. [Diagram 3] 3 is a schematic diagram showing an example of an image displayed on the walking training device of FIG. 2. [Figure 4] FIG. 2 is a schematic diagram showing an example of human skeleton information; [Diagram 5] 3 is a schematic diagram showing an example of an image captured by the walking training device of FIG. 2 and an example of skeletal information recognized from the image. [Figure 6] 6 is a diagram showing user skeletal information extracted from the skeletal information of FIG. 5. FIG. [Figure 7] 3 is a flow diagram illustrating an example of a skeleton estimation process in the walking training apparatus of FIG. 2. [Figure 8] 8 is a diagram showing an example of skeleton information recognized in a certain scene by the skeleton estimation process of FIG. 7. [Figure 9] 8 is a diagram showing an example of skeleton information recognized in another scene by the skeleton estimation process of FIG. 7. [Figure 10] 8 is a diagram showing an example of skeleton information recognized in still another scene by the skeleton estimation process of FIG. 7. FIG. [Figure 11]FIG. 8 is a diagram showing an example of a table used in the skeleton estimation process of FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0014] (Embodiment) Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a skeleton estimation system according to the present embodiment.

[0015] As shown in FIG. 1, a skeleton estimation system 10 according to this embodiment can include an input unit 11 and a control unit 12.

[0016] The input unit 11 inputs image data obtained by photographing or sensing a user moving on a treadmill. The treadmill may be, for example, one installed in a walking training device or one installed in a training support device such as a treadmill.

[0017] The input image data may be camera image data, depth data, etc. The camera image data is data captured by a visible light or infrared camera. When camera image data is employed, such a camera may be provided in the skeleton estimation system 10 or connected to the outside. The depth data may be data obtained by mapping the depth of each position obtained by capturing an image using a depth sensor that measures the distance to the object by applying a laser light such as infrared light to the object, or a 3D camera. In this case, the depth sensor or the 3D camera may be provided in the skeleton estimation system 10 or connected to the outside. In addition, LiDAR data measured using LiDAR, which can be considered a type of depth sensor, may be input as image data.

[0018] The control unit 12 recognizes the position of the user's bones based on the image data input by the input unit 11, and obtains user bone structure information indicating the position of the bones.

[0019] In particular, when the control unit 12 is unable to recognize the position of a first predetermined part among the skeletal parts whose positions are indicated by the user skeletal information, but is able to recognize the position of a second predetermined part, the control unit 12 performs the following estimation. That is, in this case, the control unit 12 estimates the position of the first predetermined part that is not currently recognized based on information indicating the position of the second predetermined part that is currently recognized and information indicating the positional relationship between the first predetermined part and the second predetermined part that was previously recognized. A specific example of the estimation process by the control unit 12 will be described with reference to FIG. 2 onwards.

[0020] In this way, the skeleton estimation system 10 estimates the position of the first predetermined part that cannot currently be recognized based on the information indicating the above positional relationship. Therefore, the skeleton estimation system 10 can estimate the position of the necessary part even when the part necessary for analyzing human behavior is hidden by other parts or when there is a large difference in the physique of the subjects.

[0021] Furthermore, because the skeleton estimation system 10 is capable of making such estimations, it is possible to prevent unnecessary and bothersome notifications that are intended to prompt the user to change their position, thereby reducing stress for the user.

[0022] Next, a specific example of the estimation process by the control unit 12 will be described with reference to Fig. 2 to Fig. 11. In the following specific example, a treadmill is mounted on a walking training device 100 shown in Fig. 2, and a skeleton estimation system 10 is mounted on the walking training device 100, but the present invention is not limited to this example.

[0023] First, an example of a walking training device 100 in which a skeleton estimation system can be installed will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a schematic side view showing an example of a configuration of the walking training device 100 in which the skeleton estimation system 10 can be installed. Fig. 3 is a schematic view showing an example of an image displayed on the walking training device 100.

[0024] The walking training device 100 shown in Fig. 2 is a device for a trainee (user) U, who is a hemiplegic patient suffering from paralysis in one leg, to perform walking training under the guidance of a training staff PT. The training staff PT may be a therapist (physiotherapist) or a doctor, and may also be called a training instructor, training assistant, or training assistant, since the training staff PT assists the trainee in training by providing guidance or assistance. As illustrated here, the training staff PT is a person.

[0025] The walking training device 100 includes a lower frame 101, an upper frame 102, a front frame 103, and a rear frame 104, which mainly form the overall shape of the device. The walking training device 100 can also include handrails 105 for the trainee U to hold on to as needed when walking. The handrails 105 can be provided on both sides of the area where the trainee U walks, for example, on the lower frame 101.

[0026] The walking training device 100 may also include an attachment 106 for attaching a training monitor 115 and a camera 117, which will be described later. In order to attach the training monitor 115 and the camera 117 so as to be located in front of the trainee U, the attachment 106 may be disposed on the front frame 103. The upper frame 102 also supports a front pulling part 111f near the front of the upper part of the trainee U's head, a harness pulling part 111w near the upper part of the head, and a rear pulling part 111b near the rear of the upper part of the head.

[0027] The walking training device 100 may also include a control panel 114 that houses an overall control unit configured, for example, by an MPU (Micro Processing Unit) that controls various motors and various sensors. In other words, the overall control unit of the control panel 114 also controls the controllable parts of the walking training device 100. As an example, the overall control unit of the control panel 114 drives a belt drive unit 108 (described later) to adjust the rotation speed of the belt 107 according to the walking speed set by the training staff PT.

[0028] The control panel 114 can be supported, for example, by the lower frame 101. The overall control section of the control panel 114 can execute control of the entire apparatus by executing a control program read from the system memory.

[0029] The walking training device 100 also includes a treadmill that encourages the trainee U to walk. The treadmill can include an endless belt 107, i.e., a ring-shaped belt 107, and a belt drive unit 108. The belt drive unit 108 can include a motor that drives and rotates the belt 107 and a drive circuit for the motor. The trainee U who is undergoing walking training gets on the belt 107 and tries to walk in accordance with the movement of the belt 107. Note that the training staff PT can stand on the belt 107 behind the trainee U and walk together with the trainee U, for example, as shown in FIG. 2, but it is usually preferable for the training staff PT to be in a position that makes it easy to assist the trainee U, such as standing astride the belt 107.

[0030] The walking training device 100 may also include a fall prevention harness device as a safety device, the main components of which are an attachment 112, a harness wire 110w, and a harness tensioning unit 111w. The attachment 112 can be connected to one end of the harness wire 110w, which is a sling, and may also be called a hanger belt. The trainee U wears the attachment 112 so that the harness wire 110w is connected at the rear of the trainee. One end of the harness wire 110w is connected to the attachment 112, and the other end is connected to the harness tensioning unit 111w. The harness tensioning unit 111w may include a motor for tensioning the harness wire 110w and a drive circuit for the motor. The overall control unit of the control panel 114 controls the winding or unwinding of the harness wire 110w by sending a drive signal to the drive circuit of the harness tensioning unit 111w.

[0031] With this configuration, when the trainee U is about to fall, the fall prevention harness device winds up the harness wire 110w in accordance with an instruction from the overall control unit of the control panel 114 which detects the movement, and supports the upper body of the trainee U with the harness 112, preventing the trainee U from falling. The movement can be detected by a posture sensor (not shown) provided on the harness 112 for detecting the posture of the trainee U. This posture sensor is, for example, a combination of a gyro sensor and an acceleration sensor, and outputs the inclination angle of the abdomen to which the harness 112 is attached with respect to the direction of gravity.

[0032] The walking training device 100 may also include a walking assist device 109 that is attached to the affected leg, which is the paralyzed leg, of the trainee U. The walking assist device 109 is attached to the affected leg of the trainee U, and assists the trainee U in walking by reducing the load of extension and flexion at the knee joint of the affected leg. The walking assist device 109 includes a sensor that measures the load on the sole of the foot, and transmits various data related to the walking to an overall control unit of the control panel 114 via wired or wireless communication.

[0033] The walking assist device 109 may include, for example, a motor (not shown) that generates a driving force to assist the extension and flexion of the knee joint, a control unit (not shown) that controls the motor, and a plurality of frames that support each part of the affected leg. Although these frames are not shown in detail, adjacent frames among these frames may be rotatably connected, and a front wire 110f and a rear wire 110b may be connected to some of the frames. The front wire 110f and the rear wire 110b are connected to the front tension portion 111f and the rear tension portion 111b, respectively, and can be wound up or unwound by the front tension portion 111f and the rear tension portion 111b.

[0034] Furthermore, the walking assistance device 109 may include a load sensor (not shown) embedded in a frame supporting the soles of the feet to detect the load acting on the soles of the feet. This load sensor may be configured to detect the magnitude and distribution of the vertical load acting on the soles of the trainee U, and to detect, for example, the COP (Center Of Pressure). This load sensor is, for example, a resistance change detection type load detection sheet in which electrodes are arranged in a matrix.

[0035] Moreover, the front pulling unit 111f and the rear pulling unit 111b may include a motor for pulling the front wire 110f and the rear wire 110b, respectively, and a drive circuit for the motor. The overall control unit of the control panel 114 sends drive signals to the drive circuits of the front pulling unit 111f and the rear pulling unit 111b to control the winding or unwinding of the front wire 110f and the rear wire 110b, respectively. The overall control unit of the control panel 114 can assist the swinging motion of the affected leg by, for example, identifying the timing at which the affected leg switches from a stance state to a swing state from the detection result of the load sensor, and increasing or decreasing the pulling force of the front wire 110f and the rear wire 110b in synchronization with the timing.

[0036] In this way, the coordinated operation of the front pulling unit 111f and the rear pulling unit 111b cancels out the load of the walking assist device 109 so that the load does not become a burden on the affected leg, and further assists the swinging motion of the affected leg according to the setting level. For example, the training staff PT, as an operator, sets a high assist level for the trainee U who has severe paralysis. When the assist level is set high, the front pulling unit 111f winds up the front wire 110f with a relatively large force in accordance with the swinging timing of the affected leg. When the training progresses and assistance is no longer necessary, the training staff PT sets the assist level to the minimum. When the assist level is set to the minimum, the front pulling unit 111f winds up the front wire 110f with a force that is sufficient to cancel the weight of the walking assist device 109 in accordance with the swinging timing of the affected leg.

[0037] The walking training device 100 also includes a camera 117. The camera 117 functions as an imaging unit for capturing an image of the entire body of the trainee U. The camera 117 is installed near the training monitor 115 so as to face the trainee U during training. Although an example in which the camera 117 is attached to the lower side of the attachment tool 106 is shown in FIG. 2, the attachment position of the camera 117 is not limited thereto.

[0038] The camera 117 takes still images and videos of the trainee U during training, and transmits the obtained image signals to the overall control unit of the control panel 114 via wired or wireless communication. The camera 117 includes a lens and imaging element set that provides an angle of view that can capture the entire body of the trainee U when the trainee U is walking within an appropriate walking range. This angle of view is, for example, exemplified by the range between two dashed lines in FIG. 2. The imaging element can be, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and converts an optical image formed on an imaging surface into an image signal.

[0039] The walking training device 100 may also include a training monitor 115. The training monitor 115 is attached to the front frame 103, and may be, for example, a display panel such as a liquid crystal panel. The training monitor 115 is a display device that mainly allows the trainee U to visually check his / her own walking state and notifications such as advice and warnings given by the walking training device 100. Here, the advice may be advice that indicates what points to pay attention to when walking by feeding back the results of gait analysis.

[0040] The training monitor 115 can also be used by the training staff PT to visually check the walking condition of the trainee U. Images and the like displayed on the training monitor 115 can be transmitted from the overall control unit of the control panel 114. The images for visually checking the walking condition include real-time images captured by the camera 117, and can also include information showing the progress of the training and information for the notification.

[0041] The walking training device 100 may also include a management monitor 116. The management monitor 116 is attached to, for example, the housing of the control panel 114, and may be, for example, a display panel such as a liquid crystal panel. The management monitor 116 may also be provided with a touch panel on the display panel. The management monitor 116 is a display input device that is mainly used for monitoring and operation by the training staff PT. The management monitor 116 displays various menu items related to training settings, various parameter values ​​during training, training results, etc., and can accept settings of various menu items, various parameter values, etc. Images, etc. displayed on the management monitor 116 are transmitted from the overall control unit of the control panel 114, and operation signals received by the touch panel of the management monitor 116 are transmitted to the overall control unit of the control panel 114, and control corresponding to the operation signals is performed by the overall control unit.

[0042] The walking training device 100 may also include an audio output unit (not shown). The contents displayed on the display units exemplified by the training monitor 115 and the management monitor 116 may be notified by audio output from this audio output unit, or may be notified by both image output on the display unit and audio output from the audio output unit.

[0043] The control panel 114 can include the skeleton estimation system 10 of FIG. 1, that is, the input unit 11 and the control unit 12, as a part of the overall control unit.

[0044] In the following, an example will be described in which the image data input for estimating skeletal information is camera image data input from the camera 117, but as described above, the image data may be depth data, etc. In other words, the camera 117 can be used as a camera for obtaining the above-mentioned camera image data.

[0045] Input unit 11 receives an image signal from camera 117 and passes it to control unit 12. Control unit 12 generates image data indicated by the image signal received from input unit 11 from the image signal. Note that camera 117 can also be configured to generate image data from the image signal. In this case, camera 117 passes the image data to input unit 11 of the overall control unit of control panel 114, and input unit 11 passes the image data to control unit 12.

[0046] The control unit 12 controls the display of the image data thus generated or received on the training monitor 115. As a result, an image of the walking state of the trainee U, such as the camera image 20, is displayed in real time on the training monitor 115.

[0047] For the sake of convenience, in Fig. 3, images of some components, such as the training staff PT, the management monitor 116, wires, and harnesses, among the images captured of the walking training device 100 in Fig. 2 are omitted. Also, in Fig. 3, an image in which notification information 20a and user skeletal information 30 are superimposed on a camera image 20 is illustrated, but such superimposition may not be performed. The notification information 20a and skeletal image 30 will be described later.

[0048] Then, based on the image data thus generated or received, the control unit 12 recognizes the position of the skeleton of the trainee U, who is the user, and obtains user skeleton information indicating the position of the skeleton. The user skeleton information will be described later. The control unit 12 can generate the obtained user skeleton information 30 and display it on the training monitor 115.

[0049] The frequency at which the user skeletal information 30 is obtained may differ from the frequency at which the camera image 20 displayed on the training monitor 115 is updated. For example, while the camera image 20 is displayed on the training monitor 115 in real time at the shooting rate, the user skeletal information 30 may be superimposed on the camera image 20 every time user skeletal information is obtained and displayed on the training monitor 115. Also, it is possible not to superimpose the user skeletal information 30 on the camera image 20. The control unit 12 may also cause the management monitor 116 to display the user skeletal information 30.

[0050] Here, the user skeletal information will be described with reference to Fig. 4 to Fig. 6. First, the skeletal information will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of human skeletal information.

[0051] Skeleton information is information used for gait analysis, and indicates the position of a person's skeleton. Skeleton information can include, for example, a plurality of joint points as shown by circles in Fig. 4, but can also include lines connecting adjacent joint points, or can include the lines instead of the joint points.

[0052] In the following example, an example will be described in which 10 joints indicated by large circles, SR, SC, SL, HR, HC, HL, KR, KL, AR, and AL, among the joints of the person HU illustrated in Fig. 4, are recognized as skeletons to be used in gait analysis. However, depending on the part to be trained or the extent to which analysis is performed, some or all of the other joints indicated by small circles in Fig. 4 or other joints not shown in Fig. 4 can also be analyzed.

[0053] In Figure 4, joint SR is the right shoulder joint, joint SC is the shoulder part of the spine, joint SL is the left shoulder joint, joint HR is the right hip joint, joint HC is the base part of the spine, joint HL is the left hip joint, joint KR is the right knee joint, joint KL is the left knee joint, joint AR is the right ankle joint, and joint AL is the left ankle joint. Note that these names of joints are given only as examples.

[0054] The control unit 12 can detect (recognize) the positions of the joints as the positions of the skeleton by analyzing the image data that is the subject of the gait analysis. The method for recognizing the position of the skeleton from the image data does not matter, and the position of the skeleton can also be recognized from a plurality of image data that are continuous in time. A simpler recognition method is to add an identification marker to the position of the skeleton on the person or the prosthetic worn by the person and detect the marker, but it is possible to recognize the position of the skeleton without adding a marker. For example, the skeleton information can be recognized as the output result of inputting the image data into a learning model that has undergone machine learning such as deep learning.

[0055] Next, the user skeletal information will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a schematic diagram showing an example of an image captured by the walking training device 100 and an example of skeletal information recognized in the image. Fig. 6 is a diagram showing user skeletal information (skeletal information of a subject) extracted from the skeletal information in Fig. 5.

[0056] For example, in the walking training device 100, when the training staff PT is assisting the trainee U from behind, the control unit 12 performs image analysis on image data shown as an image 50 including the trainee U and the training staff PT. The control unit 12 detects the positions of the skeletons through this image analysis, and recognizes the positions of the skeleton groups illustrated as user skeleton information 30, which is the skeleton information of the trainee U in Fig. 5, and the positions of the skeleton groups illustrated as skeleton information 40 of the training staff PT in Fig. 5. The position of each skeleton can be recognized as coordinates on the image data.

[0057] However, for convenience, Fig. 5 also depicts the left foot and left ankle joint 30AL outside the range of the image data, but the image data subject to image analysis in this example does not include data showing an image of the left foot of the trainee U, and therefore the control unit 12 is in a situation where it is unable to recognize the position of the left ankle joint AL of the trainee U. Also, for convenience, each component of the walking training device 100 is omitted from the image 50 in Fig. 5.

[0058] The method of distinguishing between the user skeletal information 30 and the skeletal information 40 in recognizing the positions of such skeletal groups is not limited, but for example, the distinction can be made as follows. That is, the control unit 12 can distinguish the detected skeletal groups for each person by connecting adjacent skeletons among the detected skeletal groups that can be determined to be the skeletons of the same person according to predetermined skeletal information such as that in FIG. 4. Then, as a result of this distinction, the control unit 12 specifies the skeletal information consisting of the skeletal group in the front as the user skeletal information 30, which is the skeletal group of the trainee U. For example, the skeletal information consisting of the skeletal group in the front can be determined by specifying the skeleton with the largest distance between the right shoulder and the left shoulder among the skeletons.

[0059] Alternatively, the control unit 12 can determine, among the skeletal information obtained by connecting the skeletons together, the one with the most complete skeletons as the skeleton of the trainee U and exclude the other skeletons (skeleton of the training staff PT), thereby generating the user skeletal information 30. Note that, when the image data includes depth data, the control unit 12 can specify the skeletal information with the shallower depth as the user skeletal information 30.

[0060] However, without being limited to these examples, if the control unit 12 can distinguish and recognize the skeletal position of the trainee U from the skeletal positions of persons other than the trainee U based on the image data that is the subject of gait analysis, it can extract the skeletal group of the trainee U, that is, extract it in distinction from the skeletal groups of other persons such as the training staff PT.

[0061] In this manner, the control unit 12 can specify skeletal information of the trainee U, that is, user skeletal information 30. The resultant specified user skeletal information 30 includes the positions of nine joints 30SR, 30SC, 30SL, 30HR, 30HC, 30HL, 30KR, 30KL, and 30AR, as shown as user skeletal information 60 in Fig. 6. As described above, the position of each joint can be recognized as a coordinate on the image data.

[0062] As described above, in this example, the position of the left ankle joint AL, which cannot be recognized, is not included in the user skeleton information 60. However, the position of the left ankle joint AL is often necessary for gait analysis. When the position of the left ankle joint 30AL is necessary for gait analysis, the control unit 12 estimates the position of the left ankle joint 30AL. In this embodiment, a skeleton estimation process is performed to estimate the skeleton that could not be recognized.

[0063] A specific example of the skeleton estimation process in the walking training device of Fig. 2 will be described with reference to Fig. 7 to Fig. 10. Fig. 7 is a flow diagram for explaining an example of the skeleton estimation process in the walking training device 100. Figs. 8 to 10 are diagrams showing examples of skeleton information recognized in different scenes by the skeleton estimation process of Fig. 7.

[0064] First, the control unit 12 detects a skeleton from image data that is the subject of gait analysis (step S1), and identifies the skeleton of the trainee U, who is the subject, from the detected skeleton (step S2). The processing example of steps S1 and S2 is as described above, and this identifies user skeletal information such as the user skeletal information 60.

[0065] Next, the control unit 12 determines whether or not all of the skeletons necessary for gait analysis have been recognized (step S3). In step S3, information indicating the skeletons necessary for gait analysis is stored in advance, and the information is compared with the detected skeleton information to determine whether or not all of the skeletons have been recognized.

[0066] A case where step S3 is YES is a case where information on all joints is included, for example, as in the user skeletal information 80 shown in Fig. 8. On the other hand, in the example of the user skeletal information 60 in Fig. 6, the left ankle joint 30AL is a skeleton that is necessary for gait analysis but cannot be recognized, so that step S3 is determined as NO. As in this example, in a situation where the trainee U is performing walking training facing the camera 117, the right ankle joint or the left ankle joint may be out of the frame toward the bottom of the image and cannot be recognized.

[0067] If the answer is YES in step S3, the control unit 12 calculates and stores in the internal storage device the maximum value of the distance from the hip joint to the vertical coordinate (hereinafter, Y coordinate) within a recent fixed time period. In the example of the user skeleton information 80, the maximum value within a recent fixed time period of the distance H1 between the left hip joint 30HL and the left ankle joint 30AL is stored as the Y coordinate distance. Similarly, for the left foot, the distance between the left hip joint 30HL and the left ankle joint 30AL can be stored. The maximum value stored in step S3 is referred to in the estimation process in step S8, which will be described later. The meaning of this maximum value will be described later together with the explanation of step S8.

[0068] The above-mentioned certain time can be set as a fixed value such as 5 seconds or 10 seconds, or can be set as a time that is a predetermined multiple of a walking cycle. In the latter case, it is sufficient to detect the time indicating one walking cycle by walking analysis. In the processing example of FIG. 7, since the information required for walking analysis is available in step S3, it is possible to detect the time indicating one walking cycle. Unlike the processing example of FIG. 7, even if an unrecognized skeleton is included in one walking cycle, it is possible to detect the time indicating one walking cycle by performing a walking analysis only from a recognizable skeleton, for example, a walking analysis from the rotation state of the joints 30HR, 30HC, and 30HL as a provisional analysis.

[0069] In step S3, it is determined whether all of the bones necessary for gait analysis can be recognized. Alternatively, the control unit 12 may simply determine whether or not both the ankle joint position and the hip joint position can be recognized. In this case, in step S4, the above maximum value may be retained if both are recognized.

[0070] 7, the control unit 12 can store the user skeletal information 80 for which the result in step S3 is YES in association with time information, and can perform gait analysis based on the time-series user skeletal information 80. Any method of gait analysis can be used. The results of gait analysis can also be displayed on the training monitor 115 or the management monitor 116.

[0071] Following step S4, the control unit 12 determines whether the current position of at least one of the right ankle joint 30AR and the left ankle joint 30AL is equal to or lower than the warning line, i.e., whether it is located below the warning line (step S5). Thus, the object of determination in step S5 following step S4 is the current positions of the right ankle joint 30AR and the left ankle joint 30AL. The warning line can be indicated at a predetermined distance (distance of the Y coordinate) from the bottom edge of the user skeleton information 80. The predetermined distance is a threshold value for determining whether it is near the outermost contour (the bottom edge in this example) and can be determined in advance.

[0072] The bottom side of the user skeleton information 80 corresponds to the bottom side of the angle of view of the camera 117. The above-mentioned predetermined distance can be set as a fixed value, or can be set as a predetermined percentage (e.g., 3%) of the height of the image indicated by the image data. This predetermined percentage may be changed according to the height of the trainee U, and in this case, the height of the trainee U may be input in advance or measured.

[0073] If the result is NO in step S5, the control unit 12 ends the process. Note that the process example shown in Fig. 7, including the process described later, can be performed sequentially at the timing when a skeleton is detected from image data.

[0074] On the other hand, if at least one of the right ankle joint 30AR or the left ankle joint 30AL is located within a predetermined distance from the bottom edge of the user skeletal information 80, the control unit 12 determines that it is below the warning line (YES in step S5), issues a warning to the trainee U to move away from the camera 117 (step S6), and terminates the processing.

[0075] In this example, since the determination process of step S5 is adopted, even if both the right ankle 30AR and the left ankle 30AL are included within the angle of view, if they are below the warning line, a warning is issued because there is a possibility that they may go outside the angle of view. However, for example, a process can be adopted in which a warning is not issued if they are within the angle of view, in which case the warning line is made to coincide with the bottom side of the angle of view, that is, the above-mentioned predetermined distance is set to zero.

[0076] The warning in step S6 can be given, for example, by superimposing the notification information 20a shown in Fig. 3 on the camera image 20. The notification information 20a to be superimposed may be stored in advance so that it can be read out at the time of superimposition. The notification information 20a in Fig. 3 notifies the user with a sentence such as "Please step back a little," but the content of the notification is not limited to this and may be, for example, "You are stepping forward too far. Please step back and walk."

[0077] The notification information to be superimposed may include information to warn the user when a part required for gait analysis (in this case, the ankle joint) is not included in the camera image 20, that is, when the ankle joint is out of the viewing angle of the camera 117, or information to prompt the user to adjust the walking position. The notification information may be output as sound from the above-mentioned sound output unit, or may be output as both an image and sound.

[0078] On the other hand, if the answer is NO in step S3, the control unit 12 judges whether or not only the ankle joints cannot be recognized and all other joints including the hip joints can be recognized for both the left and right (step S7). If the answer is NO in step S7, that is, if the answer is not YES for both the left and right, the control unit 12 ends the process. Here, if the answer is NO in step S7, this means that the trainee U's skeleton has not been recognized to the extent necessary for estimation, and even if the trainee U is asked to step back, it is not necessarily possible to recognize the skeleton, so the process ends and no notification is given to encourage the trainee U to step back. The judgments in steps S3 and S7 can be performed substantially simultaneously.

[0079] If step S7 is YES, the control unit 12 estimates the position of the ankle joint that cannot be currently recognized based on the previously recognized positional relationship between the ankle joint and the hip joint and the information indicating the currently recognized position of the hip joint (step S8), and proceeds to step S5. Step S8 is executed for the ankle joint on the side for which step S7 is YES, and if neither the left ankle joint 30AL nor the right ankle joint 30AR can be recognized and both the left hip joint 30HL and the right hip joint 30HR can be recognized, then both the position of the left ankle joint AL and the position of the right ankle joint AR are estimated.

[0080] The estimation process in step S8 will be described. During walking, the distance between the hip joint and the ankle joint periodically changes on the image data for both the left and right sides. Therefore, in the estimation process for estimating the positions of the left ankle joint 30AL and the right ankle joint 30AR in step S8, the estimation is performed using the maximum value of the distance (Y coordinate distance) between the left hip joint 30HL and the left ankle joint 30AL over a certain period of time, and the maximum value of the distance (Y coordinate distance) between the right hip joint 30HR and the right ankle joint 30AR over a certain period of time. In order to use this estimation process, in step S4, the above maximum values ​​obtained when the left hip joint 30HL and the left ankle joint AL, the right hip joint 30HR and the right ankle joint AR, which are skeletons required for estimating the left ankle joint AL and the right ankle joint AR, respectively, can be recognized are stored.

[0081] In the estimation process of step S8, the control unit 12 estimates that the Y coordinate position of the left ankle 30AL is the Y coordinate position of the left hip joint 30HL indicated by the user's skeletal information (i.e., the current position) extended vertically downward by the maximum value held in step S4. Similarly, the control unit 12 estimates that the Y coordinate position of the right ankle 30AR is the Y coordinate position of the right hip joint 30HR indicated by the user's skeletal information (i.e., the current position) extended vertically downward by the maximum value held in step S4. An example of this estimation process will be described later with reference to Figures 9 and 10.

[0082] Also, as described above, in step S4, the left and right can be distinguished, and the maximum value can be found and stored. However, the same value can be used for the distance between the hip joint and the ankle joint on the left and right sides. In that case, the maximum value can be stored for the distance between the left hip joint 30HL and the left ankle joint 30AL and the distance between the right hip joint 30HR and the right ankle joint 30AR at a certain time. Also, in step S4, other statistical values ​​such as the median, mode, and average value can be calculated and stored instead of the maximum value, and in that case, the statistical value is used in the estimation in step S8. Note that, if there are multiple modes, for example, the largest mode can be stored and used in the estimation. Also, instead of the mode, the center value of the mode range can be stored and used in the estimation. Here, the mode range refers to the range that appears most frequently among the ranges obtained by dividing the distance value into predetermined ranges.

[0083] Following step S8, the control unit 12 performs the determination of step S5. That is, the control unit 12 determines whether or not the estimated position of at least one of the right ankle 30AR and the left ankle 30AL is equal to or lower than the warning line, i.e., whether or not it is located below the warning line. Thus, the subject of the determination in step S5 following step S8 is the position of the right ankle 30AR and the left ankle 30AL estimated in step S8.

[0084] If the answer is YES in step S5, as described above, the control unit 12 causes the notification information 20a to be displayed on the training monitor 115. The notification information in this case can also be output as sound from the above-mentioned sound output unit, or as both an image and sound. In this way, the control unit 12 can cause the notification unit to execute a notification urging the user to adjust the position when the position of the ankle cannot be recognized and the estimated position of the ankle is outside the predetermined area (below the warning line in the above example).

[0085] For example, if the left ankle joint is hidden behind the right foot, the ankle joint may not be hidden because the trainee U has gotten too close to the camera 117, but in this embodiment, it is not necessary to issue a warning notification in such a case.

[0086] Next, the estimation process in step S8 will be specifically described by taking an example of a situation where estimation of the ankle joint is required. Fig. 9 shows an example in which the distance H2 on the left foot side is stored in step S4, and user skeletal information 91 is obtained. Distance H2 indicates the maximum value in a certain period of distance H1 obtained in user skeletal information 80 in Fig. 8, for example.

[0087] In the skeleton recognition state indicated by the user skeleton information 91, the left ankle joint 30AL cannot be recognized, and the result of step S7 is YES. Therefore, the control unit 12 executes the estimation process of step S8 for the left ankle joint 30AL. In this case, the estimation process can estimate the Y coordinate position of the left ankle joint 30AL by extending the Y coordinate position vertically downward from the position of the left hip joint 30HL indicated by the user skeleton information 91 (i.e., the current position) by the distance H2 held in step S4. In FIG. 9, the estimated position of the left ankle joint 30AL is illustrated by a black circle. In this manner, the vertical position of the left ankle joint 30AL relative to the left hip joint 30HL can be estimated. The horizontal position of the left ankle joint 30AL (assumed to be the X coordinate position) may be estimated to be the same as the X coordinate position of the left knee joint 30KL as illustrated in FIG. 9, or may be estimated to be the same as the X coordinate position of the left hip joint 30HL, or may be estimated to be the X coordinate position obtained from the X coordinate positions of both the left knee joint 30KL and the left hip joint 30HL by a predetermined function.

[0088] In the example of Figure 9, the control unit 12 determines in step S5 whether the estimated position of the left ankle joint 30AL is located below the warning line indicated by a predetermined distance (threshold value) TH, and as a result, it is determined that it is located below, so executes a warning notification in step S6.

[0089] FIG. 9 shows an example in which the estimated Y coordinate position of the left ankle joint 30AL is below the bottom edge (outside the angle of view). However, if the estimated Y coordinate position of the left ankle joint 30AL is outside the warning line, a warning notification will be issued since there is a possibility that it may be outside the angle of view even if it is not outside the angle of view.

[0090] 10 shows an example in which the distance H3 on the left foot side is stored in step S4, and user skeletal information 92 is obtained. Like the distance H2, the distance H3 indicates the maximum value for a certain period of the distance H1 obtained in the user skeletal information 80 in FIG.

[0091] In the skeleton recognition state indicated by the user skeleton information 92, the left ankle joint 30AL cannot be recognized, and the result of step S7 is YES, so that the control unit 12 executes the estimation process of step S8 for the left ankle joint 30AL. The scene in Fig. 10 may occur, for example, when the left ankle joint 30AL is hidden behind the right foot.

[0092] In this case, the estimation process can estimate the Y coordinate position of the left ankle 30AL by extending the Y coordinate position vertically downward from the position of the left hip joint 30HL indicated by the user skeleton information 92 (i.e., the current position) by the distance H3 held in step S4. In FIG. 10, the estimated position of the left ankle 30AL is illustrated by a black circle. In this manner, the vertical position of the left ankle 30AL relative to the left hip joint 30HL can be estimated. In the example of FIG. 10, the X coordinate position of the left ankle 30AL can be estimated in the same manner as in the example described in FIG. 9. In the above-mentioned scene where the left ankle 30AL is hidden behind the right foot, the estimated position will differ from the actual position, but it is at least useful as an estimation for determining whether or not a warning notice is required to be sent to the trainee U.

[0093] In the example of Figure 10, the control unit 12 determines in step S5 whether the estimated position of the left ankle joint 30AL is located below the warning line indicated by a predetermined distance (threshold value) TH, and as a result determines that it is not located below (located above), so ends the processing without issuing a warning notification in step S6.

[0094] In the above, the processing example of FIG. 7 has been described on the assumption that the control of step S6 is included, but is not limited thereto. For example, when the position of the ankle cannot be recognized and the estimated position of the ankle is below the warning line (outside the predetermined area), the control unit 12 may transmit an instruction to adjust the angle of view to the source of image data. Specifically, instead of the control of step S6, the control unit 12 may control the lens of the camera 117 to be adjusted (zoomed out) so that an image can be captured at a wider angle. Then, after performing such control, only when the result of step S5 is YES even in a state where the image can be captured at the widest angle, a notification may be made in step S6.

[0095] In addition, in this embodiment, estimation of the position of the ankle joint is not limited to the combination of the ankle joint and the hip joint (or the hip joint and the knee joint when the knee joint is also used in the case where the X-coordinate position of the ankle joint is also estimated) as in the example described in Figures 2 to 10, but estimation of the position of the first predetermined part can also be performed for a combination of the first predetermined part and the second predetermined part other than this combination. Here, in the example described in Figures 2 to 10, the first predetermined part and the second predetermined part correspond to the ankle joint and the ankle hip joint, respectively.

[0096] That is, as described above, when the control unit 12 cannot recognize the position of the first predetermined part among the parts of the skeleton whose position is indicated by the user skeleton information, but can recognize the position of the second predetermined part, the control unit 12 estimates the position of the first predetermined part that cannot be currently recognized based on information indicating the position of the currently recognized second predetermined part and information indicating the positional relationship between the previously recognized first predetermined part and the second predetermined part (positional relationship information). The second predetermined part may be two or more parts. In the example described in FIG. 2 to FIG. 10, the ankle joint is given as an example of the first predetermined part, and the hip joint is given as an example of the second predetermined part, but the second predetermined part may be, for example, the knee joint, or the knee joint and the hip joint.

[0097] Thus, for the information used in the generalized example for the first predetermined part and the second predetermined part, an example will be described in which the first predetermined part is the ankle joint and the second predetermined part is the hip joint. In the process of FIG. 7, the control unit 12 can execute the process with reference to the two parts used in the determination process of step S7, the information indicating the positional relationship (positional relationship information) used in step S8, and the threshold value used in the determination process of step S5.

[0098] These pieces of information can be stored as a table as shown in FIG. 11 in a storage device inside or outside the control unit 12 and can be read out during processing. FIG. 11 is a diagram showing an example of the table used in the skeleton estimation process of FIG. 7.

[0099] The table shown in FIG. 11 stores the first predetermined part, the second predetermined part, their positional relationship information, and the threshold value. The first predetermined part, the second predetermined part, etc. are in accordance with the examples in FIGS. 2 to 10, but are not limited to this. Also, as can be seen from the examples given for the left and right feet, a plurality of sets of the first predetermined part and the second predetermined part can also be described in the table.

[0100] Also, in FIG. 11, a predetermined distance (threshold value) from the bottom side is shown as TH. However, as described above, this predetermined distance is not limited to a fixed value. Also, a plurality of threshold values of the predetermined distance can be provided. In that case, for example, if the ankle joint recognized or estimated is located below the first threshold value Th1, a warning can be given to lower by 0.2 m backward, and if the ankle joint recognized or estimated is located below the second threshold value Th2 (<Th1), a warning can be given to lower by 0.4 m. Note that appropriate information can be selected from the pre-stored notification information to be superimposed.

[0101] In addition, in the processing example of Fig. 7, the vertical position of the ankle joint is estimated from the current hip joint position and the maximum value when the ankle joint cannot be recognized, so the positional relationship information is exemplified as information showing a statistical value of the vertical distance between the first and second predetermined parts for a certain period of time, but is not limited to this. That is, Fig. 11 shows an example in which the vertical distance, i.e., the Y coordinate distance, is used as the positional relationship information, but the positional relationship information is not limited to the vertical distance.

[0102] For example, the positional relationship information may be information indicating a statistical value of the horizontal distance between the first and second predetermined parts over a certain period of time. Alternatively, the positional relationship information may be information on both the vertical and horizontal distances. In this case, thresholds may be set for both distances. As an equivalent process, the positional relationship information may be information indicating a statistical value of the direction and distance of a straight line connecting the first and second predetermined parts over a certain period of time. In this case, thresholds may be set for both the direction and the distance. Depending on the configuration of the walking training device 100, the camera 115 may be installed above the trainee U, and the first predetermined part may be other than the ankle, so such an application example is useful.

[0103] The above describes an example of the configuration and processing of the walking training device 100 equipped with the skeleton estimation system 10 of Fig. 1, and an application example thereof. As described above, when the walking training device 100 is unable to recognize a desired first predetermined part, the walking training device 100 estimates the position of the desired first predetermined part from previously acquired positional relationship information between the position of the first predetermined part to be recognized and another part (second predetermined part) and the position of the second predetermined part recognized this time. Therefore, the walking training device 100 can estimate the position of the necessary part even when a part required for analyzing human behavior is hidden by another part or when there is a large difference in physique between subjects.

[0104] This effect will be supplemented. In fact, in the walking training device 100, in order to detect the skeleton of the trainee U and use it for gait analysis such as gait analysis and feedback to the trainee U, the trainee U needs to train within the shooting range of the camera 117 required to detect the skeleton, but there may be a situation where the trainee U walks outside the shooting range. Examples of such a situation include a situation where a part required for analyzing human behavior is hidden by another part or a situation where the physique difference between the subjects is large. In this embodiment, since it is possible to estimate the first predetermined part in response to such a situation, it is possible to estimate the position of the necessary first predetermined part even in such a situation. In addition, as a comparative example, in an example in which a logic for determining based on the range in which the center of gravity position of the detected skeleton exists is adopted, it is difficult to determine whether the joint that cannot be detected is out of frame because the physique difference between the subjects cannot be absorbed, or whether it cannot be detected due to another factor. On the other hand, in this embodiment, unlike such logic, it is possible to estimate the position of the necessary first predetermined part.

[0105] In addition, when there are multiple necessary skeletons, even if the parts outside the field of view are brought into the field of view in a situation where the skeletons of parts other than the part outside the field of view cannot be recognized, the parts necessary for analysis are not collected and analysis cannot be performed. In other words, even if a warning notification is generated to prompt the trainee U to adjust his / her walking position in such a situation, analysis cannot be performed. Therefore, if a warning notification is generated only for the event that the necessary skeleton is not detected or the event that the center of gravity position is not within a specified range according to the above logic, it will not necessarily be caused by the body being outside the field of view, and as a result, an unnecessary warning will be generated, which will cause unnecessary stress to the trainee U, or a necessary warning will not be generated.

[0106] In contrast, in this embodiment, when the first target part is outside the angle of view or near the outermost periphery of the angle of view, it is considered that the target part may not be accurately acquired or may not be acquired in the future. Therefore, only when the recognition result or estimated position of the first predetermined part is near the outermost periphery or outside the acquireable range and the following condition is satisfied, a warning is issued indicating how to move to return to the acquireable range. The above condition is that a desired part other than the part is recognized, and the target part is allowed to be analyzed by entering the acquireable range. In this way, in this embodiment, a warning is issued only when the gait analysis functions normally due to the warning, so that the stress of the trainee U caused by unnecessary warnings can be reduced.

[0107] <Alternative examples, etc.> In the above embodiment, the training staff PT is assumed to be one person, but the present invention may be applied to a case where there are multiple training staff or no training staff. Also, a non-human training assistant (mechanical, i.e., artificial training assistant) may be used instead of or in addition to the training staff PT. Examples of artificial training assistants include various types, such as humanoid robots, voice assistant programs for providing voice notifications, and display assistant programs for providing notifications by display.

[0108] When the training assistant is a program, it can be executablely incorporated in the walking training device 100, but it can also be executablely incorporated in a portable terminal such as a mobile phone (including those called smartphones) or a mobile PC, or an external server, which can communicate with the walking training device 100. The artificial training assistant can also be equipped with an artificial intelligence program (AI program).

[0109] In the above embodiment, the trainee U is assumed to be a hemiplegic patient suffering from paralysis in one of his legs, but the walking training device 100 may also be applied to a patient suffering from paralysis in both legs. In that case, the trainee performs training by wearing the walking assistance device 109 on both legs.

[0110] Furthermore, the configuration and shape of the walking training device 100 described in the above embodiment do not matter as long as the skeleton estimation system 10 is mounted or connected. Furthermore, the skeleton estimation system described in the embodiment can be configured with multiple devices by distributing functions. Similarly, the walking training device can be configured with multiple devices as a walking training system, and the training support device can be configured with multiple devices as a training support system. In either system, when the system is configured with multiple devices, the devices may be connected to each other by wire or wirelessly.

[0111] As in the above-described embodiment, a configuration example in which the skeleton estimation system 10 is incorporated in the walking training device 100 equipped with the camera 117 has been given, the present disclosure may also take the form of a walking training system equipped with a skeleton estimation system, an image acquisition unit, and a treadmill body. This walking training system may also be called a mobility training system, a mobility assistance system, or the like. In this walking training system, the control unit exemplified by the control unit 12 may be located anywhere, and may be located in the treadmill body, for example.

[0112] Furthermore, any one or more devices constituting the above-mentioned walking training device, training support device, or skeleton estimation system or walking training system may have a hardware configuration including, for example, a processor, a memory, a communication interface, etc. These devices are realized by the processor reading and executing a program stored in the memory.

[0113] Such a program includes instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, a transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0114] 10 Skeleton Estimation System 11 Input section 12 Control section 20 Camera Images 20a Notification Information 30, 60, 80, 91, 92 User skeleton information 40 Skeletal information of training staff 50 images 100 Walking training device 101 Lower Frame 102 Upper Frame 103 Front Frame 104 Rear Frame 105 Handrail 106 Mounting fixture 107 Belt 108 Belt drive unit 109 Walking aids 110b Rear wire 110f Front Wire 110w harness wire 111b Rear tension part 111f Front tension section 111w Harness tension part 112 Orthotics 114 Control Panel 115 Training Monitor 116 Management Monitor 117 Camera U Trainer PT Training Staff

Claims

[Claim 1] an input unit for inputting image data obtained by photographing or sensing a user moving on a treadmill; a control unit that recognizes a position of a skeleton of the user based on the image data and obtains user skeleton information indicating the position of the skeleton; Equipped with when the control unit cannot recognize a position of a first predetermined part among parts of the skeleton whose position is indicated by the user skeleton information, but can recognize a position of a second predetermined part, the control unit estimates the position of the first predetermined part that cannot be currently recognized based on information indicating the position of the second predetermined part that is currently recognized and information indicating a positional relationship between the first predetermined part and the second predetermined part that was previously recognized. Skeleton estimation system.

Images