Function improvement support device and function improvement support method

Abstract

To achieve a function improvement support device and a function improvement support method capable of achieving functional improvement in complexed aspects of a physical aspect, a mental aspect, and a social aspect of a subject by further improving a degree of self-supporting in daily life.SOLUTION: Mechanism switching, deformation state, and power application state of a state holding variable mechanism part are controlled on the basis of discrimination results and measurement results while discriminating a current situation of a body function and a cognitive function of a subject on the basis of person information including brain / nervus system information, muscle / skeleton system information, physiological information, mental information, operation information, and action information of the subject, and measuring a normal force at a contact portion of the state holding variable mechanism part with the subject in real time.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a function improvement support device and a function improvement support method, and is particularly suitable for application in improving the functions of daily life of a subject in a bedridden state for a relatively long time.

Background Art

[0002] Conventionally, once physical strength weakens and a person becomes bedridden, frailty (muscle strength decline) and cognitive ability decline due to bedridden state accelerate, and it has been very difficult to lead an independent daily life since the stage of becoming bedridden.

[0003] Such bedridden people are in a state of wearing diapers all day long, and it is also very difficult to go to the toilet by themselves and excrete in their daily life. In reality, they can only see the ceiling, the delirious state intensifies, and there is a very high possibility of completely falling into a severe care state both physically and cognitively, and there is a problem that it becomes very difficult to lead an independent daily life such as shopping, walking, and communication.

[0004] In order to overcome such problems, conventionally, a walking assist chair that can be used as both a wheelchair and a walker has been proposed to improve the mobility and walking ability of bedridden people (see Patent Document 1).

[0005] The walking assist chair in this Patent Document 1, when used as a wheelchair, forms a wheelchair seat surface from a central elevating seat surface and a pair of retractable seat surfaces on its left and right. On the other hand, when used as a walker, while retracting the pair of retractable seat surfaces, only the elevating seat surface is raised so that the user can walk while putting weight on the elevating seat surface by simply lowering the waist a little.

[0006] In addition, this walking assist chair is configured such that when used as a walker, the vehicle part can be moved in the walking direction in accordance with the movement of the user, thereby reducing the force with which the user pushes the walking assist chair.

[0007] And, an improved version of this walking assist chair has been proposed by the same inventor as in Patent Document 1 (see Patent Document 2). The walking assist chair in this Patent Document 2 not only operates as both an electric wheelchair and a walking aid, but also serves as a standing assist device that aids the user seated in the electric wheelchair to stand up when transitioning from the electric wheelchair to the walking aid, and as a seating assist device that aids the user walking with the walking aid to sit down on the electric wheelchair when transitioning from the walking aid to the electric wheelchair.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, the walking assist chairs of Patent Documents 1 and 2 both perform the transition from the electric wheelchair to the walking aid and the transition from the walking aid to the electric wheelchair based on the user's operation, and almost all of their operations are only assisted according to the user's operation except for following the user in accordance with the user's walking speed.

[0010] For this reason, with these walking assist chairs, the target users are limited to those with weak legs and waists who require a general wheelchair, and it is very difficult for those with a relatively low degree of self-reliance, such as bedridden individuals, to actively perform operations for movement and walking on their own.

[0011] In order for a person with relatively low self-sufficiency to improve their mobility and walking ability, it is necessary to improve their physical and neurological functions while linking the vestibular and visual information with the body functions by repeating the standing and sitting postures and performing walking movements.

[0012] However, in the walking assist chairs of Patent Documents 1 and 2 described above, it was practically insufficient for a bedridden person to improve their physical and neurological functions and thereby improve their ADL (Activities of Daily Living) and QOL (Quality of Life).

[0013] The present invention has been made in consideration of the above points, and aims to propose a function improvement support device and a function improvement support method that can significantly improve the self-sufficiency in daily life and realize the comprehensive function improvement of the physical, psychological, and social aspects of the subject.

Means for Solving the Problems

[0014] In order to solve such problems, in the present invention, while holding the subject in a seated posture, a wheelchair function that travels according to the operation by the subject, and while holding the subject in a standing posture, a walking assist function that applies an assisting force according to the walking movement of the subject are provided. There is a posture holding variable mechanism part that can be deformed while holding the subject from the first mechanism corresponding to the wheelchair function to the second mechanism corresponding to the walking assist function, or from the second mechanism to the first mechanism, a human information acquisition part that acquires human information including the subject's brain and nervous system information, muscle and skeletal system information, physiological information, psychological information, motion information, and behavior information, a current situation function discrimination part that discriminates the current situation of the subject's physical function and cognitive function based on the human information obtained by the human information acquisition part, a vertical resistance measurement part provided in the state holding variable mechanism part that measures the vertical resistance in the contact part with the subject in real time, and a control part that controls the mechanism switching, deformation state, and power application state of the state holding variable mechanism part based on the discrimination result by the current situation function discrimination part and the measurement result by the vertical resistance measurement part.

[0015] As a result, in the functional improvement support device, it is possible to safely move in a large space while remaining in the seated state by the wheelchair function, and if necessary, to realize daily life in the standing or walking state in the living space by the walking assistance function. In particular, for a subject with a relatively long bedridden state, by repeatedly performing the sitting position and the standing position, it is possible to improve the physical and neurological functions by the interaction between the vestibular semicircular canals / visual information and the body functions, and it is possible to significantly improve the independence in daily life.

[0016] Also, in the present invention, the control unit detects the transition of the center of gravity position of the subject and the load distribution to a specific part at the contact part with the subject by the vertical ground reaction force measurement unit, and while judging the forceful state and the relaxed state at the time of the subject's sitting position, standing position, and the transition between both postures, after that, according to the forceful state and the relaxed state, the control content of the state holding variable mechanism unit is adjusted.

[0017] As a result, in the functional improvement support device, while judging the forceful state and the relaxed state at the time of the subject's posture transition, by adjusting the control regarding the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, it becomes possible to more safely guide the subject from the sitting state to the standing state and from the standing state to the sitting state.

[0018] Furthermore, in the present invention, it further includes an environmental information acquisition unit that acquires environmental information representing the surrounding environment of the subject and the floor surface state on which the state holding variable mechanism unit moves, and when controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, the control unit gives top priority to ensuring the safety of the subject based on the environmental information obtained from the environmental information acquisition unit.

[0019] As a result, in the functional improvement support device, while recognizing the surrounding floor surface state and the somatic sensation state, it becomes possible to significantly reduce the risk of falling while ensuring the safety of the subject's traveling and walking.

[0020] Furthermore, in the present invention, there is provided a wheel drive unit provided in the posture holding variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms, an imaging unit that images the lower body of the subject and sequentially acquires an RGB image and a depth image, and in a state where the posture holding variable mechanism unit is deformed into the second mechanism according to the wheelchair function, based on the RGB image sequentially acquired from the imaging unit, while estimating the posture of the lower body of the subject, an image coordinate setting unit that sets a plurality of skeletal representative points attached to the lower body of the subject as image coordinates, a coordinate group extraction unit that sequentially extracts an image coordinate group related to the gait state of the subject based on the transition state of the subject's posture from among the image coordinate groups set by the image coordinate setting unit, a skeletal information acquisition unit that time-synchronizes the image coordinate group sequentially extracted by the coordinate group extraction unit and the depth image sequentially acquired from the imaging unit to acquire temporally continuous three-dimensional skeletal information centered on the lower body of the subject, and a gait state detection unit that detects a gait state, which is a connection of a plurality of local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit. The control unit generates power synchronized with the gait state detected by the gait state detection unit in the wheel drive unit.

[0021] As a result, the function improvement support device can detect the gait state of the subject in real time and assist the walking motion while synchronizing with the gait state.

[0022] Furthermore, in the present invention, there is provided a wheel drive unit provided in the posture holding variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms, a biological signal detection unit disposed on the body surface part of the subject and having an electrode group for detecting a biological signal of the subject, and in a state where the posture holding variable mechanism unit is deformed into the first mechanism according to the wheelchair function, a motion intention recognition unit that recognizes the motion intention of the subject with respect to a specific operation content among the operation contents in the wheelchair function based on the biological signal obtained from the biological signal detection unit, and an operation content understanding unit that understands the operation content, which is the motion intention of the subject recognized by the motion intention recognition unit. The control unit generates power according to the operation content understood by the operation content understanding unit in the wheel drive unit.

[0023] As a result, in the function improvement support device, it becomes possible to travel while reflecting the operation content according to the subject's own movement intention without the subject operating by himself / herself during the wheelchair function.

[0024] Furthermore, in the present invention, while imaging the space corresponding to the subject's visual field, a gaze detection unit is further provided that recognizes the movement of the subject's eyeballs within the imaging range by image recognition and detects the position of the subject's line of sight. The movement intention recognition unit recognizes, as the movement intention of the subject, that when the position of the line of sight detected by the gaze detection unit continues for a predetermined time or more, the position of the line of sight is set as the movement target.

[0025] As a result, in the function improvement support device, by recognizing the movement intention with respect to the operation content of the subject not only based on the biological signal of the subject but also adding the line-of-sight position, it becomes possible to travel while reflecting the operation content according to the subject's own movement intention even more.

[0026] Furthermore, in the present invention, a sound collection unit that collects the voice of the subject and a speech content recognition unit that recognizes the speech content based on the voice collected by the sound collection unit are further provided. The movement intention recognition unit recognizes, as the movement intention of the subject, the operation content corresponding to the speech content based on the speech content recognized by the speech content recognition unit.

[0027] As a result, in the function improvement support device, by recognizing the movement intention with respect to the operation content of the subject not only based on the biological signal of the subject but also adding the subject's own speech content, it becomes possible to travel while reflecting the operation content according to the subject's own movement intention even more.

[0028] Furthermore, in the present invention, while holding the subject in a sitting position, a wheelchair function that travels according to the operation by the subject, and while holding the subject in a standing position, a walking assistance function that applies an assisting force according to the walking motion of the subject are provided. The posture holding variable mechanism unit is configured to be deformable from the first mechanism corresponding to the wheelchair function to the second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism while holding the subject. The method includes a first step of acquiring human information including the subject's brain and nervous system information, muscle and skeletal system information, physiological information, psychological information, motion information, and behavior information; a second step of determining the current status of the subject's physical function and cognitive function based on the human information obtained in the first step; a third step of measuring in real time the vertical resistance force at the contact part between the subject and the state holding variable mechanism unit; and a fourth step of controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit based on the determination result in the second step and the measurement result in the third step.

[0029] As a result, in the function improvement support device, it is possible to perform safe movement in a wide space while in the sitting state by the wheelchair function, and to realize daily life in the standing state or walking state in the living space by the walking assistance function as needed. In particular, for a subject with a relatively long bedridden state, by repeatedly performing the sitting posture and the standing posture, it is possible to improve the physical and neurological functions by the linkage between the vestibular system / visual information and the physical function, and it is possible to significantly improve the self-sufficiency in daily life.

Effects of the Invention

[0030] According to the present invention, it is possible to realize a function improvement support device and a function improvement support method that can significantly improve the degree of self-sufficiency in daily life and realize a comprehensive function improvement in the physical, psychological, and social aspects of the subject.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Mode for Carrying Out the Invention

[0032] The following describes a detailed embodiment of the present invention with reference to the drawings.

[0033] (1) Configuration of the Function Improvement Support Device According to the Present Embodiment FIGS. 1(A) and (B) show a function improvement support device 1 according to the present embodiment. The function improvement support device 1 combines a wheelchair function (FIG. 1(A)) that travels in response to an operation by the subject while holding the subject in a sitting posture, and a walking assistance function (FIG. 1(B)) that applies an assisting force in response to the walking motion of the subject while holding the subject in a standing posture.

[0034] The function improvement support device 1 has a posture holding variable mechanism unit 2 that can switch between a first mechanism (FIG. 1(A)) corresponding to the wheelchair function and a second mechanism (FIG. 1(B)) corresponding to the walking assistance function according to the motion of the subject. This posture holding variable mechanism unit 2 is configured to be deformable while holding the subject either from the first mechanism corresponding to the wheelchair function to the second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism.

[0035] (2) Configuration of the Posture Holding Variable Mechanism Unit As shown in FIGS. 2(A) and (B), the posture-holding variable mechanism unit 2 has a frame structure 10 that is generally U-shaped as a whole and has a bilaterally symmetric structure. The frame structure 10 includes a footrest portion 11 having a footrest function at the center of the curve, and a left frame 12A and a right frame 12B that are bilaterally symmetric with respect to the footrest portion 11.

[0036] The left frame 12A and the right frame 12B in the frame structure 10 each have a shape that is bent at a predetermined angle around the middle along the longitudinal direction.

[0037] Also, in the left frame 12A and the right frame 12B, one end of a front side arm portion 20A and 20B provided with front wheel portions 13A and 13B, respectively, in the vicinity of the footrest portion 11 is fixed, and at the bent portion, one end of a rear side arm portion 21A and 21B provided with rear wheel portions 14A and 14B, respectively, is movably provided.

[0038] Furthermore, in the left frame 12A and the right frame 12B, at the end portions, the other ends of support arm portions 31A and 31B having armrest portions 30A and 30B attached to one end thereof, respectively, are movably provided. These left and right support arm portions 31A and 31B are engaged integrally with corresponding drive mechanism systems (support drive portions 32A and 32B and parallel holding drive portions 33A and 33B shown in FIG. 3 described later).

[0039] And in the left frame 12A and the right frame 12B, at a predetermined portion between the support arm portions 31A and 31B and the rear side arm portions 21A and 21B, fixed holding portions 41A and 41B to which a part of the seat portion 40 is movably attached are provided, respectively. These left and right fixed holding portions 41A and 41B are engaged with corresponding opening and closing drive portions 42A and 42B (FIG. 3), respectively.

[0040] The left and right front wheel parts 13A and 13B are composed of a pair of left and right omnidirectional wheel structures (wheel structures in which a plurality of rollers are rotatably attached so as to be orthogonal to the axle and are arranged in the circumferential direction so that the curvature of the circle centered on the axle of each roller coincides) that can rotate in the traveling direction or a direction intersecting the traveling direction according to the traveling of the apparatus main body.

[0041] The left and right front wheel parts 13A and 13B are respectively engaged with front wheel drive parts 43A and 43B (FIG. 3) through front side arm parts 20A and 20B. The front wheel drive parts 43A and 43B pivotally drive the corresponding front wheel parts 13A and 13B in the traveling direction and a direction intersecting the traveling direction respectively, with the drive shafts provided at the other ends of the front side arm parts 20A and 20B as the rotation centers.

[0042] The left and right rear wheel arm parts 21A and 21B are respectively connected to rear wheel parts 14A and 14B at their other ends and are integrally engaged with the corresponding drive mechanism systems (rear wheel drive parts 44A and 44B and attitude deformation drive parts 45A and 45B shown in FIG. 3 described later). The rear wheel drive parts 44A and 44B pivotally drive the corresponding rear wheel parts 14A and 14B with the drive shafts provided at the other ends of the rear side arm parts 21A and 21B as the rotation centers.

[0043] The attitude deformation drive parts 45A and 45B (FIG. 3) pivotally drive one ends of the rear side arm parts 21A and 21B respectively, with the rotation shafts provided at the bent parts of the above-mentioned left frame 12A and right frame 12B as the rotation centers. A lock mechanism for suppressing rotation is built in the attitude deformation drive parts 45A and 45B (see Japanese Patent No. 4997416 and Japanese Patent No. 5344501 by the inventor of the present application), and the rotation in the direction opposite to the drive direction of one ends of the left and right rear side arm parts 21A and 21B is blocked to ensure the safety of the subject.

[0044] The support drive units 32A and 32B (FIG. 3) rotate the other ends of the support arm portions 31A and 31B about the rotation axes provided at the ends of the left and right frames 12A and 12B described above as the rotation centers. Further, the parallel holding drive units 33A and 33B (FIG. 3) rotate the armrest portions 30A and 30B about one ends of the left and right support arm portions 31A and 31B as the rotation centers.

[0045] The left and right armrest portions 30A and 30B each have a flat surface for placing the upper arm of the subject, and controllers 46A and 46B for operation by the subject are provided at the front ends in the longitudinal direction, respectively. At the rear ends in the longitudinal direction, curved frame bodies 47A and 47B corresponding to the backrests of the subject are rotatably attached, respectively.

[0046] These left and right curved frame bodies 47A and 47B are in a pair relationship, and their shapes are designed in advance so as to hold the waist of the subject from the back side in a closed state (a state where the tips face each other) (a backrest forming state). Built-in the left and right armrest portions 30A and 30B are backrest drive units 48A and 48B (FIG. 3), and the backrest drive units 48A and 48B are configured to drive the pair of curved frame bodies 47A and 47B to open and close simultaneously.

[0047] The support drive units 32A and 32B, the parallel holding drive units 33A and 33B, and the backrest drive units 48A and 48B are interlocked with each other to drive so that the flat surfaces of the left and right armrest portions 30A and 30B always maintain a substantially horizontal state with respect to the floor surface which is the traveling surface, according to the control of the overall control unit 50 described later.

[0048] The seat portion 40 is configured to form a seating surface by being integrally combined from a state of being divided into left and right halves. The seat portion 40 is provided with a left seat half body 40A corresponding to the left frame 12A and a right seat half body 40B corresponding to the right frame 12B so as to be rotatable within a predetermined angle range with the fixed holding portions 41A and 41B as the rotation reference, respectively.

[0049] The opening / closing drive units 42A and 42B (Fig. 3) rotate the left seat half 40A and the right seat half 40B that form the seat portion 40 within a predetermined angular range about the fixed holding portions 41A and 41B provided on the left frame 12A and the right frame 12B described above, thereby forming the seat portion 40 of the subject, while opening in a substantially parallel relationship with the left frame 12A and the right frame 12B.

[0050] The opening / closing drive units 42A and 42B are configured to drive so as to adjust the rotation angles of the left seat half 40A and the right seat half 40B that form the seat portion 40 while interlocking with the posture deformation drive units 45A and 45B in accordance with the control of the overall control unit 50.

[0051] In the function improvement support device 1, in the posture holding variable mechanism unit 2, a control unit 60 is provided inside the footrest portion 11 of the frame structure 10, and vertical resistance measurement units 61 are provided on the flat surfaces of the left and right armrest portions 30A and 30B and the seating surfaces of the left seat half 40A and the right seat half 40B. Furthermore, a human information acquisition unit 62 is attached to a part of the posture holding variable mechanism unit 2 and the subject (Fig. 3). Further, as will be described later, the posture holding variable mechanism unit 2 is provided with an environmental information acquisition unit 130 so that the subject can recognize the surrounding floor surface state and the somatic sensation state.

[0052] Fig. 3 shows the circuit configuration of the control system in the function improvement support device 1. In the function improvement support device 1, an overall control unit 50 composed of an MCM (Multi-Chip Module) equipped with a CPU (Central Processing Unit) and a memory is built in the control unit 60 in the posture holding variable mechanism unit 2, and the support drive units 32A and 32B, the parallel holding drive units 33A and 33B, the opening / closing drive units 42A and 42B, the front wheel drive units 43A and 43B, the rear wheel drive units 44A and 44B, the posture deformation drive units 45A and 45B, and the backrest drive units 48A and 48B are overall controlled.

[0053] The support drive units 32A and 32B, the parallel holding drive units 33A and 33B, the opening / closing drive units 42A and 42B, the front-wheel drive units 43A and 43B, the rear-wheel drive units 44A and 44B, the posture deformation drive units 45A and 45B, and the backrest drive units 48A and 48B each have a drive actuator (not shown), and are configured to drive each drive actuator in accordance with control by the overall control unit 50 to transmit an output.

[0054] Further, the function improvement support device 1 incorporates a relatively large-capacity drive battery 63 composed of a secondary battery or a capacitor inside the frame structure 10, and can supply power to the drive actuators of the support drive units 32A and 32B, the parallel holding drive units 33A and 33B, the opening / closing drive units 42A and 42B, the front-wheel drive units 43A and 43B, the rear-wheel drive units 44A and 44B, the posture deformation drive units 45A and 45B, and the backrest drive units 48A and 48B.

[0055] As shown in FIG. 4, the human information acquisition unit 62 acquires human information including the brain / nervous system information, muscle / skeletal system information, physiological information, psychological information, motion information, and behavior information of the subject. Specifically, the human information acquisition unit 62 includes a vital measurement device (measurement means such as electrocardiogram, pulse, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, and electromyogram) 70, a face imaging camera 71, a sound collection microphone 72, a motion capture device (any one of optical, inertial sensor type, mechanical, and magnetic type or a combination thereof) 73, and a peripheral measurement device (thermometer, hygrometer, ambient sound collector for audible sound and ultrasonic wave, etc.) 74, and an information detection unit 75 that detects human information to be described later based on the outputs of these sensor groups and devices.

[0056] In this human information acquisition unit 62, the information detection unit 75 detects the brain / nervous system information, muscle / skeletal system information, and physiological information of the subject based on the detection results by the vital measurement device 70, analyzes the facial expression and voice of the subject by the face imaging camera 71 and the sound collection microphone 72 to detect psychological information, and further detects the motion information and behavior information of the subject by the motion capture device 73.

[0057] The overall control unit 50 serves as a current situation function determination unit, and based on the human information obtained by the human information acquisition unit 62, determines the current situation of the physical function and cognitive function of the target person.

[0058] In addition, the vertical resistance measurement unit 61 is built into the seat part 40 and the armrest parts 30A and 30B of the state holding variable mechanism part 2, and measures the vertical resistance at the contact part with the target person in real time. Specifically, the vertical resistance measurement unit 61 has a structure in which a plurality of strain gauges (strain detection elements) are surface-arranged on the seat part 40 and the armrest parts 30A and 30B in a predetermined group array, and is configured to measure all or part of the external force in the three-dimensional direction acting on the contact part of the target person.

[0059] As the plurality of strain gauges constituting the vertical resistance measurement unit 61, in order to detect minute strain changes with high sensitivity, strain detection elements formed by partially doping impurities in a thin silicon substrate to form piezoresistors, or strain detection elements formed by depositing a metal thin film on a silicon substrate are used.

[0060] The vertical resistance measurement unit 61 divides the contact part of the target person into a plurality of parts in mesh units based on the output signals from the respective strain gauges, and analyzes the stress at each node of the element to analyze the stress strain of the contact part (finite element method). As a result, the vertical resistance measurement unit 61 measures the three-dimensional external force acting on the contact part with the target person as the vertical resistance in real time based on the analyzed stress strain distribution state and transition state.

[0061] The overall control unit 50 comprehensively determines the brain / nervous system information, muscle / skeletal system information, and physiological information of the target person, the psychological information of the target person, and the motion information and behavior information of the target person based on the human information obtained from the human information acquisition unit 62, and thus determines the current situation of the physical function (the degree of comparison by element with respect to the reference physical function) and cognitive function (the degree of comparison with respect to a predetermined cognitive judgment criterion) of the target person.

[0062] (2-1) Modes of wheelchair function and walking assistance function In the function improvement support device 1, the posture holding variable mechanism unit 2 holds the subject in a sitting posture and, in accordance with an operation by the subject, has a first mechanism (FIG. 1(A)) corresponding to a wheelchair function that travels, and holds the subject in a standing posture and, in accordance with a walking motion of the subject, has a second mechanism (FIG. 1(B)) corresponding to a walking assistance function that applies an assisting force, and both are configured to be deformable with respect to each other while holding the subject.

[0063] FIGS. 5(A) and (B) show a state in which the posture holding variable mechanism unit 2 is the first mechanism corresponding to the wheelchair function. In the first mechanism, the rear side arm portions 21A and 21B are positioned with respect to the left side frame 12A and the right side frame 12B, respectively, such that the rear wheel portions 14A and 14B are at the positions farthest from the front wheel portions 13A and 13B.

[0064] In this first mechanism, the left seat half body 40A and the right seat half body 40B engaged with the fixed holding portions 41A and 41B form the seat portion 40 such that the seating surface is substantially horizontal, and while the flat surfaces of the armrest portions 30A and 30B are maintained substantially horizontally, the pair of curved frame bodies 47A and 47B are positioned to form a backrest state.

[0065] Thus, in the function improvement support device 1, by setting the posture holding variable mechanism unit 2 to the state of the first mechanism corresponding to the wheelchair function, the subject can sit on the seat portion 40, place both arms on the armrest portions 30A and 30B, place both feet on the footrest portion 11, and perform a traveling operation using the controllers 46A and 46B provided at the front ends of the armrest portions 30A and 30B while maintaining a sitting posture equivalent to that of a wheelchair.

[0066] On the other hand, FIGS. 6(A) and (B) show a state in which the posture holding variable mechanism unit 2 is the second mechanism corresponding to the walking assistance function. In the second mechanism, the rear side arm portions 21A and 21B are positioned with respect to the left side frame 12A and the right side frame 12B, respectively, such that the rear wheel portions 14A and 14B are at the positions closest to the front wheel portions 13A and 13B.

[0067] In this second mechanism, the left seat half 40A and the right seat half 40B engaged with the fixed holding parts 41A and 41B are positioned such that the seating surface is in a substantially parallel relationship with the left frame 12A and the right frame 12B, and the pair of curved frame bodies 47A and 47B are positioned in an open state while the flat surfaces of the armrest parts 30A and 30B are maintained substantially horizontally.

[0068] Thus, in the function improvement support device 1, by setting the posture holding variable mechanism part 2 to the state of the second mechanism according to the walking assistance function, the subject can hold the left and right armrest parts 30A and 30B in the standing state and walk while applying an assisting force according to the walking motion in the standing posture.

[0069] (2-2) Mutual deformation between wheelchair function and walking assistance function In the function improvement support device 1, the posture holding variable mechanism part 2 is configured to be deformable from the first mechanism according to the wheelchair function to the second mechanism according to the walking assistance function, or from the second mechanism to the first mechanism while holding the subject.

[0070] In the state of the first mechanism, the overall control unit 50 travels in a desired direction according to the operation of the subject, and based on the discrimination result as the current situation function discrimination unit and the real-time measurement result of the vertical reaction force applied to the seat part 40 and the armrest parts 30A and 30B by the vertical reaction force measurement unit 61, determines whether or not the subject starts to shift from the seated posture to the standing posture (FIG. 7(A)).

[0071] Then, when the subject starts to shift to the standing posture, the overall control unit 50 controls the posture deformation drive parts 45A and 45B to start rotating the rear arm parts 21A and 21B so that the rear wheel parts 14A and 14B approach the front wheel parts 13A and 13B.

[0072] Subsequently, as the subject transitions from a seated posture to a standing posture, the overall control unit 50 controls the opening and closing drive units 42A and 42B to start opening the left seat half 40A and the right seat half 40B that form the seat portion 40. At the same time, the backrest drive units 48A and 48B are controlled to start opening the pair of curved frame bodies 47A and 47B. At this time, the overall control unit 50 controls the parallel holding drive units 33A and 33B so that the flat surfaces of the left and right armrests 30A and 30B always maintain a horizontal state (Fig. 7(B)).

[0073] When the subject has completely transitioned to a standing posture, the overall control unit 50 sets the posture holding variable mechanism units 45A and 45B to the state of the second mechanism described above. That is, the overall control unit 50 controls the posture deformation drive units 45A and 45B to position the rear wheel units 14A and 14B at the position closest to the front wheel units 13A and 13B. At the same time, the left seat half 40A and the right seat half 40B are positioned so that the seating surfaces are in a substantially parallel relationship with the left frame 12A and the right frame 12B, and the pair of curved frame bodies 47A and 47B are positioned so as to be in a completely open state (Fig. 7(C)).

[0074] On the other hand, in the state of the second mechanism, the posture holding variable mechanism unit 2, while the overall control unit 50 performs an auxiliary operation in synchronization with the subject's walking motion, based on the discrimination result as the current situation function discrimination unit and the real-time measurement result of the vertical reaction force applied to the seat portion 40 and the armrests 30A and 30B by the vertical reaction force measurement unit 61, determines whether or not to start the transition of the subject from the standing posture to the seated posture (Fig. 7(C)).

[0075] Then, when the subject starts to transition to the seated posture, the overall control unit 50 controls the posture deformation drive units 45A and 45B to start rotating the rear arm portions 21A and 21B so that the rear wheel units 14A and 14B move away from the front wheel units 13A and 13B.

[0076] Subsequently, as the subject transitions from a standing posture to a sitting posture, the overall control unit 50 controls the opening and closing drive units 42A and 42B to start forming the seat portion 40 by opposing the left seat half 40A and the right seat half 40B. At the same time, the backrest drive units 48A and 48B are controlled to start opposing the pair of curved frame bodies 47A and 47B to form the backrest. At this time, the overall control unit 50 controls the parallel holding drive units 33A and 33B so that the flat surfaces of the left and right armrests 30A and 30B always maintain a horizontal state (FIG. 7(B)).

[0077] When the subject has completely transitioned to a sitting posture, the overall control unit 50 sets the posture holding variable mechanism unit 2 to the state of the first mechanism described above. That is, the overall control unit 50 controls the posture deformation drive units 45A and 45B to position the rear wheel units 14A and 14B at the position farthest from the front wheel units 13A and 13B. At the same time, the left seat half 40A and the right seat half 40B are opposed to form a seat portion 40 with a substantially horizontal seating surface, and the pair of curved frame bodies 47A and 47B are positioned to be in the backrest formation state (FIG. 7(A)).

[0078] In addition, when deforming the subject while holding from the first mechanism to the second mechanism or from the second mechanism to the first mechanism, the overall control unit 50 controls the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit 2 based on the discrimination result as the current situation function discrimination unit and the measurement result by the vertical resistance measurement unit 61.

[0079] That is, the overall control unit 50 is based on the current situation of the subject's physical function and cognitive function, which is the discrimination result as the current situation function discrimination unit, and the vertical resistance applied to the seat portion 40 and the armrests 30A and 30B, which is the measurement result by the vertical resistance measurement unit 61. The state holding variable mechanism unit 2 switches from the state of the first mechanism corresponding to the wheelchair function or the second mechanism corresponding to the walking assistance function, and controls the deformation state (deformation speed, posture position, reverse mechanism switching according to the situation) during the process of the mechanism switching, and also controls the power application state in the deformation state.

[0080] As a result, in the functional improvement support device 1, it is possible to move safely in a wide space while in the seated state by the wheelchair function, and if necessary, to realize daily life in the standing or walking state in the living space by the walking assistance function. In particular, for a subject who is bedridden for a relatively long time, by repeatedly changing between the sitting position and the standing position, it is possible to improve the physical and neurological functions due to the linkage between the vestibular system / visual information and the body functions, and it is possible to significantly improve the independence in daily life.

[0081] (3) Detailed configuration and functions of the human information acquisition unit (3-1) Configuration of the vital measurement device Specifically, as shown in FIG. 8(A), in the vital measurement device 70, an electrode terminal group ET and a photoelectric probe window PW are arranged in a predetermined pattern on the measurement surface 70X of the device main body 70A, and the biological information of the subject is measured through the measurement surface 70X. The biological information targeted in the vital measurement device 70 is at least one measurement result among electrocardiogram, pulse, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, and electromyogram. That is, in the vital measurement device 70, the same device main body is configured to be usable as a measurement means for various types of biological information.

[0082] Also, as shown in FIG. 8(B), in the vital measurement device 70, a sound collection microphone 72 (FIG. 4) is detachably connected to a connection terminal CT provided on the side surface of the device main body 70A, and when connected, it can collect the vibrations generated from the larynx and pharynx of the subject as acoustic signals. As a result, it becomes possible to acquire information on vibrations (inaudible range) associated with the heart and breathing that cannot be heard by the ears for the subject.

[0083] (3-2) Method for detecting brain / nervous system information In this human information acquisition unit 62, as an additional configuration of the vital measurement device 70, it has a brain activity measurement unit 80 that is worn on the head of the subject and measures the electroencephalogram and cerebral blood flow in the measurement area of the head. And the information detection unit 75 converts the transmission state of the central nervous system centered on the head based on the measurement data of the electroencephalogram and cerebral blood flow obtained from the brain activity measurement unit 80 into quantitative nervous system data.

[0084] The brain activity measurement unit 80 according to this embodiment follows the same path as and uses the same principle as the content described in Japanese Patent No. 5283700 by the inventor of the present application.

[0085] As shown in FIG. 9(A), the brain activity measurement unit 80 includes a blood flow measurement unit 81, an electrocardiograph 82, a first control unit 83, and a wireless communication device 84. The blood flow measurement unit 81 measures the state of blood flow based on the change in the optical path and transmitted light amount of laser light by blood flow, and further measures the brain activity state from the blood flow state in the brain.

[0086] That is, when the laser light irradiated on the blood enters the blood layer, it travels through the blood as light of both components, namely, the reflected and scattered light component by normal red blood cells and the reflected and scattered light component by adherent thrombi. Since the influence received by the laser light during the process of passing through the blood layer changes moment by moment depending on the state of the blood, it is possible to observe various changes in the state of the blood by continuously measuring the transmitted light amount (reflected light amount) and observing the change in the light amount.

[0087] The blood flow measurement unit 81 includes a net-shaped base 85 formed in a hemispherical shape according to the outer shape of the head so as to be mounted on the head, and a number of sensor units 86A to 86N. Each of the sensor units 86A to 86N is supported at predetermined intervals by the net-shaped base 85, and outputs a detection signal of the transmitted light amount measured at each measurement point on the head to the first control unit 83. The electrocardiograph 82 measures the electrocardiogram generated according to the movement of the heart by the electrodes E attached to the skin of the subject.

[0088] The first control unit 83 derives the displacement of blood vessels and tissues around the blood vessels due to blood flow based on the light intensity when the light emitted from the light emitting unit 87 of each of the sensor units 86A to 86N is received by the light receiving unit 88 (FIG. 9(B)), and measures the activity state of the brain (distribution of red blood cells). The first control unit 83 stores a control program that performs arithmetic processing to cancel the component due to the oxygen saturation included in the signals obtained from at least two or more light receiving units 88.

[0089] Furthermore, the first control unit 83 derives the displacement of the inner wall of the blood vessel based on the displacement of the blood vessel and the surrounding tissue. Further, the first control unit 83 obtains the pulse wave propagation velocity at each measurement position based on the phase difference between the waveform of the electrocardiogram signal of the electrocardiograph 82 and the waveform of the detection signal obtained from the light receiving unit, and derives the displacement state of the inner wall of the blood vessel from the pulse wave propagation velocity.

[0090] The wireless communication device 84 wirelessly transmits the measurement results (blood flow data) output from the first control unit 83 to the vital measurement device 70. Since the brain activity measurement unit 80 has the optical sensor units 86A to 86N arranged in a net-like base 85, it can simultaneously measure the blood flow in the entire head.

[0091] When measuring the blood vessel characteristics of the brain, as shown in FIG. 9(B), the first control unit 83 selects an arbitrary sensor unit 86D from among the multiple arranged sensor units 86A to 86N, and causes the light emitting unit 87 of the sensor unit 86D to emit laser light. At this time, the laser light emitted from the light emitting unit 87 is output at a wavelength λ (λ≒805 nm) that is not affected by oxygen saturation.

[0092] When the blood flow measurement unit 81 of the brain activity measurement unit 80 is attached to the head of the subject, due to the elasticity of the net-like base 85, the plurality of sensor units 86A to 86N are positioned at each measurement point on the head, and each measurement surface is held in a state facing the head surface. In a state where the blood flow measurement unit 81 is attached to the head, the plurality of sensor units 86A to 86N irradiate light on the surface of the brain and measure the pulse wave propagation velocity in the cerebral arteries from the change in the amount of received light of the light propagated through the brain, and can measure the blood vessel characteristics (the ratio of blood vessel elasticity, the amount of plaque in the blood vessel, the ratio of arteriosclerosis) of the middle cerebral artery and the anterior cerebral artery.

[0093] Next, the principle of measuring the pulse wave propagation velocity of the cerebral arteries will be described. For example, each sensor unit 86A to 86N arranged corresponding to the measurement position for receiving the light propagated through the brain when irradiated with light detects the pulse wave propagation velocity due to the blood flow flowing through the middle cerebral artery and the anterior cerebral artery. As a measurement method, the waveform of the electrocardiogram obtained from the electrocardiograph 82 is compared with the waveform of the signal output from each sensor unit 86A to 86N at the measurement position, and the pulse wave propagation velocity is obtained from the phase difference, and a method for deriving the vascular characteristics corresponding to the pulse wave propagation velocity is used.

[0094] Also, the principle in the case of detecting vascular characteristics from the cerebral blood flow will be described. As shown in FIG. 9(B), the brain BR is covered by cerebrospinal fluid BR1, skull BR2, and scalp BR3. The laser light emitted from the light emitting unit 87 of each sensor unit 86A to 86N of the blood flow measurement unit 81, part of the light is reflected by the scalp BR3, but the remaining light passes through the scalp BR3, skull BR2, and cerebrospinal fluid BR1 and travels inside the brain BR. Then, the light that has traveled into the brain among the light irradiated on the head propagates in the radial direction (depth direction and radial direction) in an arc pattern CP as shown by the broken line in the figure.

[0095] The propagation of light passing through the brain decreases in light transmittance as the optical path length increases as the light is separated in the radial direction from the base point BP irradiated with the laser light. For this reason, the light reception level (transmitted light amount) of the sensor unit 86B adjacent to the sensor unit 86A at a predetermined distance on the light emitting side is strongly detected. And the light reception level (transmitted light amount) of the sensor unit 86C provided at a predetermined distance adjacent to the sensor unit 86B is detected weaker than the light reception level of the sensor unit 86B. Also, the light receiving unit 88 of the sensor unit 86A on the light emitting side also receives light from the brain.

[0096] The first control unit 83 uses the detection signal corresponding to the light intensity received by the plurality of sensor units 86A to 86N as measurement data, compares its waveform with the waveform of the electrocardiogram signal from the electrocardiograph 82, and derives the vascular characteristics of each measurement position. Also, by performing mapping processing on these detection results, it is possible to obtain graphic data showing the distribution of arteriosclerosis according to the pulse wave propagation velocity.

[0097] Therefore, it becomes possible to measure the changes in the blood flow flowing through the middle cerebral artery and the anterior cerebral artery based on the detection signal waveforms of the respective sensor units 86A to 86N, and to detect the pulse wave propagation velocity in the brain from the measurement data of the blood flow changes.

[0098] In this way, in the vital measurement device 70, a brain activity measurement unit 80 is provided which is attached to the head of the subject and measures the brain waves and cerebral blood flow in the measurement area of the head. The information detection unit 75 also adds, as nervous system data, the transmission state of the central nervous system centered on the head based on the measurement data of the brain waves and cerebral blood flow obtained from the brain activity measurement unit 80. In this way, the transmission state of the central nervous system from the brain of the subject can also be quantitatively converted into data and acquired, and the progress state and transition state of the subject can be monitored.

[0099] As a result, the transmission state of the central nervous system from the brain of the subject can also be quantitatively converted into data and acquired, and the progress state and transition state of the subject can be monitored.

[0100] (3-3) Method for detecting musculoskeletal system information The human information acquisition unit 62 has, as an additional configuration of the vital measurement device 70, a biological signal measurement attachment 90 worn by the subject and an indirect surrounding detection unit (not shown). The indirect surrounding detection unit has an actuator that is actively or passively driven in conjunction with the limb movement of the subject, and based on the output signal from the actuator, detects the physical quantity around the joint associated with the limb movement of the subject.

[0101] The physical quantity around the joint includes at least one or more parameter identification results among the generated muscle force generated in the musculoskeletal system, the movable range, movable speed and reaction speed of each joint, the self-regulatory control characteristics against disturbances, the moment of inertia of the frame, the mass and the center of gravity, the impedance adjustment result (viscous characteristics due to friction) of the joint system including flexor muscles and extensor muscles, the electrical physical quantity (command signal), etc.

[0102] The biological signal measurement wearable device 90 worn by the subject is in line with the content described in Japanese Patent No. 5409637 by the inventor of the present application and uses the same principle. Fig. 10(A) shows the biological signal measurement wearable device 90 in this embodiment. This biological signal measurement wearable device 90 is formed to cover the body surface of the subject and has a wearable device main body 90A that is worn by the subject. On the inner surface of this wearable device main body 90A (the surface that contacts the body surface of the subject when worn), there is a biological signal sensor 91 that detects a bioelectrical potential signal at at least one position where a biological signal can be measured from the body of the subject.

[0103] A biological signal is a signal caused by electricity generated in the body of the subject, a signal that can be measured from the body, and a signal that changes in time series along with the movement of the body or signals from the brain. For example, biological signals include nerve potentials, myoelectric potentials, electroencephalograms, electrocardiograms, and further, potentials generated by biochemical reactions such as potentials generated by motion artifacts (effects of movement), and vibrations such as pulse waves generated by the pulsation of the heart, which means signals generated by the activities of the living body. The generated potential (amplitude, density, time delay) and generation site of the biological signal are also taken into account for quantitative data conversion.

[0104] In Fig. 10(A), since the biological signal measurement wearable device 90 of this embodiment has a symmetric structure, the left half of the figure shows the structure on the back side of the left foot, and the right half of the figure shows the structure on the front side of the left foot. The wearable device main body 90A is provided with biological signal sensor groups 92A to 92F composed of a plurality of biological signal sensors 91 that detect the biological signals of the subject. The biological signal sensors 91 are arranged at equal intervals along the flow of the muscles of the subject's leg.

[0105] A plurality of biological signal sensors 91 are arranged along the flow of the gluteus maximus muscle in a portion corresponding to the buttocks of the subject (biological signal sensor group 92A) on the wearing device body 90A. Similarly, in a portion corresponding to the back side of the thigh of the subject (biological signal sensor group 92B), they are arranged along the flow of the biceps femoris muscle, the semimembranosus muscle, and the semitendinosus muscle, and in a portion corresponding to the calf (biological signal sensor group 92C), they are arranged along the flow of the triceps surae muscle. Also, in a site corresponding to the front side of the hip joint (biological signal sensor group 92D), they are arranged along the flow of the adductor longus muscle and the iliopsoas muscle, in a site corresponding to the front side of the thigh (biological signal sensor group 92E), they are arranged along the flow of the quadriceps femoris muscle, and in a site corresponding to the tibia (biological signal sensor group 92F), they are arranged along the flow of the tibialis anterior muscle, the soleus muscle, and the extensor digitorum longus muscle.

[0106] Here, FIG. 10(B) is a diagram showing an example of a measurement module connected to the biological signal sensor group provided on the wearing device body 90A. Since all the biological signal sensor groups 92A to 92F provided on the wearing device body 90A are similarly connected to the measurement module 93, the biological signal sensor group 92A will be described as an example here.

[0107] FIG. 10(B) is a diagram schematically showing the biological signal sensor group 92A provided on the wearing device body 90A and the measurement module 93 connected to this biological signal sensor group 92A. Each biological signal sensor 91 constituting the biological signal sensor group 92A is provided in a mutually insulated state, and each is connected to the measurement module 93 via a conductive wiring. And addresses are assigned to these biological signal sensors 91 respectively.

[0108] The measurement module 93 is connected to a plurality of biological signal sensors 91 that constitute the biological signal sensor group 92A. It selects at least two biological signal sensors 91 from these biological signal sensors 91, takes the difference between the detection signals detected by these selected biological signal sensors 91 to obtain a biological signal, and includes a measurement module controller 94, a memory 95 that records the obtained biological signal, and a communication unit 96 that transmits the sequentially obtained biological signal and / or the biological signal recorded in the memory 95 to the outside. Note that the measurement module 93 is provided for each of the biological signal sensor groups 92A to 92F, and each biological signal sensor 91 is connected thereto.

[0109] This measurement module controller 94 has an electronic circuit that can sequentially select at least two biological signal sensors 91 from the plurality of connected biological signal sensors 91 according to a command signal input via the communication unit 96 and obtain the biological signal between these two selected biological signal sensors 91.

[0110] Furthermore, the measurement module controller 94 further has signal processing means such as a filter that removes or extracts a predetermined frequency component from the biological signal thus obtained, and an amplifier that amplifies the obtained biological signal. The biological signal thus obtained is output from the measurement module controller 94 to the communication unit 96 and / or the memory 95. When selecting the biological signal sensor 91, the measurement module controller 94 may sequentially operate each biological signal sensor 91 in a preset order, or may select the biological signal sensor 91 at the address specified by the specified signal based on the specified signal input via the communication unit 96.

[0111] The communication unit 96 consists of a thin antenna and a communication circuit connected to this antenna. The communication unit 96 transmits measurement information, which includes a signal containing the biological signal output from the measurement module controller 94 and information such as an address indicating the position information of the biological signal sensor 91 that detected the biological signal, and / or a signal containing the biological signal read from the memory 95 and information such as an address indicating the position information of the biological signal sensor 91 that detected the biological signal, to the second control unit 97 via the antenna.

[0112] This second control unit 97 generates a designation signal for designating which biological signal sensor 91 to select from among the respective biological signal sensor groups 92A to 92F provided in the wearing device main body 90A, signals such as signals for starting and ending data acquisition, etc., and selects the designated biological signal sensor 91 according to the signal to measure the biological signal.

[0113] Thus, according to the biological signal measurement wearing device 90 of the present invention, since the wearing device main body 90A is provided with biological signal sensor groups 92A to 92F each consisting of a plurality of biological signal sensors 91, a plurality of biological signal sensors 91 can be arranged at a predetermined site at once simply by wearing the wearing device main body 90A, and it becomes possible to detect biological signals at respective points while being in close contact with the skin surface. In this way, even when monitoring biological signals at a large number of points on the body surface of the subject, it is possible to save the trouble of attaching and detaching the biological signal sensors 91 one by one, and the biological signals can be easily measured.

[0114] Also, by detecting biological signals with such a plurality of biological signal sensors 91, it is possible to measure biological signals at a plurality of respective points within the region where the biological signal sensor groups 92A to 92F are arranged. Then, by mapping the measurement data at each point according to the address assigned to each biological signal sensor 91, it is possible to measure the distribution of biological signals on the body of the subject.

[0115] The second control unit 97 acquires, from the biological signals acquired by the measurement module 93, an electromyogram signal associated with the muscle activity of the subject and a neurotransmission signal for operating the musculoskeletal system of the subject.

[0116] The information detection unit 75 (Fig. 4) converts the activity state of the musculoskeletal system centered on the joint and / or the transmission state of the peripheral nervous system into quantitative musculoskeletal system data and / or nervous system data based on either one or both of the physical quantity around the joint described above and the electromyogram signal and the neurotransmission signal obtained from the second control unit 97, and also converts the transmission state of either one or both of the central nervous system and the peripheral nervous system centered on the desired site into quantitative nervous system data.

[0117] As a result, by performing limb movements, the subject can quantitatively obtain data on the activity state of the musculoskeletal system and the transmission state of the peripheral nervous system based on the physical quantity around the joint and the biological signals obtained from the body surface site based on the joint, and can monitor the progress state and transition state of the subject.

[0118] Also, by acquiring biological signals from body surface sites other than the site based on the joint of the subject, it is possible to quantitatively obtain data on the transmission states of not only the peripheral nervous system around the joint but also the central nervous system and the peripheral nervous system from the desired site, and to monitor the progress state and transition state of the subject.

[0119] Note that, for example, as shown in Fig. 11, by arranging the biological signal sensor groups 92A to 92F on the whole body of the subject, it becomes possible to visualize the state transition of the information transmission path of the cranial nerve system. In Fig. 11, if the whole body of the subject is divided into units of predetermined regions (for example, regions dominated by sensory nerves such as dermatomes), the nerve connections of the whole body can be grasped from an overhead view and for each region site. Therefore, if it is possible to detect where the problem has occurred in nerve transmission, it is possible to grasp not only the defective part but also the degree of influence on either the central side or the peripheral side.

[0120] (3-4) Method for Detecting Activity State of Physiological System In the human information acquisition unit 62, as an additional configuration of the vital measurement device 70, as shown in FIG. 12, a near-infrared detection unit 100 and a far-infrared detection unit 101 are provided, and the activity states of the physiological system and the nervous system of the subject are detected using a non-contact method.

[0121] That is, the human information acquisition unit 62 irradiates infrared rays centering on the face of the subject using the near-infrared detection unit 100 and the far-infrared detection unit 101, and simultaneously and non-invasively measures the fluctuations in the pulse rate and the changes in skin temperature of the subject in daily life.

[0122] The near-infrared detection unit 100 irradiates near-infrared light centering on the region of interest including the cheek of the subject's face, receives the reflected light from the face, and generates a near-infrared image. The far-infrared detection unit 101 irradiates far-infrared light centering on the region of interest of the subject's face and detects the skin temperature of the face.

[0123] The third control unit 102 estimates the pulse rate of the subject based on the change period of the intensity waveform in the region of interest among the reflected light received by the near-infrared detection unit 100. The third control unit 102 extracts, from the near-infrared image generated by the near-infrared detection unit 100, the parts of the subject's face that are easily affected by the activity of the autonomic nervous system and the parts that are not easily affected, respectively, and calculates the temperature difference of the skin temperature of each part from the detection result of the far-infrared detection unit 101.

[0124] A. Pulse measurement method based on infrared light Substances with light-absorbing properties in living tissues are water and hemoglobin in the blood. Water has strong absorption characteristics for infrared rays with wavelengths longer than 1350 [nm], while hemoglobin has strong absorption characteristics for visible light with wavelengths shorter than 650 [nm]. In non-invasive biological diagnosis using light, light in the infrared to near-infrared region with wavelengths from 650 [nm] to 1350 [nm], which has high biological permeability, is often irradiated, and the reflected light and transmitted light containing biological information are measured.

[0125] Pulse is a physiological phenomenon in which the volume of blood vessels changes due to the work of blood pressure caused by cardiac output. The change in blood vessel volume is caused by the change in blood flow. Since the change in blood flow causes the change in the amount of light absorbed by hemoglobin in the blood, the pulse rate can be estimated from the change cycle of the transmitted light or reflected light of the light irradiated on the living body.

[0126] In the present invention, in order to perform measurement regardless of day or night without asking the subject to wear a measuring device, the change cycle of the reflectance of near-infrared light irradiated on the living body is measured with a near-infrared camera, and the pulse rate is estimated. Since near-infrared light is invisible, measurement can be performed on the subject without burden even at night. In order to estimate the stress state from the pulse rate, it is necessary to measure the pulse rate that varies finely due to the activity of the autonomic nervous system.

[0127] The third control unit 102 implements an algorithm for extracting the waveform period necessary for calculating the pulse rate from the waveform data obtained from the near-infrared detection unit 100.

[0128] The near-infrared detection unit 100 measures the intensity of the reflected light of the irradiated near-infrared light by the living tissue. This near-infrared detection unit 100 measures the subject at rest, but the minute body movement and the facial movement due to breathing are superimposed on the pulse waveform as motion artifacts representing the change in the light amount of the reflected light.

[0129] In order to remove these motion artifacts, first, a digital filter is applied to the frequency band of the pulse rate. The range of the pulse rate at rest is set to 35 to 180 [bpm], and there is a possibility that motion artifacts are superimposed on the measurement waveform in the band of 0.5 to 3.0 [Hz], but the amplitude of the waveform superimposed on the pulse wave waveform is significantly large.

[0130] Therefore, when the variance value of the waveform data within a certain period exceeds a certain threshold, the third control unit 102 recognizes that the subject is in motion, and during that period, the data is removed as unreliable. The waveform after the band-pass filter has bimodality due to the arterial blood flow volume. To facilitate feature point extraction, a moving average filter is applied to convert the bimodal waveform into a unimodal waveform.

[0131] The waveforms before and after applying the band-pass filter and the moving average filter to the measured waveform are shown in FIGS. 13(A) and (B), respectively. It can be seen that the motion artifact superimposed on the pulse waveform has been removed by filtering.

[0132] Subsequently, there are generally two major methods known for calculating the pulse rate from the intensity waveform of light measured by a near-infrared camera.

[0133] In the first method, the face part is set as the region of interest (ROI), and the frequency that seems to be the pulse wave component in the frequency spectrum obtained by the spectrum analyzer for the intensity data of the reflected light of the cheek part measured by the RGB camera for 30 seconds is compared with the frequency obtained from the pulse rate indicated by the pulse oximeter. However, for the pulse wave waveform, the noise due to body movement is extremely large. When attempting to calculate the pulse rate by such frequency analysis, it is necessary to greatly restrict the actions of the measurement subject.

[0134] On the other hand, in the second method, the time required for one pulse beat is calculated from the period of the feature points of the detected waveform in the time-series data of the measured waveform, and the pulse rate is calculated by dividing 60 by that time. Compared with the method of calculating the pulse rate from frequency analysis that requires continuous measurement data for a certain period, the method of calculating the pulse rate from the period of the measured feature points can remove the motion artifact by removing the period of the feature points misdetected by the noise due to body movement. Also, the minimum measurement time required is shorter than that of frequency analysis, and the pulse rate can be calculated from several beats, so it is possible to calculate the pulse rate at shorter time intervals.

[0135] However, since the resting pulse rate is 60 to 80 [bpm] in healthy individuals, the cycle of the feature points obtained by measuring for 5 seconds is 5 to 7 times, and there is a problem that the amount of data obtained is small in order to calculate a more accurate pulse rate. Also, as shown in Fig. 12(A), since the waveform of the intensity change has two peaks in synchronization with the arterial blood flow volume, it causes misdetection of the cycle of the feature points, which becomes a factor causing an error in the calculated pulse rate.

[0136] For daily measurement, the biological site to be measured needs to be exposed regardless of day or night. Therefore, in the vital measurement device 70 according to the present invention, the ROI is set on the cheek portion of a person. The location of the set ROI is shown in Fig. 14(A). The average of the intensity values measured by the pixels within the set ROI is used as the measurement value, and the time change of the measurement value is recorded as pulse waveform data. As the cycle of the feature points of the waveform data, not only the upper end interval t tp but also the lower end interval t bp is added as another feature point to increase the feature points serving as reference data for calculating the pulse rate.

[0137] Next, in order to extract the feature point period necessary for calculating the pulse rate, among the detected t tp , t bp , the misdetected t tp , t bp that does not indicate the pulse period is removed by the algorithm shown in the following (a) to (f). Here, the set before removal is denoted as S0.

[0138] (a) For each of t tp , t bp , if there is a variation of 30 [%] or more with respect to a certain peak interval t i from the immediately preceding peak interval t i-1 , that peak interval t i is removed from the set S0. The selected set S1 of peak intervals is represented by the following formula (1).

Equation

[0139] (b) Calculate the standard deviation σ0 of set S0, and the peak intervals t that vary within the range of 2σ0 i are removed from set S1. The set S2 of the selected peak intervals is represented by the following formula (2).

Number

[0140] (c) Calculate the standard deviation σ2 of set S2, and the peak intervals t that vary within the range of σ2 i are removed from set S2. The set S3 of the selected peak intervals is represented by the following formula (3).

Number

[0141] (d) Calculate the standard deviation σ3 of set S3, and the peak intervals t that vary within the range of σ3 i are removed from S3. The set S4 of the selected peak intervals is represented by the following formula (4).

Number

[0142] (e) Convert the peak intervals remaining in set S4 to pulse rates, and create a histogram with a pulse rate interval of 5 [bpm]. At this time, the width of one interval of the histogram in terms of pulse rate is equal, but the width of the peak intervals corresponding to that pulse rate is different. Therefore, create a weighted histogram using the following formula (5).

Number

[0143] Here, t 30-35 is the time width of the peak interval from a pulse rate of 30 [bpm] to 35 [bpm], t i-i+5 is the time width of the peak interval from a pulse rate of i [bpm] to i + 5 [bpm], h i-i+5is the frequency, h’, of the peak intervals from the pre-correction pulse rate i [bpm] to i + 5 [bpm]. i-i+5 represents the correction value of the frequency of the peak intervals from the pulse rate i [bpm] to i + 5 [bpm].

[0144] (f) For the values of the weighted histogram, a window with a width of 20 [bpm] is provided such that the sum of the frequencies of four adjacent intervals is maximized, and the peak interval t i outside that range is removed.

[0145] According to the above algorithm, the average value of the extracted peak intervals is taken as the peak width t within the measurement time, and the value obtained by dividing 60 seconds by the peak interval t is taken as the pulse rate.

[0146] In this way, the third control unit 102 sets characteristic points at the upper end and the upper end of the intensity waveform of the reflected light obtained from the near-infrared detection unit 100, and estimates the pulse rate based on the peak intervals between the upper ends and between the lower ends, which are the periods of the respective characteristic points. Further, the third control unit 102 calculates the standard deviation of a plurality of peak intervals, and removes the peak intervals that do not indicate the pulse period based on the standard deviation. Furthermore, the third control unit 102 creates a weighted histogram in a predetermined time unit from the pulse rate converted based on the remaining peak intervals after removal, and corrects the frequency of the peak intervals based on the histogram.

[0147] Thus, the third control unit 102 can calculate an accurate pulse rate from the measurement waveform of the pulse waveform data for a relatively short time based on the pixel group within the set ROI.

[0148] B. Skin temperature measurement by the far-infrared detection unit The heat generated inside the living body is transported to the body surface by conduction and convection. However, the heat conduction by the living tissue itself is poor and acts as a heat insulator, so most of the heat transport to the skin is considered to be due to skin blood flow. The skin blood flow changes due to the activities of the autonomic nervous system centered on the vasoconstriction and dilation actions of the sympathetic and parasympathetic nervous systems.

[0149] In conventional prior research, attempts have been made to estimate the activity of the autonomic nervous system by measuring changes in skin temperature. However, since the heat distribution on the face is greatly affected by the hairstyle of the person being measured and the presence or absence of glasses, the human information acquisition unit according to the present invention sets the measurement site appropriate for estimating the activity of the autonomic nervous system from the near-infrared image.

[0150] In order to estimate the stress state from changes in skin temperature, it is necessary to set the ROI to measure the facial part where the temperature change due to stress appears significantly. Among the facial parts, especially in the nose, arteriovenous anastomoses (AVA) that are easily affected by the activity of the autonomic nervous system are concentrated, which is suitable for measuring skin temperature for the purpose of estimating the stress state.

[0151] In addition, since skin temperature is affected by the outside air temperature, it is necessary to record the change in skin temperature as the relative temperature. At this time, the ROI measured as the reference for the change in skin temperature of the nose needs to be set at a site that is not easily affected by the activity of the autonomic nervous system. The forehead is an example of a facial part where the density of AVA is low and it is not easily affected by the activity of the autonomic nervous system.

[0152] From the above, the third control unit 102 according to the present invention records the change in skin temperature as the temperature difference between the region of interest ROI_n of the nose and the region of interest ROI_fh of the forehead. The locations of the respective ROIs are shown in FIG. 14(B). The formula for calculating the temperature change is shown in formula (6).

Equation

[0153] Here, T n is the average value of the skin temperature measured for all pixels within ROI_n, T fh is the average value of the skin temperature measured for all pixels within ROI_fh, and T r is the relative temperature between the forehead and the nose. When setting each ROI, the coordinates of the nose and the forehead in the near-infrared image are converted into the coordinates on the far-infrared image.

[0154] Specifically, when converting coordinates from a near-infrared (NIR) image to a far-infrared (FIR) image, since the resolution and field of view angles differ between the near-infrared camera and the far-infrared camera, the coordinate information of the near-infrared image is made to correspond to the pixels of the far-infrared image.

[0155] That is, the x coordinate on the far-infrared image is X fir where θ is the horizontal field of view angle of the near-infrared camera nir_h and θ is the horizontal field of view angle of the far-infrared camera fir_h Then, it is expressed as the following formula (7).

Equation

[0156] Also, the y coordinate on the far-infrared image is Yfir, where θ is the vertical field of view angle of the near-infrared camera nir_v and θ is the vertical field of view angle of the far-infrared camera fir_v Then, it is expressed as the following formula (8). Here, d represents the difference in the optical axes, and L represents the distance to the object.

Equation

[0157] Thus, following the automatic detection result of the measurement target site using the image information of the near-infrared detection unit 100, the third control unit 102 can accurately calculate the relative temperature difference of the skin temperature between the forehead and nose of the subject based on the image information of the far-infrared detection unit 101.

[0158] C. Respiration measurement based on skin temperature Respiration is related to the function of the autonomic nervous system and immunity and is used for the diagnosis of sleep disorders. There are nasal respiration (Fig. 15(A)) and oral respiration (Fig. 15(B)), and nasal respiration is preferably the original respiration. In the case of oral respiration, inhalation does not pass through the cilia and mucous membranes of the nasal cavity. Also, oral respiration causes various diseases, including sleep apnea syndrome (SAS).

[0159] As shown in FIGS. 16(A) and (B), the nostrils and the oral cavity are warmed by exhaled air (35 - 36[°C]) and cooled by inhaled air (23 - 28[°C]). Therefore, a region of interest (ROI) is set for the nostrils and the oral cavity of the subject, and the temperature changes of the nostrils and the oral cavity due to breathing are measured by a far-infrared detection unit (far-infrared camera). Since the surface temperature of the skin is affected by the outside air temperature, it is measured as the relative temperature between the nostrils and the oral cavity.

[0160] In practice, since it is difficult to automatically detect the measurement target site from the skin temperature distribution map, as described above, the measurement target site (nostrils and oral cavity) is automatically detected using the image information of the near-infrared detection unit (camera), and a region of interest (ROI) is set. The temperature of the nostrils during nasal breathing is relatively high, while the temperature of the oral cavity during mouth breathing is relatively low.

[0161] After actually measuring the nasal breathing and mouth breathing of the subject continuously for 30 seconds each, a 1-minute break was taken, and the same breathing measurement was repeated 4 times. As a result, as shown in FIG. 17(A), a decrease in the temperature of the nostrils due to inhalation during nasal breathing was confirmed. Also, as shown in FIG. 17(B), a decrease in the temperature of the oral cavity corresponding to mouth breathing was confirmed.

[0162] Furthermore, as shown in FIG. 18(A), the opening and closing of the mouth due to the change in the breathing method from nasal breathing to mouth breathing was also confirmed based on the distance between the upper lip and the lower lip. When the graphs of FIGS. 17(A), (B) and FIG. 17(A) are superimposed and displayed, it is represented as shown in FIG. 18(B).

[0163] As a method for extracting the inhalation timing, first, high-frequency noise is removed from the measurement results of the relative temperature between the nostrils and the oral cavity using a moving average filter, and then a first derivative is executed to extract the temperature decrease timing due to exhalation.

[0164] Also, when the temperatures of the nostrils and the oral cavity decrease at the same timing, since the alveolar pressure becomes negative with respect to the atmospheric pressure and air flows in through the nasal passage, it becomes possible to discriminate mouth breathing.

[0165] In this way, it is possible to calculate the respiratory rate and discriminate the breathing method by a non-contact measurement method. Although it is possible to simultaneously measure nasal breathing and mouth breathing, accurate calculation of the respiratory rate and discrimination of the breathing method can be realized by executing an algorithm considering the characteristics of temperature changes during each breathing.

[0166] D. Oxygen saturation using two types of near-infrared wavelengths Conventionally, as a measuring device that utilizes the light absorption characteristics of hemoglobin in the blood, a pulse oximeter can be mentioned. A pulse oximeter is a medical device that measures blood oxygen saturation and pulse rate by attaching the device to the fingertip.

[0167] In the present invention, since the blood oxygen saturation has the property of changing depending on the respiratory rate, the degree of binding between hemoglobin and oxygen, and the cardiac output, attention is paid to the difference in the light absorption characteristics between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0168] In the conventional measurement method, infrared light (near a wavelength of 660 [nm]) and near-infrared light (near a wavelength of 900 [nm]) are irradiated from a light-emitting element to the fingertip, and the blood oxygen saturation is estimated from the light quantity ratio of each transmitted light measured by a light-receiving element. At the same time, a method of estimating the pulse rate by utilizing the periodic change in the light quantity of the transmitted light is used. Infrared light, which is visible light, has been used because of the large difference in the light absorption rate between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0169] However, in the present invention, non-contact measurement is performed using only near-infrared (NIR) light, which is invisible light (Fig. 19). As a result, while alternately flashing a light-emitting element that irradiates two types of near-infrared light with wavelengths of 800 (or 760) [nm] and 900 [nm], the irradiation site (ROI) of each light-emitting element is photographed by a single near-infrared camera (near-infrared detection unit 100), so that the blood oxygen saturation centered on the ROI of the subject can be measured non-contact even in a dark environment.

[0170] (3 - 5) Method for detecting the activity state of the nervous system The autonomic nervous system includes the sympathetic nervous system that functions when the living body is in a tense or active state, and the parasympathetic nervous system that functions when the body is at rest. Since blood pressure and pulse rate increase in a state where the sympathetic nervous system is dominant, while blood pressure and pulse rate decrease in a state where the parasympathetic nervous system is dominant, the autonomic nervous system function and cardiac function have a high correlation.

[0171] As a method for measuring autonomic nervous system function, there is a method for measuring the function of the sympathetic nerve of the heart using heart rate variability. Specifically, there is the CVR-R (Coefficient of Variation of R-R intervals) method in which the coefficient of variation is obtained and evaluated using the R-R interval of the electrocardiogram waveform, and secondly, a method in which the variation power ratio of the frequency components (high-frequency component and low-frequency component) of heart rate variability is used as an index of sympathetic nerve activity.

[0172] There is also a method for measuring autonomic nervous system function by measuring the function of the sympathetic nerve of the vascular system using the pulse wave. As this measurement method, there is a method of calculating the magnitude of the amplitude variation of the photoplethysmogram (PPG) waveform in particular as an evaluation value of autonomic nervous system function.

[0173] In the present invention, in the third control unit 102, based on all or some combinations of the pulse rate of the subject detected using the near-infrared detection unit 100 and the far-infrared detection unit 101 constituting the human information acquisition unit 62, the relative temperature difference between the skin temperatures of the forehead and nose of the subject, the respiration rate and respiration method of the subject, and the oxygen saturation of the subject, it is possible to contribute to the evaluation (such as the presence or absence of disturbance) of the autonomic nervous system function of the subject.

[0174] (3-6) Detection method of psychological information In the human information acquisition unit 62, the information detection unit 75 analyzes the facial expression and voice of the subject by the face imaging camera 71 and the sound collection microphone 72 to detect psychological information.

[0175] Specifically, the information detection unit 75 detects the facial expressions of the target person (including not only expressions such as sadness, joy, anger, empathy, and frivolity, but also blushing skin, goosebumps, eyelid opening and closing frequency, and tears) by executing a face analysis algorithm, and at the same time, detects the tone and pitch of the target person's voice (such as voice tone and excitement) by executing a voice analysis algorithm.

[0176] This face analysis algorithm is designed to approximate human interpretation (experience, prejudice, and prediction), and is an algorithm obtained from the accumulated results of how emotions and psychology are reflected in facial expressions based on the state and changes of the human face.

[0177] Similarly, the voice analysis algorithm is designed to approximate human interpretation (experience, prejudice, and prediction), and is an algorithm obtained from the accumulated results of how emotions and psychology are reflected in the voice based on the tone and pitch of the human voice.

[0178] Using this voice analysis algorithm, based on keywords used in the target person's utterances, intonation at the time of speaking, voice volume, narrative composition status, communication continuity and discontinuity, etc., and differences from these accumulated data, it is possible to estimate the transformation of the target person's mental state through the target person's daily communication, its analysis, and narrative analysis.

[0179] In addition to this, the information detection unit 75 can also estimate the mental state of the target person based on the human information of the target person (heartbeat, electrocardiogram, pulse, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweating, and odor substances, etc.) which is the detection result by the above-mentioned vital measurement device 70, and the environmental information of the target person which is the detection result by the peripheral measurement device 74 described later.

[0180] In this way, the information detection unit 75 can comprehensively estimate and grasp the psychological information of a person from both the human information obtained by sensing and the human information obtained by communication analysis and narrative analysis.

[0181] (3-7) Method for Detecting the Activity State (Motion Information and Behavior Information) of the Motor System In the human information acquisition unit 62 (Fig. 4), the information detection unit 75 detects the motion data recognized by the motion capture device 73 as the activity state of the motor system of the subject.

[0182] Here, as shown in Fig. 20, the motion capture device 73 is arranged in indoor facilities such as the subject's home and is composed of an RGB-D sensor 111 and an IMU (Inertial Measurement Unit) sensor 112 connected to the above-mentioned control unit 60. The IMU sensor 112 consists of a three-axis acceleration sensor, a three-axis angular velocity sensor, and a geomagnetic sensor, and can measure the three-axis acceleration, angular velocity, and geomagnetism.

[0183] In addition to the RGB color camera function, the RGB-D sensor 111 has a depth sensor that can measure the distance to the object seen from the camera and can perform a three-dimensional scan of the object. When, for example, the LiDAR camera L515 of RealSense (a trademark of Microsoft, USA) is applied as the RGB-D sensor 111, the depth sensor consists of a LiDAR sensor, irradiates laser light, measures the time until it hits the object and bounces back, and measures the distance and direction to the object.

[0184] Actually, the RGB-D sensor 111 is arranged to image the whole body of the subject based on the standing position of the subject in the indoor facilities, and sequentially acquires the RGB image and the depth image as the imaging results.

[0185] The control unit 60 has a fourth control unit 113 under the control of the above-mentioned overall control unit 50, and a data storage unit 114 in which various data are stored in a database so that they can be read and written according to the commands of the fourth control unit 113.

[0186] The fourth control unit 113 is composed of an image coordinate setting unit 120, a coordinate group extraction unit 121, a skeleton information acquisition unit 122, and an action recognition unit 123. The image coordinate setting unit 120 sets a plurality of skeleton representative points attached to the whole body of the subject as image coordinates while estimating the posture of the whole body of the subject based on the RGB images sequentially acquired from the RGB-D sensor 111.

[0187] Specifically, the image coordinate setting unit 120 sets an image coordinate group in a two-dimensional coordinate system with either the left or right shoulder of the subject as the coordinate origin, the direction of the line connecting the left and right shoulders as the X-axis, and the vertical direction with respect to the X-axis as the Z-axis. That is, the coordinate origin is moved to the right shoulder or the left shoulder, and the XZ plane is rotated to coincide with the line connecting the left and right shoulders. As a result, it is possible to enhance the robustness and perform action recognition regardless of the position of the RGB-D sensor 111.

[0188] The coordinate group extraction unit 121 sequentially extracts an image coordinate group related to the action of the subject based on the transition state of the posture of the subject from among the image coordinate groups set by the image coordinate setting unit 120. Actually, the coordinate group extraction unit 121 needs to extract key points of the whole body from the actions, assuming that the clothes of the subject and the surrounding environment may change.

[0189] Therefore, the coordinate group extraction unit 121 adopts a lightweight OpenPose architecture (a system for estimating the skeleton of a person by deep learning) using a learned model to obtain the posture estimation of the whole body and the image coordinates of the skeleton representative points of both hands. For example, for the upper body of the subject, the coordinate group extraction unit 121 sequentially extracts a total of 60 skeleton representative points, including 18 body points and 42 finger points (21 finger points per hand) of the upper body of the subject, as an image coordinate group. By using only stages 1 and 2 of the six stages of the OpenPose architecture, the amount of computation can be reduced by 32.4 [%].

[0190] The skeletal information acquisition unit 122 synchronizes in time the image coordinate groups sequentially extracted by the coordinate group extraction unit 121 and the depth images sequentially acquired from the RGB-D sensor 111, and acquires three-dimensional skeletal information that is temporally continuous with both hands of the subject as the center.

[0191] Specifically, the skeletal information acquisition unit 122 configures a three-dimensional coordinate system in combination with the two-dimensional coordinate system of the image coordinate groups sequentially extracted by the coordinate group extraction unit 121, with the depth direction in the depth images sequentially acquired from the RGB-D sensor 111 as the Y axis, and acquires three-dimensional skeletal information.

[0192] The conversion formula from the image coordinate system of the two-dimensional coordinate system to the world coordinate system of the three-dimensional coordinate system is expressed as the following formula (1).

Equation

[0193] This K, R, and t were calculated by camera calibration using Zhang's method ("A flexible new technique for camera calibration". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.). For the calibration to calculate K, a calibration board with circular dots printed in a grid pattern was used. The center coordinates of each dot on the calibration plate were calibrated up to 1 / 1000.

[0194] As a result, in the motion capture device 73, as shown in FIG. 21, for the entire body of the subject, the relationship between the position in the three-dimensional space and the position of the captured image is clarified, and high robustness is maintained regardless of the imaging position by the RGB-D sensor 111, and three-dimensional skeleton information can be acquired.

[0195] The action recognition unit 123 recognizes an action, which is a connection of a plurality of local actions, from the transition state of the subject's posture based on the three-dimensional skeleton information sequentially acquired by the skeleton information acquisition unit 122.

[0196] The action recognition unit 123 sequentially recognizes the corresponding local actions from the three-dimensional skeleton information sequentially acquired by the skeleton information acquisition unit 122 while referring to the handwashing action recognition model constructed by deep learning, using the action recognition pattern set for each local action as teacher data.

[0197] Specifically, in order to recognize a plurality of local actions, the action recognition unit 123 applies an action recognition model composed of three modules, namely, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as shown in FIG. 22, and a fully connected layer.

[0198] In the action recognition model, it is possible to input the skeleton information of the current frame and the two frames before it, and obtain the likelihood for each of a plurality of types of local actions related to the action.

[0199] Furthermore, as a recognition result, the action recognition unit 123 post-processes the most frequent action among the actions observed in 15 frames (the current frame and the previous 14 frames) as the current local action. In this way, the action recognition unit 123 learned the action recognition model by supervised learning using its own dataset. For optimization, a cross-entropy loss function and Adam (Adaptive movement estimation) with a learning rate of 0.001 were used.

[0200] As a result, in the motion capture device 73, it is possible to significantly improve the recognition accuracy and recognition speed of local movements based on three-dimensional skeleton information.

[0201] (4) Control adjustment according to the subject's tensing state Also, in the function improvement support device 1, the overall control unit 50 detects the transition of the subject's center of gravity position and the load distribution to specific parts at the contact parts (the seat part 40 and the pair of armrest parts 30A, 30B) with the subject based on the measurement results of the vertical resistance force by the vertical resistance force measurement unit 61 (Fig. 3), and determines the tensing state and the relaxing state of the subject when changing to the sitting posture, standing posture, and the transition between both postures of the subject.

[0202] That is, the overall control unit 50 detects the transition of the subject's center of gravity position and the load distribution to specific parts (the load concentration parts at the seat part 40 and the pair of armrest parts 30A, 30B) at the contact parts of the subject, and while determining whether the current posture of the subject is a sitting posture, a standing posture, or a posture during the transition between both postures, it also determines how much the subject is applying a load to the specific parts at the contact parts, thereby determining the tensing state and the relaxing state of the subject.

[0203] Then, the overall control unit 50 adjusts the control content of the state holding variable mechanism unit according to the determined tensing state and relaxing state of the subject.

[0204] As a result, in the function improvement support device 1, while determining the tensing state and the relaxing state during the transition of the subject's posture, by adjusting the control regarding the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit 2, it becomes possible to more safely guide the subject from the sitting state to the standing state and from the standing state to the sitting state.

[0205] (5) Method for detecting environmental information Furthermore, in the function improvement support device 1, the environment information acquisition unit 130 (Fig. 3) acquires environment information representing the surrounding environment of the subject and the floor surface state on which the state holding variable mechanism unit 2 moves. Specifically, the environment information acquisition unit 130 includes a thermometer, a hygrometer, a barometer, a surrounding sound collector 74 for audible sound and ultrasonic waves, etc., and acquires the surrounding environment information centered on the subject. This environment information mainly includes the temperature, humidity, and surrounding sound around the subject.

[0206] When the overall control unit 50 controls the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit 2, it gives top priority to ensuring the safety of the subject based on the environment information obtained from the environment information acquisition unit 130. As a result, in the function improvement support device 1, it is possible to significantly reduce the risk of falling while ensuring the safety of the subject's running and walking while recognizing the surrounding floor surface state and somatic sensation state.

[0207] (6) Gait state synchronization control during the walking assistance function Furthermore, in the function improvement support device 1, the overall control unit 50 controls the fourth control unit 113 while using the above-described motion capture device 73 (Fig. 20), and sequentially acquires the RGB image and depth image of the lower body of the subject captured by the RGB-D sensor (imaging unit) 111, so as to detect the gait state of the subject in real time.

[0208] That is, in a state where the posture holding variable mechanism unit 2 is deformed into a second mechanism corresponding to the wheelchair function, the image coordinate setting unit 120 estimates the posture of the lower body of the subject based on the RGB image sequentially acquired from the RGB-D sensor (imaging unit) 111, and sets a plurality of skeletal representative points attached to the lower body of the subject as image coordinates.

[0209] Subsequently, the coordinate group extraction unit 121 sequentially extracts, from the image coordinate group set by the image coordinate setting unit 120, an image coordinate group related to the walking state of the subject based on the transition state of the subject's posture. The skeleton information acquisition unit 122 acquires three-dimensional skeleton information that is temporally continuous with the lower body of the subject by synchronizing the image coordinate group sequentially extracted by the coordinate group extraction unit 121 with the depth images sequentially acquired from the RGB-D sensor (imaging unit) 111 in terms of time.

[0210] Then, the action recognition unit (walking state detection unit) 123 sequentially recognizes a plurality of local actions from the transition state of the subject's posture based on the three-dimensional skeleton information sequentially acquired by the skeleton information acquisition unit 122, and detects the walking state (walking posture and movement forms of both lower limbs), which is the connection of the plurality of local actions.

[0211] The overall control unit 50 generates power synchronized with the walking state detected by the action recognition unit (walking state detection unit) 123 in the front wheel drive units 43A, 43B and the rear wheel drive units 44A, 44B (wheel drive units), and drives the pair of front wheel units 13A, 13B and the pair of rear wheel units 14A, 14B independently.

[0212] As a result, the function improvement support device 1 can detect the walking state of the subject in real time and assist the walking motion while synchronizing with the walking state.

[0213] (7) Hybrid control during wheelchair function Furthermore, in the function improvement support device 1, the overall control unit 50 is configured to be able to travel while reflecting the operation content according to the subject's own movement intention based on the muscle and skeletal system information of the subject acquired by the human information acquisition unit 62 without the subject operating using the controllers 46A, 46B.

[0214] In the function improvement support device 1, as shown in FIG. 23, in a state where the posture holding variable mechanism unit 2 is deformed into a first mechanism according to the wheelchair function under the control of the overall control unit 50, the motion intention recognition unit 140 recognizes the motion intention of the subject with respect to a specific operation content among the operation contents in the wheelchair function based on the biological signal obtained from the biological signal measurement wearing device (biological signal detection unit) 90.

[0215] Actually, the motion intention recognition unit 140 calculates the probability (certainty factor) that the signal characteristics of the biological signal match each of a plurality of predetermined motion intentions based on the biological signal obtained from the biological signal measurement wearing device (biological signal detection unit) 90.

[0216] Specifically, the motion intention recognition unit 140 obtains a feature vector representing the signal characteristics of the biological signal by a known vector analysis method. Since the predetermined motion intentions indicate predetermined regions in the vector space, the motion intention recognition unit 140 calculates the distance between the position corresponding to the signal characteristics of the biological signal in the vector space and the plurality of predetermined motion intentions.

[0217] Since the position in the vector space is determined by vectorizing the signal characteristics of the biological signal, the motion intention recognition unit 140 obtains the distance between the position and the centroid position of each motion intention region.

[0218] After the motion intention recognition unit 140 selects a predetermined number of positions close to the signal characteristics of the biological signal in the vector space, for the selected number of motion intentions, based on the distance in the vector space, the motion intention recognition unit 140 calculates the certainty factor representing the probability that the signal characteristics of the biological signal match the motion intention.

[0219] The motion intention recognition unit 140 calculates the certainty factor by normalizing it so that the sum (total) of the predetermined number of certainty factors becomes 1. Further, in the distribution shape of the certainty factor, normalization may be performed so that the kurtosis (third moment) and skewness (fourth moment) become predetermined values.

[0220] Subsequently, under the control of the overall control unit 50, the operation content understanding unit 141 understands the operation content, which is the motion intention of the target person recognized by the motion intention recognition unit 140. Specifically, the operation content understanding unit 141 stores, as a correspondence table, the correspondence between a specific motion intention and the operation content corresponding to the motion intention in data, and selects the corresponding operation content based on the motion intention of the target person recognized by the motion intention recognition unit 140 with reference to the correspondence table.

[0221] Then, the overall control unit 50 generates power according to the operation content understood by the operation content understanding unit 141 in the front wheel drive units 43A, 43B and the rear wheel drive units 44A, 44B (wheel drive units).

[0222] As a result, in the function improvement support device 1, it is possible to travel while reflecting the operation content corresponding to the self-motion intention without the target person operating by themselves during the wheelchair function.

[0223] (8) Movement control based on the line of sight target Furthermore, the function improvement support device 1 further includes a line of sight detection unit 142 (Fig. 23) that detects the position of the target person's line of sight by imaging the space corresponding to the target person's field of view and recognizing the movement of the target person's eyeballs within the imaging range.

[0224] In addition to the above-described motion intention recognition method, the motion intention recognition unit 140 recognizes, as the motion intention of the target person, that when the position of the line of sight detected by the line of sight detection unit 142 continues for a predetermined time or longer, the position of the line of sight is set as the movement target.

[0225] As a result, in the function improvement support device 1, by recognizing the motion intention regarding the operation content of the target person not only based on the biological signal of the target person but also based on the line of sight position, it becomes possible to travel while further reflecting the operation content corresponding to the self-motion intention.

[0226] (9) Voluntary operation by voice Furthermore, the function improvement support device 1 further includes a speech content recognition unit 143 (Fig. 23) that recognizes the speech content of the target person based on the speech collected by the sound collection microphone (sound collection unit) 72.

[0227] In addition to the above-described method for recognizing the operation intention, the operation intention recognition unit 140 recognizes the operation content corresponding to the speech content as the operation intention of the target person based on the speech content recognized by the speech content recognition unit 143.

[0228] As a result, in the function improvement support device 1, not only the biological signal of the target person but also the speech content of the target person are added, and by recognizing the operation intention with respect to the operation content of the target person, it is possible to run while further reflecting the operation content according to the own operation intention.

[0229] (10) Other embodiments As described above, in this embodiment, the case where the posture holding variable mechanism unit 2 of the function improvement support device 1 is configured as shown in Figs. 1(A) and (B) has been described. However, the present invention is not limited to this. For example, a configuration of a posture holding variable mechanism unit 150 as shown in Figs. 24(A) to (C) may be applied.

[0230] In Fig. 24, the posture holding variable mechanism unit 150 is different from the posture holding variable mechanism unit 2 shown in Fig. 1 described above. The frame structure 160 has a traveling base unit 160A formed in a substantially U shape as a whole together with a pair of rear wheel units 14A and 14B so that a pair of front wheel units 13A and 13B are bridged by the footrest portion 11. The posture holding variable mechanism unit 150 is composed of a pair of left and right support frames 160B and 160C movably supported in the front-rear direction from the traveling base unit 160A. These pair of support frames 160B and 160C are configured to be tiltable in the front-rear direction with respect to the traveling base unit 160A and to be extendable and retractable in the longitudinal direction.

[0231] According to the function improvement support device 1 as described above, while holding the subject in the sitting posture, the posture holding variable mechanism unit 150 has a first mechanism (FIG. 24(A)) corresponding to the wheelchair function that travels according to the operation by the subject, and while holding the subject in the standing posture, a second mechanism (FIG. 24(B)) corresponding to the walking assistance function that applies an assisting force according to the walking motion of the subject are both configured to be mutually deformable while holding the subject.

[0232] Also, as described above in this embodiment, regarding the case where the task-oriented method of calculating the confidence level corresponding to the motion intention of the subject is applied as the method of recognizing the motion intention of the subject by the motion intention recognition unit 140 (FIG. 23) in the function improvement support device 1, the present invention is not limited to this, and the task thinking method of learning the motion intention understanding may be applied.

[0233] That is, the motion intention recognition unit 140 determines the motion intention from the biological signal obtained by the biological signal measurement wearing device (biological signal detection unit) 90 while referring to the recognition model constructed by deep learning, using the correlation data in which the correspondence between the signal characteristics of the biological signal and the motion intention is set as the teacher data.

[0234] Specifically, the motion intention recognition unit 140 applies a recognition model composed of three modules of a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function) and a fully connected layer to recognize each of a plurality of motion intentions. The motion intention recognition unit 140 uses this recognition model to evaluate whether or not the motion intention is a desired selection result as the degree of likelihood based on the current state of the subject and the determined motion intention, and based on the evaluated result, while correcting the correlation data (eliminating misrecognition), by learning the recognition model by supervised learning, it becomes possible to recognize the motion intention of the subject with relatively high accuracy.

Explanation of Signs

[0235] 1…Function improvement support device, 2, 150…Posture maintaining variable mechanism part, 10, 160…Frame structure, 11…Footrest part, 12A…Left frame, 12B…Right frame, 13A, 13B…Front wheel part, 14A, 14B…Rear wheel part, 20A, 20B…Front arm part, 21A, 21B…Rear arm part, 30A, 30B…Elbow rest part, 31A, 31B…Support arm part, 32A, 32B…Support driving part, 33A, 33B…Parallel maintaining driving part, 40…Seat part, 40A…Left seat half body, 40B…Right seat half body, 41A, 41B…Fixing and holding part, 42A, 42B…Opening and closing driving part, 43A, 43B…Front wheel driving part, 44A, 44B…Rear wheel driving part, 45A, 45B…Posture deformation driving part, 46A, 46B…Controller, 47A, 47B…Curved frame body, 48A, 48B…Backrest driving part, 50…Overall control part, 60…Control unit, 61…Vertical resistance measuring part, 62…Human information acquisition part, 63…Driving battery, 70…Vital measurement device, 71…Face imaging camera, 72…Sound collecting microphone, 73…Motion capture device, 74…Peripheral measurement device, 75…Information detection device, 80…Brain activity measurement part, 81…Blood flow measurement part, 82…Electrocardiograph, 83…First control part, 84…Wireless communication device, 90…Biological signal measurement wearing device, 91…Biological signal sensor, 92A~92F…Biological signal sensor group, 93…Measurement module, 94…Measurement module controller, 95…Memory, 96…Communication part, 97…Second control part, 100…Near-infrared detection part, 101…Far-infrared detection part, 102…Third control part, 111…RGB-D sensor, 112…IMU sensor, 113…Fourth control part, 120…Image coordinate setting part, 121…Coordinate group extraction part, 122…Skeleton information acquisition part, 123…Action recognition part, 130…Environmental information acquisition part, 140…Motion intention recognition part, 141…Operation content understanding part, 142…Gaze detection part, 143…Speech content recognition part, 160A…Traveling base part, 160B, 160C…Support frame.

Claims

1. A wheelchair function that travels in response to an operation by the subject while holding the subject in a sitting position, and a walking assistance function that applies assistance force in response to the walking motion of the subject while holding the subject in a standing position, and a posture holding variable mechanism unit that can be deformed while holding the subject from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, and A human information acquisition unit that acquires human information including the subject's brain / nervous system information, muscle / skeletal system information, physiological information, psychological information, motion information, and behavior information, and A current situation function discrimination unit that discriminates the current situation of the subject's physical function and cognitive function based on the human information obtained by the human information acquisition unit, and A vertical resistance force measurement unit provided in the state holding variable mechanism unit that measures the vertical resistance force at the contact part with the subject in real time, and A control unit that controls the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit based on the discrimination result by the current situation function discrimination unit and the measurement result by the vertical resistance force measurement unit A function improvement support device characterized by comprising the above.

2. Based on the measurement result of the vertical resistance force at the contact part with the subject by the vertical resistance force measurement unit, the control unit detects the transition of the center of gravity position of the subject and the load distribution to a specific part at the contact part, and then determines the forceful state and the relaxed state during the sitting position, standing position, and transition between both postures of the subject. After that, according to the forceful state and the relaxed state, the control content of the state holding variable mechanism unit is adjusted The function improvement support device according to claim 1, characterized by the above.

3. Further comprising an environmental information acquisition unit that acquires environmental information representing the surrounding environment of the subject and the floor surface state where the state holding variable mechanism unit moves, and When controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, the control unit gives top priority to ensuring the safety of the subject based on the environmental information obtained from the environmental information acquisition unit The function improvement support device according to claim 1 or 2, characterized by the above.

4. A wheel drive unit provided in the posture holding variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms, and An imaging unit that images the lower body of the subject and sequentially acquires an RGB image and a depth image, and In a state where the posture holding variable mechanism unit is deformed into the second mechanism according to the wheelchair function, based on the RGB images sequentially acquired from the imaging unit, while estimating the posture of the lower body of the subject, an image coordinate setting unit that sets a plurality of skeleton representative points attached to the lower body of the subject as image coordinates; A coordinate group extraction unit that sequentially extracts an image coordinate group related to the gait state of the subject based on the transition state of the subject's posture from among the image coordinate groups set by the image coordinate setting unit; A skeleton information acquisition unit that acquires temporally continuous three-dimensional skeleton information centered on the lower body of the subject by synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit with the depth images sequentially acquired from the imaging unit; A gait state detection unit that detects a gait state, which is a connection of a plurality of local movements, from the transition state of the subject's posture based on the three-dimensional skeleton information sequentially acquired by the skeleton information acquisition unit; The control unit generates power synchronized with the gait state detected by the gait state detection unit in the wheel drive unit. The function improvement support device according to claim 1 or 2, characterized in that.

5. A wheel drive unit provided in the posture holding variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms; A biological signal detection unit disposed on the body surface part of the subject and having an electrode group for detecting a biological signal of the subject; In a state where the posture holding variable mechanism unit is deformed into the first mechanism according to the wheelchair function, based on the biological signal obtained from the biological signal detection unit, an operation intention recognition unit that recognizes the operation intention of the subject with respect to a specific operation content among the operation contents in the wheelchair function; An operation content understanding unit that understands the operation content, which is the operation intention of the subject recognized by the operation intention recognition unit; The control unit generates power according to the operation content understood by the operation content understanding unit in the wheel drive unit. The function improvement support device according to claim 1 or 2, characterized in that.

6. Further provided with a gaze detection unit that detects the position of the subject's gaze by image recognition of the movement of the subject's eyeballs within the imaging range while imaging the space corresponding to the subject's field of view; Based on the position of the line of sight detected by the line-of-sight detection unit, when the position of the line of sight continues for a predetermined time or more, the motion intention recognition unit recognizes the position of the line of sight as a movement target as the motion intention of the subject. The functional improvement support device according to claim 5, characterized in that.

7. The apparatus further includes a sound collection unit that collects the voice of the subject, and a speech content recognition unit that recognizes the speech content based on the voice collected by the sound collection unit. Based on the speech content recognized by the speech content recognition unit, the motion intention recognition unit recognizes the operation content corresponding to the speech content as the motion intention of the subject. The functional improvement support device according to claim 5, characterized in that.

8. A posture holding variable mechanism unit having a wheelchair function that travels in response to an operation by the subject while holding the subject in a seated posture, and a walking assistance function that applies an assisting force in response to the walking motion of the subject while holding the subject in a standing posture, is configured to be deformable from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, while holding the subject. A first step of acquiring human information including the subject's brain and nervous system information, muscle and skeletal system information, physiological information, psychological information, motion information, and behavior information. A second step of determining the current status of the subject's physical function and cognitive function based on the human information obtained in the first step. A third step of measuring in real time the vertical resistance force at the contact part between the subject and the state holding variable mechanism unit. A fourth step of controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit based on the determination result in the second step and the measurement result in the third step. A functional improvement support method characterized by comprising the above steps.

9. In the third step, based on the measurement result of the vertical resistance force at the contact part with the subject, while detecting the transition of the center of gravity position of the subject and the load distribution to a specific part at the contact part, the forceful state and the relaxed state at the time of the subject's seated posture, standing posture, and transition between both postures are judged. In the fourth step, the control content of the state holding variable mechanism unit is adjusted according to the forceful state and the relaxed state of the subject. The functional improvement support method according to claim 8, characterized in that.

10. It further includes a fifth step of acquiring environmental information representing the surrounding environment of the subject and the road surface condition on which the state holding variable mechanism unit moves. In the fourth step, when controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, based on the environmental information obtained from the fifth step, ensuring the safety of the subject is given top priority. The method for assisting in improving functions according to claim 8 or 9, characterized in that.

11. The wheel drive units provided in the posture holding variable mechanism unit independently drive a pair of left and right drive wheels common to the first and second mechanisms, and the imaging unit images the lower body of the subject to sequentially acquire an RGB image and a depth image. In a state where the posture holding variable mechanism unit is deformed into the second mechanism corresponding to the wheelchair function, while estimating the posture of the lower body of the subject based on the RGB image sequentially acquired from the imaging unit, a sixth step of setting a plurality of skeleton representative points attached to the lower body of the subject as image coordinates. A seventh step of sequentially extracting an image coordinate group related to the walking posture state of the subject based on the transition state of the posture of the subject from among the image coordinate groups set in the sixth step. An eighth step of synchronizing the image coordinate groups sequentially extracted in the seventh step with the depth images sequentially acquired from the imaging unit to acquire temporally continuous three-dimensional skeleton information centered on the lower body of the subject. A ninth step of detecting a walking posture state, which is a connection of a plurality of local movements, from the transition state of the posture of the subject based on the three-dimensional skeleton information sequentially acquired in the eighth step. It is provided with, and in the fourth step, power synchronized with the walking posture state detected in the ninth step is generated in the wheel drive unit. The function improvement support device according to claim 8 or 9, characterized in that.

12. The wheel drive units provided in the posture holding variable mechanism unit independently drive a pair of left and right drive wheels common to the first and second mechanisms, and using an electrode group arranged on the body surface part of the subject, detect the biological signal of the subject. In a state where the posture holding variable mechanism unit is deformed into the first mechanism corresponding to the wheelchair function, a tenth step of recognizing the movement intention of the subject with respect to a specific operation content among the operation contents in the wheelchair function based on the biological signal of the subject. An eleventh step of understanding the operation content, which is the motion intention of the target person recognized in the seventh step and, in the fourth step, generating power according to the operation content understood in the eleventh step to the wheel drive unit The method for assisting in improving functions according to claim 8 or 9, characterized in that

13. Further comprising a twelfth step of detecting the position of the line of sight of the target person by image recognition of the movement of the eyeballs of the target person within the imaging range while imaging the space corresponding to the field of view of the target person and, in the tenth step, recognizing, as the motion intention of the target person, that when the position of the line of sight continues for a predetermined time or more based on the position of the line of sight detected in the twelfth step, the position of the line of sight is set as a movement target The method for assisting in improving functions according to claim 12, characterized in that

14. Further comprising a thirteenth step of collecting the voice of the target person and recognizing the speech content based on the voice and, in the tenth step, recognizing, as the motion intention of the target person, the operation content corresponding to the speech content based on the speech content recognized in the thirteenth step The method for assisting in improving functions according to claim 12, characterized in that

Images