Function improvement assistance device and function improvement assistance method

Abstract

The present invention determines the current state of the physical function and the cognitive function of a subject on the basis of human information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, and behavior information of the subject, and controls, while measuring in real time the normal force exerted on a state holding variable mechanism unit at a site of contact with the subject, mechanism switching, deformation state, and power application state of the state holding variable mechanism unit on the basis of the result of the determination and the result of the measurement.

Description

Function improvement support device and function improvement support method

[0001] The present invention relates to a function improvement support device and a function improvement support method, and is particularly suitable for application to improve the daily life functions of a subject who has been bedridden for a relatively long time.

[0002] Traditionally, once a person's physical strength weakens and they become bedridden, the resulting frailty (muscle weakness) and decline in cognitive ability accelerate, making it extremely difficult for them to lead an independent daily life.

[0003] Such bedridden people wear diapers all day long, and have great difficulty with everyday activities such as going to the toilet and defecating on their own. In reality, they can only see the ceiling, and their delirium worsens, and there is a very high possibility that they will become completely dependent on care both physically and cognitively. This makes it extremely difficult for them to carry out independent daily activities such as shopping, walking, and communicating.

[0004] In order to overcome these problems and improve the mobility and walking ability of bedridden people, walking aid chairs that can be used as both wheelchairs and walkers have been proposed (see Patent Document 1).

[0005] When the walking aid chair in Patent Document 1 is used as a wheelchair, the wheelchair seat is formed from a central liftable seat and a pair of retractable seat surfaces on either side of it, while when it is used as a walker, the pair of retractable seat surfaces are retracted while only the liftable seat surface is raised, allowing the user to walk by simply lowering their hips slightly and putting their weight on the liftable seat surface.

[0006] Furthermore, when this walking aid chair is used as a walker, the vehicle section moves in the walking direction in accordance with the user's movements, thereby reducing the force with which the user pushes the walking aid chair.

[0007] A further development of this walking aid chair has been proposed by the same inventor as in Patent Document 1 (see Patent Document 2). The walking aid chair in Patent Document 2 functions as both an electric wheelchair and a walking aid, and also functions as a standing aid that helps a user sitting in the electric wheelchair to stand up when transferring from the electric wheelchair to the walking aid, and also functions as a sitting aid that helps a user walking with the walking aid to sit down in the electric wheelchair when transferring from the walking aid to the electric wheelchair.

[0008] JP 2013-85716 A JP 2019-208561 A

[0009] However, the walking aid chairs in Patent Documents 1 and 2 both transition from an electric wheelchair to a walking aid and from a walking aid to an electric wheelchair based on the user's operation, and other than following the user at the user's walking speed, almost all movements are assisted in accordance with the user's operation.

[0010] For this reason, the target users of these walking aid chairs are limited to people with weak legs and hips who require a regular wheelchair, and it is very difficult for people with relatively low independence, such as those who are bedridden, to actively operate the chair to move and walk on their own.

[0011] In order for such people with a relatively low level of independence to improve their mobility and walking ability, it is necessary to improve physical and neurological functions by linking semicircular canals and visual information with physical functions through repeated standing and sitting postures and walking movements.

[0012] However, the walking assistance chairs of Patent Documents 1 and 2 mentioned above were insufficient in practical terms to improve the physical and neurological functions of bedridden people and to enhance their ADL (Activities of Daily Living) and QOL (Quality of Life).

[0013] The present invention has been made in consideration of the above points, and aims to propose a function improvement support device and a function improvement support method that can significantly improve the degree of independence in daily life and achieve comprehensive functional improvement in the subject's physical, psychological, and social aspects.

[0014] In order to solve these problems, the present invention provides a wheelchair function that holds a subject in a seated position and moves in response to operation by the subject, and a walking assistance function that holds the subject in an upright position and applies an assist force in response to the walking movement of the subject, and is equipped with a posture-holding variable mechanism that can change from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, all while holding the subject, a human information acquisition unit that acquires human information including information on the brain and nervous system, musculoskeletal system, physiological information, psychological information, movement information, and behavioral information of the subject, a current function determination unit that determines the current status of the subject's physical functions and cognitive functions based on the human information obtained by the human information acquisition unit, a normal force measurement unit provided in the state-holding variable mechanism that measures normal force at the site of contact with the subject in real time, and a control unit that controls mechanism switching, deformation state, and power application state of the state-holding variable mechanism based on the determination result by the current function determination unit and the measurement result by the normal force measurement unit.

[0015] As a result, the functional improvement support device allows the user to move safely in a large space while remaining in a seated position using the wheelchair function, and, if necessary, allows the user to perform daily activities in a standing or walking position in the living space using the walking assistance function. In particular, for subjects who have been bedridden for a relatively long time, by repeatedly switching between a sitting and standing position, it is possible to improve physical and neurological functions by linking the semicircular canals and visual information with physical functions, thereby significantly improving independence in daily life.

[0016] Furthermore, in the present invention, the control unit detects the shift in the position of the subject's center of gravity and the distribution of load on specific parts at the contact site based on the measurement results of the normal force at the contact site with the subject by the normal force measurement unit, and determines the subject's state of tension and relaxation when in a sitting position, standing position, or when transitioning between both postures, and then adjusts the control content of the state-holding variable mechanism unit according to the state of tension and relaxation.

[0017] As a result, the functional improvement support device can more safely guide the subject from a sitting position to a standing position and from a standing position to a sitting position by determining whether the subject is tense or relaxed when their posture changes and adjusting the control over the mechanism switching, deformation state, and power application state of the state-holding variable mechanism unit.

[0018] Furthermore, the present invention further includes an environmental information acquisition unit that acquires environmental information representing the subject's surrounding environment and the condition of the floor on which the state-holding variable mechanism unit moves, and when controlling the mechanism switching, deformation state, and power application state of the state-holding variable mechanism unit, the control unit gives top priority to ensuring the safety of the subject based on the environmental information acquired from the environmental information acquisition unit.

[0019] As a result, the functional improvement support device can significantly reduce the risk of falling while ensuring the safety of the subject's running and walking, while recognizing the surrounding floor conditions and bodily sensations.

[0020] Furthermore, in the present invention, the posture maintaining variable mechanism includes a wheel drive unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms; an imaging unit for imaging the lower body of the subject and sequentially acquiring RGB images and depth images; an image coordinate setting unit for setting a plurality of skeletal representative points attached to the lower body of the subject as image coordinates while estimating the posture of the lower body of the subject based on the RGB images sequentially acquired from the imaging unit when the posture maintaining variable mechanism is transformed into the second mechanism corresponding to the wheelchair function; and a position determining unit for determining the position of the subject based on the image coordinates set by the image coordinate setting unit. The system is equipped with a coordinate group extraction unit that sequentially extracts image coordinate groups related to the gait state of the subject based on the transition state of the subject's posture, a skeletal information acquisition unit that acquires three-dimensional skeletal information that is continuous over time, centered on the lower body of the subject, by time-synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit with the depth images sequentially acquired from the imaging unit, and a gait state detection unit that detects the gait state, which is a connection of multiple local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit, and the control unit generates power in the wheel drive unit that is synchronized with the gait state detected by the gait state detection unit.

[0021] As a result, the function improvement assistance device can detect the gait state of the subject in real time, and can assist the walking movement in synchronization with the gait state.

[0022] Furthermore, the present invention includes a wheel drive unit provided in the posture maintenance variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms; a biosignal detection unit arranged on a surface part of the subject's body and having a group of electrodes for detecting the subject's biosignals; an action intention recognition unit for recognizing the subject's action intention for a specific operation content among the operation contents of the wheelchair function based on the biosignal obtained from the biosignal detection unit when the posture maintenance variable mechanism unit is transformed into the first mechanism corresponding to the wheelchair function; and an operation content understanding unit for understanding the operation content that is the subject's action intention recognized by the action intention recognition unit, and the control unit generates power in the wheel drive unit according to the operation content understood by the operation content understanding unit.

[0023] As a result, the function improvement support device allows the subject to move around in wheelchair function while the operation content reflects the subject's intentions, without the subject having to operate the device himself.

[0024] Furthermore, the present invention further includes a gaze detection unit that detects the position of the subject's gaze by performing image recognition of the movement of the subject's eyeballs within the imaging range while imaging a space corresponding to the subject's field of vision, and the action intention recognition unit recognizes, based on the gaze position detected by the gaze detection unit, that the gaze position is to be used as a movement target if the gaze position continues for a predetermined period of time or more.

[0025] As a result, the functional improvement support device recognizes the subject's intention regarding the operation content by taking into account not only the subject's biological signals but also their gaze position, making it possible for the subject to drive while reflecting the operation content in accordance with their own intention to operate the vehicle.

[0026] Furthermore, the present invention further includes a sound collection unit that collects the voice of the subject, and a speech content recognition unit that recognizes the speech content based on the voice collected by the sound collection unit, and the action intention recognition unit recognizes the operation content corresponding to the speech content as the action intention of the subject based on the speech content recognized by the speech content recognition unit.

[0027] As a result, the functional improvement support device recognizes the subject's intentions regarding the operation content by taking into account not only the subject's biological signals but also the subject's own speech, making it possible for the subject to drive while reflecting the operation content that is more in line with the subject's intentions.

[0028] Furthermore, in the present invention, the posture-holding variable mechanism has a wheelchair function that moves in response to operation by the subject while holding the subject in a seated position, and a walking assistance function that applies an assist force in response to the subject's walking movement while holding the subject in an upright position, and is configured to be able to change from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, both while holding the subject, and is provided with a first step of acquiring personal information including brain / nervous system information, musculoskeletal system information, physiological information, psychological information, movement information, and behavioral information of the subject, a second step of determining the current state of the subject's physical functions and cognitive functions based on the personal information obtained in the first step, a third step of measuring in real time the normal force at the contact point between the subject and the state-holding variable mechanism, and a fourth step of controlling mechanism switching, deformation state, and power application state of the state-holding variable mechanism based on the determination result in the second step and the measurement result in the third step.

[0029] As a result, the functional improvement support device allows the user to move safely in a large space while remaining in a seated position using the wheelchair function, and, if necessary, allows the user to perform daily activities in a standing or walking position in the living space using the walking assistance function. In particular, for subjects who have been bedridden for a relatively long time, by repeatedly switching between a sitting and standing position, it is possible to improve physical and neurological functions by linking the semicircular canals and visual information with physical functions, thereby significantly improving independence in daily life.

[0030] According to the present invention, it is possible to realize a function improvement support device and a function improvement support method that can significantly improve the degree of independence in daily life and achieve comprehensive functional improvement in the physical, psychological, and social aspects of the subject.

[0031] 1 is a conceptual diagram showing the external configuration of a function improvement support device according to the present embodiment. FIG. 1 is a conceptual diagram showing the configuration of the posture maintenance variable mechanism shown in FIG. 1. FIG. 2 is a conceptual diagram showing the circuit configuration of a control system in the function improvement support device shown in FIG. 1. FIG. 3 is a conceptual diagram used to explain a person information acquisition unit. FIG. 4 is a conceptual diagram showing a state in which the posture maintenance variable mechanism is a first mechanism corresponding to a wheelchair function. FIG. 5 is a conceptual diagram showing a state in which the posture maintenance variable mechanism is a second mechanism corresponding to a walking assistance function. FIG. 6 is a conceptual diagram used to explain mutual transformation between the wheelchair function and the walking assistance function. FIG. 7 is a perspective view and six-sided view showing the external configuration of a vital sign measuring device. FIG. 8 is a conceptual diagram showing the configuration of a brain activity measuring unit. FIG. 9 is a conceptual diagram showing the configuration of a biosignal measurement garment. FIG. 10 is a conceptual diagram used to explain the wearing state of the biosignal measurement garment. FIG. 11 is a conceptual diagram showing an additional configuration of a vital sign measuring device. FIG. 12 is a graph showing pulse waveforms before and after filter application. FIG. 13 is a conceptual diagram showing the ROI of a near-infrared image and the ROI of a far-infrared image. FIG. 14 is a conceptual diagram used to explain nasal breathing and mouth breathing. FIG. 15 is a conceptual diagram used to explain the temperature states of nasal breathing and mouth breathing. 1 is a graph showing the change in nostril temperature due to nasal breathing and the change in oral cavity temperature due to mouth breathing. FIG. 2 is a graph showing the change in breathing technique from nasal breathing to mouth breathing and the superimposition of each state content. FIG. 3 is a conceptual diagram explaining non-contact measurement of oxygen saturation using light. FIG. 4 is a block diagram showing the internal configuration of a motion capture device. FIG. 5 is a conceptual diagram explaining a method for acquiring three-dimensional skeletal information about the entire body of a subject. FIG. 6 is a conceptual diagram showing the structure of a behavior recognition model. FIG. 7 is a conceptual diagram explaining hybrid control during wheelchair function. FIG. 8 is a conceptual diagram showing another embodiment of a posture holding variable mechanism.

[0032] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0033] 1(A) and 1(B) show a function improvement support device 1 according to this embodiment. The function improvement support device 1 has both a wheelchair function (FIG. 1(A)) that moves in response to operation by a subject while holding the subject in a seated position, and a walking assistance function (FIG. 1(B)) that applies an assist force in response to the subject's walking motion while holding the subject in a standing position.

[0034] The functional improvement assistance device 1 has a posture-maintaining variable mechanism 2 that can switch between a first mechanism (FIG. 1A) corresponding to the wheelchair function and a second mechanism (FIG. 1B) corresponding to the walking assistance function according to the movement of the subject. This posture-maintaining variable mechanism 2 is configured to be able to change from the first mechanism corresponding to the wheelchair function to the second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, while supporting the subject.

[0035] (2) Configuration of the Posture Holding Variable Mechanism As shown in Figures 2(A) and (B), the posture holding variable mechanism 2 has a frame structure 10 that is generally U-shaped and has a bilaterally symmetrical structure. The frame structure 10 is composed of a footrest section 11 that functions as a footrest in the center of the curve, and a left frame 12A and a right frame 12B that are bilaterally symmetrical with respect to the footrest section 11.

[0036] The left frame 12A and the right frame 12B of the frame structure 10 each have a shape that is bent at a predetermined angle near the middle along the longitudinal direction.

[0037] In addition, in the left frame 12A and the right frame 12B, one end of the front arm portions 20A and 20B, which are respectively provided with the front wheel portions 13A and 13B, is fixed near the footrest portion 11, and one end of the rear arm portions 21A and 21B, which are respectively provided with the rear wheel portions 14A and 14B, is freely movable at the bent portion.

[0038] Furthermore, support arms 31A and 31B, each with an armrest 30A or 30B attached to one end thereof, are movably mounted on the other end of the left and right frames 12A and 12B. These left and right support arms 31A and 31B are integrally engaged with corresponding drive mechanisms (support drive units 32A and 32B and parallel holding drive units 33A and 33B shown in FIG. 3, which will be described later).

[0039] Fixed holders 41A and 41B to which a portion of the seat 40 is movably attached are provided at predetermined positions on the left and right frames 12A, 12B between the support arms 31A, 31B and the rear arms 21A, 21B. These left and right fixed holders 41A, 41B are engaged with corresponding opening / closing drive units 42A, 42B (FIG. 3), respectively.

[0040] The left and right front wheel sections 13A, 13B are composed of a pair of left and right drive wheels with an omni-wheel structure (a wheel structure in which multiple rollers are rotatably attached so as to be perpendicular to the axle, and each roller is arranged circumferentially so as to match the curvature of a circle centered on the axle) that can rotate freely in the running direction or in a direction intersecting the running direction depending on the running of the device main body.

[0041] The left and right front wheels 13A, 13B are engaged with front wheel drive units 43A, 43B (FIG. 3) via front arm units 20A, 20B, respectively. The front wheel drive units 43A, 43B rotate the corresponding front wheels 13A, 13B in the traveling direction and in a direction intersecting the traveling direction, about drive shafts provided at the other ends of the front arm units 20A, 20B.

[0042] The left and right rear wheel arm sections 21A, 21B have rear wheel sections 14A, 14B connected to the other ends thereof, respectively, and are integrally engaged with the corresponding drive mechanism system (rear wheel drive sections 44A, 44B and posture deformation drive sections 45A, 45B shown in FIG. 3, which will be described later). The rear wheel drive sections 44A, 44B rotate the corresponding rear wheel sections 14A, 14B around drive shafts provided at the other ends of the rear arm sections 21A, 21B.

[0043] The posture deformation drive units 45A, 45B (FIG. 3) rotate one end of the rear arm units 21A, 21B around the rotation axes provided at the bent portions of the left frame 12A and the right frame 12B. These posture deformation drive units 45A, 45B have built-in locking mechanisms for inhibiting rotation (see Japanese Patent Application Laid-Open Nos. 4997416 and 5344501 filed by the present inventor), which prevent rotation of one end of the left and right rear arm units 21A, 21B in the direction opposite to the drive direction, thereby ensuring the safety of the subject.

[0044] The support drive units 32A, 32B (FIG. 3) rotate the other ends of the support arms 31A, 31B around the rotation axes provided at the ends of the left frame 12A and the right frame 12B. The parallelism drive units 33A, 33B (FIG. 3) rotate the armrests 30A, 30B around one end of the left and right support arms 31A, 31B.

[0045] The left and right armrests 30A, 30B each have a flat surface for the subject's upper arms to rest on, and at the front longitudinal ends thereof are provided controllers 46A, 46B for operation by the subject, while at the rear longitudinal ends thereof are rotatably attached curved frames 47A, 47B which correspond to the backrests of the subject.

[0046] These left and right curved frames 47A, 47B form a pair, and are designed in advance to hold the subject's lower back from the back side (backrest forming state) when closed (with their tips facing each other). Backrest drive units 48A, 48B (FIG. 3) are built into the left and right seating sections 30A, 30B, and the backrest drive units 48A, 48B drive the pair of curved frames 47A, 47B to open and close simultaneously.

[0047] The support drive units 32A, 32B, the parallel holding drive units 33A, 33B and the backrest drive units 48A, 48B are driven in conjunction with each other under the control of the integrated control unit 50 described later so that the flat surfaces of the left and right armrest units 30A, 30B are always maintained in an approximately horizontal position relative to the floor surface, which is the running surface.

[0048] The seat 40 is divided into left and right halves that are joined together to form a seating surface. The seat 40 has a left seat half 40A corresponding to the left frame 12A and a right seat half 40B corresponding to the right frame 12B, which are rotatable within a predetermined angle range using fixed holders 41A and 41B as rotation bases, respectively.

[0049] The opening / closing drive units 42A, 42B (Figure 3) rotate the left seat half 40A and the right seat half 40B that make up the seat 40 within a predetermined angle range, using the fixed holding units 41A, 41B provided on the above-mentioned left frame 12A and right frame 12B, respectively, as the rotation center, thereby forming the seat 40 for the subject while opening it in a substantially parallel relationship with the left frame 12A and the right frame 12B.

[0050] The opening and closing drive units 42A, 42B are configured to operate in conjunction with the posture deformation drive units 45A, 45B in accordance with the control of the overall control unit 50 to adjust the rotation angle of the left seat half 40A and the right seat half 40B that make up the seat 40.

[0051] In the function improvement assistance device 1, the posture-maintaining variable mechanism 2 is provided with a control unit 60 inside the footrest 11 of the frame structure 10, and normal force measuring units 61 are provided on the flat surfaces of the left and right armrests 30A, 30B and on the seating surfaces of the left and right seat halves 40A, 40B. Furthermore, a human information acquiring unit 62 is attached to part of the posture-maintaining variable mechanism 2 and to the subject (FIG. 3). Furthermore, as will be described later, the posture-maintaining variable mechanism 2 is provided with an environmental information acquiring unit 130 so that the subject can recognize the condition of the surrounding floor and their bodily sensations.

[0052] 3 shows the circuit configuration of the control system in the function improvement support device 1. In the function improvement support device 1, a central control unit 50 consisting of an MCM (Multi-Chip Module) equipped with a CPU (Central Processing Unit) and memory is built into the control unit 60 in the attitude holding and variable mechanism 2, and is configured to centrally control the support drive units 32A, 32B, parallel holding drive units 33A, 33B, opening and closing drive units 42A, 42B, front wheel drive units 43A, 43B, rear wheel drive units 44A, 44B, attitude deformation drive units 45A, 45B, and backrest drive units 48A, 48B.

[0053] The support drive units 32A, 32B, parallel holding drive units 33A, 33B, opening / closing drive units 42A, 42B, front wheel drive units 43A, 43B, rear wheel drive units 44A, 44B, posture deformation drive units 45A, 45B and backrest drive units 48A, 48B each have a drive actuator (not shown), and are configured to drive each drive actuator in accordance with the control of the integrated control unit 50 to transmit output.

[0054] In addition, the functional improvement support device 1 has a relatively large capacity driving battery 63 made of a secondary battery or a capacitor built into the frame structure 10, and is able to supply power to each of the driving actuators of the support driving units 32A, 32B, parallel holding driving units 33A, 33B, opening and closing driving units 42A, 42B, front wheel driving units 43A, 43B, rear wheel driving units 44A, 44B, posture deformation driving units 45A, 45B and backrest driving units 48A, 48B.

[0055] 4, the human information acquisition unit 62 acquires human information including brain and nervous system information, musculoskeletal system information, physiological information, psychological information, motion information, and behavior information of the subject. Specifically, the human information acquisition unit 62 has vital sign measuring devices (measuring means for electrocardiogram, pulse rate, pulse pressure, oxygen saturation, body temperature, brain waves, blood glucose level, blood pressure, electromyogram, etc.) 70, a face imaging camera 71, a sound collecting microphone 72, motion capture devices (optical, inertial sensor, mechanical, and magnetic types, or a combination thereof) 73, and peripheral measuring devices (thermometer, hygrometer, audible and ultrasonic ambient sound collector, etc.) 74, and an information detection unit 75 that detects human information (described later) based on outputs from these sensors and devices, etc.

[0056] In this human information acquisition unit 62, the information detection unit 75 detects the subject's brain and nervous system information, musculoskeletal system information, and physiological information based on the detection results from the vital sign measuring device 70, and also detects psychological information by analyzing the subject's facial expressions and voice using a facial imaging camera 71 and a sound collecting microphone 72, and further detects the subject's movement information and behavior information using a motion capture device 73.

[0057] The general control unit 50 serves as a current function determination unit, and determines the current state of the physical functions and cognitive functions of the subject based on the person information obtained by the person information acquisition unit 62.

[0058] Furthermore, the normal force measuring unit 61 is built into the seat 40 and armrests 30A, 30B of the state-holding variable mechanism 2, and measures the normal force at the contact points with the subject in real time. Specifically, the normal force measuring unit 61 has a structure in which a plurality of strain gauges (strain sensing elements) are arranged in a predetermined array on the surface of the seat 40 and armrests 30A, 30B, respectively, and is configured to measure all or part of the external forces in three-dimensional directions acting on the contact points of the subject.

[0059] The multiple strain gauges that make up this normal force measuring unit 61 are strain detection elements in which impurities are partially doped into a thin silicon substrate to form piezoresistors, or strain detection elements in which a thin metal film is formed on a silicon substrate, in order to detect minute strain changes with high sensitivity.

[0060] Based on the output signals from each strain gauge, the normal force measurement unit 61 divides the contact area of ​​the subject into multiple mesh units and analyzes the stress strain at the contact area by analyzing the stress at each node of the element (finite element method). As a result, the normal force measurement unit 61 measures in real time the three-dimensional external force acting on the contact area with the subject as a normal force, based on the analyzed distribution and transition state of the stress strain.

[0061] Based on the human information obtained from the human information acquisition unit 62, the general control unit 50 comprehensively judges the subject's brain / nervous system information, musculoskeletal system information, and physiological information, as well as the subject's psychological information, and the subject's movement information and behavioral information, to determine the current state of the subject's physical function (degree of comparison of each element with a reference physical function) and cognitive function (degree of comparison with a specified cognitive judgment standard).

[0062] (2-1) Aspects of wheelchair function and walking assistance function In the functional improvement support device 1, the posture holding variable mechanism unit 2 is capable of transforming into either a first mechanism (Fig. 1(A)) corresponding to the wheelchair function, which holds the subject in a seated position and moves in response to operation by the subject, or a second mechanism (Fig. 1(B)) corresponding to the walking assistance function, which holds the subject in a standing position and applies an assist force in response to the subject's walking movement, while both mechanisms are capable of transforming into each other while holding the subject.

[0063] 5A and 5B show the state in which the attitude-maintaining variable mechanism 2 is in the first mechanism corresponding to the wheelchair function. In the first mechanism, the rear arms 21A and 21B are positioned relative to the left frame 12A and the right frame 12B, respectively, so that the rear wheels 14A and 14B are located at the farthest positions relative to the front wheels 13A and 13B.

[0064] In this first mechanism, the left seat half 40A and the right seat half 40B, which are engaged with the fixed holding portions 41A and 41B, form the seat portion 40 so that the seating surface is approximately horizontal, and the pair of curved frame bodies 47A and 47B are positioned so that they form a backrest while the flat surfaces of the armrest portions 30A and 30B are maintained approximately horizontal.

[0065] In this way, in the functional improvement support device 1, by setting the posture holding variable mechanism unit 2 to the first mechanism state corresponding to the wheelchair function, the subject can sit on the seat unit 40, place both arms on the armrest units 30A, 30B, and place both feet on the footrest unit 11, and operate the wheelchair while remaining in a seated position similar to that of a wheelchair using the controllers 46A, 46B provided at the front ends of the armrest units 30A, 30B.

[0066] 6A and 6B show the state in which the attitude-maintaining variable mechanism 2 is in the second mechanism corresponding to the walking assist function. In the second mechanism, the rear arms 21A and 21B are positioned relative to the left frame 12A and the right frame 12B, respectively, so that the rear wheels 14A and 14B are closest to the front wheels 13A and 13B.

[0067] In this second mechanism, the left seat half 40A and the right seat half 40B engaged with the fixed holding portions 41A and 41B are positioned so that their seating surfaces are approximately parallel to the left frame 12A and the right frame 12B, respectively, and the pair of curved frame bodies 47A and 47B are positioned in an open state while the flat surfaces of the armrest portions 30A and 30B are maintained approximately horizontal.

[0068] In this way, in the functional improvement support device 1, by setting the posture maintaining variable mechanism unit 2 to the second mechanism state corresponding to the walking assist function, the subject can grasp the left and right armrest units 30A, 30B while standing, and walk while receiving assist force according to the walking movement while remaining in a standing position.

[0069] (2-2) Mutual transformation between wheelchair function and walking assistance function In the functional improvement support device 1, the posture holding variable mechanism unit 2 is designed to be able to transform from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, both while holding the subject.

[0070] In the first mechanism state, the posture holding variable mechanism unit 2 has the overall control unit 50 move in the desired direction in response to the subject's operation, and determines whether the subject will begin to transition from a seated position to a standing position based on the discrimination results of the current function discrimination unit and the real-time measurement results of the normal force acting on the seat 40 and armrests 30A, 30B by the normal force measurement unit 61 (Figure 7 (A)).

[0071] Then, when the subject begins to transition to a standing posture, the overall control unit 50 controls the posture transformation drive units 45A, 45B to begin rotating the rear arm units 21A, 21B so that the rear wheel units 14A, 14B approach the front wheel units 13A, 13B.

[0072] Next, as the subject transitions from a seated position to a standing position, the integrated control unit 50 controls the opening / closing drive units 42A, 42B to begin opening the left seat half 40A and the right seat half 40B that form the seat 40, and simultaneously controls the backrest drive units 48A, 48B to begin opening the pair of curved frames 47A, 47B. At this time, the integrated control unit 50 controls the parallelism drive units 33A, 33B so that the flat surfaces of the left and right armrests 30A, 30B are always kept horizontal ( FIG. 7B ).

[0073] When the subject has completely transitioned to an upright position, the integrated control unit 50 causes the posture maintaining and variable mechanisms 45A, 45B to enter the second mechanism state. That is, the integrated control unit 50 controls the posture transformation drive units 45A, 45B to position the rear wheels 14A, 14B at positions closest to the front wheels 13A, 13B, and simultaneously position the left and right seat half bodies 40A, 40B so that their seating surfaces are substantially parallel to the left and right frames 12A, 12B, respectively, and position the pair of curved frames 47A, 47B so that they are completely open ( FIG. 7C ).

[0074] On the other hand, when the posture holding variable mechanism unit 2 is in the second mechanism state, the overall control unit 50 performs an assisting operation in synchronization with the walking movement of the subject, and determines whether the subject will begin to transition from a standing position to a sitting position based on the discrimination results of the current function discrimination unit and the real-time measurement results of the normal force acting on the seat unit 40 and armrest units 30A, 30B by the normal force measurement unit 61 (Figure 7 (C)).

[0075] Then, when the subject begins to transition to a seated position, the overall control unit 50 controls the posture deformation drive units 45A, 45B to begin rotating the rear arm units 21A, 21B so that the rear wheel units 14A, 14B move away from the front wheel units 13A, 13B.

[0076] Next, as the subject transitions from a standing position to a seated position, the integrated control unit 50 controls the opening / closing drive units 42A, 42B to move the left and right seat half units 40A, 40B toward each other to form the seat 40, and simultaneously controls the backrest drive units 48A, 48B to move the pair of curved frames 47A, 47B toward each other to form the backrests. At this time, the integrated control unit 50 controls the parallelism drive units 33A, 33B so that the flat surfaces of the left and right armrests 30A, 30B remain horizontal at all times (FIG. 7B).

[0077] When the subject has completely shifted to a seated position, the integrated control unit 50 causes the posture-maintaining and variable mechanism 2 to assume the first mechanism state. That is, the integrated control unit 50 controls the posture transformation drive units 45A and 45B to position the rear wheels 14A and 14B at the farthest position from the front wheels 13A and 13B, and simultaneously positions the left and right seat half bodies 40A and 40B to face each other to form the seat 40 with a substantially horizontal seating surface, and positions the pair of curved frames 47A and 47B to form a backrest (FIG. 7A).

[0078] When transforming the subject from the first mechanism to the second mechanism, or from the second mechanism to the first mechanism while holding the subject, the integrated control unit 50 controls the mechanism switching, transformation state, and power application state of the state holding variable mechanism unit 2 based on the discrimination results of the current function discrimination unit and the measurement results of the normal force measurement unit 61.

[0079] That is, based on the current state of the subject's physical and cognitive functions, which is the result of the discrimination made by the current function discrimination unit, and the normal force acting on the seat 40 and armrests 30A, 30B, which is the measurement result made by the normal force measurement unit 61, the overall control unit 50 switches the state holding variable mechanism unit 2 from the state of the first mechanism corresponding to the wheelchair function or the state of the second mechanism corresponding to the walking assistance function, and controls the transformation state during the mechanism switching process (the speed of transformation, posture position, and mechanism switching in the opposite direction depending on the situation), and also controls the power application state in that transformed state.

[0080] As a result, with the function improvement support device 1, the wheelchair function allows the user to move safely in a large space while remaining in a seated position, and the walking assistance function allows the user to perform daily activities in a standing or walking position in the living space, as needed. In particular, for subjects who have been bedridden for a relatively long time, repeated sitting and standing postures can improve physical and neurological functions through the linkage of semicircular canals / visual information with physical functions, thereby making it possible to significantly improve independence in daily life.

[0081] (3) Detailed Configuration and Function of the Human Information Acquisition Unit (3-1) Configuration of the Vital Sign Measuring Device Specifically, as shown in FIG. 8A , the vital sign measuring device 70 has an electrode terminal group ET and a photoelectric probe window PW arranged in a predetermined pattern on a measurement surface 70X of a device main body 70A, and measures the subject's biological information via the measurement surface 70X. The biological information measured by the vital sign measuring device 70 is at least one measurement result of electrocardiogram, pulse, pulse pressure, oxygen saturation, body temperature, electroencephalogram, blood glucose level, blood pressure, and electromyogram. In other words, the vital sign measuring device 70 is configured so that the same device main body can be used as a means for measuring all types of biological information.

[0082] 8(B), the vital sign measuring device 70 has a sound collecting microphone 72 (FIG. 4) detachably connected to a connection terminal CT provided on the side of the device body 70A, and when connected, can collect vibrations generated from the subject's larynx and pharynx as acoustic signals. As a result, it is possible to obtain information about the subject's vibrations (inaudible range) associated with the heart and breathing that cannot be heard by the human ear.

[0083] (3-2) Method for detecting brain and nervous system information The human information acquisition unit 62 has a brain activity measurement unit 80 that is attached to the subject's head and measures brain waves and cerebral blood flow in a measurement region of the head, as an additional component of the vital sign measurement device 70. The information detection unit 75 converts the transmission state of the central nervous system centered on the head into quantitative nervous system data based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measurement unit 80.

[0084] The brain activity measuring section 80 according to this embodiment is in line with the content described in Japanese Patent Application Laid-Open No. 5283700 by the inventor of the present application, and uses the same principle.

[0085] 9(A), the brain activity measuring unit 80 has a blood flow measuring unit 81, an electrocardiograph 82, a first control unit 83, and a wireless communication device 84. The blood flow measuring unit 81 measures the state of blood flow from changes in the optical path and transmitted light amount of laser light caused by blood flow, and further measures the state of brain activity from the state of blood flow in the brain.

[0086] That is, when the laser light irradiated onto the blood enters the blood layer, it passes through the blood as both light components: a reflected and scattered light component by normal red blood cells and a reflected and scattered light component by adherent thrombi. The influence of the laser light on the process of passing through the blood layer changes from moment to moment depending on the state of the blood. Therefore, by continuously measuring the amount of transmitted light (amount of reflected light) and observing the change in the amount of light, it is possible to observe various changes in the state of the blood.

[0087] The blood flow measurement unit 81 comprises a net-like base 85 formed in a hemispherical shape corresponding to the external shape of the head so as to be worn on the head, and a number of sensor units 86A to 86N. Each of the sensor units 86A to 86N is supported at predetermined intervals by the net-like base 85, and outputs a detection signal of the amount of transmitted light measured at each measurement point on the head to a first control unit 83. The electrocardiograph 82 measures cardiac potentials generated in response to cardiac movement using electrodes E attached to the subject's skin.

[0088] The first control unit 83 derives the displacement of blood vessels and tissues around the blood vessels due to blood flow and measures the state of brain activity (distribution of red blood cells) based on the light intensity when light (FIG. 9B) emitted from the light-emitting unit 87 of each sensor unit 86A to 86N is received by the light-receiving unit 88. The first control unit 83 also stores a control program that performs arithmetic processing to cancel components due to oxygen saturation included in signals obtained from at least two or more light-receiving units 88.

[0089] The first control unit 83 further derives the displacement of the inner wall of the blood vessel based on the displacement of the blood vessel and the tissue surrounding the blood vessel. The first control unit 83 further calculates the pulse wave velocity at each measurement position from the phase difference between the waveform of the electrocardiogram signal from the electrocardiograph 82 and the waveform of the detection signal obtained from the light receiving unit, and derives the displacement state of the inner wall of the blood vessel from the pulse wave velocity.

[0090] The wireless communication device 84 wirelessly transmits the measurement results (blood flow data) output from the first control unit 83 to the vital sign measuring device 70. The brain activity measuring unit 80 has optical sensor units 86A to 86N arranged on a net-like base 85, and therefore can simultaneously measure blood flow throughout the entire head.

[0091] 9(B), when measuring cerebral blood vessel characteristics, the first control unit 83 selects an arbitrary sensor unit 86D from the array of multiple sensor units 86A to 86N and causes the light-emitting unit 87 of that sensor unit 86D to emit laser light. At this time, the laser light emitted from the light-emitting unit 87 has a wavelength λ (λ≈805 nm) that is not affected by oxygen saturation.

[0092] When the blood flow measurement unit 81 of the brain activity measurement unit 80 is attached to the subject's head, the elasticity of the net-like base 85 positions the multiple sensor units 86A to 86N at each measurement point on the head and holds each measurement surface facing the head surface. With the blood flow measurement unit 81 attached to the head, the multiple sensor units 86A to 86N can measure the vascular characteristics of the middle cerebral artery and anterior cerebral artery (the rate of vascular elasticity, the amount of plaque in the blood vessels, and the rate of arteriosclerosis) by measuring the pulse wave velocity in the cerebral arteries from changes in the amount of light received after irradiating the surface of the brain with light and propagating within the brain.

[0093] The principle of measuring the pulse wave velocity of cerebral arteries will now be described. For example, the pulse wave velocity due to blood flowing through the middle cerebral artery and the anterior cerebral artery is detected by each of the sensor units 86A-86N, which are arranged corresponding to the measurement positions and receive light that has propagated through the brain when irradiated with light. The measurement method involves comparing the waveform of the cardiac potential obtained from the electrocardiograph 82 with the waveform of the signal output from each of the sensor units 86A-86N at the measurement positions, determining the pulse wave velocity from the phase difference, and deriving the vascular characteristics corresponding to the pulse wave velocity.

[0094] Next, the principle of detecting vascular characteristics from cerebral blood flow will be described. As shown in Figure 9(B), the brain BR is covered by cerebrospinal fluid BR1, skull BR2, and scalp BR3. A portion of the laser light emitted from the light-emitting unit 87 of each sensor unit 86A to 86N of the blood flow measurement unit 81 is reflected by the scalp BR3, while the remaining light passes through the scalp BR3, skull BR2, and cerebrospinal fluid BR1 and travels into the brain BR. The light irradiated onto the head and traveling into the brain propagates in the radial direction (depth direction and radial direction) along an arc-shaped pattern CP as shown by the dashed lines in the figure.

[0095] The propagation of light through the brain becomes longer the farther radially from the base point BP where the laser light is irradiated, and the lower the light transmittance. Therefore, the light reception level (amount of transmitted light) of sensor unit 86B, which is adjacent to sensor unit 86A on the light-emitting side at a predetermined distance, is detected as being strong. The light reception level (amount of transmitted light) of sensor unit 86C, which is located a predetermined distance next to sensor unit 86B, is detected as being weaker than the light reception level of sensor unit 86B. The light-receiving unit 88 of sensor unit 86A on the light-emitting side also receives light from the brain.

[0096] The first control unit 83 uses detection signals corresponding to the light intensities received by the plurality of sensor units 86A to 86N as measurement data, and derives the vascular characteristics of each measurement position by comparing the waveforms of the detection signals with the waveform of the cardiac potential signal from the electrocardiograph 82. Furthermore, by performing mapping processing on these detection results, it is possible to obtain graphic data showing the distribution of arteriosclerosis according to the pulse wave velocity.

[0097] Therefore, it is possible to measure changes in blood flow through the middle cerebral artery and anterior cerebral artery based on the detection signal waveforms of each sensor unit 86A to 86N, and to detect the pulse wave velocity in the brain from the measurement data of the changes in blood flow.

[0098] Thus, the vital sign measuring device 70 is equipped with a brain activity measuring unit 80 that is attached to the subject's head and measures brain waves and cerebral blood flow in the measurement area of ​​the head, and the information detecting unit 75 also adds as nervous system data the state of communication in the central nervous system centered on the head based on the measurement data of brain waves and cerebral blood flow obtained from the brain activity measuring unit 80. In this way, the state of communication in the central nervous system from the subject's brain can also be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transitional state.

[0099] As a result, the state of transmission from the subject's brain to the central nervous system can be quantitatively converted into data and acquired, making it possible to monitor the subject's progress and transitional state.

[0100] (3-3) Method for detecting musculoskeletal system information The human information acquisition unit 62 has a biological signal measurement garment 90 worn by the subject and a joint circumference detection unit (not shown) as additional components of the vital sign measuring device 70. The joint circumference detection unit has an actuator that is driven actively or passively in conjunction with the limb movement of the subject, and detects physical quantities around the joints that accompany the limb movement of the subject based on the output signal from the actuator.

[0101] The physical quantities around the joints include the muscle forces generated in the musculoskeletal system, the range of motion of each joint, the speed of motion and reaction speed, the autonomous control characteristics against disturbances, the parameter identification results of at least one of the frame's moment of inertia, mass and center of gravity, the impedance adjustment results of the joint system including flexor and extensor muscles (viscous characteristics due to friction), electrical physical quantities (command signals), etc.

[0102] The biosignal measurement wearing device 90 worn by the subject is consistent with and uses the same principles as the device described in Japanese Patent Application Publication No. 5409637 filed by the present inventor. Fig. 10(A) shows the biosignal measurement wearing device 90 of this embodiment. This biosignal measurement wearing device 90 is formed to cover the subject's body surface and has a wearing device main body 90A worn by the subject. The inner surface of this wearing device main body 90A (the surface that comes into contact with the subject's body surface when worn) has a biosignal sensor 91 for detecting biopotential signals at at least one position where a biosignal can be measured from the subject's body.

[0103] Biosignals are signals generated by electricity within a subject's body. They are measurable signals that change over time in response to bodily movements and signals from the brain. For example, biosignals refer to signals generated by biological activity, such as nerve potentials, muscle potentials, electroencephalograms, cardiac potentials, and potentials generated by motion artifacts (movement effects), potentials generated by biochemical reactions, and vibrations such as pulse waves generated by heartbeats. The potentials (amplitude, density, time delay) and locations of biosignals are also taken into account when quantifying the data.

[0104] 10A, the biosignal measurement garment 90 of this embodiment has a bilaterally symmetrical structure, so the left half of the figure shows the structure of the back of the left leg, and the right half of the figure shows the structure of the front of the left leg. The garment main body 90A is provided with a group of biosignal sensors 92A to 92F, each consisting of multiple biosignal sensors 91 that detect the biosignals of the subject. The biosignal sensors 91 are arranged at equal intervals along the flow of the muscles in the subject's leg.

[0105] The wearing device main body 90A has multiple biosignal sensors 91 arranged along the gluteus maximus muscle in the area corresponding to the subject's buttocks (biosignal sensor group 92A). Similarly, biosignal sensors 91 are arranged along the biceps femoris, semimembranosus, and semitendinosus muscles in the area corresponding to the back of the subject's thighs (biosignal sensor group 92B). Similarly, biosignal sensors 91 are arranged along the triceps surae muscle in the area corresponding to the calves (biosignal sensor group 92C). Furthermore, biosignal sensors 91 are arranged along the adductor longus and iliopsoas muscles in the area corresponding to the anterior hip joints (biosignal sensor group 92D). Biosignal sensors 91 are arranged along the quadriceps femoris muscle in the area corresponding to the anterior thighs (biosignal sensor group 92E). Biosignal sensors 91 are arranged along the tibialis anterior, soleus, and extensor digitorum longus muscles in the area corresponding to the shins (biosignal sensor group 92F).

[0106] 10(B) is a diagram showing an example of a measurement module connected to a group of biosignal sensors provided on the wearing device main body 90A. Note that all of the groups of biosignal sensors 92A to 92F provided on the wearing device main body 90A are connected to the measurement module 93 in the same way, so the description here will be made using the group of biosignal sensors 92A as an example.

[0107] 10(B) is a diagram schematically illustrating a biosignal sensor group 92A provided on the wearing device main body 90A and a measurement module 93 connected to the biosignal sensor group 92A. The biosignal sensors 91 constituting the biosignal sensor group 92A are provided insulated from one another and connected to the measurement module 93 via conductive wiring. Each of the biosignal sensors 91 is assigned an address.

[0108] The measurement module 93 is connected to a plurality of biosignal sensors 91 that make up the biosignal sensor group 92A, and includes a measurement module controller 94 that selects at least two of these biosignal sensors 91 and acquires biosignals by calculating the difference between the detection signals detected by these selected biosignal sensors 91, a memory 95 that records the acquired biosignals, and a communication unit 96 that transmits to the outside the sequentially acquired biosignals and / or the biosignals recorded in the memory 95. Note that a measurement module 93 is provided for each of the biosignal sensor groups 92A to 92F, and each biosignal sensor 91 is connected to the measurement module 93.

[0109] This measurement module controller 94 has an electronic circuit that can sequentially select at least two biosignal sensors 91 from the multiple connected biosignal sensors 91 in response to a command signal input via the communication unit 96, and acquire the biosignals between these two selected biosignal sensors 91.

[0110] The measurement module controller 94 further includes signal processing means such as a filter that removes or extracts predetermined frequency components from the acquired biosignal, and an amplifier that amplifies the acquired biosignal. The acquired biosignal is output from the measurement module controller 94 to the communication unit 96 and / or the memory 95. When selecting a biosignal sensor 91, the measurement module controller 94 may operate each biosignal sensor 91 sequentially in a predetermined order, or may select a biosignal sensor 91 at an address specified by a designation signal input via the communication unit 96.

[0111] The communication unit 96 includes a thin antenna and a communication circuit connected to the antenna, and transmits, via the antenna to the second control unit 97, measurement information including a signal containing the biosignal output from the measurement module controller 94 and information such as an address indicating the positional information of the biosignal sensor 91 that detected the biosignal, and / or a signal containing the biosignal read from the memory 95 and information such as an address indicating the positional information of the biosignal sensor 91 that detected the biosignal.

[0112] This second control unit 97 generates a designation signal that specifies which biosignal sensor 91 to select from the group of biosignal sensors 92A to 92F provided on the attachment main body 90A, as well as signals such as the start and end of data acquisition, and selects the designated biosignal sensor 91 to measure the biosignal in accordance with the signal.

[0113] As described above, according to the biosignal measurement wearing device 90 of the present invention, the wearing device main body 90A is provided with biosignal sensor groups 92A-92F each consisting of a plurality of biosignal sensors 91, so that simply by wearing the wearing device main body 90A, the plurality of biosignal sensors 91 can be placed at predetermined locations at once, and can be brought into close contact with the skin surface to detect biosignals at each point. In this way, even when monitoring biosignals at multiple points on the subject's body surface, the effort of attaching and removing the biosignal sensors 91 one by one can be eliminated, and biosignals can be easily measured.

[0114] Furthermore, by detecting biosignals using the plurality of biosignal sensors 91 in this manner, biosignals can be measured at a plurality of points within the area where the biosignal sensors 92A to 92F are arranged. Then, by mapping the measurement data at each point according to the address assigned to each biosignal sensor 91, it is possible to measure the distribution of biosignals in the subject's body.

[0115] The second control unit 97 acquires, from the biosignals acquired by the measurement module 93, myoelectric potential signals associated with the subject's muscle activity and nerve transmission signals for operating the subject's musculoskeletal system.

[0116] The information detection unit 75 (FIG. 4) converts the activity state of the musculoskeletal system centered on the joint and / or the transmission state of the peripheral nervous system into quantitative musculoskeletal system data and / or nervous system data based on the physical quantities around the joint described above and either or both of the myoelectric potential signal and the nerve transmission signal obtained from the second control unit 97, and also converts the transmission state of either or both of the central nervous system and the peripheral nervous system centered on a desired location into quantitative nervous system data.

[0117] As a result, by having the subject perform limb movements, the activity state of the musculoskeletal system and the transmission state of the peripheral nervous system can be quantitatively converted into data based on the physical quantities around the joints and the biological signals obtained from the surface parts of the body relative to the joints, and the subject's progress and transition state can be monitored.

[0118] In addition, biological signals can be obtained from areas on the body surface other than those based on the subject's joints, and quantitative data can be obtained not only on the peripheral nervous system around the joints, but also on the transmission state of the central nervous system and peripheral nervous system from desired areas, making it possible to monitor the subject's progress and transitional state.

[0119] For example, as shown in Figure 11, by disposing biosignal sensors 92A-92F all over the subject's body, it is possible to visualize the state transitions of the information transmission pathways in the cranial nervous system. In Figure 11, by dividing the subject's body into predetermined regions (e.g., regions innervated by sensory nerves such as dermatomes), it is possible to grasp the neural connections of the entire body from a bird's-eye view and for each region. Therefore, if it is possible to detect in which region a problem has occurred in neural transmission, it is possible to grasp not only the location of the problem but also the degree of impact on either the central or peripheral side.

[0120] (3-4) Method for detecting the activity state of the physiological system In the human information acquisition unit 62, as shown in FIG. 12, a near-infrared detection unit 100 and a far-infrared detection unit 101 are provided as additional components of the vital sign measuring device 70, and the activity state of the physiological system and nervous system of the subject is detected using a non-contact method.

[0121] That is, the human information acquisition unit 62 uses the near-infrared detection unit 100 and the far-infrared detection unit 101 to irradiate infrared rays mainly onto the face of the subject, thereby simultaneously and non-invasively measuring fluctuations in the subject's pulse rate and skin temperature in daily life.

[0122] The near-infrared detection unit 100 irradiates a region of interest on the face of the subject, including the cheeks, with near-infrared light and receives light reflected from the face to generate a near-infrared image. The far-infrared detection unit 101 irradiates a region of interest on the face of the subject with far-infrared light and detects the skin temperature of the face.

[0123] The third control unit 102 estimates the pulse rate of the subject based on the period of change in the intensity waveform in the region of interest of the reflected light received by the near-infrared detection unit 100. The third control unit 102 extracts areas of the subject's face that are susceptible to and less susceptible to the activity of the autonomic nervous system from the near-infrared image generated by the near-infrared detection unit 100, and calculates the temperature difference in the skin temperature of each area from the detection result of the far-infrared detection unit 101.

[0124] A. Pulse measurement method based on infrared light The light-absorbing substances in biological tissue are water and hemoglobin in the blood. Water has strong absorption properties for infrared light with wavelengths longer than 1350 nm, while hemoglobin has strong absorption properties for visible light with wavelengths shorter than 650 nm. Non-invasive biomedical diagnosis using light often involves irradiating the body with light in the infrared to near-infrared range with wavelengths between 650 nm and 1350 nm, which has high biological penetration, and measuring the reflected and transmitted light, which contains biological information.

[0125] Pulse is a physiological phenomenon in which the volume of blood vessels changes due to the work of blood pressure caused by cardiac output. Changes in blood vessel volume are brought about by changes in blood flow. Changes in blood flow change the amount of light absorbed by hemoglobin in the blood, so the pulse rate can be estimated from the periodic changes in transmitted and reflected light when light is irradiated onto a living body.

[0126] In this invention, the subject is not required to wear a measuring device and measurements can be performed day or night. Therefore, the pulse rate is estimated by measuring the periodic change in reflectance of near-infrared light irradiated onto the living body using a near-infrared camera. Since near-infrared light is invisible, measurements can be made without burdening the subject, even at night. In order to estimate stress state from the pulse rate, it is necessary to measure the pulse rate, which fluctuates minutely due to activity of the autonomic nervous system.

[0127] The third control unit 102 implements an algorithm for extracting the waveform period required for calculating the pulse rate from the waveform data obtained from the near-infrared detection unit 100 .

[0128] The near-infrared detection unit 100 measures the intensity of light reflected from biological tissue when irradiated with near-infrared light. This near-infrared detection unit 100 measures when the subject is at rest, but minute body movements and facial movements due to breathing are superimposed on the pulse waveform as motion artifacts that represent changes in the amount of reflected light.

[0129] To remove these motion artifacts, a digital filter is first applied to the pulse rate frequency band. The resting pulse rate range is set to 35 to 180 bpm, and motion artifacts may also be superimposed on the measured waveform in the 0.5 to 3.0 Hz band, but the amplitude of the waveform superimposed on the pulse waveform is significantly large.

[0130] Therefore, if the variance value of the waveform data within a certain period of time exceeds a certain threshold, the third control unit 102 recognizes that the subject is moving and removes the data during that period as unreliable. The waveform after the bandpass filter has a bimodal feature due to arterial blood flow. To facilitate feature point extraction, a moving average filter is applied to reduce the two-peak waveform to a single peak.

[0131] 13A and 13B show the waveforms before and after applying a band-pass filter and a moving average filter to the measured waveform, respectively. It can be seen that the motion artifacts superimposed on the pulse waveform have been removed by filtering.

[0132] Next, generally, there are two known methods for calculating the pulse rate from the light intensity waveform measured by a near-infrared camera.

[0133] In the first method, the facial area was set as the region of interest (ROI), and the intensity data of reflected light from the cheek area over a 30-second period, obtained by measuring with an RGB camera, was used to obtain a frequency spectrum using a spectrum analyzer. The frequency that was thought to represent the pulse wave component was then compared with the frequency determined from the pulse rate indicated by a pulse oximeter. However, the noise caused by body movement was significantly large in the pulse waveform. When attempting to calculate the pulse rate using this type of frequency analysis, it was necessary to significantly restrict the subject's movements.

[0134] On the other hand, the second method calculates the time required for one pulse beat from the period of the waveform feature points detected from the time-series data of the measured waveform, and then calculates the pulse rate by dividing that time by 60. Compared to methods that calculate the pulse rate from frequency analysis, which requires continuous measurement data for a certain period of time, methods that calculate the pulse rate from the period of the measured feature points can remove motion artifacts by removing feature point periods that are falsely detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that required by frequency analysis, and the pulse rate can be calculated from several beats, making it possible to calculate the pulse rate over shorter intervals.

[0135] However, since the pulse rate of a healthy person at rest is 60 to 80 bpm, the period of the characteristic points obtained in a 5-second measurement is 5 to 7 times, which is a problem in that the amount of data obtained is insufficient to calculate a more accurate pulse rate. Also, as shown in Figure 12(A), the waveform of the intensity change has two peaks synchronized with the arterial blood flow, which can cause erroneous detection of the period of the characteristic points, resulting in an error in the calculated pulse rate.

[0136] For daily measurements, the body part to be measured needs to be exposed day and night. Therefore, in the vital sign measuring device 70 according to the present invention, the ROI is set on the cheek of a person. The location of the set ROI is shown in Figure 14(A). The average of the intensity values ​​measured at the pixels within the set ROI was taken as the measurement value, and the time change of the measurement value was recorded as pulse waveform data. The period of the characteristic points of the waveform data was determined by the interval t between the top ends of the waveform. tp Not only that, but also the bottom interval t bp is added as another feature point, the number of feature points that can be used as reference data for calculating the pulse rate is increased.

[0137] Next, in order to extract the characteristic point period necessary for calculating the pulse rate, the detected t tp , t bp Among them, there are falsely detected t tp , t bp The algorithm for removing S is shown below in (a) to (f). ０ Let's say.

[0138] (a) t tp , t bp For each of the peaks, a certain peak interval t i is the previous peak interval t i-1 If there is a fluctuation of 30% or more, the peak interval t i Set S ０ The set of selected peak intervals S １ is expressed by the following equation (1).

[0139] (b) Set S ０ Standard deviation σ ０ Calculate 2σ ０ The peak interval t varies in the range i Set S １ The set of selected peak intervals S ２ is expressed by the following equation (2).

[0140] (c) Set S ２ Standard deviation σ ２ Calculate and σ ２ The peak interval t varies in the range iSet S ２ The set of selected peak intervals S ３ is expressed by the following equation (3).

[0141] (d) Set S ３ Standard deviation σ ３ Calculate and σ ３ The peak interval t varies in the range i S ３ The set of selected peak intervals S ４ is expressed by the following equation (4).

[0142] (e) Set S ４ The remaining peak interval blood pressure is converted to pulse rate, and a histogram is created with pulse rate intervals of 5 bpm. In this case, the pulse rate width for each section of the histogram is the same, but the width of the peak intervals corresponding to that pulse rate varies. Therefore, a weighted histogram is created using the following equation (5).

[0143] Here, t 30-35 is the time interval between peaks of pulse rate 30 [bpm] and 35 [bpm], t i-i+5 is the time interval between peaks of pulse rate i [bpm] to i+5 [bpm], h i-i+5 is the frequency of the peak interval between pulse rates i [bpm] and i+5 [bpm] before correction, and h' i-i+5 represents the correction value of the frequency of the peak intervals from pulse rate i [bpm] to i+5 [bpm].

[0144] (f) Regarding the values ​​of the weighted histogram, a window of width 20 [bpm] is set so that the sum of the frequencies of the four adjacent sections is maximized, and the peak interval t outside that range is i Remove.

[0145] According to the above algorithm, the average value of the extracted peak intervals is set as the peak width t within the measurement time, and the pulse rate is calculated by dividing 60 seconds by the peak interval t.

[0146] In this way, the third control unit 102 sets feature points at the upper and lower ends of the intensity waveform of the reflected light obtained from the near-infrared detection unit 100, and estimates the pulse rate based on the peak intervals between the upper and lower ends, which are the periods of the feature points. The third control unit 102 also calculates the standard deviation of the multiple peak intervals and removes peak intervals that do not represent a pulse period based on the standard deviation. The third control unit 102 then creates a weighted histogram for a predetermined time unit from the pulse rate converted based on the remaining peak intervals after the removal, and corrects the frequency of the peak intervals based on the histogram.

[0147] Thus, the third control unit 102 can calculate an accurate pulse rate from the measured waveform of pulse waveform data for a relatively short period based on the pixel group within the set ROI.

[0148] B. Skin temperature measurement using far-infrared detector Heat generated inside the body is carried to the body surface by conduction and convection. However, since biological tissue itself has poor thermal conductivity and acts as a heat insulator, most of the heat transport to the skin is thought to be due to skin blood flow. Skin blood flow changes depending on the activity of the autonomic nervous system, which is centered on the contraction and expansion of blood vessels by the sympathetic and parasympathetic nervous systems.

[0149] Previous studies have attempted to estimate the activity of the autonomic nervous system by measuring changes in skin temperature. However, because the heat distribution on the face is significantly affected by the subject's hairstyle and whether or not they are wearing glasses, the human information acquisition unit of the present invention uses near-infrared images to set the measurement site appropriate for estimating the activity of the autonomic nervous system.

[0150] In order to estimate stress levels from changes in skin temperature, it is necessary to set the ROI for measurement to a region of the face where temperature changes due to stress are most noticeable. Among the facial regions, the nose in particular is home to a high concentration of arteriovenous anastomoses (AVAs), which are susceptible to the activity of the autonomic nervous system, making it suitable for measuring skin temperature to estimate stress levels.

[0151] Furthermore, because skin temperature is affected by the ambient temperature, changes in skin temperature must be recorded as relative temperatures. In this case, the ROI used to measure the nasal skin temperature change as a reference must be set in a region that is less susceptible to the activity of the autonomic nervous system. The forehead is an example of a region of the face where AVAs are less dense and less susceptible to the activity of the autonomic nervous system.

[0152] From the above, the third control unit 102 according to the present invention records the change in skin temperature as the temperature difference between the region of interest ROI_n in the nose and the region of interest ROI_fh in the forehead. The locations of the ROIs are shown in FIG. 14(B). The formula for calculating the temperature change is shown in Equation (6).

[0153] Here, T n is the average skin temperature measured at all pixels in ROI_n, T fh is the average skin temperature measured at all pixels in ROI_fh, T r is the relative temperature of the forehead and nose. When setting each ROI, the coordinates of the nose and forehead in the near-infrared image are converted to coordinates on the far-infrared image.

[0154] Specifically, when converting coordinates from a near-infrared (NIR) image to a far-infrared (FIR) image, the resolution and angle of view differ between near-infrared and far-infrared cameras, so the coordinate information of the near-infrared image is made to correspond to the pixels of the far-infrared image.

[0155] That is, the x-coordinate on the far-infrared image is X fir is the horizontal angle of view of the near-infrared camera, θ nir_h , the horizontal angle of view of the far-infrared camera is θ fir_h Then, it is expressed as the following equation (7).

[0156] The y coordinate on the far-infrared image, Yfir, is the vertical angle of view of the near-infrared camera, θ nir_v , the vertical angle of view of the far-infrared camera is θ fir_v Then, it is expressed as the following equation (8): where d represents the difference between the optical axes, and L represents the distance to the object.

[0157] Thus, the third control unit 102 can accurately calculate the relative temperature difference between the skin temperatures of the subject's forehead and nose based on the image information of the far-infrared detection unit 101 following the automatic detection result of the measurement target area using the image information of the near-infrared detection unit 100.

[0158] C. Respiration measurement based on skin temperature Respiration is related to the function of the autonomic nervous system and the immune system, and is used to diagnose sleep disorders. There are two types of breathing: nasal breathing (Figure 15(A)) and mouth breathing (Figure 15(B)), with nasal breathing being the preferred form of breathing. When breathing through the mouth, the inhaled air does not pass through the cilia and mucous membranes of the nasal cavity. Mouth breathing can also cause various diseases, including sleep apnea syndrome (SAS).

[0159] As shown in Figures 16 (A) and (B), the nostrils and oral cavity are warmed by exhaled air (35-36°C) and cooled by inhaled air (23-28°C). Therefore, regions of interest (ROIs) are set at the subject's nostrils and oral cavity, and the temperature changes at the nostrils and oral cavity due to breathing are measured using a far-infrared detector (far-infrared camera). Because the surface temperature of the skin is affected by the ambient air temperature, it is measured as the relative temperature between the nostrils and oral cavity.

[0160] In practice, because it is difficult to automatically detect the measurement target area using a skin temperature distribution map, the measurement target areas (nostrils and oral cavity) are automatically detected using image information from the near-infrared detector (camera) as described above, and the region of interest (ROI) is set. The nostril temperature is relatively high during nasal breathing, while the oral cavity temperature is relatively low during mouth breathing.

[0161] We measured the subject's nasal breathing and mouth breathing for 30 seconds each, then took a one-minute break and repeated the same respiration measurement four times. As a result, as shown in Figure 17(A), we confirmed that the temperature of the nostrils decreased due to inhalation during nasal breathing. Furthermore, as shown in Figure 17(B), we confirmed that the temperature of the oral cavity also decreased due to mouth breathing.

[0162] Furthermore, as shown in Fig. 18(A), the opening and closing of the mouth due to a change in breathing method from nose breathing to mouth breathing was also confirmed based on the distance between the upper and lower lips. When the graphs in Fig. 17(A), (B) and Fig. 17(A) are superimposed, it is shown as in Fig. 18(B).

[0163] The method for extracting the timing of inhalation is to first use a moving average filter to remove high frequency noise from the measurement results of the relative temperatures of the nostrils and oral cavity, and then perform first differentiation to extract the timing of the temperature drop due to exhalation.

[0164] Furthermore, if the nostril temperature and oral temperature drop at the same time, the intra-alveolar pressure becomes negative relative to atmospheric pressure, causing air to flow into the nasal passages, making it possible to distinguish this as mouth breathing.

[0165] In this way, it is possible to calculate the respiration rate and distinguish the respiration method using a non-contact measurement method. It is possible to measure nasal breathing and mouth breathing simultaneously, and by executing an algorithm that takes into account the characteristics of temperature changes during each breath, it is possible to accurately calculate the respiration rate and distinguish the respiration method.

[0166] D. Oxygen saturation using two types of near-infrared wavelengths A pulse oximeter is a medical device that uses the light absorption characteristics of hemoglobin in the blood. A pulse oximeter is attached to the fingertip and measures blood oxygen saturation and pulse rate.

[0167] In this invention, we focused on the difference in absorption characteristics between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb), since blood oxygen saturation varies depending on respiratory rate, the degree of hemoglobin-oxygen binding, and cardiac output.

[0168] In conventional measurement methods, infrared light (wavelength around 660 [nm]) and near-infrared light (wavelength around 900 [nm]) are irradiated onto the fingertip from a light-emitting element, and the blood oxygen saturation is estimated from the ratio of the light intensity of each transmitted light measured by a light-receiving element, while the pulse rate is also estimated using the periodic change in the light intensity of the transmitted light.Infrared light, which is visible light, has been used because there is a large difference in the light absorption coefficient between oxygenated hemoglobin (HbO2) and reduced hemoglobin (Hb).

[0169] However, in the present invention, non-contact measurement is performed using only near-infrared (NIR) light, which is invisible light (Fig. 19). As a result, by alternately turning on and off light-emitting elements that irradiate two types of near-infrared light, 800 (or 760) nm and 900 nm, and capturing images of the region of interest (ROI) illuminated by each light-emitting element with a single near-infrared camera (near-infrared detection unit 100), it is possible to measure the blood oxygen saturation of the subject centered on the ROI without contact and even in a dark environment.

[0170] (3-5) Method for detecting the activity state of the nervous system The autonomic nervous system includes the sympathetic nervous system, which functions when the living body is in a state of tension or activity, and the parasympathetic nervous system, which functions when the living body is at rest. When the sympathetic nervous system is dominant, blood pressure and pulse rate increase, while when the parasympathetic nervous system is dominant, blood pressure and pulse rate decrease, so there is a high correlation between autonomic nervous function and cardiac function.

[0171] As a method for measuring autonomic nervous function, there is a method for measuring cardiac sympathetic nervous function using heart rate fluctuations. Specifically, there is the CVR-R (Coefficient of Variation of R-R intervals) method, which uses the R-R intervals of an electrocardiogram waveform to calculate and evaluate the coefficient of variation. Secondly, there is a method in which the fluctuation power ratio of the frequency components (high frequency components and low frequency components) of heart rate fluctuations is used as an index of sympathetic nervous activity.

[0172] There is also a method for measuring autonomic nervous function by measuring the function of the sympathetic nerves in the vascular system using pulse waves, particularly a method for calculating the magnitude of amplitude fluctuation in the photoplethysmogram (PPG) waveform as an evaluation value of autonomic nervous function.

[0173] In the present invention, the third control unit 102 can contribute to the evaluation of the subject's autonomic nervous function (presence or absence of disturbance, etc.) based on all or a combination of the subject's pulse rate detected using the near-infrared detection unit 100 and far-infrared detection unit 101 that constitute the human information acquisition unit 62, the relative temperature difference between the skin temperatures of the subject's forehead and nose, the subject's respiratory rate and breathing method, and the subject's oxygen saturation.

[0174] (3-6) Method for Detecting Psychological Information In the person information acquisition unit 62, the information detection unit 75 detects psychological information by analyzing the facial expression and voice of the subject using the face imaging camera 71 and the sound collection microphone 72.

[0175] Specifically, the information detection unit 75 executes a face analysis algorithm to detect the facial expressions of the subject (including not only expressions such as sadness, joy, anger, empathy, and frivolity, but also flushed skin, goosebumps, frequency of eyelid opening and closing, tears, etc.), and at the same time executes a voice analysis algorithm to detect the tone and pitch of the subject's voice (tone of voice, elation, etc.).

[0176] This facial analysis algorithm is designed to approximate human interpretation (experience, preconceptions, and expectations) and is derived from the accumulated results of how emotions and psychology are reflected in facial expressions based on the state and changes of the human face.

[0177] Similarly, voice analysis algorithms are designed to approximate human interpretation (experience, biases, and expectations) and are derived from the accumulated results of how emotions and psychology are reflected in the voice based on human tone and pitch.

[0178] Using this voice analysis algorithm, it is possible to estimate changes in the subject's psychological state through the subject's everyday communication and narrative analysis, based on the keywords used in the subject's speech, the intonation of the speech, the volume of the voice, the structure of the narrative, the continuity / discontinuity of communication, and other accumulated data.

[0179] In addition, the information detection unit 75 can also estimate the psychological state of the subject based on the subject's personal information (heart rate, electrocardiogram, pulse, blood pressure, body temperature, skin blood flow distribution, skin temperature distribution, sweating, odorous substances, etc.) detected by the vital sign measuring device 70 described above, and the subject's environmental information detected by the peripheral measuring device 74 described below.

[0180] In this way, the information detection unit 75 can comprehensively estimate and grasp a person's psychological information from both the person information obtained by sensing and the person information obtained by communication analysis and narrative analysis.

[0181] (3-7) Method for detecting the activity state of the motor system (movement information and behavior information) In the human information acquisition unit 62 (Figure 4), the information detection unit 75 detects the movement data recognized by the motion capture device 73 as the activity state of the motor system of the subject.

[0182] 20, the motion capture device 73 is placed in an indoor facility such as the subject's home, and is composed of an RGB-D sensor 111 and an IMU (Inertial Measurement Unit) sensor 112 connected to the control unit 60. The IMU sensor 112 is composed of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a geomagnetic sensor, and is capable of measuring triaxial acceleration, angular velocity, and geomagnetism.

[0183] In addition to its RGB color camera function, the RGB-D sensor 111 also has a depth sensor that can measure the distance to an object as seen from the camera, enabling it to perform 3D scanning of the object. For example, if the RealSense (a trademark of Microsoft Corporation) LiDAR camera L515 is used as the RGB-D sensor 111, the depth sensor is made up of a LiDAR sensor that measures the time it takes for laser light to irradiate the object, hit it, and bounce back, thereby measuring the distance and direction to the object.

[0184] In practice, the RGB-D sensor 111 is positioned so as to capture an image of the subject's entire body based on the subject's standing position in the indoor equipment, and sequentially acquires an RGB image and a depth image as the imaging results.

[0185] The control unit 60 has a fourth control unit 113 under the control of the above-mentioned general control unit 50, and a data storage unit 114 in which various data is stored in a database that can be read and written in response to instructions from the fourth control unit 113.

[0186] The fourth control unit 113 is composed of an image coordinate setting unit 120, a coordinate group extraction unit 121, a skeletal information acquisition unit 122, and a behavior recognition unit 123. The image coordinate setting unit 120 estimates the posture of the entire body of the subject based on the RGB images sequentially acquired from the RGB-D sensor 111, and sets a plurality of skeletal representative points attached to the entire body of the subject as image coordinates.

[0187] Specifically, the image coordinate setting unit 120 sets a group of image coordinates in a two-dimensional coordinate system in which the coordinate origin is either the left or right shoulder of the subject, the direction of the line connecting the left and right shoulders is the X-axis, and the direction perpendicular to the X-axis is the Z-axis, based on the coordinate origin. That is, the coordinate origin is moved to the right or left shoulder, and the XZ plane is rotated so that it coincides with the line connecting the left and right shoulders. As a result, it becomes possible to recognize movements with increased robustness regardless of the position of the RGB-D sensor 111.

[0188] The coordinate group extraction unit 121 sequentially extracts image coordinate groups related to the subject's behavior based on the transition of the subject's posture from the image coordinate groups set by the image coordinate setting unit 120. In practice, the coordinate group extraction unit 121 needs to extract key points of the entire body from the behavior, taking into account changes in the subject's clothing and surrounding environment.

[0189] Therefore, the coordinate group extraction unit 121 employs a lightweight OpenPose architecture (a system that estimates a person's skeleton using deep learning) that uses a trained model to estimate the posture of the entire body and obtain image coordinates of the skeletal representative points of the fingers of both hands. For example, for the subject's upper body, the coordinate group extraction unit 121 sequentially extracts a total of 60 skeletal representative points as image coordinate groups: 18 points on the torso of the subject's upper body and 42 points on the fingers of both hands (21 points on each finger). Furthermore, by using only stages 1 and 2 of the six stages of the OpenPose architecture, the amount of calculation can be reduced by 32.4%.

[0190] The skeletal information acquisition unit 122 acquires three-dimensional skeletal information that is continuous over time, centered on the fingers of both hands of the subject, by time-synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit 121 with the depth images sequentially acquired from the RGB-D sensor 111.

[0191] Specifically, the skeletal information acquisition unit 122 constructs a three-dimensional coordinate system by combining the depth direction in the depth images sequentially acquired from the RGB-D sensor 111 as the Y-axis with the two-dimensional coordinate system of the image coordinate groups sequentially extracted by the coordinate group extraction unit 121, and acquires three-dimensional skeletal information.

[0192] The transformation equation from the image coordinate system, which is a two-dimensional coordinate system, to the world coordinate system, which is a three-dimensional coordinate system, is expressed as the following equation (1). Here, X Ｗ = ｔ [X Ｗ, Y Ｗ, Z Ｗ ] indicates a coordinate in the world coordinate system, and x Ｉ = ｔ [x Ｉ ,y Ｉ ] indicates coordinates in the image coordinate system. K (3 × 3 matrix) is an intrinsic parameter of the image sensor, R (3 × 3 matrix) is a rotation matrix that performs coordinate transformation to the image coordinate system, t (3 × 1 vector) is a translation vector, Z Ｃ represents depth information corresponding to two-dimensional coordinates in the image coordinate system.

[0193] K, R, and t were calculated by camera calibration using Zhang's method ("A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.). A calibration board with circular dots printed in a grid pattern was used for the calibration to calculate K. The central coordinates of each dot on the calibration board were calibrated to 1 / 1,000.

[0194] As a result, as shown in Figure 21, the motion capture device 73 clarifies the relationship between the position in three-dimensional space and the position of the captured image for the entire body of the subject, making it possible to acquire three-dimensional skeletal information while maintaining high robustness regardless of the imaging position taken by the RGB-D sensor 111.

[0195] The behavior recognition unit 123 recognizes behaviors, which are a series of multiple local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 122.

[0196] The behavior recognition unit 123 uses the behavior recognition pattern set for each local behavior as training data, and sequentially recognizes the corresponding local behavior from the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 122, while referring to the hand-washing behavior recognition model constructed by deep learning.

[0197] Specifically, in order to recognize multiple local actions, the action recognition unit 123 applies an action recognition model consisting of three modules, namely, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as well as a fully connected layer, as shown in FIG. 22 .

[0198] In the behavior recognition model, it is possible to input skeletal information from the current frame and the frame two frames before it and calculate the likelihood of each of multiple types of local motions related to behavior.

[0199] Furthermore, the behavior recognition unit 123 performs post-processing on the most frequently occurring behavior among the behaviors observed in 15 frames (the current frame and the 14 frames preceding it) as the current local behavior as a recognition result. In this way, the behavior recognition unit 123 trained the behavior recognition model through supervised learning using a unique dataset. For optimization, a cross-entropy loss function and Adam (Adaptive Movement Estimation) with a learning rate of 0.001 were used.

[0200] As a result, the motion capture device 73 can significantly improve the accuracy and speed of recognizing local movements based on three-dimensional skeletal information.

[0201] (4) Control adjustment according to the subject's state of tension Furthermore, in the functional improvement support device 1, the general control unit 50 detects the shift in the subject's center of gravity position and the load distribution on specific parts at the contact points with the subject (seat 40 and pair of armrests 30A, 30B) based on the measurement results of the normal force at the contact points with the subject by the normal force measuring unit 61 (Figure 3), and determines the subject's state of tension and relaxation when transitioning to a sitting position, a standing position, or both positions.

[0202] That is, the integrated control unit 50 detects the transition of the subject's center of gravity and the distribution of load on specific parts of the subject's contact areas (load concentration parts of the seat 40 and the pair of armrests 30A, 30B), determines whether the subject's current posture is a sitting posture, a standing posture, or a posture in transition to either of these postures, and at the same time determines how much load the subject is applying to the specific parts of the contact areas, thereby determining the subject's state of tension or relaxation.

[0203] The integrated control unit 50 then adjusts the control content of the state-holding variable mechanism unit according to the subject's state of strain and relaxation, which is the result of this determination.

[0204] As a result, the functional improvement support device 1 can more safely guide the subject from a sitting position to a standing position and from a standing position to a sitting position by determining whether the subject is tense or relaxed when their posture changes and adjusting the control over the mechanism switching, deformation state and power application state of the state holding variable mechanism unit 2.

[0205] (5) Environmental Information Detection Method Furthermore, in the function improvement assistance device 1, the environmental information acquisition unit 130 ( FIG. 3 ) acquires environmental information representing the environment surrounding the subject and the condition of the floor on which the state-holding variable mechanism unit 2 moves. Specifically, the environmental information acquisition unit 130 has a thermometer, a hygrometer, a barometer, an audible and ultrasonic ambient sound collector 74, etc., and acquires environmental information surrounding the subject. This environmental information mainly includes the temperature, humidity, and ambient sound around the subject.

[0206] When controlling the mechanism switching, deformation state, and power application state of the state-holding variable mechanism unit 2, the integrated control unit 50 prioritizes ensuring the safety of the subject based on the environmental information obtained from the environmental information acquisition unit 130. As a result, the function improvement support device 1 can significantly reduce the risk of falling while ensuring the safety of the subject's running and walking, while recognizing the surrounding floor surface condition and bodily sensations.

[0207] (6) Gait state synchronization control during walking assistance function Furthermore, in the function improvement support device 1, the general control unit 50 uses the above-mentioned motion capture device 73 (Figure 20) to control the fourth control unit 113, and sequentially acquires RGB images and depth images of the subject's lower body captured by the RGB-D sensor (imaging unit) 111, thereby detecting the subject's gait state in real time.

[0208] That is, when the posture holding variable mechanism unit 2 is transformed into a second mechanism corresponding to the wheelchair function, the image coordinate setting unit 120 estimates the posture of the subject's lower body based on the RGB images sequentially acquired from the RGB-D sensor (imaging unit) 111, and sets multiple skeletal representative points attached to the subject's lower body as image coordinates.

[0209] Next, the coordinate group extraction unit 121 sequentially extracts image coordinate groups related to the gait state of the subject based on the transition state of the subject's posture from the image coordinate groups set by the image coordinate setting unit 120. The skeletal information acquisition unit 122 acquires temporally continuous three-dimensional skeletal information centered on the lower body of the subject by temporally synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit 121 with the depth images sequentially acquired from the RGB-D sensor (imaging unit) 111.

[0210] Then, the behavior recognition unit (gait state detection unit) 123 sequentially recognizes multiple local movements from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit 122, and detects the gait state (walking posture and movement pattern of both lower limbs) which is the connection between the multiple local movements.

[0211] The integrated control unit 50 causes the front wheel drive units 43A, 43B and the rear wheel drive units 44A, 44B (wheel drive units) to generate power synchronized with the gait state detected by the behavior recognition unit (gait state detection unit) 123, thereby independently driving the pair of front wheel units 13A, 13B and the pair of rear wheel units 14A, 14B.

[0212] As a result, the function improvement assistance device 1 can detect the gait state of the subject in real time, and can assist the walking movement in synchronization with the gait state.

[0213] (7) Hybrid control during wheelchair function Furthermore, in the functional improvement support device 1, the general control unit 50 is configured to allow the subject to drive while reflecting the operation content according to the subject's own intention for movement, based on the subject's musculoskeletal system information acquired by the human information acquisition unit 62, without the subject having to operate using the controllers 46A, 46B.

[0214] In the functional improvement support device 1, as shown in FIG. 23, under the control of the overall control unit 50, when the posture holding variable mechanism unit 2 is transformed into a first mechanism corresponding to the wheelchair function, the action intention recognition unit 140 recognizes the subject's action intention for a specific operation content among the operation contents of the wheelchair function based on the bio-signal obtained from the bio-signal measurement attachment (bio-signal detection unit) 90.

[0215] In fact, the operational intention recognition unit 140 calculates, for each of a plurality of predetermined operational intentions, the likelihood (certainty) that the signal characteristics of the biological signal match the operational intention, based on the biological signal obtained from the biological signal measurement garment (biological signal detection unit) 90.

[0216] Specifically, the motor intention recognition unit 140 obtains a feature vector representing the signal characteristics of the biosignal using a known vector analysis method. Since a predetermined motor intention indicates a predetermined region in the vector space, the motor intention recognition unit 140 calculates the distances between a position corresponding to the signal characteristics of the biosignal and a plurality of predetermined motor intentions in the vector space.

[0217] Since a position in a vector space is determined by vectorizing the signal characteristics of the biosignal, the motor intention recognition unit 140 calculates the distance between this position and the center of gravity of the region of each motor intention.

[0218] The action intention recognition unit 140 selects a predetermined number of positions in the vector space that are close to the signal characteristics of the biosignal, and then calculates a confidence level for the selected number of action intentions, which indicates the likelihood that the signal characteristics of the biosignal match the action intention, based on the distance in the vector space.

[0219] The operational intention recognition unit 140 calculates and normalizes the confidence levels so that the sum (total) of a predetermined number of confidence levels is 1. Furthermore, the distribution shape of the confidence levels may be normalized so that the kurtosis (third moment) and skewness (fourth moment) are predetermined values.

[0220] Next, under the control of the overall control unit 50, the operation content understanding unit 141 understands the operation content, which is the operation intention of the subject recognized by the operation intention recognition unit 140. Specifically, the operation content understanding unit 141 stores data as a comparison table showing the correspondence between a specific operation intention and the operation content corresponding to that operation intention, and selects the operation content corresponding to the operation intention of the subject recognized by the operation intention recognition unit 140 based on the comparison table.

[0221] The integrated control unit 50 then causes the front wheel drive units 43A, 43B and rear wheel drive units 44A, 44B (wheel drive units) to generate power according to the operation content understood by the operation content understanding unit 141.

[0222] As a result, with the function improvement support device 1, when the wheelchair is in function, the subject can move while the operation content according to the subject's own intentions is reflected without the subject having to operate the wheelchair himself.

[0223] (8) Movement control using gaze target Furthermore, the function improvement support device 1 further includes a gaze detection unit 142 (Figure 23) that captures an image of the space corresponding to the subject's field of vision and detects the position of the subject's gaze by image-recognizing the movement of the subject's eyeballs within the captured image range.

[0224] In addition to the above-mentioned method of recognizing action intentions, the action intention recognition unit 140 recognizes, based on the gaze position detected by the gaze detection unit 142, that if the gaze position continues for a predetermined period of time or more, the subject's action intention is to use the gaze position as a moving target.

[0225] As a result, the functional improvement support device 1 recognizes the subject's intention regarding the operation content by taking into account not only the subject's biological signals but also their gaze position, making it possible for the subject to drive while reflecting the operation content in accordance with their own intention to operate.

[0226] (9) Voluntary operation by voice Furthermore, the function improvement support device 1 further includes an utterance content recognition unit 143 (Figure 23) that recognizes the content of the utterance based on the subject's voice collected by the sound collection microphone (sound collection unit) 72.

[0227] In addition to the above-described method of recognizing an intention to operate, the operation intention recognition unit 140 recognizes the operation content corresponding to the utterance content recognized by the utterance content recognition unit 143 as the intention of the subject's operation.

[0228] As a result, the functional improvement support device 1 recognizes the subject's intention regarding the operation content by taking into account not only the subject's biological signals but also the subject's own speech, making it possible for the subject to drive while reflecting the operation content that is more in line with the subject's intention.

[0229] (10) Other Embodiments As described above, in this embodiment, the configuration of the posture holding and variable mechanism unit 2 of the function improvement support device 1 is described as being configured as shown in Figures 1(A) and (B). However, the present invention is not limited to this, and it is also possible to apply, for example, a posture holding and variable mechanism unit 150 configured as shown in Figures 24(A) to (C).

[0230] 24, the attitude maintenance variable mechanism 150 differs from the attitude maintenance variable mechanism 2 shown in Fig. 1 in that the frame structure 160 is configured to be roughly U-shaped overall together with the pair of rear wheels 14A, 14B so that the pair of front wheels 13A, 13B are bridged by the footrest 11, and is configured to include a running base 160A and a pair of left and right support frames 160B, 160C supported so as to be movable in the front-to-rear direction from the running base 160A. The pair of support frames 160B, 160C are configured to be able to tilt in the front-to-rear direction relative to the running base 160A and to be able to expand and contract in the longitudinal direction.

[0231] In this way, according to the functional improvement support device 1, the posture holding variable mechanism unit 150 is capable of transforming into either a first mechanism (Figure 24(A)) corresponding to a wheelchair function that holds the subject in a seated position and moves in response to operation by the subject, or a second mechanism (Figure 24(B)) corresponding to a walking assistance function that holds the subject in a standing position and applies an assist force in response to the subject's walking movement, while both mechanisms are capable of transforming into each other while holding the subject.

[0232] As described above, in this embodiment, the method for recognizing the subject's intention of movement by the intention recognition unit 140 (FIG. 23) in the function improvement support device 1 is described as applying a task-oriented method for calculating a degree of certainty corresponding to the subject's intention of movement. However, the present invention is not limited to this, and a task-based method for learning to understand the subject's intention of movement may also be applied.

[0233] That is, the action intention recognition unit 140 determines the action intention from the biosignal obtained by the biosignal measurement garment (biosignal detection unit) 90, while referring to the recognition model constructed by deep learning, using as training data the association data in which the correspondence between the signal characteristics of the biosignal and the action intention is set.

[0234] Specifically, to recognize each of the multiple motion intentions, the motion intention recognition unit 140 applies a recognition model consisting of three modules, a convolutional neural network (CNN) layer, a batch normalization layer (Batch Norm), and an activation function layer (tanh function), as well as a fully connected layer. Using this recognition model, the motion intention recognition unit 140 evaluates, based on the current state of the subject and the determined motion intention, whether the motion intention is the desired selection result as a degree of likelihood, and corrects the association data (eliminates erroneous recognition) based on the evaluation result, thereby learning the recognition model through supervised learning, thereby making it possible to recognize the motion intention of the subject with a relatively high degree of accuracy.

[0235] 1...Function improvement support device, 2, 150...Posture holding variable mechanism section, 10, 160...Frame structure, 11...Footrest section, 12A...Left side frame, 12B...Right side frame, 13A, 13B...Front wheel section, 14A, 14B...Rear wheel section, 20A, 20B...Front arm section, 21A, 21B...Rear arm section, 30A, 30B...Armrest section, 31A, 31B...Support arm section, 32A, 32B...Support drive section, 33A, 33B...Parallel holding drive section, 40...Seat section, 40A... Left seat half, 40B...right seat half, 41A, 41B...fixed holding portion, 42A, 42B...opening / closing drive portion, 43A, 43B...front wheel drive portion, 44A, 44B...rear wheel drive portion, 45A, 45B...posture deformation drive portion, 46A, 46B...controller, 47A, 47B...curved frame, 48A, 48B...backrest drive portion, 50...general control portion, 60...control unit, 61...normal force measurement portion, 62...person information acquisition portion, 63...driving battery, 70...vital measurement Equipment, 71...Facial imaging camera, 72...Sound collection microphone, 73...Motion capture equipment, 74...Peripheral measurement equipment, 75...Information detection equipment, 80...Brain activity measurement unit, 81...Blood flow measurement unit, 82...Electrocardiograph, 83...First control unit, 84...Wireless communication device, 90...Biological signal measurement wearing tool, 91...Biological signal sensor, 92A to 92F...Biological signal sensor group, 93...Measurement module, 94...Measurement module controller, 95...Memory, 96...Communication unit, 97...Second control unit, 100...Near-infrared detection unit, 101...Far-infrared detection unit, 102...Third control unit, 111...RGB-D sensor, 112...IMU sensor, 113...Fourth control unit, 120...Image coordinate setting unit, 121...Coordinate group extraction unit, 122...Skeleton information acquisition unit, 123...Action recognition unit, 130...Environmental information acquisition unit, 140...Movement intention recognition unit, 141...Operation content understanding unit, 142...Gaze detection unit, 143...Speech content recognition unit, 160A...Running base unit, 160B, 160C...Support frame.

Claims

1. While holding the subject in a seated position, it has a wheelchair function that travels in response to an operation by the subject, and while holding the subject in a standing position, it has a walking assistance function that applies an assisting force in response to the walking motion of the subject. It also has a posture holding variable mechanism unit that can be deformed while holding the subject, either from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism. It further includes a human information acquisition unit that acquires human information including the subject's brain / nervous system information, muscle / skeletal system information, physiological information, psychological information, motion information, and behavior information. A current state function discrimination unit that discriminates the current state of the subject's physical function and cognitive function based on the human information obtained by the human information acquisition unit. A vertical resistance measurement unit provided in the state holding variable mechanism unit that measures the vertical resistance at the contact part with the subject in real time. And a control unit that controls the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit based on the discrimination result by the current state function discrimination unit and the measurement result by the vertical resistance measurement unit. A function improvement support device characterized by comprising these components.

2. The control unit, based on the measurement result of the vertical resistance at the contact part with the subject by the vertical resistance measurement unit, detects the transition of the subject's center of gravity position and the load distribution to a specific part at the contact part, and after determining the forceful state and the relaxed state during the transition of the subject's seated position, standing position, and the transition between both postures, adjusts the control content of the state holding variable mechanism unit according to the forceful state and the relaxed state. The function improvement support device according to claim 1, characterized by this.

3. It further includes an environmental information acquisition unit that acquires environmental information representing the surrounding environment of the subject and the floor surface state on which the state holding variable mechanism unit moves. When controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, the control unit gives top priority to ensuring the safety of the subject based on the environmental information obtained from the environmental information acquisition unit. The function improvement support device according to claim 1 or 2, characterized by this.

4. A wheel drive unit provided in the posture maintaining variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms; an imaging unit that images the lower body of the subject and sequentially acquires an RGB image and a depth image; an image coordinate setting unit that, in a state where the posture maintaining variable mechanism unit is deformed into the second mechanism according to the wheelchair function, estimates the posture of the lower body of the subject while setting a plurality of skeletal representative points attached to the lower body of the subject as image coordinates based on the RGB image sequentially acquired from the imaging unit; a coordinate group extraction unit that sequentially extracts an image coordinate group related to the gait state of the subject based on the transition state of the subject's posture from among the image coordinate groups set by the image coordinate setting unit; a skeletal information acquisition unit that acquires temporally continuous three-dimensional skeletal information centered on the lower body of the subject by synchronizing the image coordinate groups sequentially extracted by the coordinate group extraction unit with the depth images sequentially acquired from the imaging unit; and a gait state detection unit that detects a gait state, which is a connection of a plurality of local movements, from the transition state of the subject's posture based on the three-dimensional skeletal information sequentially acquired by the skeletal information acquisition unit. The control unit generates power synchronized with the gait state detected by the gait state detection unit in the wheel drive unit. The functional improvement support device according to claim 1 or 2, characterized in that.

5. A wheel drive unit provided in the posture maintaining variable mechanism unit for independently driving a pair of left and right drive wheels common to the first and second mechanisms; a biological signal detection unit having an electrode group arranged on the body surface part of the subject for detecting a biological signal of the subject; an intention recognition unit that recognizes the intention of the subject with respect to a specific operation content among the operation contents in the wheelchair function based on the biological signal obtained from the biological signal detection unit in a state where the posture maintaining variable mechanism unit is deformed into the first mechanism according to the wheelchair function; and an operation content understanding unit that understands the operation content, which is the intention of the subject recognized by the intention recognition unit. The control unit generates power according to the operation content understood by the operation content understanding unit in the wheel drive unit. The functional improvement support device according to claim 1 or 2, characterized in that.

6. The apparatus further comprises a line-of-sight detection unit that, while imaging the space corresponding to the subject's field of view, recognizes the movement of the subject's eyeballs within the imaging range by image recognition and detects the position of the subject's line of sight. The movement intention recognition unit recognizes, as the movement intention of the subject, that when the position of the line of sight detected by the line-of-sight detection unit continues for a predetermined time or longer, the position of the line of sight is set as a movement target. The functional improvement support apparatus according to claim 5, characterized by the above.

7. The apparatus further comprises a sound collection unit that collects the voice of the subject and a speech content recognition unit that recognizes the speech content based on the voice collected by the sound collection unit. The movement intention recognition unit recognizes, as the movement intention of the subject, the operation content corresponding to the speech content based on the speech content recognized by the speech content recognition unit. The functional improvement support apparatus according to claim 5, characterized by the above.

8. A posture holding variable mechanism unit having a wheelchair function that travels in response to an operation by the subject while holding the subject in a sitting posture and a walking assistance function that applies an assisting force in response to the walking motion of the subject while holding the subject in a standing posture is configured to be deformable from a first mechanism corresponding to the wheelchair function to a second mechanism corresponding to the walking assistance function, or from the second mechanism to the first mechanism, while holding the subject. A first step of acquiring human information including the subject's brain / nervous system information, muscle / skeletal system information, physiological information, psychological information, motion information, and behavior information; a second step of determining the current state of the subject's physical function and cognitive function based on the human information obtained in the first step; a third step of measuring in real time the vertical resistance force at the contact site between the subject and the posture holding variable mechanism unit; and a fourth step of controlling the mechanism switching, deformation state, and power application state of the posture holding variable mechanism unit based on the determination result in the second step and the measurement result in the third step. The functional improvement support method is characterized by the above.

9. In the third step, while detecting the transition of the center of gravity position of the subject and the load distribution to a specific part at the contact part with the subject based on the measurement result of the vertical resistance at the contact part with the subject, the forceful state and the relaxed state at the time of the subject's sitting posture, standing posture, and the transition between both postures are judged. In the fourth step, the control content of the state holding variable mechanism unit is adjusted according to the forceful state and the relaxed state of the subject. The method for supporting functional improvement according to claim 8, characterized in that.

10. Further comprising a fifth step of acquiring environmental information representing the surrounding environment of the subject and the road surface state on which the state holding variable mechanism unit moves. In the fourth step, when controlling the mechanism switching, deformation state, and power application state of the state holding variable mechanism unit, the safety of the subject is given top priority based on the environmental information obtained from the fifth step. The method for supporting functional improvement according to claim 8 or 9, characterized in that.

11. The wheel drive unit provided in the posture holding variable mechanism unit independently drives a pair of left and right drive wheels common to the first and second mechanisms, and the imaging unit captures the lower body of the subject to sequentially acquire an RGB image and a depth image. In a state where the posture holding variable mechanism unit is deformed into the second mechanism according to the wheelchair function, based on the RGB image sequentially acquired from the imaging unit, while estimating the posture of the lower body of the subject, a sixth step of setting a plurality of skeleton representative points attached to the lower body of the subject as image coordinates; A seventh step of sequentially extracting an image coordinate group related to the gait state of the subject based on the transition state of the subject's posture from among the image coordinate groups set in the sixth step; An eighth step of obtaining temporally continuous three-dimensional skeleton information centered on the lower body of the subject by synchronizing in time the image coordinate group sequentially extracted in the seventh step and the depth image sequentially acquired from the imaging unit; A ninth step of detecting a gait state, which is a connection of a plurality of local movements, from the transition state of the subject's posture based on the three-dimensional skeleton information sequentially acquired in the eighth step. In the fourth step, power synchronized with the gait state detected in the ninth step is generated in the wheel drive unit. The function improvement support device according to claim 8 or 9, characterized in that.

12. The wheel drive unit provided in the posture holding variable mechanism unit independently drives a pair of left and right drive wheels common to the first and second mechanisms, and uses an electrode group arranged on the body surface part of the subject to detect a biological signal of the subject. In a state where the posture holding variable mechanism unit is deformed into the first mechanism according to the wheelchair function, a tenth step of recognizing the subject's intention to move with respect to a specific operation content among the operation contents in the wheelchair function based on the biological signal of the subject; An eleventh step of understanding the operation content, which is the intention of the subject recognized in the seventh step. In the fourth step, power according to the operation content understood in the eleventh step is generated in the wheel drive unit. The function improvement support method according to claim 8 or 9, characterized in that.

13. Further comprising a 12th step of detecting the position of the subject's line of sight by image-recognition of the movement of the subject's eyeballs within the imaging range while imaging the space corresponding to the field of view of the subject, and in the 10th step, based on the position of the line of sight detected in the 12th step, when the position of the line of sight continues for a predetermined time or more, recognizing the position of the line of sight as a movement target as the subject's intention of movement. The method for supporting improvement of functions according to claim 12, characterized by the above.

14. Further comprising a 13th step of collecting the voice of the subject and recognizing the utterance content based on the voice, and in the 10th step, based on the utterance content recognized in the 13th step, recognizing the operation content corresponding to the utterance content as the subject's intention of movement. The method for supporting improvement of functions according to claim 12, characterized by the above.

Images