Pose estimation display system

Abstract

It can accurately estimate the patient's posture over a long period of time and present the long-term measurement data in an optimally user-friendly interface for easy viewing. [Solution] The posture estimation display system 1 comprises a first sensor 2 attached to the user and measuring atmospheric pressure, a second sensor 3 attached to the user or placed stationary in the space where the user is located and measuring atmospheric pressure, a posture estimation unit 11 that acquires the atmospheric pressure difference between the atmospheric pressure of the first sensor and the atmospheric pressure of the second sensor and estimates the user's posture based on the atmospheric pressure difference, and a presentation unit 12 that presents the proportion of each different posture of the user estimated from the atmospheric pressure difference acquired from the first sensor and the second sensor over a predetermined period of time.

Description

[[Technical Field]]

[0001] The present disclosure relates to a posture estimation and display system. [[Background Art]]

[0002] Patent Literature 1 discloses an operating state monitoring system comprising: an acquisition unit that acquires sensing information from a sensor attached to a target site; an attachment direction detection unit that detects the attachment direction of the sensor; and a control processing unit that outputs information related to the sensing information in association with the attachment direction of the sensor. [[Prior Art Literature]] [[Patent Literature]]

[0003] [[Patent Literature 1]] Japanese Unexamined Patent Application Publication No. 2022-034450 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] However, when an acceleration sensor is worn on the patient's body over a relatively long period of time during the patient's hospitalization for rehabilitation, if the patient moves their body, the orientation of the worn acceleration sensor may unexpectedly change. In this case, accurate posture estimation cannot be performed over a long period of time. Therefore, there is a demand for accurately estimating the patient's posture over a long period of time and presenting the long-term measurement data via an optimal user interface (UI) that is easily visible to medical personnel and other relevant parties.

[0005] The present disclosure has been made to solve such problems, and an object thereof is to provide a posture estimation and display system or the like that can accurately estimate a patient's posture over a long period of time and presents long-term measurement data via an optimal UI that allows visual recognition of the data. [[Means for Solving the Problem]]

[0006] A posture estimation display system according to one aspect of the present disclosure includes: a first sensor attached to a user for measuring atmospheric pressure; a second sensor attached to the user or placed stationary in the space where the user is located for measuring atmospheric pressure; a posture estimation unit that acquires the atmospheric pressure difference between the atmospheric pressure of the first sensor and the atmospheric pressure of the second sensor and estimates the user's posture based on the atmospheric pressure difference; and a presentation unit that displays the proportion of each different posture of the user estimated from the atmospheric pressure difference acquired from the first sensor and the second sensor over a predetermined period of time.

[0007] This allows for accurate estimation of the patient's posture over a long period, and enables the presentation of long-term measurement data in an optimally user-friendly interface.

[0008] The display unit may display the proportion of the user's posture for each type of different posture in each of the intermediate sections into which the predetermined period is divided. In this case, the display order of the proportions for each type of different postures in each intermediate section is the same so that the user can visually see how the proportions for each type of different postures change across a plurality of consecutive intermediate sections.

[0009] This allows the user interface to improve visibility by structuring time hierarchically (e.g., 24 hours and 1 hour). It also makes it possible to visualize how the proportion of each different posture type changes over time.

[0010] The display unit can display at least one of the following: weather information for the surrounding area and atmospheric pressure information, corresponding to the multiple sub-divisions obtained by dividing the predetermined period.

[0011] This allows for verification of data reliability and appropriate data correction. [Effects of the Invention]

[0012] According to this disclosure, it is possible to provide a posture estimation display system that can accurately estimate a patient's posture over a long period of time and presents the long-term measurement data in an optimally user-friendly interface. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram illustrating the configuration of the posture estimation and display system. [Figure 2] This diagram illustrates two typical arrangement patterns for the first and second sensors used in a posture estimation and display system. [Figure 3] This diagram illustrates a method for estimating a patient's posture using a first sensor and a second sensor. [Figure 4] This diagram illustrates an example of a user interface for a posture estimation and display system. [Figure 5] This diagram illustrates an example of a user interface for a posture estimation and display system. [Figure 6] This diagram illustrates an example of a user interface for a posture estimation and display system. [Modes for carrying out the invention]

[0014] The present invention will be described below through embodiments of the invention, but the invention claimed is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means of solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0015] In the following embodiments, the description will be divided into multiple sections or embodiments where necessary for convenience. Unless otherwise specified, these are not unrelated, and one may be a modification, application, detailed explanation, or supplementary explanation of part or all of the other. Furthermore, in the following embodiments, when referring to the number of elements (including number, numerical value, quantity, and range), unless otherwise specified or clearly limited to a specific number in principle, it is not limited to that specific number, and may be greater than or less than that number.

[0016] Furthermore, in the following embodiments, the components (including operation steps, etc.) are not necessarily essential unless specifically stated or considered to be fundamentally essential. Similarly, in the following embodiments, when referring to the shape or positional relationship of components, etc., it shall include those substantially similar to or resembling their shape, etc., unless specifically stated or considered to be fundamentally different. The same applies to the numbers, etc. (including number, numerical value, quantity, and range) mentioned above.

[0017] Figure 1 is a block diagram illustrating the configuration of the posture estimation display system. The posture estimation display system 1 estimates the user's posture over a predetermined period and displays the estimated posture and its changes in a way that is easy for medical staff to see. The user may, for example, be a hospitalized patient undergoing rehabilitation, but is not limited to this. The predetermined period here may, for example, be a specific period during which the patient is hospitalized (including the training period and hospital life), but is not limited to this. The predetermined period may be various periods, such as 2 hours or more, 3 hours or more, 8 hours or more, 1 day or more, or 3 days or more. Even if patients with similar disabilities and requiring rehabilitation undergo similar training, there are significant differences in the degree of recovery. Therefore, the posture estimation display system can be used to visualize and analyze changes in the patient's posture and activity level during hospital life outside of the training period.

[0018] A posture estimation and display system 1 includes at least a first sensor 2, a second sensor 3, a posture estimation device 10, and a display 4. The first sensor 2 may be, or may include, an atmospheric pressure sensor that is worn on a user's body and measures atmospheric pressure. The first sensor 2 may be a wearable device, and may include an attachment portion (for example, a belt, a band, etc.) for attaching the device to any position on the body. In some embodiments, the first sensor 2 may include a function of measuring triaxial acceleration and angular velocity. This also allows calculation of the angle of each axis. Furthermore, the angle formed with an adjacent sensor can also be measured. The display is not limited to a computer, and may be a display device of various electronic devices such as a tablet or a smartphone.

[0019] The second sensor 3 is a wearable device worn on a user's body, and can measure atmospheric pressure. Alternatively, the second sensor 3 may be placed stationary in the space where the user is located, and can measure atmospheric pressure. For example, the second sensor 3 may be placed on the floor in the space where the user is located, or may be placed on a desk or a chair. The phrase "placed stationary (of a sensor)" means that the sensor may be fixed so as not to move, or may be placed separately so as not to be moved by a person. Since the second sensor 3 measures atmospheric pressure that serves as a reference for estimating the user's posture, it may also be called a reference atmospheric pressure sensor. In some embodiments, the second sensor 3 may include a function of measuring triaxial acceleration and angular velocity. This also allows calculation of the angle of each axis. Furthermore, the angle formed with an adjacent sensor can also be measured.

[0020] Both the first sensor 2 and the second sensor 3 are capable of measuring atmospheric pressure. Therefore, for example, when the sensor is worn by a patient and the patient performs training or the like, even if the orientation of the first sensor 2 or the second sensor 3 unexpectedly changes, the posture can be accurately estimated based on the atmospheric pressure difference without depending on the orientation of the sensor.

[0021] The first sensor 2 and the second sensor 3 are configured to be capable of two-way communication with the posture estimation device 10 via a wired or wireless network. Typically, the first sensor 2 and the second sensor 3 are configured to be capable of two-way communication with the posture estimation device 10 by short-range wireless communication technologies such as Bluetooth (registered trademark) and NFC (Near Field Communication).

[0022] The posture estimation device 10 is a computer having a processor, a memory and the like. A control unit of the posture estimation device 10 has a functional operation unit that executes each of divided processes by reading a stored program. Specifically, the control unit includes a posture estimation unit 11 and a presentation unit 12. The posture estimation unit 11 acquires a barometric pressure difference between the barometric pressure of the first sensor 2 and the barometric pressure of the second sensor 3, and estimates the posture of the user based on said barometric pressure difference. The presentation unit 12 presents (via the display 4) the cumulative time or cumulative ratio for each different posture of the user estimated from the barometric pressure differences acquired from the first sensor 2 and the second sensor 3 over a predetermined period. The display 4 is connected to the posture estimation device 10 via a wired or wireless network. These functions can be used on various electronic devices by installing an application.

[0023] Figure 2 is a diagram illustrating two typical arrangement patterns for arranging the first sensor and the second sensor used in the posture estimation and display system. In arrangement pattern (1), the first sensor 2 is attached to the back of a user U, and the second sensor 3a is attached to the shin of the left foot of the user U. Thereby, the barometric pressure near the back where the first sensor 2 is located can be measured, and the second sensor 3a can measure the barometric pressure near the shin of the user U. Thus, the barometric pressure difference between the first sensor 2 and the second sensor 3a can be acquired, and the posture of the user can be estimated based on this barometric pressure difference.

[0024] This arrangement pattern is merely an example, and various modifications are possible. The first and second sensors can be installed at different heights when the user is standing. For example, the first sensor 2 may be located on any part of the upper body, such as the chest, solar plexus, upper arm, shoulder, or neck. The first sensor 2 may also be located on the torso. Furthermore, the second sensor 3a, which measures the reference atmospheric pressure, may be located on any part of the lower body, such as the shin or calf of the right leg, ankle, instep, or toes. The second sensor 3a may also be located on the thigh or lower leg.

[0025] In arrangement pattern (2), the position of the first sensor 2 is the same as in arrangement pattern (1), but the second sensor 3b is placed in a location that is not a part of the body. The second sensor 3b may be built into an external terminal 30. The external terminal can be any terminal, such as a smartphone, tablet, or laptop computer. The second sensor 3b is placed stationary in the space where the user is located in order to measure a reference atmospheric pressure. The second sensor 3b may be placed on a desk, chair, floor, etc., in the space where the user is located.

[0026] Furthermore, while the first and second sensors are provided in each arrangement pattern (1) and (2) shown in Figure 2, additional sensors may be provided in other locations indicated by the dashed lines in Figure 2. In some embodiments, the first sensor 2, the second sensor 3, and the additional sensors may include functions for measuring acceleration and angular velocity in three axes, thereby enabling the calculation of the angle of each axis. It is also possible to measure the angle between adjacent sensors.

[0027] Figure 3 illustrates a method for estimating a patient's posture using a first sensor and a second sensor. The upper part of Figure 3 shows typical postures that can be determined based on the pressure difference (PD), namely standing, sitting, and lying down, but the disclosure is not limited to these. As shown in Figure 3, the pressure difference between the first and second sensors is largest when the patient is standing. The pressure difference between the first and second sensors is smallest when the patient is lying down. The pressure difference for sitting is somewhere between that of standing and lying down. The lower part of Figure 3 shows the change in the pressure difference between the first and second sensors. In this way, the posture estimation device 10 can determine whether the patient's posture is standing, sitting, or lying down based on the pressure difference between the first and second sensors over a predetermined period of time.

[0028] Figures 4 to 6 illustrate various examples of user interfaces for posture estimation and display systems. These user interfaces are designed to provide an overview of long-term measurement data, allow users to view detailed information at any given point in time, and enable them to understand the results at each time interval.

[0029] In the lower region 41 of Figure 4, the cumulative time and percentage are displayed for each different posture, such as walking, standing, sitting, and lying down, for the entire long-term measurement data (region 411 in Figure 4; in this example, the entire data represents 24 hours, or one day). In the lower region 41 of Figure 5, the cumulative time and percentage are displayed for each different posture, such as walking, standing, sitting, and lying down, for a specified period of long-term measurement data (16:00-16:59 in region 412 of Figure 5). That is, the user can select either the "entire" long-term measurement data or a "specified period" via an input device (keyboard, touch panel, mouse, etc.). In Figures 4 and 5, bar graphs are shown showing the cumulative time and percentage for each different posture. In the lower region 41 of Figure 6, a pie chart (region 413) is shown showing the cumulative time and percentage for each different posture. However, this disclosure is not limited to these, and various other preferred methods may be used to show the cumulative time and percentage for each different posture.

[0030] Above area 41 in Figure 4, area 42 is shown for viewing detailed information at any given time. This area 42 presents the proportion of the user's posture for each type of posture (e.g., walking, standing, sitting, and lying down) in a histogram format, for each sub-division (e.g., 1 hour) that divides a predetermined period (e.g., 1 day on January 16, 2025). The display order of the proportions for each type of posture in each sub-division (each 1 hour from 14:00 to 16:00) is the same so that the user can visually see how the proportions for each type of posture change over multiple consecutive sub-divisions (area 427 in Figure 4, 3 hours from 14:00 to 16:00).

[0031] Thus, the user interface described in this disclosure can be made more visually appealing by having a hierarchical structure of time (for example, 24 hours and 1 hour).

[0032] In some embodiments, hourly weather information may be displayed below region 427 (region 428 in Figure 5, in the order of sunny, cloudy, and rainy). That is, for each of the multiple sub-divisions that divide a predetermined period (each hour from 14:00 to 16:00), at least one of the surrounding area weather information and barometric pressure information can be displayed. Surrounding area weather information can be obtained from external sources such as weather information media or news. Surrounding area barometric pressure information can be obtained from external sources such as weather information media or news, without being obtained from the first and second sensors.

[0033] This weather and barometric pressure information for the surrounding area is provided to inform the user of its influence on attitude estimation based on barometric pressure differences. In particular, in the case of arrangement pattern (2), if the user moves vertically (for example, to a different floor) relative to the external terminal used to measure the reference barometric pressure, accurate attitude estimation based on barometric pressure differences may not be possible. Furthermore, it may become unclear whether the reference barometric pressure changed due to such movement or due to changes in the external environment (weather). For this reason, the barometric pressure data measured by the external terminal 30 may need to be saved in order to verify the reliability of the data and to perform data correction as appropriate. For this reason, the user interface in this disclosure displays weather and barometric pressure information for the surrounding area.

[0034] In Figure 4, above region 42, region 43 is shown, displaying further detailed information for any given time point. In region 42, the posture ratios for the selected portion (in this example, the hour starting at 16:00) are displayed in 60 entries per minute (a total of 60 minutes).

[0035] In some embodiments, as shown in region 43, the proportion of the user's posture in each sub-interval (e.g., 1 minute), obtained by further dividing an intermediate interval (e.g., 1 hour), may be presented histographically for each type of different posture. The display order of the proportions for each type of different postures in each sub-interval is the same so that it is possible to visualize how the proportions for each type of different postures change across multiple consecutive sub-intervals (i.e., in the case of Figure 4, from top to bottom: walking, standing, sitting, and lying down).

[0036] The time-series measurement data shown in region 43 can be displayed as a list for a predetermined period (e.g., 1 hour) via an input device. The list can show, for example, the start time, the estimated type of posture, and the duration. For example, the list might show: "16:00:00 Sitting (00:03:12), 16:03:12 Standing (00:00:10), 16:03:22 Sitting (00:00:01), 16:03:23 Lying down (00:05:56)" ... "16:50:00 Sitting (00:03:12), 16:53:12 Standing (00:00:10), 16:53:22 Sitting (00:00:01), 16:53:23 Lying down (00:05:56)". The starting point of the list can be selected by moving the seek bar 431.

[0037] The list may also include weather information and atmospheric pressure information for the surrounding area. In some embodiments, at least one of the weather information and atmospheric pressure information for the surrounding area may be presented for each of the multiple sub-sections obtained by dividing the central section.

[0038] In other embodiments, as shown in Figure 5, the occurrence of standing up and sitting down may be indicated by marks (e.g., triangles) in at least a portion of the region 432 of the region 43 that shows time-series measurement data. The number or frequency of standing up and sitting down may be indicated by whether or not the marks (e.g., triangles) are filled in. The occurrence of abnormal standing up or sitting down may be represented by a difference in color. The indication of the occurrence of standing up and sitting down may be shown from outside the region 43, using triangles, arrows, etc., to indicate the time of occurrence.

[0039] Above region 43 in Figure 4, region 44 is shown, displaying time-series measurement data for a predetermined period (e.g., 30 seconds) starting from an arbitrary position specified by the seek bar 431 of region 43. In region 44, the horizontal axis represents time, and the transitions from standing, sitting, walking, sitting, walking, sitting, standing, lying down, etc., can be clearly indicated by hatching and color differences. The seek bar 431 of region 43 can be moved arbitrarily horizontally via an input device, and the start of the time-series measurement data in region 44 may change accordingly.

[0040] In the left side of region 44 in Figure 4, a video showing walking can be displayed.

[0041] The upper part of Figure 4 is provided with a region 46 that shows one of the two arrangement patterns shown in Figure 2. The dashed region 461 in Figure 4 indicates that the measured atmospheric pressure data was acquired using arrangement pattern (1) of Figure 2. In particular, Figure 4 shows that a back sensor is used as the first sensor 2 and a lower leg sensor is used as the second sensor 3. In this case, it is shown that no external terminal is used.

[0042] On the other hand, in region 46 of Figure 5, the dashed area 462 indicates that the measured atmospheric pressure data was acquired using the arrangement pattern (2) in Figure 2. That is, it indicates that a back sensor is used as the first sensor 2, and a sensor built into an external terminal is used as the second sensor 3.

[0043] Furthermore, whether the measured atmospheric pressure data was acquired using arrangement pattern (1) or arrangement pattern (2) can be indicated in various ways that a person skilled in the art can understand (for example, colored when the sensor is active, and gray when the sensor is inactive).

[0044] As mentioned above, especially in arrangement pattern (2), if the user moves vertically (for example, to a different floor) relative to the external terminal used to measure the reference atmospheric pressure, the reference atmospheric pressure may change. Therefore, the reliability of the measurement data using the pressure sensor can be communicated through the UI.

[0045] In Figure 4, a back sensor is used as the first sensor 2, and a lower leg sensor is used as the second sensor 3. Therefore, to the right of the region 46 showing arrangement pattern (1), a region 47 is provided that shows the back sensor pressure and the lower leg sensor pressure.

[0046] In Figure 5, a back sensor is used as the first sensor 2, and a sensor built into the external terminal is used as the second sensor 3. Therefore, to the right of the area 46 showing arrangement pattern (2), an area 47 is provided that shows the back sensor pressure and the ambient pressure of the external terminal.

[0047] In Figure 6, barometric pressure information is shown in region 414, to the right of the pie chart (region 413) which shows the cumulative time and percentage for each different posture in region 41. In Figure 6, the barometric pressure information includes the barometric pressure of the area surrounding the external terminal, the barometric pressure of the reference sensor (lower leg sensor), and the barometric pressure of the attached sensor (back sensor). A full-body human-shaped icon is also displayed to indicate the attachment positions of the first sensor 2 and the second sensor 3. If an external terminal is being used, it may be shown in color; if an external terminal is not being used, it may be shown in gray. In addition, weather information for the region at the time the measurement data was acquired may be displayed to the right of the barometric pressure information. This barometric pressure and weather information is useful for verifying the reliability of the data obtained by using barometric pressure sensors.

[0048] If the first sensor 2 or second sensor 3 attached to the user has the function of measuring not only barometric pressure but also 3-axis acceleration, velocity, and angular velocity, then specific actions of the patient (e.g., standing up, sitting down, or abnormal movements) can be detected. In this case, a marked area 432 is provided outside area 43 in Figure 6 so that it is possible to see when the specific action (e.g., standing up, sitting down, or abnormal movements) occurred. As mentioned above, area 43 shows how the proportion of each type of different posture changes over time, but the timing of the occurrence of a specific action is indicated chronologically from outside area 43 with a mark (e.g., arrow, triangle). This improves the UI visibility of long-term measurement data and allows for analysis of the occurrence of specific actions that should be focused on in rehabilitation.

[0049] Outside of region 42 in Figure 6, the number and frequency of occurrences of specific behaviors (e.g., standing up and sitting down, or abnormal movements) may be indicated by marks (e.g., arrows, triangles) and colors. For example, white could indicate 1 to 4 occurrences of a particular behavior, yellow 5 to 9 occurrences, and red 10 or more occurrences. This allows for analysis of how often and at what times a particular behavior occurs.

[0050] In Figures 4 to 6, region 43 shows how the proportion of each different posture type has changed over time. However, it is also acceptable to show only the posture type with the highest proportion, or only the top few (for example, three) in order of proportion.

[0051] As explained above, the user interface of this disclosure is optimized to make long-term measurement data easily viewable by medical professionals and others, and to facilitate the analysis of patients' rehabilitation status.

[0052] In some embodiments, a posture estimation and display method is provided. This method obtains atmospheric pressure from a first sensor attached to the user, obtains atmospheric pressure from a second sensor attached to the user or placed stationary in the space where the user is located, obtains the pressure difference between the atmospheric pressure of the first sensor and the atmospheric pressure of the second sensor, estimates the user's posture based on the pressure difference, and presents the cumulative time or cumulative percentage for each of the user's different postures estimated from the pressure differences obtained from the first and second sensors over a predetermined period. The posture estimation and display method can incorporate some or all of the features of the posture estimation and display system described above.

[0053] In some embodiments, a program is provided that causes a computer to execute the posture estimation and display method described above. The program may incorporate some or all of the features of the posture estimation and display system and the program described above.

[0054] In the above example, the program can be stored and supplied to the computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives) and magneto-optical storage media (e.g., magneto-optical disks). Examples of non-transitory computer-readable medium further include CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM; examples of non-transitory computer-readable medium further include PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (random access memory)). Alternatively, the program may be supplied to the computer by various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels. [Explanation of Symbols]

[0055] 1. Pose Estimation and Display System 2. First Sensor 3. Second Sensor 3a Reference side pressure sensor 3b Reference side pressure sensor 4 displays 10 Posture estimation device 11 Posture estimation section 12 Presentation section 30 External terminals U User

Claims

1. A first sensor is attached to the user to measure atmospheric pressure, A second sensor, which is attached to the user or placed stationary in the space where the user is located, measures atmospheric pressure. A posture estimation unit that acquires the pressure difference between the pressure of the first sensor and the pressure of the second sensor and estimates the user's posture based on the pressure difference, A posture estimation display system comprising: a display unit that displays the proportion of each different posture of the user estimated from the pressure difference acquired from the first sensor and the second sensor over a predetermined period of time.

2. The posture estimation display system according to claim 1, wherein the display unit displays the proportion of the user's posture for each type of different posture in each of the intermediate sections into which the predetermined period is divided, and the display order of the proportions for each type of different postures in each intermediate section is the same so that the user can visually see how the proportions for each type of different postures change across a plurality of consecutive intermediate sections.

3. The posture estimation display system according to claim 2, wherein the display unit displays at least one of weather information for the surrounding area and atmospheric pressure information, corresponding to the plurality of intermediate sections into which the predetermined period has been divided.

4. The posture estimation display system according to claim 2, wherein the display unit displays the time or number of times when the user stands up or sits down, near the area where the cumulative time of the user's posture for each of the plurality of intermediate sections obtained by dividing the predetermined period is displayed for each different type of posture.

5. The display unit indicates whether the second sensor is attached to the user's body or is placed stationary in the space where the user is located. Furthermore, the display unit displays the proportion of the user's posture for each sub-section obtained by further dividing the intermediate section, for each different type of posture, and the display order of the proportions for each different type of posture in each sub-section is the same so that the user can visually see how the proportions for each different type of posture change across a plurality of consecutive sub-sections, the posture estimation display system according to claim 2.

Images