Device and system for estimating posture of legs of seated user

Abstract

Provided is a device for estimating the posture of the legs of a seated user. The device can be attached to or built into a piece of furniture on which a user can sit and includes: at least one lower leg sensor that is positioned below a seat section of the furniture and serves to measure a depth distribution, namely, an in-plane distribution of a depth distance including the depth distance to the lower legs of the seated user; and a posture estimation means that uses the measured depth distribution as a basis to determine the positions of the ankles of the seated user and the directions or orientations of the lower legs as the posture of the lower legs and uses the positions of the ankles, the directions or orientations of the lower legs, and hip joint positions set in advance or determined with another sensor as a basis to determine the positions of the knees or the positions of the knees and the directions or orientations of the upper legs of the seated user as the posture of the upper legs.

Description

Apparatus and system for estimating leg posture of a seated user

[0001] The present invention relates to a technology for devices and systems that function as user interfaces in virtual spaces, and more particularly to a technology for manipulating the posture of virtual objects, such as avatars and characters, in virtual reality spaces and game spaces.

[0002] This application claims the benefit of priority under the Paris Convention to Japanese Patent Application JP2023-122895, filed on July 27, 2023, which is incorporated herein by reference in its entirety pursuant to PCT Rule 20.6.

[0003] Virtual reality is a system that projects a three-dimensional virtual space into a user's field of vision. While viewing the virtual space, users can virtually move within the virtual space by holding and operating a stick device or game controller. In particular, users can feel immersed in the virtual space by wearing a head-mounted display (HMD) terminal on their head. This HMD terminal may access a virtual space server via a network or may be equipped with a virtual space server itself. The HMD terminal measures the user's head movement using a posture sensor and controls the user's field of view and position in the virtual space based on the measured values. It can also measure the user's hand movement in real space using a depth camera and perform operations in the virtual space based on the measured values.

[0004] The virtual space server may be a virtual space server (Dedicated Server) located on the Internet, or a virtual space server (Listen Server) located locally. Well-known mechanisms for providing virtual spaces include social VR platforms such as VRChat (registered trademark), Metaverse (registered trademark), and Second Life (registered trademark).

[0005] For example, Non-Patent Document 1 discloses a technology for reflecting the posture of a user's legs in real space in the posture of a virtual character in virtual reality space. In this technology, IMUs (Inertial Measurement Units) are attached to at least the user's torso, above each knee, and below each knee. Next, posture information of the user's torso and legs estimated using the measurements of these IMUs is transmitted to a terminal running a virtual reality application. This allows the user's posture in real space to be reflected in the posture of a virtual character corresponding to the user in the virtual reality application.

[0006] Furthermore, Non-Patent Document 2 discloses a technology that measures a user's posture using an RGB camera and a depth sensor and performs game operations. This technology uses an RGB camera, a depth sensor, a multi-array microphone, and a processor running dedicated software to recognize the player's position, movement, voice, and face. This allows the user to intuitively play video games using their own body. This technology also primarily uses motion capture, which reads the player's movements and performs synthesis processing. However, unlike conventional motion capture, this motion capture does not require the special marker-equipped suit worn during regular motion capture or the tracker used for marker detection. Specifically, a subject is captured on a camera and the distance from the player to the Kinect is measured, and the player's skeletal movements are measured. The player's movements are then reflected in real time in the movements of the in-game character.

[0007] Furthermore, Patent Document 1 discloses a chair-type device for measuring a user's posture in real space. This chair-type device is a controller chair that generates signals corresponding to the movements of the user's torso, left arm, right arm, left leg, and right leg, and includes: (a) a seat supported on a base; (b) a backrest that extends upward or diagonally upward from the rear edge of the seat and is supported on the base so as to be rotatable around an axis extending along the center line in the width direction of the seat; (c) a left arm holder that protrudes forward from the left side of the backrest and is supported on the backrest so as to be rotatable up and down around its base end; (d) a right arm holder that protrudes forward from the right side of the backrest and is supported on the backrest so as to be rotatable up and down around its base end; (e) a left leg holder that extends downward from near the left side of the front edge of the seat on the front side of the base and is supported on the base so as to be rotatable up and down around its top end; (f) a right leg holding part extending downward from near the right side of the front edge of the seat at the front side of the base and supported on the base so as to be rotatable up and down around its upper end, and (g) a first rotation angle detection part, a second rotation angle detection part, a third rotation angle detection part, a fourth rotation angle detection part and a fifth rotation angle detection part which detect the rotation angles of the backrest part, the left arm holding part, the right arm holding part, the left leg holding part and the right leg holding part, respectively. By transmitting these rotation angle signals to a terminal running a virtual reality application, the posture of the virtual character in the virtual space can be changed.

[0008] Patent No. 7189811 Patent No. 6886141 Patent No. 6778205

[0009] Shiftall, [online], [searched July 12, 2023], Internet<URL: https: / / ja.shiftall.net / products / haritorax> Wikipedia "Kinect", [online], [searched July 12, 2023], Internet<URL: https: / / en.wikipedia.org / wiki / Kinect> Wikipedia "Eular angles", [online], [Retrieved July 12, 2023], Internet<URL: https: / / en.wikipedia.org / wiki / Euler_angles> Wikipedia "Quaternion", [online], [Retrieved July 12, 2023], Internet<URL: https: / / en.wikipedia.org / wiki / Quaternion> Wikipedia "Axis-angle representation", [online], [Retrieved July 12, 2023], Internet<URL: https: / / en.wikipedia.org / wiki / Axis-angle_representation >

[0010] According to the techniques disclosed in the aforementioned Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2, the user's own posture in real space can be reflected in the posture of a virtual object and game operations.

[0011] However, the technology disclosed in Non-Patent Document 1 has a problem in that a posture detection device must be attached to a body part of the user in order to measure the posture of the user's legs. In other words, this technology cannot measure the posture of a body part to which a sensor unit is not attached. Furthermore, the technology disclosed in Non-Patent Document 2 has a problem in that the user's privacy may be violated by measuring the user's entire body using an RGB camera or a depth sensor. Furthermore, in the technology disclosed in Patent Document 1, the posture detection device can only measure the rotation angle with one degree of freedom, i.e., it cannot measure posture with multiple degrees of freedom, which causes a problem in that the user's range of motion is limited by the holding unit.

[0012] Therefore, an object of the present invention is to provide a device and system that can estimate the posture of the legs (thighs) of a seated user. For example, the present invention can provide a device and system that does not require the user to wear a sensor, reduces the possibility of violating the user's privacy by measuring the user's entire body, and does not limit the user's range of motion by a holder.

[0013] According to the present invention, there is provided a device that is built into or can be attached to furniture on which a user can sit, the device comprising: at least one lower leg sensor that is positioned under the seat of the furniture and that measures a depth distribution, which is an in-plane distribution of depth distances including the depth distance to the lower legs of a seated user; and posture estimation means that determines the position of the ankle and the direction or orientation of the lower leg of the seated user as the posture of the lower leg based on the measured depth distribution, and that determines the position of the knee, or the position of the knee and the direction or orientation of the upper leg of the seated user as the posture of the upper leg based on the position of the ankle and the direction or orientation of the lower leg and the position of the hip joint determined using a preset sensor or a separate sensor.

[0014] As one embodiment of the device of the present invention, it is also preferable that the device outputs, as the posture of the upper thigh, the position of the knee determined based on the position vector of the knee or the orientation of the upper thigh determined based on the direction vector of the upper thigh, and outputs, as the posture of the lower leg, the position of the ankle determined based on the position vector of the ankle or the orientation of the lower leg determined based on the direction vector of the lower leg.

[0015] In another embodiment of the device of the present invention, the posture estimation means includes a machine learning engine that has been pre-trained with teacher data in which information related to the depth distribution measured by the lower leg sensor is used as an explanatory variable, and information related to the position of the ankle and information related to the direction of the lower leg are used as objective variables, and by inputting the information related to the depth distribution into the machine learning engine, it is also preferable to obtain information related to the position of the ankle and information related to the direction or orientation of the lower leg as outputs.

[0016] In yet another embodiment of the device of the present invention, it is also preferable that the device further comprises one or more seat pressure sensors that measure pressure distribution in the seat portion, and the posture estimation means uses the position of the hip joint determined based on the pressure distribution.

[0017] In yet another embodiment of the device of the present invention, it is also preferable that the posture estimation means corrects the determined posture of the upper leg and the posture of the lower leg to a standing posture of the seated user and outputs the corrected posture.

[0018] In yet another embodiment of the device of the present invention, it is also preferable that the device further comprises at least one seat posture sensor that detects the position and orientation of the seat of the seat portion, and the posture estimation means performs a rotation process on the posture of the upper thigh and the posture of the lower leg according to the detected orientation of the seat, adds the detected position of the seat as an offset, and outputs the posture of the upper thigh and the posture of the lower leg as values ​​in absolute coordinates stretched in the space in which the device exists.

[0019] In yet another embodiment of the device of the present invention, it is also preferable that the posture estimation means determines the amount of movement in the virtual space of a virtual object operated by a user based on changes in the posture of the upper thighs and / or the posture of the lower thighs of a seated user.

[0020] The device of the present invention is also preferably in the form of a chair, a cushion, a mat, or a wheelchair.

[0021] The device of the present invention is preferably incorporated into furniture to form a furniture-type appliance together with the furniture, and is also preferably an accessory device that can be attached to furniture.

[0022] The present invention also provides a system having a device built into or attachable to furniture on which a user can sit, and an external device that communicates with the device, wherein the device has at least one lower leg sensor located under the seat of the furniture for measuring a depth distribution, which is an in-plane distribution of depth distances including the depth distance to the lower legs of a seated user, and the external device has posture estimation means for determining the position of the ankle and the direction or orientation of the lower leg of the seated user as the posture of the lower leg based on the measured depth distribution, and for determining the position of the knee of the seated user, or the position of the knee and the direction or orientation of the upper leg, as the posture of the upper leg based on the position of the ankle and the direction or orientation of the lower leg and the position of the hip joint determined using a preset sensor or a different sensor.

[0023] The device and system of the present invention can estimate the posture of the legs (thighs) of a seated user. For example, the present invention can provide a device and system that does not require the user to wear a sensor, reduces the possibility of violating the user's privacy by measuring the user's entire body, and does not limit the user's range of motion by a holder.

[0024] FIG. 1 is a schematic diagram showing a chair-like furniture-type device according to a first embodiment of the present invention. FIG. 2 is a schematic diagram showing a chair-like furniture-type device according to a second embodiment of the present invention. FIG. 3 is a schematic diagram for explaining each vector of the right leg of a user sitting on the furniture-type device. FIG. 4 is a schematic diagram for explaining the ankle position vector j u and the direction vector l of the lower leg u5A, 5B, and 5C are schematic diagrams illustrating vectors determined by the present invention. FIG. 6 is a schematic diagram illustrating a transition from a seated posture to a standing posture with and without correction processing. FIGS. 7A and 7B are schematic diagrams illustrating a cushion-like furniture-type device and a mat-like furniture-type device, respectively, according to other embodiments. FIG. 8 is a schematic diagram illustrating an accessory device that can be attached to seatable furniture. FIG. 9 is a schematic diagram illustrating the configuration of a system according to the present invention.

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0026] Fig. 1 is a schematic diagram showing a chair-like furniture-type device according to a first embodiment of the present invention, and Fig. 2 is a schematic diagram showing a chair-like furniture-type device according to a second embodiment of the present invention.

[0027] 1 and 2 show a furniture-type device 1 according to the present invention. The furniture-type device 1 comprises a chair-like piece of furniture on which a user can sit and a device (leg posture estimation device) installed in the furniture for estimating the posture of the user's legs. The leg posture estimation device includes at least one lower leg sensor 11 and a posture estimation unit 12. In this embodiment, the posture estimation unit 12 includes a processor and a memory, and the processor functions as the posture estimation unit 12 by executing a program according to the present invention stored in the memory. In other words, the posture estimation unit 12 can be considered a functional component realized by executing the program according to the present invention. The posture estimation unit 12 may be built into the furniture-type device 1 (leg posture estimation device), or may be included in an external device that communicates with the furniture-type device 1, as will be described later with reference to FIG. 9 .

[0028] [Lower leg sensor 11] The lower leg sensor 11 is placed under the seat of the furniture-type device 1 shown in Figure 1 and measures the depth distribution, which is the in-plane distribution of depth distances including the depth distance to the lower legs of a seated user.

[0029] According to the embodiment shown in FIG. 1 , one lower leg sensor 11 is installed on a relay axis extending from the seat of the furniture-type device 1 to the floor, with the sensor surface of the lower leg sensor 11 facing the area around the lower leg of the seated user. The lower leg sensor 11 may include, for example, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or a depth camera. This allows for measurement of a depth distribution, which is the in-plane distribution of depth distances including the depth distance to the lower leg of the seated user. The lower leg sensor 11 may also include a stereo camera or an IR (infrared) camera. This type of lower leg sensor 11 can also measure a depth distribution, which is the in-plane distribution of depth distances.

[0030] According to the embodiment shown in Fig. 2, 30 distance measuring sensors serving as lower leg sensors 11 are arranged in 5 rows and 6 columns on the front curtain surface located under the seat of the furniture-type device 1, with their sensor surfaces facing the area around the lower legs of a seated user. The distance measuring sensors measure changes in light, ultrasound, or capacitance to measure the depth distance to an object. By arranging such distance measuring sensors in, for example, two or more rows by two or more columns, it is possible to measure the depth distribution, which is the in-plane distribution of depth distance.

[0031] According to the present invention, it is sufficient to measure a depth distribution, which is an in-plane distribution of depth distances, using at least one lower leg sensor 11. Here, the depth distribution, which is an in-plane distribution of depth distances, may be a two-dimensional array of multiple depth distance values, or may be a three-dimensional point cloud constructed based on these depth distance values. Note that a depth distribution in which depth distance values ​​are arranged linearly, i.e., one-dimensionally, may not be generated by the lower leg sensor 11.

[0032] [Posture Estimation Unit 12] The posture estimation unit 12 estimates and outputs the posture of the upper thighs and / or lower thighs of each of the left and right legs of the seated user based on the depth distribution measured by the lower leg sensor 11.

[0033] FIG. 3 is a schematic diagram for explaining vectors in the right leg of a user seated on a piece of furniture-type equipment.

[0034] According to FIG. 3, the following five vectors are set in a coordinate system stretched in a space including the user's legs. Here, the ankle refers to the area around the ankle or heel. The position vectors are three-dimensional real vectors. a. Ankle position vector j u : Vector representing the posture of the lower leg b. Direction vector l of the lower leg u : Vector from the knee to the ankle, representing the posture of the lower leg c. Knee position vector k u : Vector representing the posture of the upper thigh d. Upper thigh direction vector u u :Vector from the hip joint to the knee, representing the posture of the upper thigh e. Hip joint position vector h u

[0035] These vectors may be converted into a predetermined coordinate system before output. The coordinate system may be at least one arbitrary coordinate system set during the manufacture of the furniture-type device 1, or may be a coordinate system that is dynamically set according to the posture of the seated user. Furthermore, these vector values ​​may be corrected by multiplying them by a predetermined scaling factor, rotating them, adding an offset, limiting the range of values, or the like, before output.

[0036] Figure 4 shows the ankle position vector j u and the direction vector l of the lower leg u 10 is a schematic diagram illustrating a furniture-type device as viewed from the side, for explaining a process of determining the position of the furniture-type device. FIG.

[0037] The posture estimation unit 12 estimates the posture of the user's ankle position vector j based on the depth distribution. u and the direction vector of the lower leg l u 4 shows five depth arrows that represent the measured depth distance when the front curtain surface under the seat of the furniture-type device 1 is viewed from the side. There is a difference in length (depth difference) between the two upper depth arrows and the three lower depth arrows. This means that the two upper depth arrows can be understood to represent the depth distance to the user's lower legs. In this case, the ankle position vector j ucan be a vector indicating the position of the depth distance indicated by the lowest of the two upper depth arrows. Also, the direction vector l of the lower leg u can be calculated from the difference in the depth distance positions indicated by the two depth arrows on the top. u and the direction vector of the lower leg l u can be analytically determined based on the depth distribution detected by the lower leg sensor 11 by taking the difference between adjacent or distant depth distance values ​​or by extracting feature points based on a threshold or gradient.

[0038] 4 shows the leg posture determination process for one leg in two dimensions for simplicity, but it is also preferable to determine the leg postures for both legs, and it is also preferable to perform the leg posture determination process in three dimensions. For example, if a high-resolution depth distribution can be obtained using LiDAR or a depth camera, the leg posture can be determined with higher accuracy.

[0039] [The ankle position vector j using a machine learning engine] u and the direction vector of the lower leg l u The posture estimation unit 12 uses a machine learning engine to estimate the ankle position vector j from the depth distribution measured by the lower leg sensor 11. u and the direction vector of the lower leg l u In this case, the depth distribution is used as an explanatory variable, and the ankle position vector j u and the direction vector of the lower leg l u First, training data is prepared with the objective variable as the training variable. The machine learning engine is equipped with a trained learning model that has been trained using this training data. Of course, techniques such as segmentation and clustering may be used to build this machine learning engine, or a neural network algorithm or other machine learning algorithm may be used.

[0040] 5A, 5B and 5C are schematic diagrams for explaining vectors determined by the present invention.

[0041] As shown in FIG. 5A, the posture estimation unit 12 estimates the posture of the upper thigh by estimating the position vector j of the user's ankle. u and the direction vector of the lower leg l u From the above, the knee position vector k is calculated using the given parameter t. u Specifically, the knee position vector k is estimated using the following equation: u is determined. u = -t×l u +j u (t>0)

[0042] As shown in FIG. 5B, the posture estimation unit 12 estimates the position vector j of the user's ankle. u and the direction vector of the lower leg l u and the hip joint position vector h determined in advance or using another sensor. u From this, the length of the upper leg x is calculated using the given parameter t. u and the length of the lower leg y u Specifically, the user's upper leg length x u and the length of the lower leg y u is determined by the following formula: x u =||k u -h u ||=||-t×l u +j u -h u || y u =||j u -k u ||=||t×l u ||

[0043] Next, the hip joint position vector h u is determined as the center of rotation of the upper leg based on the position and orientation of the user's hips. This hip joint position vector h u may be set in advance when the furniture-type device is designed. In addition, the pressure distribution in the seat surface is measured by at least one seat pressure sensor arranged in the seat surface of the furniture-type device 1, and the hip joint position vector h uSuch a seat pressure sensor is disclosed in, for example, Patent Document 2. As described above, the posture estimation unit 12 estimates and determines the hip joint position vector h u For example, if an embedded pressure distribution sensor sheet, which is a film-type sensor sheet with a thickness of approximately 0.1 mm, is used as the seat pressure sensor, it will be possible to measure the applied pressure in real time.

[0044] The posture estimation unit 12 calculates the determined upper leg length x u and the length of the lower leg y u The predetermined parameter t is determined based on the ratio α between x and t. The ratio α may be preset based on statistical information of human body dimensions. Alternatively, α may be set to 1 as an approximate value, or may be changeable by the user as appropriate. Using the ratio α determined in this way, the following equation is established: x u = αy u (α>0) ||-t×l u +j u -h u || 2 = α 2 ||t×l u || 2 Here, the second equation is a linear equation with respect to t when α = 1, and a quadratic equation when α ≠ 1. By analytically solving these equations with respect to t, the following equation for calculating t is derived:

[0045] According to FIG. 5C, the posture estimation unit 12 estimates the posture of the upper leg by using the position vector j of the user's ankle. u and the direction vector of the lower leg l u and the hip joint position vector h u Then, using the given parameter t, the direction vector u of the upper leg is u Specifically, the direction vector of the upper leg u is estimated using the following equation: u can be determined. u = k u -h u = -t×l u +j u -hu

[0046] Upper thigh direction vector u u and the direction vector of the lower leg l u The orientation in can be calculated based on a predetermined coordinate system. Here, this orientation may take the value of an index that expresses rotation, such as Euler angles, quaternions, or axis-angle representation, as disclosed in, for example, Non-Patent Documents 3, 4, and 5.

[0047] As shown in Figures 5(A) to (C), according to the present invention, the lower leg sensor 11 placed on the furniture-type device 1 can estimate and determine the position of the knee and the posture of the upper leg by simply observing only a portion of the user's lower leg.

[0048] <Regarding local coordinate systems> The orientation of a body part can be calculated by setting a reference coordinate system and a local coordinate system for that body part, as disclosed in, for example, Non-Patent Document 3. For example, the following xyz coordinate system can be set as the local coordinate system for the upper thigh, with the coordinate system related to the seat surface or the lower back as the reference: x-axis: direction vector u of the upper thigh u Axis including y-axis: Upper leg direction vector u u and the direction vector of the upper leg u u and the direction vector of the lower leg l u z-axis: Axis perpendicular to the x-axis and y-axis. In addition, the following xyz coordinate system can be set as the local coordinate system for the lower leg, based on the local coordinate system for the upper leg set as above. x-axis: Direction vector l of the lower leg u Axis including y-axis: Direction vector of the lower leg l u and the direction vector of the upper leg u u and the direction vector of the lower leg l u z-axis: Axis parallel to the plane defined by the x-axis and y-axis. Note that the setting of the local coordinate system for a body part may differ depending on the system, so the method for calculating the orientation of a body part is not limited to the one described below.

[0049] When the orientation of the upper leg or lower leg is expressed as Euler angles and output, the unit of the orientation value may be converted to degrees, radians, or the like, and the converted value may be output. Furthermore, only a portion of the Euler angles may be output, taking into account the degrees of freedom of the joint. For example, since the knee joint has one degree of freedom of rotation, when the orientation of the lower leg is output relative to the local coordinate system of the upper leg as a reference coordinate system, only the pitch angle of the Euler angles may be output as the orientation of the lower leg. Furthermore, the output orientation value may be corrected by multiplying it by a predetermined scaling factor, rotating it, adding an offset, limiting the range of values, or the like. For example, the pitch angle of the orientation of the lower leg relative to the local coordinate system of the upper leg as a reference coordinate system may be corrected, and the corrected value may be output as the flexion angle of the knee joint.

[0050] The postures determined and output by the furniture-type device may be, for example: (a) posture of the upper leg: position of the center of rotation of the upper leg (hip joint), position of the knee, orientation of the upper leg, orientation of the knee; and / or (b) posture of the lower leg: position of the center of rotation of the lower leg (knee), position of the ankle, orientation of the lower leg, orientation of the ankle. Note that this posture information is redundant. That is, the orientation of the knee, orientation of the ankle, orientation of the upper leg, and orientation of the lower leg can be calculated from the position of the center of rotation of the upper leg, position of the knee, and position of the ankle of the user. Furthermore, if the lengths of the upper leg and lower leg are set in advance, the position and orientation of the knee and the position and orientation of the ankle can also be calculated from the orientation of the upper leg and orientation of the lower leg. Therefore, the furniture-type device does not need to determine and output all of the above-mentioned postures, and may determine and output only a part of them.

[0051] Furthermore, in another embodiment, the posture estimation unit 12 may u and the length of the lower leg y u The user's height and / or eye height are estimated using statistical information on human body dimensions. u and the length of the lower leg y u The correlation between the above and the above may be stored in a table in advance.

[0052] FIG. 6 is a schematic diagram for explaining the change from a seated posture to a standing posture in the cases where a correction process is performed and where a correction process is not performed.

[0053] The posture estimation unit 12 corrects the posture of a seated user to a standing posture and outputs the corrected posture. The user shown in the upper left of FIG. 6 is seated on a chair-like furniture-type device 1, and the postures of the right leg and the left leg are different from each other. On the other hand, the virtual objects shown in the upper right and lower right of FIG. 6 are in standing postures determined from the user's seated posture based on the knee positions and ankle positions output by the furniture-type device 1. Here, the virtual object shown in the upper right of FIG. 6 has not been subjected to correction processing from the user's seated posture to a standing posture, so the right knee of this virtual object is bent in an unnatural direction. In contrast, the virtual object shown in the lower right of FIG. 6 has been subjected to correction processing to a standing posture taking into account the range of motion of the joints, resulting in a more natural standing posture of this virtual object.

[0054] As a correction process for the virtual object shown in the lower right of FIG. 6, the posture estimation unit 12 calculates the direction vector l of the lower leg. u and the direction vector of the upper leg u u From this, the flexion angle θ of the knee joint can be estimated and determined. In this case, a correction process is performed to set the range of motion of the knee flexion angle θ. By sending the corrected flexion angle θ of the knee joint to a virtual space application, the results of this correction process can be reflected in the movement of the virtual object.

[0055] When a furniture-type device is movable, such as the chair-type furniture-type device shown in FIG. 1, the posture (position and orientation) of the seat can be measured and determined by installing an IMU, a rotary encoder, a VR tracking system, etc., as disclosed in Patent Document 3, for example. Using information related to the posture of the seat, the positions and orientations of the user's body parts may be corrected and output. For example, when outputting the positions of the knees and ankles, the knee position vector k u and the ankle position vector j uIt is also possible to perform a rotation process according to the orientation of the seat, add the position of the seat as an offset, and output the result. For example, this type of correction makes it possible to convert the user's posture from relative coordinates based on the furniture-type device to absolute coordinates based on the space in which the furniture-type device exists, and output the result.

[0056] <Output of Movement Amount> The posture estimation unit 12 may determine and output the movement amount of a virtual object operated by a seated user in a virtual space based on changes in the user's posture. For example, the posture estimation unit 12 may: (a) determine and output a forward movement amount when the user steps while seated; (b) determine and output a left-right translation or rotational movement amount in the virtual space when the user moves their legs sideways; (c) determine and output a virtual crouching value, which is a boolean value, as a movement amount based on the user's knee flexion angle; or (d) determine and output a virtual jump amount when the user bends and straightens their knees within a threshold time. Conventionally, in virtual reality applications, instructions to crouch or jump have been given by, for example, pressing a button on a controller. In contrast, according to the present invention, a user can intuitively instruct a crouching posture or jump by simply moving their legs.

[0057] 7A and 7B are schematic diagrams showing a cushion-shaped furniture-type device and a mat-shaped furniture-type device, respectively, according to other embodiments.

[0058] Fig. 7(A) shows a cushion-like furniture-type device that can be placed on the seat of a chair. In this device, multiple lower leg sensors 11 are arranged on the thick part (front side) of the seat. Fig. 7(B) shows a mat-like furniture-type device that covers the entire chair and, as a result, comes into contact with the body of a seated user. In this device, multiple lower leg sensors 11 are arranged on the front curtain surface that comes close to the user's lower legs. Of course, the furniture-type device is not limited to the embodiments shown in Figs. 7(A) and 7(B), and may be, for example, a wheelchair-like furniture-type device.

[0059] FIG. 8 is a schematic diagram showing an accessory device that can be attached to furniture that can be seated.

[0060] 8 shows a pipe chair as an example of furniture on which a user can sit. Attached to this pipe chair is an accessory device 1' serving as a leg posture estimation device according to the present invention. Similar to the furniture-type device 1 (the leg posture estimation device incorporated therein) shown in FIG. 1, the accessory device 1' includes a lower leg sensor 11 and a posture estimation unit 12.

[0061] FIG. 9 is a schematic diagram showing the configuration of a system according to the present invention.

[0062] The system shown in FIG. 9 includes a furniture-type device 1 on which a user can sit and an external device 2 that communicates with the furniture-type device 1. The external device 2 may be an HMD terminal 21 or a virtual space server 22. The furniture-type device 1 includes at least one lower leg sensor 11 disposed under the seat and transmitting a depth distribution, which is an in-plane distribution of depth distances including the depth distances to the lower legs of the seated user, to the terminal. The external device 2 also includes a posture estimation unit 12 that estimates and determines the posture of the upper leg and / or lower leg of each of the seated user's left and right legs based on the depth distribution measured by the lower leg sensor 11. The external device 2 allows the user to visually recognize a virtual space in which the user moves as an avatar. In particular, when the HMD terminal 21 is worn on the user's head, the user can experience a sense of immersion in the virtual space.

[0063] A system including the furniture-type device 1 and the HMD terminal 21 can also be used, for example, for walking training as part of a user's rehabilitation. The display of the HMD terminal 21 displays a virtual space that shows the view in front of the user undergoing walking training. As a result, even if the user is seated in a chair in real space, the furniture-type device 1 determines and outputs the amount of movement, allowing the user to recognize their walking status in the virtual space and thereby recovering the walking nerves in the brain. Furthermore, by combining the posture of the lower body determined by the furniture-type device 1 with the hand tracking function of the HMD terminal 21, the user's entire body posture can be reflected in the posture of a virtual object in the virtual space.

[0064] As described above in detail, the device and system of the present invention can estimate the posture of the legs (thighs) of a seated user. For example, the present invention can provide a device and system that does not require the user to wear a sensor, reduces the possibility of violating the user's privacy by measuring the user's entire body, and does not limit the user's range of motion by a holder.

[0065] The above-described embodiments are merely illustrative of the present invention, and are not intended to limit the scope of the present invention, which can be embodied in various other modified and altered forms. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

[0066] REFERENCE SIGNS LIST 1 Furniture-type device (leg posture estimation device) 1' Accessory device (leg posture estimation device) 11 Lower leg sensor 12 Posture estimation unit 21 HMD terminal (external device) 22 Virtual space server (external device)

Claims

1. A device that is built into or can be attached to furniture on which a user can sit, comprising: at least one lower leg sensor located under the seat of the furniture for measuring a depth distribution, which is an in-plane distribution of depth distances including the depth distance to the lower legs of a seated user; and posture estimation means for determining the position of the ankle and the direction or orientation of the lower leg of the seated user as the posture of the lower leg based on the measured depth distribution, and for determining the position of the knee, or the position of the knee and the direction or orientation of the upper leg of the seated user as the posture of the upper leg based on the position of the ankle and the direction or orientation of the lower leg and the position of the hip joint determined using a preset sensor or another sensor.

2. The device according to claim 1, characterized in that it outputs, as the posture of the upper leg, the position of the knee determined based on the position vector of the knee or the orientation of the upper leg determined based on the direction vector of the upper leg, and outputs, as the posture of the lower leg, the position of the ankle determined based on the position vector of the ankle or the orientation of the lower leg determined based on the direction vector of the lower leg.

3. The apparatus described in claim 1, characterized in that the posture estimation means includes a machine learning engine that is pre-trained with teacher data in which information related to the depth distribution measured by the lower leg sensor is used as an explanatory variable, and information related to the position of the ankle and information related to the direction of the lower leg are used as objective variables, and by inputting the information related to the depth distribution to the machine learning engine, information related to the position of the ankle and information related to the direction or orientation of the lower leg are obtained as outputs.

4. The device according to claim 1, further comprising one or more seat pressure sensors for measuring a pressure distribution in the seat, and wherein the posture estimation means uses the position of the hip joint determined based on the pressure distribution.

5. The device according to claim 1, characterized in that the posture estimation means corrects the determined posture of the upper leg and the posture of the lower leg to a standing posture of a seated user and outputs the corrected posture.

6. The device according to claim 1, further comprising at least one seat posture sensor for detecting the position and orientation of the seat of the seat section, wherein the posture estimation means performs a rotation process on the posture of the upper leg and the posture of the lower leg in accordance with the detected orientation of the seat, adds the detected position of the seat as an offset, and outputs the posture of the upper leg and the posture of the lower leg as values ​​in absolute coordinates stretched in the space in which the device exists.

7. The furniture-type device according to claim 1, characterized in that the posture estimation means determines the amount of movement in the virtual space of a virtual object operated by a user based on changes in the posture of the user's upper leg and / or lower leg when seated.

8. The device according to claim 1, characterized in that it is in the form of a chair, a cushion, a mat, or a wheelchair.

9. The furniture-type equipment according to claim 1, wherein the device is incorporated into the furniture to form a furniture-type equipment together with the furniture, or the device is an accessory device that can be attached to the furniture.

10. A system having a device built into or attachable to furniture on which a user can sit, and an external device that communicates with the device, wherein the device has at least one lower leg sensor located under the seat of the furniture for measuring a depth distribution, which is an in-plane distribution of depth distances including the depth distance to the lower leg of a seated user, and the external device has posture estimation means for determining the position of the ankle and the direction or orientation of the lower leg of the seated user as the posture of the lower leg based on the measured depth distribution, and for determining the position of the knee, or the position of the knee and the direction or orientation of the upper leg of the seated user as the posture of the upper leg based on the position of the ankle and the direction or orientation of the lower leg and the position of the hip joint determined using a predetermined sensor or another sensor.

Images