Information processor, whole-body posture estimation method, and program

Abstract

To provide an information processor, a whole-body posture estimation method, and a program that estimate a whole-body posture of a user from a posture of a part of the body without separately adding a sensor.SOLUTION: An information processor detects at least postures (positions) of shoulders of a user to acquire them as sensing information, estimates, on the basis of the sensing information, a posture (position) of at least an upper body including upper limbs of the user, estimates a posture (position) of a lower body of the user based on the posture of the upper body, and outputs information on a whole body posture (position of each part) of the user based on the postures of the upper body and the posture of the lower body.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a whole-body posture estimation method, and a program. [Background technology]

[0002] In fitness activities that involve the whole body (such as yoga), it is important to move each part of the body to the appropriate position to maximize the fitness effect. Furthermore, with the recent development of virtual reality (VR) devices such as head-mounted displays (HMDs), fitness activities performed in 3D virtual environments such as the metaverse have emerged. Even in VR fitness, it is necessary to grasp the user's overall body posture (the position of each part) in order to exercise effectively, just as in real life.

[0003] For example, as shown in Figure 11, a full tracking system in which multiple sensors (e.g., acceleration sensors) are attached to the entire body can accurately track the movements of each part of the body, but individual sensors may have a negative impact on exercise, such as restricting the user's movements and requiring effort to wear.

[0004] For example, Patent Document 1 describes a technology for estimating the user's entire body posture based on the user's head posture based on sensor data captured by an HMD, the wrist posture measured by a controller, and the user's elbow posture obtained from an image captured by the HMD camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special table number 2023-542809 Summary of the Invention [Problem to be solved by the invention]

[0006] In the system described in Patent Document 1, in order to estimate the entire body posture of the user, it is necessary to determine the wrist posture based on the posture of the controller held by the user and the estimated relationship between the controller and the user's wrist. For this reason, a controller is required, and it is difficult to estimate the entire body posture of the user with an HMD that does not include a controller.

[0007] An object of the present disclosure is to provide an information processing device, a whole-body posture estimation method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present disclosure includes a sensing unit that detects at least the posture of a user's shoulders and acquires it as sensing information, an upper body posture estimation unit that estimates at least the posture of the user's upper body based on the sensing information, a lower body posture estimation unit that estimates the posture of the user's lower body based on the posture of the upper body, and an output unit that outputs information on the user's whole body posture based on the posture of the upper body and the posture of the lower body.

[0009] A whole-body posture estimation method according to one aspect of the present disclosure is characterized by including the steps of: detecting the posture of at least a user's shoulders and acquiring it as sensing information; estimating the posture of at least the user's upper body based on the sensing information; estimating the posture of the user's lower body based on the posture of the upper body; and outputting information on the user's whole-body posture based on the posture of the upper body and the posture of the lower body.

[0010] ○○ according to one aspect of the present disclosure causes a computer of an information processing device to function as a sensing function that detects at least the posture of a user's shoulders and acquires it as sensing information, an upper body posture estimation function that estimates at least the posture of the user's upper body based on the sensing information, a lower body posture estimation function that estimates the posture of the user's lower body based on the posture of the upper body, and an output function that outputs information on the posture of the user's whole body based on the posture of the upper body and the posture of the lower body. [Effects of the Invention]

[0011] According to the above aspect, it is possible to estimate the entire posture of the user's body from the posture of one part of the body without adding a separate sensor. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram of an information processing device according to the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating a configuration of a data storage unit of an information processing device according to the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating a configuration of an information processing unit of an information processing device according to the present disclosure. [Figure 4] 10 is a flowchart illustrating an operation of an information processing device according to the present disclosure. [Figure 5] 10 is a flowchart illustrating an operation of an information processing device according to the present disclosure. [Figure 6] FIG. 1 is a schematic diagram for explaining the exclusion effect on fine movements of a human being when immobile. [Figure 7] 1 is a conceptual diagram showing an actual posture and an estimated posture during sensing by an information processing device according to the present disclosure. [Figure 8] FIG. 10 is a block diagram illustrating another example configuration of an information processing device according to the present disclosure. [Figure 9] 10 is a flowchart illustrating an operation of an information processing device according to the present disclosure. [Figure 10] FIG. 10 is a block diagram illustrating another example configuration of an information processing device according to the present disclosure. [Figure 11] FIG. 1 is a hardware configuration diagram of an information processing device according to the present disclosure. [Figure 12] FIG. 1 is a conceptual diagram for explaining a method for estimating a whole-body posture by full tracking. DETAILED DESCRIPTION OF THE INVENTION

[0013] Each embodiment will be described below with reference to the drawings. The same or corresponding components in all drawings are denoted by the same reference numerals, and common descriptions will be omitted. In the following description, the "posture" of a body part includes not only the position of the part, but also the movement of the part obtained by tracking the continuously changing position. Therefore, acquiring the "posture" of a part at a certain point in time means acquiring the position of the part at that point in time, and continuously acquiring the "posture" of a part means acquiring the movement of the part. Furthermore, the "position" of each part refers to, for example, the relative position with respect to the head, trunk, etc.

[0014] First Embodiment Hereinafter, an embodiment according to the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram of an information processing device according to the present disclosure. As shown in FIG. 1, the information processing device (HMD) 1 includes multiple sensors 10a to 10c, a data storage unit 11, an information processing unit 12, and an output unit 13. The sensors 10a and 10b are cameras provided on the left and right lower portions of the exterior of the HMD 1, and sequentially capture images including at least the shoulders as part of the user's body. The sensor 10c is an acceleration sensor for head tracking built into the HMD 1, and sequentially detects the movement (angle, acceleration) of the user's head. The image data including a specific part of the user's body (shoulders) captured by the sensors 10a and 10b, and the data detected by the sensor 10c (hereinafter collectively referred to as sensing data) are stored in the data storage unit 11.

[0015] The sensors 10a to 10c are sensors originally installed in the HMD 1. The sensors 10a and 10b may be any type of sensor other than a camera, such as an infrared camera, a TOF (Time of Flight) sensor, or a Lidar (Light Detection and Ranging) sensor, as long as it can detect the posture (position) of a specific part (shoulder) of the user's body. In the illustrated example, two sensors 10a and 10b are used as cameras, but this is not limiting. A single sensor, or a combination of three or more sensors, such as a camera, an infrared camera, a TOF sensor, or a Lidar sensor, may also be used. The sensor 10c does not need to be an acceleration sensor, and a camera such as the sensors 10a and 10b may also be used as long as it can detect the movement of the user's head from image data captured by the surroundings. Only the sensors 10a and 10b may also be used as long as it can detect the movement of the user's head from images captured by the sensors 10a and 10b.

[0016] The information processing unit 12 reads the sensing data from the data storage unit 11, estimates the posture of the user's forearms and upper arms based on the posture (position) of the user's shoulders, further identifies the posture of the upper body from the posture of the forearms and upper arms, estimates the posture of the user's lower body from the posture of the upper body, and finally estimates the entire body posture of the user (position of each part). The information processing unit 12 supplies information related to the estimated entire body posture of the user (position of each part) to the output unit 13 and stores it in the data storage unit 11. The estimated entire body posture (position of each part) includes the posture (position) of parts such as the upper body (head, trunk, upper limbs) and lower body (lower limbs) in addition to the user's hands, which were not photographed by the sensors 10a and 10b.

[0017] The output unit 13 is a display device in the HMD 1, and displays a human figure (such as an avatar) that simulates the user based on the whole-body posture estimated by the information processing unit 12. For example, the human figure (such as an avatar) that simulates the whole-body posture of the user may be superimposed on the VR space displayed on the output unit 13.

[0018] In the first embodiment, the output unit 13 is the display device of the HMD 1, but this is not limited to this and the output may be to another information processing device via wired or wireless communication, and the other information processing device may perform processing (display, analysis, machine learning, etc.) using data on the user's whole body posture.

[0019] Fig. 2 is a block diagram showing the configuration of a data storage unit of an information processing device according to the present disclosure. As shown in Fig. 2, the data storage unit 11 is composed of a sensor information storage unit 20, a log storage unit 21, and a model storage unit 22. The sensor information storage unit 20 stores sensing information from the sensors 10a and 10b and exchanges data with the information processing unit 12. The log storage unit 21 associates and stores the sensing data from the sensors 10a to 10c with data on the whole-body posture estimated by the information processing unit 12. The model storage unit 22 stores model data used to estimate the whole-body posture.

[0020] The model data is generated by learning sensing data of a specific body part when one body part (e.g., the upper body) is moved, and sensing data of other body parts (forearm, upper arm, and other body parts). As an example, the model data is generated by learning the relationship between sensing data recording changes in the position information of the right wrist when the right wrist is stretched straight above the body and sensing data of other body parts (including the right shoulder, right upper arm, forearm, and legs). Although one model data that learns the transitions of each body part is assumed, one model data may be provided for each body part. By inputting images captured by sensors 10a and 10b into the model data, the output (states of other body parts) corresponding to the posture of the shoulder shown in the image (position, shape, etc. in the image) can be determined. If the posture of the shoulders in the captured image is known, the posture (position) of the forearms and upper arms can be determined, if the posture of the forearms and upper arms is known, the posture (position) of the upper body can be identified, and if the posture (position) of the upper body is identified, the posture (position) of the lower body can also be identified. Ultimately, the posture (position) of each part of the user's entire body can be estimated.

[0021] The model data may be generated (updated) as appropriate by the HMD 1 of the present disclosure based on data on the estimated whole-body posture, or may be generated (updated) by another information processing device. For example, when using images, the model data may be generated by learning the relationship between an image of the shoulder and the posture of the lower body corresponding to that image. Furthermore, when using information from a sensor other than a camera, the model data may be generated by learning the relationship between the angle, angular velocity, position, etc. indicated by the sensor and the posture of the lower body, etc.

[0022] Fig. 3 is a block diagram showing the configuration of an information processing unit of an information processing device according to the present disclosure. As shown in Fig. 3, the information processing unit 12 is composed of an estimation unit 30. The estimation unit 30 reads out sensing data from the sensor information storage unit 20 and estimates the user's whole body posture (the position of each part) using the above-mentioned model data.

[0023] 4 is a flowchart for explaining the operation (main routine) of the information processing device according to the present disclosure. While the VR software or the like is running, the sensors 10a and 10b sequentially capture (as video) images of the surroundings of the user, including at least the shoulders as part of the user's body, and the sensor 10c sequentially detects the movement of the head of the user wearing the HMD 1.

[0024] The information processing unit 12 determines whether or not a user's movement (change in the sensing data) is detected based on the sensing data from the sensors 10a to 10c (step S10). If a user's movement is detected (YES in step S10), the information processing unit 12 stores the sensing data from the sensors 10a to 10c in the sensor information storage unit 20 (step S12).

[0025] Next, the estimation unit 30 of the information processing unit 12 estimates the whole body posture of the user (the position of each part) using the sensing data stored in the sensor information storage unit 20 and the model data stored in the model storage unit 22 (step S14). After that, the information processing unit 12 supplies information related to the whole body posture of the user (the position of each part), which is the estimation result, to the output unit 13 (step S16).

[0026] 5 is a flowchart for explaining the operation of the information processing device according to the present disclosure (the operation of the estimation unit 30). The estimation unit 30 reads the stored sensing data from the sensor information storage unit 20 (step S20). At this time, the estimation unit 30 reads the sensing data only for a predetermined specified time (e.g., 0.5 seconds). Next, the estimation unit 30 uses the model data in the model storage unit 22 to estimate the current whole-body posture of the user (the position of each part) from the sensing data (step S22).

[0027] Next, the estimation unit 30 determines whether the estimation accuracy of the estimated body part is n% (e.g., 90%) or more (step S24). If the estimation accuracy of the estimated body part is not n% (e.g., 90%) or more (NO in step S24), the estimation unit 30 determines that the relevant part (e.g., forearm) is not moving even if there is a change due to the sensing data (step S26). This process excludes minute body movements, such as those shown in FIG. 6, from the "movement" when the user is motionless, thereby preventing the estimated whole body posture (position of each part) from being inadvertently reflected in the display. Thereafter, the estimation unit 30 associates the sensing data used for the estimation with the estimation result (the whole body posture of the user (position of each part)) and stores them in the log storage unit 21 (step S32).

[0028] On the other hand, when the user extends the right leg widely to the right and compares the estimation result with the sensing data at the start of system operation, if the estimation accuracy for the body part of the estimation result is n% (e.g., 90%) or more (YES in step S24), the estimation unit 30 determines that the relevant part (e.g., forearm) has moved (step S28). Next, the estimation unit 30 supplies the estimation result (information related to the user's entire body posture (position of each part)) to the output unit 13 (step S30). The output unit 13 displays, for example, a human figure (e.g., an avatar) simulating the user taking the estimated entire body posture (position of each part). Thereafter, the sensing data used for the estimation and the estimation result (posture (position) of each part of the user's body) are linked and stored in the log storage unit 21 (step S32).

[0029] 7(a) and 7(b) are conceptual diagrams showing the actual posture and estimated posture during sensing by an information processing device according to the present disclosure. For example, this can be used in VR fitness content, such as yoga, that requires confirmation of the posture of the entire body. As shown in FIG. 7(a), a user wears an HMD 1 equipped with an acceleration sensor (corresponding to sensor 10c in FIG. 1) for head tracking and cameras (corresponding to sensors 10a and 10b in FIG. 1) that capture the front and bottom of the body (shoulders and floor), and assumes a case where the user is using yoga software. When the user slowly raises their right foot horizontally to the right from an upright position, the head tracking sensor 10c (acceleration sensor) senses the position of the user's head, and sensors 10a and 10b (cameras) disposed below the HMD 1 sense the posture (position) of the user's shoulders, as indicated by the diagonal lines.

[0030] Next, the posture (position) of both arms (forearms and upper arms) connected to the shoulders is estimated from the sensing data. Once the position of the head, shoulders, and arms is determined, the posture (position) of the trunk (upper body) is uniquely determined. Next, the posture (position) of the lower body (lower limbs) is estimated based on the determined posture (position) of the upper body. Finally, the user's overall posture (position of each part) is estimated, as shown in Figure 7(b). The VR yoga software uses the estimated overall posture (position of each part) to determine whether the instructor's posture (position of each part) and the user's posture (position of each part) match, and outputs the result (e.g., true or false) (this may be communicated by sound, voice, etc.).

[0031] In addition, by displaying the instructor's basic movements (posture) and the whole-body posture of a human figure (such as an avatar) simulating the user, generated based on the estimated whole-body posture (position of each part), side by side on the output unit 13 within the HMD 1, the user can easily check whether the movements (posture) match the basic movements (posture) of the instructor teaching yoga poses in the VR space, and if they do not, can easily correct the posture.

[0032] According to the HMD 1 (information processing device) of the first embodiment, the posture (position) of a specific body part (shoulder) is detected by sensors 10a and 10b built into the HMD 1, the posture (position) of the forearm and upper arm is estimated from the posture (position) of the shoulder using model data previously trained through machine learning, the posture (position) of the upper body is identified from the estimated posture (position) of the forearm and upper arm, and the posture (position) of the lower body is estimated from the posture (position) of the upper body using the model data. Therefore, the entire body posture (position of each part) can be estimated without attaching individual sensors to specific body parts. Furthermore, since there is no need to attach individual sensors to specific body parts, there is no restriction on the user's movements. Furthermore, there is no need to install a separate sensor to detect the posture (position) of the shoulder, which is a body part, which leads to cost reduction.

[0033] Second Embodiment Hereinafter, an embodiment according to the present disclosure will be described with reference to the drawings. FIG. 8 is a block diagram showing another example configuration of an information processing device according to the present disclosure. As shown in FIG. 8, in the second embodiment, the data storage unit 11 includes a dictionary data storage unit 23 instead of the model storage unit 22. The dictionary data storage unit 23 holds dictionary data in which the posture (position, angle, etc.) of each linked body part is associated with the movement of each body part and recorded. For example, the angle of the forearm and upper arm relative to the shoulder angle is recorded (in increments of 1 degree), such as what is the angle between the forearm and upper arm when the shoulder angle is 0 degrees, and what is the angle between the forearm and upper arm when the shoulder angle is 1 degree. Similarly, the angle of the upper body is recorded relative to the angle of the forearm and upper arm, and the angle of the lower body is recorded relative to the angle of the upper body. As an example, assume dictionary data for the movement of the right shoulder when the right arm is raised horizontally to the right in advance. The dictionary data records the transition of sensing data for each angle of the right arm, indicating how minutely the right shoulder moves when the right arm is raised. By using this dictionary data, rather than using model data, data that is closest to the dictionary data is searched for in a dictionary search, making it possible to estimate the movement of the right arm from the movement of the right shoulder.

[0034] When the estimation unit 30 of the information processing unit 12 detects that the user's body has moved from the sensing data from the sensors 10a to 10c, it searches the sensing data before and after the movement in the dictionary data stored in the dictionary data storage unit 23, and supplies the closest result to the output unit 13 as the posture of the linked body parts.

[0035] 9 is a flowchart for explaining the operation of the information processing device according to the present disclosure. The estimation unit 30 reads sensing data for only a predetermined specified time (e.g., 0.5 seconds) stored in the sensor information storage unit 20 (step S50). Next, the estimation unit 30 searches the dictionary data in the dictionary data storage unit 23 for data obtained by subtracting the start time of the movement from the end time of the movement, i.e., the sensing data before and after the movement (step S52). If the closest sensing data is found (YES in step S52), the estimation unit 30 supplies the result closest to the search result, i.e., information regarding the user's whole body posture (the position of each part), to the output unit (step S54).

[0036] According to the HMD 1 (information processing device) of the second embodiment, the user's whole body posture (position of each part) can be estimated without using model data generated by machine learning, and therefore response performance can be improved.

[0037] Third Embodiment Hereinafter, an embodiment according to the present disclosure will be described with reference to the drawings. 10 is a block diagram showing another example configuration of an information processing device according to the present disclosure. The information processing device (HMD) 100 includes a sensor 101 that detects at least the posture (position) of the shoulders of a user and acquires it as sensing information, an upper body posture estimation unit 102 that estimates the posture (position) of at least the upper body including the upper limbs of the user based on the sensing information, a lower body posture estimation unit 103 that estimates the posture (position) of the lower body of the user based on the posture of the upper body, and an output unit 104 that outputs information on the whole body posture of the user (position of each part) based on the posture of the upper body and the posture of the lower body.

[0038] According to the information processing device (HMD) 100 of the third embodiment, the posture (position) of a specific part of the body (shoulder) is detected by the sensor 101, the posture (position) of the forearm and upper arm linked to the posture of the shoulder is estimated by the upper body posture estimation unit 102, the posture (position) of the upper body linked to the estimated posture of the forearm and upper arm is identified, and further, the posture (position) of the lower body linked to the posture (position) of the upper body is estimated by the lower body posture estimation unit 103, so that the position of each part of the entire body can be estimated without attaching an individual sensor to each movable part of the body.

[0039] FIG. 11 is a hardware configuration diagram of an information processing device according to the present disclosure. As shown in FIG. 11, the information processing device (HMD) 1 is a computer equipped with various hardware components such as a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a display unit 204, and a sensor 205.

[0040] In the above-described embodiment, the explanation is based on the assumption that an HMD worn on the user's head is used, but this is not limited to this. The posture (position) of the user's shoulders may be detected by a sensor worn on the head, and the posture (position) of other parts (upper limbs, trunk, lower limbs) may be estimated from the posture (position) of the shoulders.

[0041] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0042] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0043] (Appendix 1) An information processing device comprising: a sensing unit that detects at least the posture of a user's shoulders and acquires it as sensing information; an upper body posture estimation unit that estimates the posture of at least the user's upper body based on the sensing information; a lower body posture estimation unit that estimates the posture of the user's lower body based on the posture of the upper body; and an output unit that outputs information on the user's whole body posture based on the posture of the upper body and the posture of the lower body.

[0044] (Appendix 2) The information processing device described in Appendix 1 includes a housing that is worn on the user's head, and the sensing unit includes a sensor arranged in the housing and acquires the sensing information from the sensor.

[0045] (Appendix 3) The information processing device according to claim 2, wherein the sensor is an imaging means for capturing an image of the user's surroundings, including at least the user's shoulders.

[0046] (Appendix 4) The information processing device according to any one of claims 1 to 3, wherein the upper body posture estimation unit estimates at least the posture of the upper body of the user based on sensing data information for only a predetermined specified time period.

[0047] (Appendix 5) An information processing device as described in any one of Appendices 1 to 4, further comprising a movement determination unit that determines that a specific part has moved when the estimated accuracy of the user's whole body posture is n% or more, and wherein the output unit outputs information about the user's whole body posture when the movement determination unit determines that the specific part has moved.

[0048] (Appendix 6) The information processing device according to any one of appendices 1 to 5, wherein the upper body posture estimation unit estimates the posture of at least the user's forearm and upper arm from the posture of at least the shoulders of the user using model data that has been machine-learned in advance to determine the interconnectedness of each part of the body, and identifies the posture of the upper body from the estimated postures of the forearm and upper arm, and the lower body posture estimation unit estimates the posture of the lower body from the posture of the upper body using the model data.

[0049] (Appendix 7) The information processing device according to any one of appendices 1 to 6, wherein the upper body posture estimation unit uses dictionary data in which the interlocking relationships between each part of the body are linked in advance to estimate the posture of at least the user's forearms and upper arms from the posture of the shoulders, and estimates the posture of the upper body from the estimated postures of the forearms and upper arms, and the lower body posture estimation unit uses the dictionary data to estimate the posture of the lower body from the posture of the upper body.

[0050] (Appendix 8) 8. The information processing device according to claim 1, further comprising a log storage unit that stores the sensing information and information on the user's whole body posture in association with each other.

[0051] (Appendix 9) A whole-body posture estimation method comprising the steps of: detecting at least a posture of a user's shoulders and acquiring the posture as sensing information; estimating at least a posture of the user's upper body based on the sensing information; estimating a posture of the user's lower body based on the posture of the upper body; and outputting information on the whole-body posture of the user based on the posture of the upper body and the posture of the lower body.

[0052] (Appendix 10) 10. The whole-body posture estimation method according to claim 9, wherein the step of acquiring the sensing information acquires the sensing information from a sensor arranged in a housing worn on the user's head.

[0053] (Appendix 11) 11. The whole-body posture estimation method according to claim 10, wherein the step of acquiring sensing information involves using the sensor to capture an image of the user's surroundings, including at least the user's shoulders.

[0054] (Appendix 12) 12. The whole-body posture estimation method according to any one of appendices 9 to 11, wherein the step of estimating the posture of the user's upper body estimates the posture of at least the user's upper body based on sensing data information for only a predetermined specified time.

[0055] (Appendix 13) 13. The whole-body posture estimation method according to any one of appendices 9 to 12, further comprising a step of determining that a specific part has moved when the estimation accuracy for the user's whole-body posture is n% or more, and wherein the step of outputting information about the user's whole-body posture outputs the information about the user's whole-body posture when it is determined that the specific part has moved.

[0056] (Appendix 14) The method for estimating the posture of the user's upper body comprises estimating the posture of the forearms and upper arms from the posture of at least the shoulders of the user using model data that has been previously machine-learned to determine the interconnections between each part of the body, and identifying the posture of the upper body from the estimated postures of the forearms and upper arms; and the method for estimating the posture of the user's lower body comprises estimating the posture of the lower body from the posture of the upper body using the model data.

[0057] (Appendix 15) The method for estimating the posture of the user's upper body comprises estimating the posture of the forearms and upper arms from the posture of at least the user's shoulders using dictionary data in which interlocking relationships between each part of the body are linked in advance, and estimating the posture of the upper body from the estimated postures of the forearms and upper arms; and the step of estimating the posture of the user's lower body comprises estimating the posture of the lower body from the posture of the upper body using the dictionary data.

[0058] (Appendix 16) 16. The whole-body posture estimation method according to any one of claims 9 to 15, further comprising the step of linking the sensing information with information on the user's whole-body posture and storing the information in a log storage unit.

[0059] (Appendix 17) A program that causes a computer of an information processing device to function as a sensing function that detects at least the posture of a user's shoulders and acquires it as sensing information, an upper body posture estimation function that estimates the posture of at least the user's upper body based on the sensing information, a lower body posture estimation function that estimates the posture of the user's lower body based on the posture of the upper body, and an output function that outputs information on the posture of the user's entire body based on the posture of the upper body and the posture of the lower body.

[0060] (Appendix 18) The program described in Appendix 17 includes a housing that is worn on the user's head, and the sensing function includes a sensor arranged in the housing and acquires the sensing information from the sensor.

[0061] (Appendix 19) The program described in Appendix 18, wherein the sensing function uses the sensor to capture an image of the user's surroundings, including at least the user's shoulders.

[0062] (Appendix 20) The program described in any one of Appendices 17 to 19, characterized in that the upper body posture estimation function estimates at least the posture of the user's upper body based on sensing data information for only a predetermined specified time period.

[0063] (Appendix 21) The program described in any one of Appendices 17 to 20, further comprising a movement determination function that determines that a specific part has moved when the estimated accuracy of the user's whole body posture is n% or more, and the output function that outputs information about the user's whole body posture when the movement determination function determines that the specific part has moved.

[0064] (Appendix 22) The upper body posture estimation function estimates the posture of at least the user's forearm and upper arm from the posture of the shoulder using model data that has been machine-learned in advance to determine the interconnectedness of each part of the body, and identifies the posture of the upper body from the estimated posture of the forearm and upper arm; and the lower body posture estimation function estimates the posture of the lower body from the posture of the upper body using the model data.

[0065] (Appendix 23) The upper body posture estimation function uses dictionary data in which the interrelationships between each part of the body are linked in advance to estimate the posture of at least the user's forearms and upper arms from the posture of their shoulders, and estimates the posture of the upper body from the estimated postures of the forearms and upper arms; and the lower body posture estimation function uses the dictionary data to estimate the posture of the lower body from the posture of the upper body.

[0066] (Appendix 24) The program according to any one of appendices 17 to 23, further comprising a log storage function that links the sensing information with information on the user's whole body posture and stores the information in a log storage unit. [Explanation of symbols]

[0067] 1, 100 Information processing equipment 10a~10c, 101 sensors 11 Data storage unit 12 Information Processing Department 13 Output section 20 Sensor information storage section 21 Log storage unit 22 Model storage area 23 Dictionary data storage section 30 Estimation part 102 Upper body posture estimation section 103 Lower body posture estimation section 104 Output section

Claims

1. a sensing unit that detects at least the posture of the user's shoulders and acquires the posture as sensing information; an upper body posture estimation unit that estimates a posture of at least the upper body of the user based on the sensing information; a lower body posture estimation unit that estimates a lower body posture of the user based on the upper body posture; an output unit that outputs information about the user's whole body posture based on the upper body posture and the lower body posture; An information processing device comprising:

2. a housing to be worn on the user's head, The information processing apparatus according to claim 1 , wherein the sensing unit includes a sensor disposed in the housing, and acquires the sensing information from the sensor.

3. 3. The information processing apparatus according to claim 2, wherein the sensor is an image capturing unit that captures an image of the user's surroundings, including at least the user's shoulders.

4. The information processing apparatus according to claim 1 , wherein the upper body posture estimation unit estimates at least the posture of the upper body of the user based on sensing data information only for a predetermined specified time period.

5. a movement determination unit that determines that a specific part has moved when the estimation accuracy for the user's whole body posture is n% or more; The information processing apparatus according to claim 1 , wherein the output unit outputs information about the user's whole body posture when the movement determination unit determines that the specific part has moved.

6. the upper body posture estimation unit estimates postures of the forearms and upper arms from the posture of at least the user's shoulders using model data that has been previously machine-learned to determine the interlocking relationships between body parts, and identifies the posture of the upper body from the estimated postures of the forearms and upper arms; The information processing apparatus according to claim 1 , wherein the lower body posture estimation unit estimates the lower body posture from the upper body posture using the model data.

7. the upper body posture estimation unit estimates postures of the forearms and upper arms from postures of at least the user's shoulders using dictionary data in which interlocking relationships between body parts are linked in advance, and estimates postures of the upper body from the estimated postures of the forearms and upper arms; The information processing apparatus according to claim 1 , wherein the lower body posture estimation unit estimates the lower body posture from the upper body posture using the dictionary data.

8. The information processing apparatus according to claim 1 , further comprising a log storage unit that stores the sensing information and the information on the user's whole body posture in association with each other.

9. detecting at least the posture of the shoulders of the user and acquiring the posture as sensing information; estimating a posture of at least the upper body of the user based on the sensing information; estimating a lower body posture of the user based on the upper body posture; outputting information about the user's whole body posture based on the upper body posture and the lower body posture; A whole-body posture estimation method comprising:

10. The computer of the information processing device a sensing function that detects at least the posture of the user's shoulders and acquires the posture as sensing information; an upper body posture estimation function that estimates at least the posture of the upper body of the user based on the sensing information; a lower body posture estimation function that estimates the posture of the lower body of the user based on the posture of the upper body; an output function for outputting information on the user's whole body posture based on the upper body posture and the lower body posture; A program characterized by functioning as

Images