Pose estimation system and pose estimation method

Abstract

To estimate a user's posture, including a direction of the user's body, using a small number of sensors.SOLUTION: A posture estimation system comprises a measurement member 1 located at any part of four limbs of a user, and a posture acquisition part 520 for acquiring the posture of the measurement member. The measurement member includes an acceleration sensor 14 and a gyro sensor 15. The posture acquisition part 520 includes a reference coordinate determination part 521 for setting a reference coordinate system of the measurement member based on the user's operation of making the measurement member face a target 3, and an attitude estimation part 522 for estimating an attitude of the measurement member relative to the target by acquiring detection values Da and Dr output from the acceleration sensor and the gyro sensor in response to the user's operation of changing the attitude of the measurement member.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for estimating a user's posture.

Background Art

[0002] There is a technique for estimating a user's posture from a plurality of sensors attached to the user's body. In Non-Patent Document 1 below, the pose of the user is estimated from nine-axis sensors (three-axis acceleration sensor, three-axis gyro sensor, three-axis geomagnetic sensor) attached to six locations on the user's body.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=X]] In Non-Patent Document 1, the values of sensors attached to other parts of the user's body are normalized by a sensor attached to the user's waist. That is, the posture of each part of the user's body is normalized with the direction in which the user's waist faces as the front direction.

[0005]

[0006] ​​The objective of this invention is to estimate the user's posture, including the orientation of their body, using a small number of sensors. [Means for solving the problem]

[0007] A posture estimation system according to one aspect of the present invention comprises a measuring member located on any part of the user's limbs and a posture acquisition unit that acquires the posture of the measuring member, wherein the measuring member includes an acceleration sensor and a gyro sensor, and the posture acquisition unit includes a reference coordinate determination unit that sets a reference coordinate system for the measuring member based on the user's action of bringing the measuring member toward a target object, and a posture estimation unit that estimates the posture of the measuring member relative to a target object by acquiring detected values ​​output from the acceleration sensor and gyro sensor in response to the user's action of changing the posture of the measuring member.

[0008] A posture estimation method according to another aspect of the present invention includes the steps of setting a reference coordinate system for a measuring member based on the user's action of bringing a measuring member located on any part of the user's limbs toward a target object, and estimating the posture of the measuring member toward a target object by acquiring detected values ​​output from an acceleration sensor and a gyro sensor provided on the measuring member in response to the user's action of changing the posture of the measuring member. [Effects of the Invention]

[0009] According to the present invention, it is possible to estimate the user's posture, including the orientation of their body, with a small number of sensors. [Brief explanation of the drawing]

[0010] [Figure 1] This is an overall diagram of the game system including the posture estimation system according to this embodiment. [Figure 2] This is a functional block diagram showing a game system including a posture estimation system according to the first embodiment. [Figure 3] This is a block diagram showing the structure of the game itself. [Figure 4]This diagram shows the three-axis coordinate system of the accelerometer and gyroscope sensors included in the controller. [Figure 5] This diagram shows the user holding the controller and performing the calibration operation. [Figure 6] This figure shows the calibration coordinate system and the controller's reference coordinate system after coordinate transformation. [Figure 7] This diagram shows the controller's reference coordinate system and the controller's current coordinate system after user interaction. [Figure 8] This is a flowchart showing the posture estimation method according to the first embodiment. [Figure 9] This is a functional block diagram showing a game system including a posture estimation system according to a second embodiment. [Figure 10] This diagram shows how user behavior differs depending on their height. [Figure 11] This is a flowchart showing the posture estimation method according to the second embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, an embodiment of the attitude estimation system and an attitude estimation system according to the present invention will be described with reference to the attached drawings.

[0012] [1] Overall structure of the game system Figure 1 is an overall diagram showing a game system 10 that utilizes the posture estimation system according to this embodiment. In this embodiment, the case in which the posture estimation system is used in the game system 10 will be described as an example.

[0013] As shown in FIG. 1, the game system 10 includes a controller 1, a game main body 2, and a display 3. The controller 1 includes a controller 1L held by the user's left hand and a controller 1R held by the user's right hand. The controllers 1L and 1R can communicate with the game main body 2 via wireless communication. The controllers 1L and 1R may be configured to communicate with the game main body 2 via wired communication. The display 3 can use, for example, a home television monitor. The display 3 is connected to the game main body 2 by an audio / video cable. The display 3 can also be configured to receive audio and video from the game main body 2 via wireless communication.

[0014] In the following description, when explaining the common functions of the left and right controllers 1L and 1R, the controller 1 is appropriately referred to. The game system 10 using the posture estimation system shown in FIG. 1 has a configuration common to the following first embodiment and second embodiment.

[0015] [2] First Embodiment (1) Functional Configuration of Game System FIG. 2 is a functional block diagram showing the game system 10 according to the first embodiment. The controller 1 includes a control unit 11, a communication unit 12, an operation unit 13, an acceleration sensor 14, and a gyro sensor 15. The controllers 1L and 1R are held by the user's left and right hands, and the progress of the game is carried out according to the operations of the controllers 1L and 1R.

[0016] The control unit 11 performs overall control of the controller 1. The communication unit 12 performs wireless communication with the game main body 2. As the wireless communication, a protocol such as Bluetooth (registered trademark) is used. The operation unit 13 is composed of various operators such as buttons and a cross cursor. The user performs various operations related to game settings, start / end instructions of the game, and progress of the game by operating the operation unit 13.

[0017] The acceleration sensor 14 is a sensor that measures the acceleration of the controller 1 in three axes. In other words, the acceleration sensor 14 can measure the acceleration in the three axes of the X, Y, and Z. The gyro sensor 15 is a sensor that measures the rotational speed of the controller 1 in three axes. In other words, the gyro sensor 15 can measure the rotational angular velocity in the three axes of the yaw, pitch, and roll. In this embodiment, the acceleration sensor 14 and the gyro sensor 15 of the controller 1 are built into the controller 1 so that their three axes coincide. These three axes are the device coordinate system of the controller 1. The acceleration sensor 14 provides the measured acceleration detection value Da to the control unit 11. The gyro sensor 15 provides the measured rotational speed detection value Dr to the control unit 11.

[0018] The game unit 2 comprises a control unit 50, a communication unit 21, an operation unit 22, and an output unit 23. Figure 3 is a block diagram showing the configuration of the game unit 2. As shown in Figure 3, the game unit 2 comprises a CPU (Central Processing Unit) 51, RAM (Random Access Memory) 52, ROM (Read Only Memory) 53, and a storage device 54. The storage device 54 stores the program GP. The CPU 51 uses the RAM 52 as a working area and executes the program GP to run the game. The images of the run game are output to the display 3 via the output unit 23. The ROM 53 stores control programs and the like. The CPU 31, RAM 52, ROM 53, and storage device 54 constitute the control unit 50.

[0019] Refer to Figure 2 again. As shown in Figure 2, the control unit 50 comprises a game control unit 510 and a posture acquisition unit 520. The control unit 50 is a functional unit realized by the CPU 51 using RAM 52 as a work area and executing the program GP. The game control unit 510 advances the game based on user operations using the controller 1. The posture acquisition unit 520 acquires the posture of each part of the user's body based on the left and right controllers 1L and 1R held by the user. The posture acquisition unit 520 comprises a reference coordinate determination unit 521, a posture estimation unit 522, a common coordinate transformation unit 523, a normalization unit 524, and an other part estimation unit 525. The controller 1 and the posture acquisition unit 520 constitute the posture estimation system according to this embodiment.

[0020] The reference coordinate determination unit 521 determines the reference coordinate system of the controller 1 when the controller 1 is calibrated by the user. Figure 4 shows the coordinate system (device coordinate system) common to the acceleration sensor 14 and gyro sensor 15 provided in the controller 1. The controller 1L on the left has a coordinate system of XL axis, YL axis, and ZL axis. The controller 1R on the right has a coordinate system of XR axis, YR axis, and ZR axis. The front surface 1F of the controller 1 is mainly the surface on which the operation unit 13 of the controller 1 is located. However, the operation unit 13 may be located on surfaces other than the front surface 1F. As shown in Figure 4, for the controller 1L on the left, the ZL axis extends perpendicularly forward relative to the front surface 1F of the controller 1L. On the other hand, for the controller 1R on the right, the ZR axis extends perpendicularly backward relative to the front surface 1F of the controller 1L. The XL axis and YL axis of the controller 1L on the left, and the XR axis and YR axis of the controller 1R on the right, extend in a plane parallel to the front surface 1F.

[0021] Figure 5 shows a user performing the calibration of controller 1. To perform the calibration of controller 1, the user holds the left and right controllers 1L and 1R with their left and right hands, as shown in the figure, and positions the front surfaces 1F of the left and right controllers 1L and 1R facing the display 3. The user also adjusts the left and right controllers 1L and 1R so that they are parallel to the vertical direction. In this embodiment, the calibration operation requires the user to perform a so-called I-pose. This positions the display 3 so that it is in the positive direction of the ZL axis of the left controller 1L and the negative direction of the ZR axis of the right controller 1R. In other words, the user aligns the left and right controllers 1L and 1R so that the front surfaces 1F are parallel to the screen of the display 3. Then, with the controllers aligned, the user operates the operation unit 13 to give a calibration command. When the reference coordinate determination unit 521 receives a calibration execution command, it sets the coordinate system of the left controller 1L (XL axis, YL axis, ZL axis) and the coordinate system of the right controller 1R (XR axis, YR axis, ZR axis) as the calibration coordinate system. In addition to being commanded by user operation, calibration may also be automatically commanded by the system. For example, calibration may be performed when an acceleration of -1G (a value that balances the acceleration due to gravity) is detected in the X-axis direction, and the acceleration in the Y-axis and Z-axis directions remains at 0 for a certain period of time.

[0022] Suppose the user precisely aligns the left and right controllers 1L and 1R so that their front surfaces 1F are parallel to the screen of display 3, and the XL and XR axes point vertically downwards. In this state, the ZL and ZR axes are perpendicular to the screen of display 3, the XL and XR axes point vertically downwards, and the YL and YR axes are parallel to the screen of display 3 and extend horizontally. The calibration coordinate systems of the left and right controllers 1L and 1R are rotated 180 degrees relative to each other within the YZ plane (horizontal plane). While this state ideally allows for calibration, in reality, each axis usually deviates slightly from the ideal state due to the user's holding position.

[0023] Next, the reference coordinate determination unit 521 performs a coordinate transformation to set the reference coordinate system, as shown in Figure 6. Figure 6 shows the coordinate transformation from the calibration coordinate system to the reference coordinate system. For the left controller 1L, the reference coordinate determination unit 521 rotates the ZL axis approximately 90 degrees counterclockwise around the YL axis so that the ZL axis points vertically upward. Similarly, for the right controller 1R, the reference coordinate determination unit 521 rotates the ZR axis approximately 90 degrees counterclockwise around the YR axis so that the ZR axis points vertically upward. If calibration is performed ideally, the reference coordinate determination unit 521 will rotate the calibration coordinate system by 90 degrees. If the calibration operation is slightly off, the reference coordinate determination unit 521 will rotate the ZL axis and ZR axis by an angle of approximately 90 degrees until they point vertically upward.

[0024] As described above, if user calibration is performed ideally, the ZL and ZR axes in the reference coordinate system will point vertically upward. The XL and XR axes will be perpendicular to the screen of display 3, and the YL and YR axes will be parallel to the screen of display 3 and extend horizontally. The reference coordinate systems of the left and right controllers 1L and 1R are rotated 180 degrees relative to each other in the XY plane (horizontal plane). If there is any deviation in the user calibration, the reference coordinate systems will also be set to a slightly different state accordingly.

[0025] The attitude estimation unit 522 estimates the attitude of the controller 1 relative to the display 3 based on the acceleration detection value Da obtained from the acceleration sensor 14 and the rotational speed detection value Dr obtained from the gyro sensor 15. Figure 7 shows the state in which the attitudes of controllers 1L and 1R change after the reference coordinate system has been set during calibration, and the controllers 1L and 1R are operated individually by the user. The coordinate system after the attitude of controllers 1L and 1R has changed is referred to as the current coordinate system. After calibration (after the reference coordinate system has been set), the attitude estimation unit 522 estimates the current attitude of controller 1 by integrating the acceleration detection value Da and the rotational speed detection value Dr output from controller 1. In other words, the attitude estimation unit 522 estimates how much the current coordinate system of controllers 1L and 1R (a coordinate system into which the reference coordinate system has been rotated due to the change in attitude) has changed relative to the reference coordinate system at the time of calibration.

[0026] Theoretically, the attitude (tilt) of controller 1 in the reference coordinate system can be calculated based on the rotational speed detection value Dr. However, estimating the attitude based solely on the rotational speed detection value Dr results in large errors. Therefore, the attitude estimation unit 522 improves the accuracy of the estimation by estimating the attitude based on the acceleration detection value Da in addition to the rotational speed detection value Dr.

[0027] The common coordinate transformation unit 523 performs a process to make the coordinate systems of the left and right controllers 1L and 1R the same. As described above, the reference coordinate systems of the left and right controllers 1L and 1R are rotated 180 degrees relative to each other in the XY plane (horizontal plane) during calibration. Therefore, by rotating the attitude of either controller 1L or 1R estimated by the attitude estimation unit 522 by 180 degrees in the XY plane of the reference coordinate system (in the plane spanned by the XR axis and YR axis or the plane spanned by the XL axis and YL axis), the coordinate systems of both can be made the same. In this embodiment, by rotating the attitude of controller 1R in the XY plane of the reference coordinate system, the attitudes of controllers 1L and 1R are represented in a common coordinate system.

[0028] Here, if we let Q(t) be the attitude estimation result of controller 1R at time t in the attitude estimation unit 522, Ps be the correction quaternion of controller 1R, and Qa(t) be the corrected attitude of controller 1R at time t, then the coordinate-transformed attitude Qa(t) of controller 1R on the right is expressed by the following equation 1. Qa(t) = Ps * Q(t) ... (Equation 1)

[0029] The normalization unit 524 normalizes the orientation of the controller 1, which has been transformed by the common coordinate transformation unit 523, according to the coordinate system of the other part estimation unit 525. In this embodiment, the reference coordinate system determined by the reference coordinate determination unit 521 matches the coordinate system of the other part estimation unit 525 due to the common coordinate transformation by the common coordinate transformation unit 523, so normalization is unnecessary. In other words, in this embodiment, the orientation of the display 3 is obtained by calibration, and the front direction (display direction) in the coordinate system of the other part estimation unit 525 becomes known, so the orientation relative to the display direction can be estimated. When a learning engine that has been machine-trained in a different coordinate system is prepared as the other part estimation unit 525, normalization processing is performed according to that coordinate system.

[0030] Here, let Qa(t) be the attitude of controllers 1L and 1R at time t after coordinate transformation, Pn be the normalized quaternion of controller 1, and Qb(t) be the attitude of controllers 1L and 1R at time t after normalization. Then, the normalized attitudes Qb(t) of the left and right controllers 1L and 1R are expressed by the following equation 2. Qb(t) = Pn * Qa(t) ... (Equation 2)

[0031] The other body part estimation unit 525 estimates the posture of other parts of the user's body based on the posture of the controllers 1L and 1R. In this embodiment, as shown in Figure 5, the user holds the left and right controllers 1L and 1R with the palms of both hands. Therefore, the posture estimation unit 522 estimates the posture of the left and right palms by calculation. The other body part estimation unit 525 estimates the posture of other parts of the user's body based on the posture of the left and right palms. In this embodiment, the other body part estimation unit 525 estimates the posture of the eight joints of the user's upper body. As a result, the other body part estimation unit 525 can obtain the posture of the user's upper body.

[0032] In this embodiment, the other-body part estimation unit 525 is composed of a machine learning model. For example, a deeply trained neural network is used as the other-body part estimation unit 525. The other-body part estimation unit 525, composed of a machine learning model, estimates the posture of two joints of the torso and three joints of the left and right arms (shoulder, upper arm, and forearm) using the posture of the left and right palms as input data. To this end, the other-body part estimation unit 525 is trained using training data in which the posture of the left and right palms is input data and the posture of two joints of the torso and three joints of the left and right arms (shoulder, upper arm, and forearm) is output data.

[0033] (2) Posture estimation method The posture estimation method in the game system 10 that utilizes the posture estimation system described above will now be explained with reference to Figure 8. Figure 8 is a flowchart of the posture estimation method according to the first embodiment. The flowchart in Figure 8 is a process realized when the CPU 51 executes the program GP. In other words, it is a process executed by the control unit 50 when the CPU 51 executes the program GP.

[0034] Before the process shown in Figure 8 begins, the user turns on the power to the game console 2. This executes the program GP. First, in step S11, the posture acquisition unit 520 requests the user to perform a calibration operation. For example, a message such as "Hold the controllers in both hands and assume an I-pose." is displayed on the display 3. Then, the reference coordinate determination unit 521 enters a waiting state for a calibration setting command.

[0035] As shown in Figure 5, the user positions the front surfaces F of the left and right controllers 1L and 1R facing the display 3, and adjusts them so that they are parallel to the vertical direction. When the user determines that the controllers 1 are in the correct position, they operate the control unit 13 of either the left or right controller 1 to give a calibration setting command. The calibration setting command is sent to the control unit 11 of the controller 1. The control unit 11 transmits the setting command to the game unit 2 via the communication unit 12. When the communication unit 21 of the game unit 2 receives the setting command, it sends the setting command to the reference coordinate determination unit 521.

[0036] When the calibration setting command is input, the reference coordinate determination unit 521 sets the reference coordinate system for controllers 1L and 1R in step S12. The reference coordinate determination unit 521 sets the reference coordinate system as explained with reference to Figure 6. Specifically, the reference coordinate determination unit 521 first sets the coordinate system of controllers 1L and 1R at the time the calibration setting command is input as the calibration coordinate system. Furthermore, the reference coordinate determination unit 521 transforms the calibration coordinate system into the reference coordinate system.

[0037] The calibration is completed with the above process. The control unit 50 displays a message on the display 3, for example, "Controller calibration is complete. You can start the game." The user starts the game by operating the control unit 13. When the user starts the game and operates the left and right controllers 1L and 1R, the acceleration sensor 14 and gyro sensor 15 output acceleration detection value Da and rotation speed detection value Dr, respectively, according to the user's movements. These output detection values ​​are transmitted wirelessly from the controllers 1L and 1R to the game unit 2.

[0038] In step S13, the attitude estimation unit 522 obtains the acceleration detection value Da and the rotational speed detection value Dr via the communication unit 21. Subsequently, in step S14, the attitude estimation unit 522 estimates the current attitude of the controllers 1L and 1R relative to the reference coordinate system by integrating the acceleration detection value Da and the rotational speed detection value Dr obtained from the time of calibration setting (when the reference coordinate system was set). In other words, the attitude estimation unit 522 estimates the current attitude of the controllers 1L and 1R with the direction of the display 3 as the forward direction.

[0039] Next, in step S15, the common coordinate transformation unit 523 rotates the orientation of the right controller 1R by 180 degrees in the XY plane of the reference coordinate system, thereby making the coordinate systems of the left and right controllers 1L and 1R the same. Subsequently, in step S16, the normalization unit 524 performs the necessary normalization processing.

[0040] Next, in step S17, the other body part estimation unit 525 receives the estimated posture of the controllers 1L and 1R relative to the display 3 and outputs the posture of the user's other body parts. In other words, based on the estimated postures of the left and right controllers 1L and 1R, the other body part estimation unit 525 estimates the posture of, for example, two joints of the trunk and three joints of the left and right arms (shoulder, upper arm, and forearm).

[0041] Through the above process, the posture acquisition unit 520 acquires the posture of each part of the user's body and provides this information to the game control unit 510. As a result, the game control unit 510 displays game images that reflect the user's movements on the display 3 and proceeds with the game.

[0042] As described above, according to the posture estimation system of this embodiment, the posture estimation system estimates the posture of the controller 1 relative to the display based on the user's calibration operation. In other words, the posture estimation system estimates the user's posture with the direction of the display 3 as the front direction. This makes it possible to estimate the user's posture including orientation (direction in the horizontal plane). For example, with conventional methods that normalize the direction that a sensor attached to the waist is facing as the front direction, it was not possible to estimate the user's posture including orientation. According to this embodiment, it is possible to estimate the user's posture including orientation, so it is highly useful.

[0043] [3] Second embodiment Next, a second embodiment of the present invention will be described. Figure 9 is a functional block diagram showing a game system 10 including a posture estimation system according to the second embodiment. The posture acquisition unit 520A of the second embodiment differs from the posture acquisition unit 520 of the first embodiment in that it includes an acceleration correction unit 526. The configuration of the game system 10 in the second embodiment is the same as the configuration of the first embodiment shown in Figure 2, except that it includes an acceleration correction unit 526. The second embodiment describes a system and method for performing more appropriate posture estimation by taking into account differences in the user's body shape.

[0044] Even when multiple users with different body types perform the same action, the acceleration detection value Da and rotational velocity detection value Dr measured based on that action will be different. For example, consider the case shown in Figure 10, where child A with arm length r(m) and adult B with arm length 2r(m) perform the same action of raising their arms from a vertical downward position to a horizontal position. In this case, the movement trajectory (rotation trajectory) of child A's arm is πr / 2(m), and the movement trajectory of adult B's arm is πr(m). In other words, even if the same action is performed over the same amount of time, the acceleration of adult B's arm is twice that of child A's arm.

[0045] Therefore, for example, if the posture of each part of a child's body is estimated using the multi-part estimation unit 525, which has been trained based on the body shape of an adult, the estimation accuracy will decrease. In order for the multi-part estimation unit 525 to perform estimations according to the body shape of users with various body shapes, it is necessary to prepare training data for each user. However, constructing the multi-part estimation unit 525 based on training data for users with various body shapes is costly and inefficient. Therefore, in the second embodiment, the multi-part estimation unit 525 is constructed by performing machine learning using training data for a user of standard height (standard user). When inferring the posture of a user, the magnitude of acceleration is corrected according to the height of each user.

[0046] The acceleration correction unit 526 corrects the acceleration detection value Da of the controller 1 received from the controller 1 based on the user's height. The control unit 50 prompts the user for their height at the start of the game and stores the user's height data in the storage device 54 or RAM 52. The acceleration correction unit 526 corrects the acceleration detection value Da according to the user's height using formula 3. Ka(t)=β*K(t) (Formula 3)

[0047] In Equation 3, K(t) is the acceleration value (magnitude) obtained from Controller 1. In other words, K(t) is the acceleration detection value Da. Ka(t) is the corrected acceleration value. β is the correction coefficient, which is the value obtained by dividing the standard height by the user's height, as shown in Equation 4. β = (reference height) / (user height) ... (Equation 4)

[0048] For example, if the standard height is 170 cm and the user's height is 100 cm, then β = 1.7. In this case, the acceleration correction unit 526 corrects the acceleration detection value Da by 1.7 times and then provides the acceleration value to the posture estimation unit 522.

[0049] Figure 11 is a flowchart showing the posture estimation method according to the second embodiment. Steps S21 to S23 are the same as steps S11 to S13 described using Figure 8. In step S24, the acceleration correction unit 526 corrects the magnitude of the acceleration output from the acceleration sensor 14 according to the user's height. The acceleration correction unit 526 provides the corrected acceleration value to the posture estimation unit 522. In step S25, the posture estimation unit 522 estimates the posture of the controller 1 relative to the display 3 based on the corrected acceleration and the rotational speed detection value Dr obtained from the gyro sensor 15. Steps S26 to S28 are the same as steps S15 to S17 described using Figure 8.

[0050] According to the second embodiment, by correcting the acceleration value based on the user's height, it is possible to utilize the other body part estimation unit 525 that has been machine-learned based on a standard height. Since machine learning only needs to be performed using data of users with a standard height, the amount of data required for learning can be significantly reduced. In addition, the time required for learning can also be significantly reduced.

[0051] [4] Modification of the second embodiment Next, a modified version of the second embodiment will be described. In the second embodiment described above, the correction coefficient β was determined by having the user input their height in advance. In this modified version, the user is asked to perform a predetermined action before the inference process begins, and the correction coefficient is determined based on the data obtained from that predetermined action.

[0052] As a standard operation, for example, the user is requested to perform the action shown in Figure 10. The user holds controllers 1L and 1R in both hands and performs the action of raising their arms, which are pointing vertically downwards, to a horizontal position. At this time, even if the speed at which the arms are raised is the same (the acceleration is the same), the time required to complete the action and the amount of change in posture will differ among multiple users of different heights. Also, even if the rotation speed is the same (i.e., the time required to complete the action and the amount of change in posture are the same), the acceleration for raising the arms will differ among multiple users of different heights. Therefore, the acceleration correction unit 526 calculates the acceleration correction coefficient β using the following formula 5. β=(Pb / Pu) / (τb / τu) (Formula 5)

[0053] In equation 5, τb is the reference time (reference time) required for a specified action. The reference time is the time measured by a user of standard height, or the time set assuming a user of standard height. τu is the time required for the user to perform the specified action. In other words, τb is a preset value, and τu is the value obtained by the specified action. (τb / τu) is the first ratio. Pb is the reference time average (reference time average) of the magnitude of the vector obtained by subtracting the gravitational acceleration component from the acceleration vector detected during the specified action (hereinafter referred to as acceleration power). The reference time average is the reference value measured by a user of standard height, or the reference value set assuming a user of standard height. Pu is the time average of the user's acceleration power during the specified action. In other words, Pb is a preset value, and Pu is the value obtained by the specified action. (Pb / Pu) is the second ratio. Thus, the correction coefficient β is expressed as the ratio of the first ratio and the second ratio. Using the correction coefficient β calculated in this way, the corrected acceleration Ka(t) can be determined by equation 3.

[0054] Here, the prescribed action was one in which the user, holding controllers 1L and 1R in both hands, raises their arms from a vertically downward position to a horizontal position. However, the prescribed action is not limited to this. Any action that reflects the user's body type in terms of the time required and the average acceleration power should be considered the prescribed action.

[0055] [5] Correspondence between each component of the claim and each element of the embodiment The following describes examples of the correspondence between each component of the claims and each element of the embodiments, but the present invention is not limited to the following examples. In the above embodiments, controllers 1L and 1R are examples of measuring members in the present invention, and controller 1L is an example of the first measuring member in the present invention. Also, display 3 is an example of a target object in the present invention.

[0056] Various elements having the configuration or function described in the claim may be used as each component of the claim.

[0057] [6] Variant In the above embodiment, the use of the posture estimation system in the execution of a game program was described as an example. As another example, the posture estimation system can be used in various application systems that operate by acquiring the user's posture.

[0058] In the above embodiment, the posture estimation system requested the user to assume an I-pose as a calibration operation. In other words, the posture estimation system requested the user to assume an I-pose as an action to face the controller 1 toward the display. The I-pose is just one example, and other poses may be used, including a pose in which the user extends their arms to the left and right.

[0059] In the above embodiment, the case in which the game body 2 is equipped with posture acquisition units 520 and 520A was described as an example. In other words, an example was described in which the posture of each part of the user's body is acquired in the game body 2 based on the acceleration detection value Da and rotational speed detection value Dr acquired from the controller 1. In another embodiment, the controller 1 may be configured to be equipped with posture acquisition units 520 and 520A.

[0060] In the above embodiment, the posture of each part of the user's body was acquired by holding controllers 1L and 1R in the user's left and right hands. In other words, the example described was when measuring members equipped with acceleration sensors and gyro sensors were positioned in the user's left and right hands. As another example, it is also possible to configure the system so that measuring members equipped with acceleration sensors and gyro sensors are positioned in a total of four locations: two locations on the user's left and right hands and two locations on the left and right feet. In this case, the measuring member held in both hands is controller 1, and measuring members can be attached to the left and right feet. This makes it possible to estimate the posture of the lower body in addition to the upper body. In the posture estimation system of this embodiment, the location of the measuring members equipped with acceleration sensors and gyro sensors is not particularly limited as long as it is in any of the user's limbs (i.e., a location other than the torso). In this embodiment, since it is not necessary to normalize the direction of the user's body using a sensor attached to the waist as in the conventional system, it is not necessary to attach sensors to the user's torso.

[0061] In the above embodiment, the axial directions of the calibration coordinate system and the reference coordinate system described with reference to Figure 6 are merely examples, and the axial directions of these coordinate systems are not particularly limited.

[0062] In this embodiment, the posture estimation system sets the calibration coordinate system and reference coordinate system of the controller 1 to the direction of the display 3 (forward direction). Therefore, if the user's body orientation changes, the same movement will be recognized as a different movement. For this reason, training data corresponding to the user's body orientation is required to construct the other body part estimation unit 525. Therefore, training data may be created based on the user's movements relative to a predetermined pseudo-direction of the display 3 (forward direction). For example, training data can be created by pseudo-assuming that the user's rightward direction is the forward direction. This reduces the time and cost required to create training data corresponding to the user's body orientation.

[0063] [7] Embodiments The posture estimation system and posture estimation method described in the above embodiment are clearly characterized by the following features.

[0064] (First aspect) The posture estimation system of the first embodiment includes a measuring member located on any part of the user's limbs and a posture acquisition unit that acquires the posture of the measuring member, the measuring member including an acceleration sensor and a gyro sensor, and the posture acquisition unit includes a reference coordinate determination unit that sets a reference coordinate system for the measuring member based on the user's action of bringing the measuring member toward a target object, and a posture estimation unit that estimates the posture of the measuring member relative to a target object by acquiring detected values ​​output from the acceleration sensor and gyro sensor in response to the user's action of changing the posture of the measuring member.

[0065] With a small number of sensors, it is possible to estimate the user's posture, including the orientation of their body.

[0066] (Second aspect) A posture estimation system according to the first embodiment, wherein measuring members are positioned on any multiple parts of the user's limbs, and the posture estimation unit estimates the posture of the multiple measuring members with respect to the target object.

[0067] It is possible to estimate the orientation of multiple measuring elements located on any of the user's limbs relative to a target object.

[0068] (Third aspect) The posture estimation system according to the second embodiment, wherein the plurality of measuring members may include two measuring members located on each of the user's left and right hands.

[0069] The orientation of the measuring members located on the left and right hands relative to the target object can be estimated.

[0070] (Fourth aspect) The posture estimation system according to the second embodiment may include two measuring members located on each of the user's left and right hands and two measuring members located on each of the left and right feet.

[0071] The orientation of the measuring members located on the left and right hands and left and right feet relative to the target object can be estimated.

[0072] (Fifth aspect) The attitude estimation system according to the second embodiment may include a common coordinate transformation unit that transforms the reference coordinate system of each of the plurality of measuring members into the reference coordinate system of a first measuring member selected from the plurality of measuring members, thereby transforming the attitude of the plurality of measuring members with respect to a target object, estimated by the attitude estimation unit, into the reference coordinate system of the first measuring member.

[0073] This makes it possible to process the orientation of multiple measurement components using a common coordinate system.

[0074] (Sixth aspect) The posture estimation system according to the first embodiment may include an other-part estimation unit that inputs the posture of the measuring member with respect to a target object estimated by the posture estimation unit and outputs the posture of other parts other than the part where the measuring member is located.

[0075] It is possible to estimate the user's posture while reducing the number of measurement elements.

[0076] (Seventh aspect) The posture estimation system according to the sixth embodiment is characterized in that measuring members are positioned on any multiple parts of the user's limbs, and the other part estimation unit receives the posture of the multiple measuring members estimated by the posture estimation unit relative to a target object and outputs the posture of other parts other than the multiple parts on which the multiple measuring members are located.

[0077] It is possible to estimate the user's posture while reducing the number of measurement elements.

[0078] (Eighth aspect) The posture estimation system according to the sixth embodiment, wherein the other body part estimation unit may be a machine learning model trained using training data.

[0079] By using machine learning models, it is possible to estimate the user's posture while reducing the number of measurement components.

[0080] (Ninth aspect) The posture estimation system according to the eighth embodiment further comprises a normalization unit that converts the reference coordinate system of the measurement member to the coordinate system of the other part estimation unit, in accordance with the coordinate system of the other part estimation unit.

[0081] It is possible to utilize the other-part estimation unit that has been trained in a different coordinate system.

[0082] (Tenth aspect) The posture estimation system according to the eighth embodiment, wherein the other body part estimation unit is trained using learning data from a reference user, which is a reference height, and the posture acquisition unit may further include an acceleration correction unit that corrects the magnitude of acceleration output from an acceleration sensor according to the user's height.

[0083] This eliminates the need to prepare training data tailored to the user's height, reducing the time and cost required for training.

[0084] (The 11th aspect) The posture estimation system according to the tenth embodiment, wherein the acceleration correction unit may correct the acceleration according to the ratio of the user's height to the reference height.

[0085] Estimation based on the user's height enables posture estimation.

[0086] (The 12th aspect) The attitude estimation system according to the tenth embodiment, wherein the acceleration correction unit may correct the acceleration based on the ratio of the first ratio to the second ratio, where the first ratio is the ratio of the time required for a specified operation by the user to the reference time required for the specified operation, and the second ratio is the ratio of the time average of the acceleration power during the specified operation to the reference time average of the acceleration power during the specified operation.

[0087] Estimation based on the user's height enables posture estimation.

[0088] (The 13th aspect) The posture estimation system according to the first embodiment, wherein the target object includes a display, and the image output to the display may change based on the posture of the measuring member relative to the display estimated by the posture estimation unit.

[0089] It is possible to estimate the user's posture toward the display.

[0090] (Aspect 14) A posture estimation system according to the 13th embodiment, wherein the measuring member includes a game controller, and the display may show a game image that changes based on the orientation of the controller relative to the display.

[0091] You can enjoy games that utilize the user's posture towards the display.

[0092] (The 15th aspect) The posture estimation method according to the 15th embodiment includes the steps of setting a reference coordinate system for a measuring member based on the user's action of bringing a measuring member located on any part of the user's limbs toward a target object, and estimating the posture of the measuring member toward a target object by acquiring detected values ​​output from an acceleration sensor and a gyro sensor provided on the measuring member in response to the user's action of changing the posture of the measuring member.

[0093] With a small number of sensors, it is possible to estimate the user's posture, including the orientation of their body.

[0094] (The 16th aspect) The posture estimation method according to the 15th embodiment may include the steps of inputting the estimated posture of the measuring member with respect to a target object and outputting the posture of a part other than the part where the measuring member is located.

[0095] It is possible to estimate the user's posture while reducing the number of measurement elements.

[0096] (Pattern 17) The posture estimation method according to the 15th embodiment, wherein the target object includes a display, and the image output to the display may change based on the estimated posture of the measuring member relative to the target object.

[0097] It is possible to estimate the user's posture toward the display.

[0098] (Aspect 18) A posture estimation method according to the 15th embodiment, which may include a step of correcting the magnitude of acceleration output from an acceleration sensor according to the user's height.

[0099] The user's posture is estimated based on their height. [Explanation of Symbols]

[0100] 10...Game system, 1(1L,1R)...Controller, 14...Accelerometer, 15...Gyroscope, 2...Game unit, 50...Control unit, 510...Game control unit, 520...Attitude acquisition unit, 521...Reference coordinate determination unit, 522...Attitude estimation unit, 523...Common coordinate transformation unit, 524...Normalization unit, 525...Other part estimation unit, 3...Display

Claims

1. A measuring element located on one of the user's limbs, A posture acquisition unit that acquires the posture of the measuring member, Equipped with, The measuring member is Accelerometer and gyroscope, Includes, The aforementioned posture acquisition unit, Based on the user's action of aligning one axis of the measuring member with the planar portion of the target object, a reference coordinate determination unit sets the reference coordinate system of the measuring member, A posture estimation unit that acquires detected values ​​output from the acceleration sensor and the gyro sensor in response to the user's action of changing the posture of the measuring member, and estimates the posture of the measuring member with respect to the target object by integrating the detected values, Includes, The measuring members are positioned on any multiple parts of the user's limbs. The attitude estimation unit estimates the attitude of the plurality of measuring members with respect to the target object. The aforementioned posture acquisition unit, A common coordinate transformation unit transforms the reference coordinate system of each of the plurality of measuring members into the reference coordinate system of the first measuring member selected from the plurality of measuring members, thereby transforming the attitude of the plurality of measuring members with respect to the target object, as estimated by the attitude estimation unit, into the reference coordinate system of the first measuring member. A posture estimation system, including the following.

2. The posture estimation system according to claim 1, wherein the plurality of measuring members include two measuring members located on each of the user's left and right hands.

3. The posture estimation system according to claim 1, wherein the plurality of measuring members include two measuring members located on each of the user's left and right hands and two measuring members located on each of the left and right feet.

4. A measuring element located on one of the user's limbs, A posture acquisition unit that acquires the posture of the measuring member, Equipped with, The measuring member is Accelerometer and gyroscope, Includes, The aforementioned posture acquisition unit, Based on the user's action of aligning one axis of the measuring member with the planar portion of the target object, a reference coordinate determination unit sets the reference coordinate system of the measuring member, A posture estimation unit that acquires detected values ​​output from the acceleration sensor and the gyro sensor in response to the user's action of changing the posture of the measuring member, and estimates the posture of the measuring member with respect to the target object by integrating the detected values, A body part estimation unit receives the posture of the measuring member relative to the target object estimated by the posture estimation unit and outputs the posture of other parts of the user's body other than the part where the measuring member is located. Includes, The aforementioned other body part estimation unit is a posture estimation system that uses a machine learning model trained with training data that takes the posture of the aforementioned body part as input and outputs the posture of the aforementioned other body part.

5. The measuring members are positioned on any multiple parts of the user's limbs. The posture estimation system according to claim 4, wherein the other part estimation unit receives the posture of the plurality of measuring members with respect to the target object estimated by the posture estimation unit and outputs the posture of other parts other than the plurality of parts where the plurality of measuring members are located.

6. The aforementioned posture acquisition unit, A normalization unit that converts the reference coordinate system of the measuring member to the coordinate system of the other part estimation unit by rotating the reference coordinate system of the measuring member so that it matches the coordinate system of the other part estimation unit. The posture estimation system according to claim 4, further comprising:

7. The aforementioned other body part estimation unit is trained using training data in which the posture of the aforementioned body part of a standard user, which is the standard height, is taken as input and the posture of the aforementioned other body part is taken as output. The aforementioned posture acquisition unit, An acceleration correction unit corrects the magnitude of acceleration output from the acceleration sensor by multiplying the acceleration estimated by the machine learning model corresponding to the aforementioned reference user by a coefficient corresponding to the user's height. The posture estimation system according to claim 4, further comprising:

8. The acceleration correction unit is, The posture estimation system according to claim 7, wherein the acceleration is corrected according to the ratio of the user's height to the standard height.

9. The acceleration correction unit is, The posture estimation system according to claim 7, wherein the ratio of the time required for a specified operation by the user to a reference time required for the specified operation is defined as a first ratio, and the ratio of the time average of the acceleration power during the specified operation by the user to the reference time average of the acceleration power during the specified operation is defined as a second ratio, and the acceleration is corrected based on the ratio of the first ratio to the second ratio.

10. A measuring element located on one of the user's limbs, A posture acquisition unit that acquires the posture of the measuring member, Equipped with, The measuring member is Accelerometer and gyroscope, Includes, The aforementioned posture acquisition unit, Based on the user's action of aligning one axis of the measuring member with the planar portion of the target object, a reference coordinate determination unit sets the reference coordinate system of the measuring member, A posture estimation unit that acquires detected values ​​output from the acceleration sensor and the gyro sensor in response to the user's action of changing the posture of the measuring member, and estimates the posture of the measuring member with respect to the target object by integrating the detected values, Includes, The aforementioned target object includes a display, The display shows images of a game that progresses in response to the user's operation of the measuring member. A posture estimation system in which the image output to the display changes based on the posture of the measuring member relative to the display estimated by the posture estimation unit.

11. The measuring member includes a game controller, The posture estimation system according to claim 10, wherein the display shows a game image that changes based on the posture of the controller relative to the display.

12. A method for estimating posture performed by a computer, A step of setting the reference coordinate system of the measuring member based on the user's action of aligning one axis of the measuring member, located on any part of the user's limbs, with the planar surface of the target object; The process involves acquiring detected values ​​output from the acceleration sensor and gyro sensor provided on the measuring member in response to the user's action of changing the orientation of the measuring member, and estimating the orientation of the measuring member with respect to the target object by integrating the detected values. Includes, The measuring members are positioned on any multiple parts of the user's limbs. The step of estimating the orientation involves estimating the orientation of the plurality of measuring members with respect to the target object, A posture estimation method that converts the reference coordinate system of each of the plurality of measuring members to the reference coordinate system of a first measuring member selected from the plurality of measuring members, thereby converting the posture of the plurality of measuring members with respect to the target object, which was estimated in the posture estimation step, to the reference coordinate system of the first measuring member.

13. The process includes inputting the estimated orientation of the measuring member relative to the target object and estimating other body parts, which output the orientation of other body parts of the user other than the part where the measuring member is located. The posture estimation method according to claim 12, wherein the step of estimating the other body part uses a machine learning model trained with training data that takes the posture of the body part as input and outputs the posture of the other body part.

14. The aforementioned target object includes a display, The display shows images of a game that progresses in response to the user's operation of the measuring member. The posture estimation method according to claim 12, wherein the image output to the display changes based on the estimated posture of the measuring member relative to the target object.

15. The posture estimation method is A step of estimating other body parts, inputting the posture of the measuring member relative to the target object estimated by the posture estimation step, and outputting the posture of other body parts of the user other than the part where the measuring member is located. Includes, The process of estimating the other body part involves a machine learning model that takes the posture of the body part as input and estimates the posture of the other body part, and is trained using training data that takes the posture of the body part of a reference user, whose height is a standard, as input and outputs the posture of the other body part. The aforementioned posture estimation method is: A step of correcting the magnitude of acceleration output from the acceleration sensor by multiplying the acceleration estimated by the machine learning model corresponding to the aforementioned reference user by a coefficient corresponding to the user's height. The posture estimation method according to claim 12, including the method described in claim 12.

16. A method for estimating posture performed by a computer, A step of setting the reference coordinate system of the measuring member based on the user's action of aligning one axis of the measuring member, located on any part of the user's limbs, with the planar surface of the target object; The process involves acquiring detected values ​​output from the acceleration sensor and gyro sensor provided on the measuring member in response to the user's action of changing the orientation of the measuring member, and estimating the orientation of the measuring member with respect to the target object by integrating the detected values. A step to estimate other body parts, inputting the posture of the measuring member relative to the target object estimated in the posture estimation step, and outputting the posture of other body parts of the user other than the part where the measuring member is located, Includes, The step of estimating the other body part is a posture estimation method that uses a machine learning model trained with training data that takes the posture of the body part as input and outputs the posture of the other body part.

17. A method for estimating posture performed by a computer, A step of setting the reference coordinate system of the measuring member based on the user's action of aligning one axis of the measuring member, located on any part of the user's limbs, with the planar surface of the target object; The process involves acquiring detected values ​​output from the acceleration sensor and gyro sensor provided on the measuring member in response to the user's action of changing the orientation of the measuring member, and estimating the orientation of the measuring member with respect to the target object by integrating the detected values. Includes, The aforementioned target object includes a display, The display shows images of a game that progresses in response to the user's operation of the measuring member. A posture estimation method wherein the image output to the display changes based on the posture of the measuring member relative to the display estimated in the step of estimating the posture.

Images