Analytical device, analytical method, storage medium storing program, and calibration method

Abstract

The present invention provides an analysis device, an analysis method, a storage medium storing a program, and a correction method. The analysis device includes: an acquisition unit that acquires an image captured by a camera unit, the camera unit capturing one or more first markers assigned to an estimated object; and a correction unit that corrects a transformation rule from a sensor coordinate system to a segment coordinate system based on the image. The first marker has a form in which its relative posture relative to at least one of a plurality of inertial measurement sensors does not change, and its posture relative to the camera unit can be identified by analyzing the captured image. The correction unit derives the posture of the first marker relative to the camera unit, derives a transformation matrix from the sensor coordinate system to the camera coordinate system based on the derived posture, and uses the derived transformation matrix from the sensor coordinate system to the camera coordinate system to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

Description

Technical Field

[0001] The present invention relates to an analysis device, an analysis method, a storage medium storing a program, and a correction method. Background Art

[0002] Conventionally, the following technology (motion capture) has been disclosed: multiple inertial measurement units (IMU sensors) capable of measuring angular velocity and acceleration are attached to the body to estimate body posture and its changes (movement) (see, for example, Patent Document 1).

[0003] [Prior art literature]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2020-42476 Summary of the Invention

[0006] [Problems to be solved by the invention]

[0007] Inference technology using IMU sensors sometimes uses a calibration rule to transform the IMU sensor's output into a specific coordinate system in an initial posture when the IMU sensor is attached to the subject's body. However, depending on the subject's subsequent movements, the IMU sensor's attachment position or posture may change after calibration, making the conversion rule inappropriate.

[0008] The present invention has been made in consideration of such circumstances, and one object of the present invention is to provide an analysis device, an analysis method, a program, and a calibration method that can appropriately perform calibration related to posture estimation using an IMU sensor.

[0009] [Technical means to solve the problem]

[0010] The analyzing device, analyzing method, storage medium storing a program, and calibration method of the present invention have the following configurations.

[0011] (1): An analysis device according to an embodiment of the present invention includes: a posture estimation unit for performing posture estimation of an estimated object, wherein the posture estimation of the estimated object includes a process of transforming the output of an inertial measurement sensor represented by a sensor coordinate system into a segment coordinate system, wherein the sensor coordinate system is based on the respective positions of a plurality of inertial measurement sensors installed at a plurality of parts of the estimated object and detecting angular velocity and acceleration, and the segment coordinate system represents the posture of each segment corresponding to the position of the estimated object at which the inertial measurement sensor is installed; and an acquisition unit for acquiring an image captured by a camera unit, wherein the camera unit captures one or more first marks assigned to the estimated object. ; and a correction unit, which corrects the transformation rule from the sensor coordinate system to the segment coordinate system based on the image, the first marker has the following form: the relative posture relative to at least one of the multiple inertial measurement sensors does not change, and the posture relative to the camera unit can be identified by analyzing the captured image, the correction unit derives the posture of the first marker relative to the camera unit, derives the transformation matrix from the sensor coordinate system to the camera coordinate system based on the derived posture, and uses the derived transformation matrix from the sensor coordinate system to the camera coordinate system to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

[0012] (2): In the embodiment of (1), the camera unit also captures a second marker, which is stationary in the space where the estimated object exists, and the second marker has the following form: by analyzing the captured image, the posture relative to the camera unit can be identified, and the correction unit derives the posture of the second marker relative to the camera unit, and derives the transformation matrix from the global coordinate system representing the space to the camera coordinate system based on the derived posture, and regards the segment coordinate system and the global coordinate system as the same, thereby deriving the transformation matrix from the sensor coordinate system to the segment coordinate system based on the transformation matrix from the sensor coordinate system to the camera coordinate system and the transformation matrix from the global coordinate system to the camera coordinate system, and correcting the transformation rule from the sensor coordinate system to the segment coordinate system based on the derived transformation matrix from the sensor coordinate system to the segment coordinate system.

[0013] (3): In the embodiment of (1) or (2), the camera unit also captures a third mark assigned to the estimated object, and the third mark has the following form: the relative posture relative to at least one of the segments does not change, and the posture relative to the camera unit can be identified by analyzing the captured image, the correction unit derives the posture of the third mark relative to the camera unit, derives the transformation matrix from the segment coordinate system to the camera coordinate system based on the derived posture, derives the transformation matrix from the sensor coordinate system to the segment coordinate system based on the transformation matrix from the sensor coordinate system to the camera coordinate system, and the transformation matrix from the segment coordinate system to the camera coordinate system, and corrects the transformation rule from the sensor coordinate system to the segment coordinate system based on the derived transformation matrix from the sensor coordinate system to the segment coordinate system.

[0014] (4): In another embodiment of the analysis method of the present invention, a computer performs the following operations: performing posture estimation of an estimated object, the posture estimation of the estimated object includes processing of converting the output of an inertial measurement sensor represented by a sensor coordinate system into a segment coordinate system, the sensor coordinate system is based on the respective positions of a plurality of inertial measurement sensors installed at a plurality of parts of the estimated object and detecting angular velocity and acceleration, the segment coordinate system represents the posture of each segment corresponding to the position of the estimated object at which the inertial measurement sensor is installed; acquiring an image taken by a camera unit, the camera unit taking one or more first marks assigned to the estimated object; And based on the image, the transformation rule from the sensor coordinate system to the segment coordinate system is corrected, and the first marker has the following form: the relative posture relative to at least one of the multiple inertial measurement sensors does not change, and the posture relative to the camera unit can be identified by analyzing the captured image. In the correction process, the posture of the first marker relative to the camera unit is derived, and the transformation matrix from the sensor coordinate system to the camera coordinate system is derived based on the derived posture. The derived transformation matrix from the sensor coordinate system to the camera coordinate system is used to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

[0015] (5): Another embodiment of the present invention is a storage medium storing a program, wherein the program causes a computer to perform the following operations: performing posture estimation of an estimated object, wherein the posture estimation of the estimated object includes processing of converting the output of an inertial measurement sensor represented by a sensor coordinate system into a segment coordinate system, wherein the sensor coordinate system is based on the respective positions of a plurality of inertial measurement sensors installed at a plurality of parts of the estimated object and detecting angular velocity and acceleration, and the segment coordinate system represents the posture of each segment corresponding to the position of the estimated object at which the inertial measurement sensor is installed; acquiring an image captured by a camera unit, wherein the camera unit captures one or more first segments assigned to the estimated object. a marker; and based on the image, correcting the transformation rule from the sensor coordinate system to the segment coordinate system, and the first marker has the following form: the relative posture relative to at least one of the multiple inertial measurement sensors does not change, and the posture relative to the camera unit can be identified by analyzing the captured image, in the correction process, the posture of the first marker relative to the camera unit is derived, and the transformation matrix from the sensor coordinate system to the camera coordinate system is derived based on the derived posture, and the transformation rule from the sensor coordinate system to the segment coordinate system is corrected using the derived transformation matrix from the sensor coordinate system to the camera coordinate system.

[0016] (6): The correction method of another embodiment of the present invention is to photograph one or more first marks assigned to the estimated object by the camera unit mounted on the unmanned aerial vehicle, and the analysis device of the embodiments (1) to (3) obtains the image photographed by the camera unit and corrects the transformation rule from the sensor coordinate system to the segment coordinate system.

[0017] (7): The correction method of another embodiment of the present invention is to photograph one or more first marks assigned to the estimated object by the camera unit installed on the stationary object, and the analysis device of the embodiments (1) to (3) obtains the image photographed by the camera unit and corrects the transformation rule from the sensor coordinate system to the segment coordinate system.

[0018] (8): The correction method of another embodiment of the present invention is to photograph one or more first marks assigned to the estimated object by the camera unit installed on the estimated object, and the analysis device of the embodiments (1) to (3) obtains the image photographed by the camera unit and corrects the transformation rule from the sensor coordinate system to the segment coordinate system.

[0019] [Effects of the Invention]

[0020] According to the embodiments (1) to (8) described above, the IMU sensor can be calibrated appropriately. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 FIG. 1 is a diagram showing an example of a usage environment of the analysis device 100 .

[0022] Figure 2 2 is a diagram showing an example of the arrangement of the IMU sensor 40 .

[0023] Figure 3 2 is a diagram showing an example of a more detailed configuration and function of the posture estimating unit 120 .

[0024] Figure 4 This is a diagram for explaining the plane assumption processing performed by the correction unit 160 .

[0025] Figure 5 3 is a diagram for explaining the process of defining the direction vector vi performed by the correction unit 160 .

[0026] Figure 6 This is a diagram showing a situation in which the direction vector vi rotates due to a change in the posture of the estimation target TGT.

[0027] Figure 7 This is a diagram for explaining an overview of correction processing performed by the analysis device 100 .

[0028] Figure 8 3 is a diagram showing an example of the configuration of the whole body correction amount calculation unit 164 .

[0029] Figure 9 It is a diagram showing another example of the configuration of the whole body correction amount calculation unit 164 .

[0030] Figure 10 It is a diagram schematically showing the entirety of the whole-body correction amount calculation unit 164 .

[0031] Figure 11 It is a diagram for explaining the processing flow of the whole body correction amount calculation unit 164 in stages.

[0032] Figure 12 It is a diagram for explaining the processing flow of the whole body correction amount calculation unit 164 in stages.

[0033] Figure 13 It is a diagram for explaining the processing flow of the whole body correction amount calculation unit 164 in stages.

[0034] Figure 14 A diagram showing an example of the appearance of the first marker Mk1.

[0035] Figure 15 IM1 is a diagram showing an example of a captured image IM1.

[0036] Figure 16This is a diagram for explaining the processing contents of the correction unit 180 .

[0037] Figure 17 IM2 is a diagram showing an example of a captured image IM2.

[0038] Figure 18 A diagram for explaining a modification example (first) of the method for acquiring a captured image.

[0039] Figure 19 A diagram for explaining a second modification of the method for acquiring a captured image.

[0040] [Explanation of Symbols]

[0041] 10: Terminal device

[0042] 30: Measurement equipment

[0043] 40: Inertial Measurement Sensor

[0044] 50, 50A, 50B: Camera device

[0045] 100: Analytical device

[0046] 110: Ministry of Communications

[0047] 120: Posture estimation unit

[0048] 130: First Acquisition Section

[0049] 140: Primary conversion unit

[0050] 150: Integral

[0051] 160: Correction Department

[0052] 170: Second Acquisition Section

[0053] 180: Correction Department

[0054] 190: Storage DETAILED DESCRIPTION

[0055] Hereinafter, embodiments of the analysis device, analysis method, program, and calibration method of the present invention will be described with reference to the accompanying drawings.

[0056] The analysis device is implemented by at least one processor. For example, the analysis device is a service server that communicates with a user's terminal device via a network. Alternatively, the analysis device may be a terminal device with an application installed. The following description assumes that the analysis device is a server.

[0057] The analysis device acquires detection results from multiple inertial sensors (IMU sensors) attached to an estimated object, such as a human body, and uses these results to estimate the object's posture. The estimated object is not limited to the human body, as long as it includes segments (such as arms, hands, legs, and feet, which are considered rigid bodies in analytical mechanics, in other words, links) and joints connecting two or more segments. In other words, the estimated object can be humans, animals, or robots with limited ranges of motion of their joints.

[0058] <First embodiment>

[0059] Figure 1 1 is a diagram showing an example of a usage environment of the analysis device 100. The terminal device 10 is a smartphone, a tablet terminal, a personal computer, or the like. The terminal device 10 communicates with the analysis device 100 via a network NW. The network NW includes a wide area network (WAN) or a local area network (LAN), the Internet, a cellular network, or the like. The imaging device 50 is, for example, an unmanned aerial vehicle (UAV) equipped with a camera (camera). The imaging device 50 is operated by, for example, the terminal device 10, and sends the captured image to the analysis device 100 via the terminal device 10. The image captured by the imaging device 50 is used by the correction unit 180. This will be described below.

[0060] The IMU sensor 40 is attached to, for example, the measurement equipment 30 worn by the user serving as the estimation target. The measurement equipment 30 may be, for example, a piece of easily movable sportswear with multiple IMU sensors 40 attached. Alternatively, the measurement equipment 30 may be a simple wearable device such as a gum band, swimsuit, or supporter with multiple IMU sensors 40 attached.

[0061] The IMU sensor 40 is, for example, a sensor that detects acceleration and angular velocity along three axes. The IMU sensor 40 includes a communication device that wirelessly transmits the acceleration or angular velocity detected in conjunction with the application to the terminal device 10. If the measurement equipment 30 is worn by the user, the corresponding part of the user's body for each IMU sensor 40 is automatically determined (hereinafter referred to as configuration information).

[0062] [About the Analyzing Device 100]

[0063] The analysis device 100 includes, for example, a communication unit 110, a posture estimation unit 120, a second acquisition unit 170, and a correction unit 180. The posture estimation unit 120 includes, for example, a first acquisition unit 130, a primary transformation unit 140, an integration unit 150, and a correction unit 160. These components are implemented by executing a program (software) on a hardware processor such as a central processing unit (CPU). Some or all of these components may also be implemented by hardware (circuitry, including circuitry) such as a large-scale integrated circuit (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be implemented through the collaboration of software and hardware. The program can be pre-stored on a storage device (including non-transitory storage media) such as a hard disk drive (HDD) or flash memory, or can be stored on a removable storage medium (non-transitory storage media) such as a digital versatile disc (DVD) or a compact disc read-only memory (CD-ROM) and installed by attaching the storage medium to a drive. Furthermore, the analysis device 100 includes a storage unit 190. Storage unit 190 is implemented by an HDD, flash memory, random access memory (RAM), or the like.

[0064] The communication unit 110 is a communication interface such as a network card for accessing the network NW.

[0065] [Posture Estimation Processing]

[0066] An example of the posture estimation process performed by the posture estimation unit 120 will be described below. Figure 2 This diagram shows an example of the configuration of the IMU sensors 40. For example, IMU sensors 40-1 through 40-N (N is the total number of IMU sensors) are attached to multiple locations on the user's head, chest, pelvic area, and limbs. Hereinafter, the user wearing the measurement equipment 30 will sometimes be referred to as the estimated target TGT. Furthermore, the parameter i is used to represent any of 1 through N, and is referred to as the IMU sensor 40-i, etc. Figure 2In the example shown in FIG. 3 , a heart rate sensor or a temperature sensor is also installed in the measuring equipment 30 .

[0067] For example, the IMU sensors 40 are arranged so that IMU sensor 40-1 is located on the right shoulder, IMU sensor 40-2 is located on the right upper arm, IMU sensor 40-8 is located on the left thigh, and IMU sensor 40-9 is located below the left knee. Furthermore, IMU sensor 40-p is attached to the periphery of a reference site. The reference site is, for example, a portion of the user's torso, such as the pelvis. In the following description, the site where one or more IMU sensors 40 are attached and whose movement is measured is referred to as a "segment." A segment includes the reference site and sensor attachment sites other than the reference site (hereinafter referred to as the reference site).

[0068] In the following description, components corresponding to the IMU sensors 40 - 1 to 40 -N are described by adding hyphens to the reference numerals.

[0069] Figure 3 This figure shows an example of a more detailed structure and function of the posture estimation unit 120. The first acquisition unit 130 acquires angular velocity and acceleration information from the plurality of IMU sensors 40. The primary conversion unit 140 converts the information acquired by the first acquisition unit 130 from the three-axis coordinate system of each IMU sensor 40 (hereinafter referred to as the sensor coordinate system) into information in the segment coordinate system and outputs the conversion result to the correction unit 160.

[0070] The primary conversion unit 140 includes, for example, a segment angular velocity calculation unit 146-i and an acceleration aggregation unit 148 corresponding to each segment. The segment angular velocity calculation unit 146-i converts the angular velocity of the IMU sensor 40i output by the first acquisition unit 130 into information in a segment coordinate system. The so-called segment coordinate system is a coordinate system that represents the posture of each segment. The processing result obtained by the segment angular velocity calculation unit 146-i (based on the detection result of the IMU sensor 40, which is information representing the posture of the estimated target TGT) is stored in the form of a quaternion, for example. In addition, expressing the measurement result of the IMU sensor 40-i in the form of a quaternion is only an example, and other expression methods such as the rotation matrix of the three-dimensional rotation group SO3 can also be used.

[0071] The acceleration aggregation unit 148 aggregates the accelerations detected by the IMU sensors 40 - i corresponding to the segments, and converts the aggregated results into the acceleration of the entire body of the estimated target TGT (hereinafter sometimes referred to as total IMU acceleration).

[0072] Integrator 150 integrates the angular velocity corresponding to the segment, which has been converted to the reference coordinate system by segment angular velocity calculator 146-i, to calculate the orientation of the segment in the estimation target TGT where IMU sensor 40-i is mounted as part of the estimated target posture. Integrator 150 outputs the integration result to correction unit 160 and storage unit 190.

[0073] In addition, in the integration unit 150, when the processing cycle is the first time, the angular velocity output by the primary transformation unit 140 (the angular velocity that has not been corrected by the correction unit 160) is input, but thereafter, the angular velocity reflecting the correction is input, and the correction is derived by the correction unit 160 described later based on the processing result of the previous processing cycle.

[0074] The integration unit 150 includes, for example, an angular velocity integration unit 152-i corresponding to each segment. The angular velocity integration unit 152-i integrates the angular velocity of the segment output by the segment angular velocity calculation unit 146-i to calculate the orientation of the reference portion of the estimated target, to which the IMU sensor 40-i is attached, as part of the estimated target's posture.

[0075] The correction unit 160 assumes a representative plane passing through the reference part included in the estimation target and corrects the transformed angular velocity of the reference part so that the normal of the representative plane is approximately orthogonal to the orientation of the reference part calculated by the integration unit 150. The representative plane will be described below.

[0076] The correction unit 160 includes, for example, an estimated posture aggregation unit 162 , a whole-body correction amount calculation unit 164 , a correction amount decomposition unit 166 , and angular velocity correction units 168 - i corresponding to each segment.

[0077] The estimated posture aggregation unit 162 aggregates the calculation results obtained by the angular velocity integration unit 152 - i , that is, the quaternions representing the posture of each segment, into one vector.

[0078] The whole-body correction amount calculation unit 164 calculates correction amounts for the angular velocities of all segments based on the total IMU acceleration output by the acceleration aggregation unit 148 and the estimated whole-body posture vector output by the estimated posture aggregation unit 162. Furthermore, the correction amounts calculated by the whole-body correction amount calculation unit 164 are adjusted by taking into account the relationships between the segments so that the whole-body posture being estimated does not appear unnatural. The whole-body correction amount calculation unit 164 outputs the calculation results to the correction amount decomposition unit 166.

[0079] The correction amount decomposition unit 166 decomposes the correction amount calculated by the whole-body correction amount calculation unit 164 into a correction amount for the angular velocity of each segment so that it can be reflected in the angular velocity of each segment. The correction amount decomposition unit 166 outputs the decomposed correction amount for each segment's angular velocity to the angular velocity correction unit 168-i for the corresponding segment.

[0080] Angular velocity correction unit 168-i reflects the decomposition result of the correction amount for the corresponding segment's angular velocity, output by correction amount decomposition unit 166, on the calculation result of the angular velocity for each segment, output by segment angular velocity calculation unit 146-i. As a result, in the next processing cycle, the object of integration by integrator 150 becomes the angular velocity reflecting the correction performed by correction unit 160. Angular velocity correction unit 168-i outputs the correction result to angular velocity integrator 152-i.

[0081] The result of estimation of the posture of each segment, which is the integration result obtained by the integration unit 150 , is transmitted to the terminal device 10 .

[0082] Figure 4 160 is a diagram for explaining the plane assumption processing performed by the correction unit 160. Figure 4 As shown in the left figure of FIG, when the reference part is the pelvis of the estimated target TGT, the correction unit 160 assumes that the median sagittal plane (sagittal plane in the figure) passing through the center of the pelvis is used as the representative plane. The so-called median sagittal plane refers to a plane that is parallel to the center of the body of the estimated target TGT, which is symmetrical on both sides and divides the body into left and right. Figure 4 As shown in the right figure of FIG, the correction unit 160 sets the normal line n (the arrow in the figure represents the normal vector) of the assumed median sagittal plane.

[0083] Figure 5 This diagram illustrates the process of defining the direction vector vi performed by the correction unit 160. The correction unit 160 uses the output of a particular IMU sensor 40-i as the initial state and defines its orientation as horizontal and parallel to the representative plane (first correction process). The direction vector then rotates in three directions along the three-directional rotation obtained by integrating the output of the IMU sensor 40-i.

[0084] like Figure 5As shown, when the reference parts of the estimated target TGT include the chest, left and right thighs, and left and right below the knees, the correction unit 160 estimates the installation posture of the IMU sensor 40 based on the results of the first correction process. The transformed angular velocities of the reference parts are corrected so that the normal n and the orientation of the reference parts calculated by the integration unit 150 are approximately orthogonal to each other. As shown in the figure, direction vectors v1 to v5 (forward vectors in the figure) for the orientation of the reference parts are derived. As shown in the figure, direction vector v1 represents the direction vector of the chest, direction vectors v2 and v3 represent the direction vectors of the thighs, and direction vectors v4 and v5 represent the direction vectors of the below the knees. Furthermore, the x-axis, y-axis, and z-axis in the figure are examples of directions of a reference coordinate system.

[0085] Figure 6 This diagram illustrates the situation in which the direction vector vi rotates due to changes in the posture of the estimation target TGT. When the output of the IMU sensor 40-p at a certain reference location is set to the initial state, the representative plane rotates in the yaw direction along the displacement in the yaw direction obtained by integrating the output of the IMU sensor 40-p. The correction unit 160 increases the degree of correction of the converted angular velocity of the reference location as the orientation of the reference location calculated by the integrator 150 in the previous cycle continues to deviate from the orientation perpendicular to the normal n to the midsagittal plane.

[0086] [Posture Estimation]

[0087] The correction unit 160 is, for example, Figure 5 When the inner product of the direction vector vi of the reference part and the normal line n is 0 as shown, it is determined that the orientation of the reference part does not deviate from the orientation perpendicular to the normal line n of the midsagittal plane, and the posture of the home position is determined as follows. Figure 6 If the inner product of the direction vector vi and the normal n is greater than 0, as shown, the reference part is determined to be oriented away from an orientation perpendicular to the normal n of the midsagittal plane. The so-called home position refers to the basic posture of the estimated target TGT (however, it is a relative posture with respect to the representative plane) obtained as a result of the first calibration process after the IMU sensor 40 is attached to the estimated target TGT. For example, it is a static upright position. The correction unit 160 defines the home position based on the measurement results of the IMU sensor 40 obtained by performing a predetermined action (calibration action) on the estimated target TGT.

[0088] Thus, the correction unit 160 estimates that the subject rarely maintains a posture that deviates from the orientation perpendicular to the normal line n with respect to the midsagittal plane for a long time (ie, Figure 6The correction is made based on the assumption that the deviation becomes smaller (closer to) as time passes, based on the assumption that the body is twisted as shown in FIG. 1 ), or that the body is moved in a posture that rarely deviates from the normal line n to the midsagittal plane. Figure 5 original position as shown).

[0089] Figure 7 This figure is used to explain the outline of the correction process performed by the analysis device 100. The analysis device 100 defines different optimization problems for the pelvis of the estimation target TGT and other segments. First, the analysis device 100 calculates the pelvic posture of the estimation target TGT and uses the pelvic posture to calculate the postures of other segments.

[0090] If the pelvic posture is calculated separately from the postures of other segments besides the pelvis, the pelvic posture is estimated using only gravity correction. The analysis device 100 simultaneously estimates the pelvic posture and the postures of other segments, allowing the pelvic posture to be estimated while also considering the postures of the other segments. This aims to optimize the estimation by taking into account the mutual influence of all IMU sensors 40.

[0091] [Calculation Example]

[0092] A specific calculation example when estimating the posture will be described below using mathematical formulas.

[0093] The following describes how to express a quaternion for expressing a posture. If the rotation from a coordinate system frame A to a coordinate system frame B is expressed as a quaternion, it becomes the following equation (1): Frame B is rotated by θ around a normalized axis relative to Frame A.

[0094] [Number 1]

[0095]

[0096] In the following description, the number obtained by annotating the quaternion q with a hat symbol (the unit quaternion representing the rotation) is represented as "q(h)." The so-called unit quaternion is a quaternion divided by its norm. q(h) is a column vector with four real-valued elements, as shown in Equation (1). If the estimated whole-body posture vector Q of the estimated target TGT is represented using this representation method, it can be expressed as shown in the following Equation (2).

[0097] [Number 2]

[0098]

[0099] also, S E q(h)i (i is an integer from 1 to N representing a segment, or p representing a reference position.) The rotation from the reference position of the IMU sensor 40's coordinate system S (segment coordinate system) to the reference coordinate position E (for example, a coordinate system definable based on the direction of Earth's gravity) is expressed as a quaternion. The estimated whole-body posture vector Q of the estimated target TGT is a column vector having 4(N+1) real-valued elements, which is a combination of all the unit quaternions representing the segment posture.

[0100] In order to estimate the posture of the estimation target TGT, first, the posture estimation of a certain segment on which the IMU sensor 40 is attached is considered.

[0101] [Number 3]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] Equation (3) is an example of an update equation for an optimization problem. It is used to derive correction amounts for the roll and pitch directions by deriving the minimum value of ½ of the norm of the function derived from Equation (4). The right side of Equation (4) is an equation that subtracts the reference direction measured by the IMU sensor 40, expressed in the sensor coordinate system, from information indicating the direction in which the reference should exist (e.g., the direction of gravity or the Earth's magnetic field), expressed in the sensor coordinate system, from information expressing the direction in which the reference should exist, obtained from the estimated posture.

[0108] As shown in formula (5), S E q is the unit quaternion expressed in matrix form S E An example of q(h). In addition, as shown in formula (6), E d(h) is a vector representing the reference direction (e.g., the direction of gravity or the earth's magnetic field) used to correct the yaw direction. In addition, as shown in equation (7), S s(h) represents a vector of a reference direction measured by the IMU sensor 40 expressed in the sensor coordinate system.

[0109] Furthermore, when gravity is used as a reference, equations (6) and (7) can be expressed as equations (8) and (9) below. ax, ay, and az represent the acceleration in the x-axis direction, the acceleration in the y-axis direction, and the acceleration in the z-axis direction, respectively.

[0110] E d(h)=〔0 0 0 1〕

[0111] S S(h)=〔0 a x a y a z 〕…(9)

[0112] The relationship shown in formula (3) can be solved by, for example, the gradient descent method. In this case, the updated formula for the estimated posture can be expressed by formula (10). In addition, the gradient of the objective function is expressed using the following formula (11). In addition, formula (11) representing the gradient can be calculated using the Jacobian formula as shown in formula (12). In addition, the Jacobian formula shown in formula (12) is a matrix obtained by partially differentiating the gravity error term and the yaw direction error term with each element of the direction vector vi of the whole body. The gravity error term and the yaw direction error term will be described below.

[0113] [Number 4]

[0114]

[0115]

[0116]

[0117] As shown on the right side of formula (10), the unit quaternion S E q(h) k+1 The current estimated posture can be represented by a unit quaternion S Eq (h) k The gradient is derived by subtracting the product of the coefficient μ (a constant equal to or less than 1). In addition, as shown in equations (11) and (12), the gradient can be derived with a relatively small amount of calculation.

[0118] Furthermore, actual calculation examples of equations (4) and (12) using gravity as a reference are shown in the following equations (13) and (14).

[0119] [Number 5]

[0120]

[0121]

[0122] Using the method represented by equations (3) to (7) and (10) to (12) in the figure above, the posture can be estimated by calculating the update equation once for each sampling. Furthermore, when gravity is used as a reference, as in equations (8), (9), (13), and (14), corrections can be made in the roll and pitch axis directions.

[0123] [Calculation of whole body correction]

[0124] The following describes a method for deriving a whole-body correction amount (particularly, a correction amount in the yaw direction) for the estimated posture. Figure 8 164 is a diagram showing an example of the configuration of the whole-body correction amount calculator 164. The whole-body correction amount calculator 164 includes, for example, a yaw direction error term calculator 164a, a gravity error term calculator 164b, an objective function calculator 164c, a Jacobian equation calculator 164d, a gradient calculator 164e, and a correction amount calculator 164f.

[0125] The yaw direction error term calculation unit 164 a calculates a yaw direction error term for correcting the yaw angle direction based on the estimated whole-body posture.

[0126] The gravity error term calculation unit 164 b calculates a gravity error term for achieving correction in the roll axis direction and the pitch axis direction based on the estimated whole body posture and the acceleration detected by the IMU sensor 40 .

[0127] The objective function calculation unit 164c calculates an objective function based on the estimated whole-body posture, the acceleration detected by the IMU sensor 40, the calculation results of the yaw error term calculation unit 164a, and the calculation results of the gravity error term calculation unit 164b. This objective function is used to correct the estimated target object TGT so that its midsagittal plane is parallel to the direction vector vi. The objective function is the sum of the squares of the gravity error term and the yaw error term. Details of the objective function will be described below.

[0128] The Jacobian equation calculation unit 164 d calculates the Jacobian equation obtained by partial differentiation of the estimated whole-body posture vector Q based on the estimated whole-body posture and the acceleration detected by the IMU sensor 40 .

[0129] The gradient calculation unit 164 e uses the calculation results obtained by the objective function calculation unit 164 c and the calculation results obtained by the Jacobian equation calculation unit 164 d to derive a solution to the optimization problem and calculate a gradient.

[0130] The correction amount calculation unit 164 f derives a whole-body correction amount to be applied to the estimated whole-body posture vector Q of the estimation target TGT using the calculation result of the gradient calculation unit 164 e .

[0131] Figure 9 It is a diagram showing another example of the configuration of the whole body correction amount calculation unit 164 . Figure 9 The whole body correction amount calculation unit 164 shown in FIG. 1 uses the midsagittal plane and the direction vector vi of each segment to derive the whole body correction amount, except Figure 8 In addition to the structural elements shown, a representative plane normal calculation unit 164g and a segment vector calculation unit 164h are also included.

[0132] The representative plane normal calculation unit 164g calculates the normal n of the median sagittal plane as the representative plane based on the estimated whole body posture. The segment vector calculation unit 164h calculates the direction vector vi of the segment based on the estimated whole body posture.

[0133] [Example of deriving the whole body correction amount]

[0134] An example of deriving the whole-body correction amount will be described below.

[0135] The yaw direction error term calculation unit 164a uses the following equation (15) to perform inner product calculation of the yaw direction error term fb for performing correction so that the median sagittal plane and the direction vector of the segment become parallel.

[0136] [Number 6]

[0137]

[0138] Yaw direction error term f b is the unit quaternion representing the estimated posture of segment i S E q(h) i , and the unit quaternion representing the estimated posture of the pelvis as the reference part S E q(h) p The right side of equation (15) derives the inner product of the normal n of the median sagittal plane expressed in the sensor coordinate system calculated by the representative plane normal calculation unit 164g and the direction vector vi of the segment expressed in the sensor coordinate system calculated by the segment vector calculation unit 164h. Thus, in the case of the state of the body twisting of the estimated object TGT, the correction content can be increased to eliminate the twist (close to Figure 5 The original position as shown in the figure) is corrected.

[0139] Next, the gravity error term calculation unit 164b calculates a reference correction (eg, gravity correction) for each segment as shown in equation (16).

[0140] [Number 7]

[0141]

[0142] Formula (16) is the unit quaternion representing the estimated posture of any segment i S E q(h) i The relationship between , and the acceleration (gravity) measured by the IMU sensor 40-i is shown on the right side of equation (16) and can be derived as follows: the direction in which gravity should exist in the sensor coordinate system (the assumed gravity acceleration direction) obtained based on the estimated posture is subtracted from the measured gravity direction (the measured gravity acceleration direction) expressed in the sensor coordinate system. S a i (h).

[0143] Here, the measured gravity direction S a i A specific example of (h) is shown in equation (17). E d g (h) can be expressed by constants such as those shown in formula (18).

[0144] [Number 8]

[0145]

[0146]

[0147] Next, the objective function calculation unit 164c calculates equation (19) which is a correction function for segment i by integrating the gravity error term and the yaw direction error term.

[0148] [Number 9]

[0149]

[0150] Here, c i is the weight coefficient representing the plane correction. If the equation (19) representing the correction function of segment i is formalized as an optimization problem, it can be expressed as equation (20).

[0151] [Number 10]

[0152]

[0153] Furthermore, equation (20) is equivalent to equation (21) which represents a correction function that can be expressed as the sum of the objective functions of the gravity correction and the representative plane correction.

[0154] [Number 11]

[0155]

[0156] The objective function calculation unit 164c performs posture estimation for all segments in the same manner, defining an optimization problem that integrates the objective functions for the entire body. Equation (22) is the correction function F(Q, α) that integrates the objective functions for the entire body. α is the total IMU acceleration measured by the IMU sensor and can be expressed as in Equation (23).

[0157] [Number 12]

[0158]

[0159]

[0160] Furthermore, the first row on the right side of Equation (22) represents the correction function corresponding to the pelvis, and the second row and subsequent rows on the right side represent the correction functions corresponding to each segment other than the pelvis. The optimization problem for correcting the whole-body posture of the estimated target TGT using the correction function shown in Equation (22) can be defined as shown in Equation (24) below. Equation (24) can be modified in the same form as Equation (21) described above as the correction function for each segment, as shown in Equation (25).

[0161] [Number 13]

[0162]

[0163]

[0164] Next, the gradient calculation unit 164e uses the Jacobian J obtained by the partial differentiation of the estimated whole body posture vector Q F , the gradient of the objective function is calculated as follows (26). In addition, the Jacobian formula J F As shown in formula (27).

[0165] [Number 14]

[0166]

[0167]

[0168] The size of each element shown in formula (27) becomes as shown in the following formulas (28) and (29).

[0169] [Number 15]

[0170]

[0171]

[0172] That is, the Jacobian J shown in formula (27) FIt becomes a large matrix of (3+4N)×4(N+1) (N is the total number of IMU sensors other than the IMU sensors used for reference part measurement), but in fact, the elements shown in the following equations (30) and (31) become 0, so the calculation can be omitted, and even a low-speed computing device can perform real-time posture estimation.

[0173] [Number 16]

[0174]

[0175]

[0176] If equations (30) and (31) are substituted into equation (27) described above, they can be expressed as equation (32).

[0177] [Number 17]

[0178]

[0179] The gradient calculation unit 164e can use the calculation result of equation (32) to calculate the gradient shown in equation (26).

[0180] [Processing image of the whole body correction amount calculation unit]

[0181] Figures 10 to 13 It is a diagram schematically showing the flow of calculation processing by the whole-body correction amount calculation unit 164 . Figure 10 : is a diagram schematically showing the entirety of the whole body correction amount calculation unit 164. Figures 11 to 13 It is a diagram for explaining the processing flow of the whole body correction amount calculation unit 164 in stages.

[0182] like Figure 10 As shown, the acceleration aggregation unit 148 aggregates the acceleration of each IMU sensor 40-i measured at time t. S a i,t The acquisition result obtained by the first acquisition unit 130 is converted into the total IMU acceleration α of the estimated target TGT as a result of the aggregation. t In addition, the angular velocity of each IMU sensor 40-i measured at time t obtained by the first acquisition unit 130 is S ω i,t The values are respectively output to the corresponding angular velocity integration units 152 - i.

[0183] in addition, Figure 10 The Z shown in the upper right part -1 The processing blocks up to β indicate that the correction unit 160 derives the correction amount for the next processing cycle.

[0184] also, Figures 10 to 13 In the equation (33), if the gradient of the objective function is set to ΔQ t , then the angular velocity Q at time t t (·)(A dot is marked as Q t The characters above are the estimated whole body posture vector Q at time t t The feedback of the time differential result of the correction amount can be expressed as shown in the following equation (34). In the equation (34), β is a real number for adjusting the gain of the correction amount, and 0≦β≦1.

[0185] [Number 18]

[0186]

[0187]

[0188] The whole body correction amount calculation unit 164 calculates the angular velocity Q as shown in equation (34). t (·) Normalizes the gradient ΔQ, and as a result, an arbitrary real number β is reflected as a correction amount.

[0189] like Figure 11 As shown in FIG. 1 , the integration unit 150 integrates the angular velocity of each segment. Figure 12 As shown, the correction unit 160 calculates the gradient ΔQ using the angular velocity and estimated posture of each segment. Figure 13 As shown, the correction unit 160 feeds back the derived gradient ΔQ to the angular velocity of each IMU sensor. If the first acquisition unit 130 acquires the next measurement result of the IMU sensor 40, the integration unit 150 Figure 11 As shown in FIG, the angular velocity of each segment is integrated again. The analyzing device 100 repeatedly performs Figures 11 to 13 The illustrated process estimates the posture of the estimation target TGT, thereby reflecting human body characteristics or empirical rules in the estimation results of the posture of each segment, thereby improving the accuracy of the estimation results of the analysis device 100.

[0190] By repeatedly Figures 11 to 13By performing the processing shown in FIG. 1 and aggregating the angular velocity integration results of the integration unit 150, the estimated posture aggregation unit 162 averages the errors in the measured angular velocities of the IMU sensors 40, thereby deriving the estimated whole-body posture vector Q of equation (2). The estimated whole-body posture vector Q reflects the result of calculating the yaw direction correction amount based on the whole-body posture using the human body characteristics or empirical rules. By using this method to estimate the posture of the estimation target TGT, it is possible to estimate a seemingly reasonable whole-body posture of the person with suppressed yaw angle drift without using the Earth's magnetic field. Therefore, even in the case of long-term measurement, whole-body posture estimation with suppressed yaw direction drift can be performed.

[0191] The analysis device 100 stores the whole-body posture estimation result as an analysis result in the storage unit 190 , and provides information indicating the analysis result to the terminal device 10 .

[0192] [Correction Processing]

[0193] An example of the correction processing performed by the correction unit 180 is described below. The second acquisition unit 170 acquires an image captured by the camera unit of the camera device 50 (hereinafter referred to as a captured image). The camera device 50 is controlled to capture the estimated object TGT by, for example, the control from the terminal device 10 (which may be automatic control or manual control). One or more first marks are assigned to the estimated object TGT. The first mark can be printed on the measuring equipment 30 or attached as a seal. The first mark includes an image that can be easily recognized by a machine, and the position and posture change in conjunction with the segment to which the position is assigned. The image preferably includes an image representing a spatial direction.

[0194] Figure 14 1 is a diagram showing an example of the appearance of the first marker Mk1. The first marker Mk1 is drawn with a contrast that allows easy extraction from a captured image, and has a two-dimensional shape such as a rectangle.

[0195] Figure 15 1 is a diagram showing an example of a captured image IM1. The imaging device 50 is controlled so that the captured image IM1 includes a second marker Mk2 in addition to the first marker Mk1. The second marker Mk2 is a marker assigned to a stationary object such as the ground. Like the first marker Mk1, the second marker Mk2 is drawn with a contrast that allows for easy extraction from the captured image and has a two-dimensional shape such as a rectangle.

[0196] The posture of the first marker Mk1 conforms to the sensor coordinate system. The first marker Mk1 is given, for example, in a form in which the relative posture does not change relative to the posture of the IMU sensor 40. For example, the first marker Mk1 is printed or attached to the rigid body component constituting the IMU sensor 40. The correction unit 180 corrects the transformation rule from the sensor coordinate system to the segment coordinate system based on the first marker Mk1 and the second marker Mk2 in the captured image IM. The "transformation unit" of the claim includes at least a primary transformation unit 140, and may also include an integration unit 150 or a correction unit 160. Therefore, the so-called transformation rule may refer to a rule by which the primary transformation unit 140 transforms the angular velocity of the IMU sensor 40-i into information of the segment coordinate system, or may refer to a rule that also includes processing performed by the integration unit 150 or the correction unit 160.

[0197] Here, the sensor coordinate system is defined as <M>, the segment coordinate system is defined as <S>, the camera coordinate system with the position of the imaging device 50 as the origin is defined as <E>, and the global coordinate system, which is a stationary coordinate system, is defined as <G>. The global coordinate system <G> is, for example, a terrestrial coordinate system with the direction of gravity as one axis. The object of calibration is the transformation rule (hereinafter referred to as the transformation matrix) from the sensor coordinate system <M> to the segment coordinate system <S>. M S R.

[0198] Figure 16 This is a diagram for explaining the processing contents of the correction unit 180. At the above-mentioned setting time point t0 of the original position, the correction unit 180 obtains Figure 15 The captured image IM shown in the figure is used to derive the posture of the first marker Mk1 relative to the imaging unit based on the position of the vertex of the first marker Mk1. The rotation angle between the coordinate systems is calculated based on the derived posture, and the transformation matrix from the sensor coordinate system <M> to the camera coordinate system <E> is derived. M E R. This technology is well known as a function of the Open Source Computer Vision Library (OpenCV), for example. In addition, the calibration unit 180 derives the posture of the second marker Mk2 relative to the imaging unit based on the position of the vertex of the second marker Mk2, and calculates the rotation angle between the coordinate systems based on the derived posture, thereby deriving the transformation matrix from the global coordinate system <G> to the camera coordinate system <E> G E At this time, when the estimated target TGT is in an upright position, it can be assumed that the segment coordinate system <S> and the global coordinate system <G> are consistent. Therefore, it can be assumed that the transformation matrix S E R = transformation matrix GE R. At this time, the transformation matrix from the sensor coordinate system <M> to the segment coordinate system <S> is set to M S R.

[0199] If the position and posture of the IMU sensor 40 relative to the estimation target TGT deviate at the calibration time point t1 after the original position setting time point t0, the transformation matrix from the sensor coordinate system <M> to the segment coordinate system <S> changes to M S R#. At this time, the transformation matrix M S R# is obtained from formula (35). As mentioned above, it can be assumed that S E R= G E R, so when the estimated object TGT takes the same upright posture as the set time point t0 of the original position, the relationship of formula (36) can be obtained. Therefore, by transforming the transformation matrix from the global coordinate system <G> to the camera coordinate system <E> G E The inverse matrix of R E G R, and the transformation matrix from the sensor coordinate system <M> to the camera coordinate system <E> M E R multiplied to derive the transformation matrix from the sensor coordinate system <M> to the segment coordinate system <S> M S R#.

[0200] M S R#= S E R T · M E R…(35)

[0201] M S R#= G E R T · M E R

[0202] =( E G R T ) T · M E R

[0203] = E G R. ME R…(36)

[0204] If the transformation matrix from the sensor coordinate system <M> to the segment coordinate system <S> is obtained as described above M S R#, the correction unit 180 is based on the transformation matrix M S R#, calibrate the transformation rule from the sensor coordinate system to the segment coordinate system. This allows for appropriate calibration of the posture estimation using the IMU sensor 40 at the calibration time point t1 after the original position setting time point t0.

[0205] According to the first embodiment described above, correction related to posture estimation using the IMU sensor 40 can be appropriately performed.

[0206] <Second embodiment>

[0207] The second embodiment will be described below. The second embodiment differs from the first embodiment in that the processing contents of the correction unit 180 are different. Therefore, the description will focus on the difference.

[0208] In the second embodiment, one or more third markers Mk3 are assigned to the estimated target TGT. Unlike the first marker Mk1, the third marker Mk3 represents an axis graphic that indicates the axial direction of the segment coordinate system. While the second marker Mk2 is not essential in the second embodiment, its presence can be expected to improve accuracy.

[0209] Figure 17 : is a diagram showing an example of a captured image IM2. The imaging device 50 is controlled so that the captured image IM2 includes a third marker Mk3 in addition to the first marker Mk1. Figure 17 In the example of , the second marker Mk2 is captured. The third marker Mk3 is also drawn with a contrast that allows easy extraction from the captured image, and has a two-dimensional shape such as a rectangle.

[0210] The posture of the third marker Mk3 conforms to the segment coordinate system. For example, the third marker Mk3 is printed or affixed to the measurement device 30 so that it contacts a rigid portion of a body such as the pelvis or spine near the estimated target TGT. The calibration unit 180 calibrates the transformation rule from the sensor coordinate system to the segment coordinate system based on the axis graphics of the first marker Mk1 and the third marker Mk3 in the captured image IM.

[0211] The same definitions as those in the first embodiment will be used for explanation. At the above-mentioned setting time point t0 of the original position and the subsequent correction time point t1, the correction unit 180 obtains Figure 17The captured image IM shown in FIG. 1 is converted from the sensor coordinate system <M> to the camera coordinate system <E> based on the position of the vertex of the first marker Mk1. M E R. In addition, the calibration unit 180 derives the posture of the third marker Mk3 relative to the imaging unit based on the position of the vertex of the third marker Mk3, and calculates the rotation angle between the coordinate systems based on the derived posture, thereby deriving the transformation matrix from the segment coordinate system <S> to the camera coordinate system <E> S E R. Correction matrix of the transformation from the sensor coordinate system <M> to the segment coordinate system <S> at time point t1 M S R# is directly obtained from the above formula (35).

[0212] If the transformation matrix from the sensor coordinate system <M> to the segment coordinate system <S> is obtained as described above M S R#, the correction unit 180 is based on the transformation matrix M S R#, calibrate the transformation rule from the sensor coordinate system to the segment coordinate system. This allows for appropriate calibration of the posture estimation using the IMU sensor 40 at the calibration time point t1 after the original position setting time point t0.

[0213] According to the second embodiment described above, correction related to posture estimation using the IMU sensor 40 can be appropriately performed.

[0214] <Modification of Second Embodiment>

[0215] In the second embodiment, the correction unit 180 derives the transformation matrix from the segment coordinate system <S> to the camera coordinate system <E> based on the third marker Mk3 included in the captured image IM2. S E Alternatively, the correction unit 180 may derive the position and posture of the segment of the estimated target TGT by analyzing the captured image, thereby deriving the transformation matrix from the segment coordinate system <S> to the camera coordinate system <E> S E For example, the position and posture of the head in the segment can be estimated using a technique that estimates facial orientation based on facial feature points. In this case, the imaging device 50 is preferably a time-of-flight (TOF) camera capable of measuring distance, since it can acquire a three-dimensional outline of the estimation target TGT.

[0216] <Modification of the method for acquiring a captured image>

[0217] Next, a method for acquiring captured images other than the method using a drone will be described. Figure 18 This figure illustrates a first variation of the captured image acquisition method. As shown, for example, one or more imaging devices 50A can be installed at a door or other location through which the estimated target TGT passes, capturing one or more captured images as the estimated target TGT passes. In this case, the imaging device 50A is stationary, so the global coordinate system <G> and the camera coordinate system <E> can be considered identical. Therefore, even if the third marker Mk3 is not present, the second marker Mk2 can be omitted.

[0218] Figure 19 This figure illustrates a second variation of the method for acquiring captured images. As shown in the figure, for example, one or more imaging devices 50B (micro-camera rings) attached to a wrist band or ankle band may be attached to the estimated subject TGT to acquire one or more captured images. In this case, the second marker Mk2 is preferably present, and it is appropriate to instruct the estimated subject TGT to assume a predetermined posture when the imaging device 50B is used to capture the image.

[0219] Instead of the above, one or more camera devices may be installed on the ground, wall, ceiling, etc. to obtain captured images.

[0220] As mentioned above, although embodiment of the present invention was demonstrated using embodiment, this invention is not limited to this embodiment at all, Various deformation|transformation and substitution are possible within the range which does not deviate from the summary of this invention.

Claims

1. An analysis device, characterized in that include: A posture estimating unit estimates the posture of an estimated object, wherein the posture estimation of the estimated object includes: converting an output of an inertial measurement sensor represented in a sensor coordinate system into a segment coordinate system, wherein the sensor coordinate system is based on the positions of a plurality of inertial measurement sensors installed at a plurality of locations of the estimated object and detecting angular velocity and acceleration, the segment coordinate system representing a coordinate system of the posture of each segment corresponding to the position of the estimated object at which the inertial measurement sensors are installed, wherein the posture estimating unit includes: a first acquiring unit configured to acquire information on the angular velocity and the acceleration from the plurality of inertial measurement sensors; a primary conversion unit that converts the information acquired by the first acquisition unit from the sensor coordinate system of each of the inertial measurement sensors into information in the segment coordinate system; an integrating unit that integrates the angular velocity of the segment output by the primary transform unit to calculate, as a part of the posture of the estimated object, an orientation of a reference portion of the estimated object on which the inertial measurement sensor is mounted; and a correction unit that assumes a representative plane passing through a reference portion included in the estimation target and corrects the transformed angular velocity of the reference portion so that a normal to the representative plane and the orientation of the reference portion calculated by the integration unit become close to a perpendicular direction; a second acquiring unit that acquires an image captured by an imaging unit that captures one or more first markers assigned to the estimation target and a second marker assigned to a stationary body other than the estimation target; and a correction unit that corrects a transformation rule from the sensor coordinate system to the segment coordinate system based on the first mark and the second mark of the image; The first marker has a form in which a relative posture relative to at least one of the plurality of inertial measurement sensors does not change, and the posture relative to the imaging unit can be identified by analyzing the captured image. The correction unit derives the posture of the first marker relative to the camera unit, derives the transformation matrix from the sensor coordinate system to the camera coordinate system based on the derived posture, and uses the derived transformation matrix from the sensor coordinate system to the camera coordinate system to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

2. The analysis device according to claim 1, characterized in that The second marker has a form in which a posture relative to the imaging unit can be recognized by analyzing the captured image. The correction unit derives the posture of the second marker relative to the camera unit, derives the transformation matrix from the global coordinate system representing the space to the camera coordinate system based on the derived posture, regards the segment coordinate system and the global coordinate system as the same, thereby deriving the transformation matrix from the sensor coordinate system to the segment coordinate system based on the transformation matrix from the sensor coordinate system to the camera coordinate system and the transformation matrix from the global coordinate system to the camera coordinate system, and corrects the transformation rule from the sensor coordinate system to the segment coordinate system based on the derived transformation matrix from the sensor coordinate system to the segment coordinate system.

3. The analysis device according to claim 1 or 2, characterized in that The imaging unit further captures a third marker assigned to the estimated target. The third marker has a form in which the relative posture relative to at least one of the segments does not change, and the posture relative to the imaging unit can be identified by analyzing the captured image. The correction unit derives the posture of the third marker relative to the camera unit, derives the transformation matrix from the segment coordinate system to the camera coordinate system based on the derived posture, derives the transformation matrix from the sensor coordinate system to the segment coordinate system based on the transformation matrix from the sensor coordinate system to the camera coordinate system and the transformation matrix from the segment coordinate system to the camera coordinate system, and corrects the transformation rule from the sensor coordinate system to the segment coordinate system based on the derived transformation matrix from the sensor coordinate system to the segment coordinate system.

4. An analysis method, characterized in that The computer of the analysis device according to any one of claims 1 to 3 is caused to execute the following operations: performing posture estimation of the estimation target, the posture estimation of the estimation target including: converting an output of the inertial measurement sensor represented by the sensor coordinate system into the segment coordinate system, the sensor coordinate system being based on respective positions of a plurality of inertial measurement sensors mounted at a plurality of locations of the estimation target and detecting the angular velocity and the acceleration, the segment coordinate system being a coordinate system representing the posture of each segment corresponding to the position of the estimation target at which the inertial measurement sensor is mounted; acquiring an image captured by the imaging unit, the imaging unit capturing one or more first markers assigned to the estimation target and a second marker assigned to a stationary object other than the estimation target; as well as Correcting a transformation rule from the sensor coordinate system to the segment coordinate system based on the first and second markers of the image, The first marker has a form in which a relative posture relative to at least one of the plurality of inertial measurement sensors does not change, and the posture relative to the imaging unit can be identified by analyzing the captured image. In the correction process, the posture of the first marker relative to the camera unit is derived, and the transformation matrix from the sensor coordinate system to the camera coordinate system is derived based on the derived posture. The derived transformation matrix from the sensor coordinate system to the camera coordinate system is used to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

5. A storage medium storing a program, characterized in that: The program causes a computer of the analysis device according to any one of claims 1 to 3 to execute the following operations: performing posture estimation of the estimation target, the posture estimation of the estimation target including: converting an output of the inertial measurement sensor represented by the sensor coordinate system into the segment coordinate system, the sensor coordinate system being based on respective positions of a plurality of inertial measurement sensors mounted at a plurality of locations of the estimation target and detecting the angular velocity and the acceleration, the segment coordinate system being a coordinate system representing the posture of each segment corresponding to the position of the estimation target at which the inertial measurement sensor is mounted; acquiring an image captured by the imaging unit, the imaging unit capturing one or more first markers assigned to the estimation target and a second marker assigned to a stationary object other than the estimation target; as well as Correcting a transformation rule from the sensor coordinate system to the segment coordinate system based on the first and second markers of the image, The first marker has a form in which a relative posture relative to at least one of the plurality of inertial measurement sensors does not change, and the posture relative to the imaging unit can be identified by analyzing the captured image. In the correction process, the posture of the first marker relative to the camera unit is derived, and the transformation matrix from the sensor coordinate system to the camera coordinate system is derived based on the derived posture. The derived transformation matrix from the sensor coordinate system to the camera coordinate system is used to correct the transformation rule from the sensor coordinate system to the segment coordinate system.

6. A calibration method, characterized in that: include: The imaging unit mounted on the unmanned aerial vehicle captures one or more first markers assigned to the estimated object and a second marker assigned to a stationary object other than the estimated object. The analyzing device according to any one of claims 1 to 3 acquires the first marker and the second marker of the image captured by the imaging unit, and calibrates the transformation rule from the sensor coordinate system to the segment coordinate system.

7. A calibration method, characterized in that: include: The imaging unit attached to the stationary object captures one or more first markers assigned to the estimation target and a second marker assigned to a stationary object other than the estimation target, The analyzing device according to any one of claims 1 to 3 acquires the first marker and the second marker of the image captured by the imaging unit, and calibrates the transformation rule from the sensor coordinate system to the segment coordinate system.

8. A calibration method, characterized in that: include: The imaging unit attached to the estimated object captures one or more first markers assigned to the estimated object and a second marker assigned to a stationary object other than the estimated object. The analyzing device according to any one of claims 1 to 3 acquires the first marker and the second marker of the image captured by the imaging unit, and calibrates the transformation rule from the sensor coordinate system to the segment coordinate system.

Images