Data processing device, method and program

Abstract

To reset an integrated error in inertial data.SOLUTION: A data processing device includes: an optical data processing part that estimates optically derived data showing a position or posture of a moving body on the basis of data obtained from an optical type sensor; an inertia data processing part that estimates inertial data showing the position or posture of the moving body on the basis of data obtained from an inertia type sensor; a synthesis part synthesizing the optically derived data and the inertial data based on reliability of the optically derived data or the inertial data; an initialization processing part that repeatedly executes initialization processing setting initial value of the position or posture of the moving body to be used for estimating the inertial data in the inertia data processing part; and a reliability calculation part that calculates reliability of the inertial data so that it becomes low as time passes after the initialization processing and becomes high as new initialization processing is performed.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a data processing apparatus, method, and program.

Background Art

[0002] In recent years, with the development of technologies such as VR (Virtual Reality), motion capture technology has also been developing. In terms of measurement methods, motion capture technology is mainly divided into two types: optical motion capture and inertial motion capture.

[0003] Optical motion capture is a motion capture technology that uses optical sensors such as RGB cameras, infrared cameras, or Depth sensors. If it is possible to use a large number of sensors and prepare a large-scale measurement environment such as attaching markers to the measurement target, optical motion capture is considered the most accurate measurement method. On the other hand, there is also markerless optical motion capture that can be measured with a single sensor. Such optical motion capture is easy to measure, but occlusion often occurs where the measurement site is hidden from the measurement range of the sensor, and in such cases, the accuracy significantly decreases.

[0004] Inertial motion capture is a technology that measures the body's movements by using inertial sensors such as acceleration sensors or gyro sensors. Generally, inertial motion capture defines the position and posture of a person at the start of measurement, and calculates the position and posture of the joints to which the sensors are attached by integrating the output values of the acceleration sensors or gyro sensors. Therefore, there is a problem that integration errors gradually accumulate. However, in inertial motion capture, there is no deterioration in accuracy due to occlusion like in optical sensors.

[0005] As described above, optical motion capture and inertial motion capture each have their own advantages and disadvantages. Therefore, many methods have been proposed to combine optical motion capture, inertial motion capture, and other measurement methods to solve their respective disadvantages.

[0006] For example, Patent Document 1 proposes a method of synthesizing data measured by sensors with different characteristics, such as an optical sensor and an inertial sensor, to reduce the integration error of the inertial sensor. More specifically, Patent Document 1 discloses a technique for reducing the integration error in the measurement value based on the inertial sensor by setting the degree of approximation (reflection degree) of the position and posture of a human joint or an object measured by the inertial sensor to the position and posture measured by the optical sensor to be between 0 and 1. When a value close to 1 is set for the reflection degree, the final measurement value approaches the measurement value of the optical sensor. Patent Document 1 describes that this reduces the integration error in the measurement value based on the inertial sensor.

[0007] Also, Patent Document 2 discloses a method of improving the final measurement accuracy by synthesizing two data, namely, regression estimation based on a pre-prepared model and inertial-derived data obtained by integrating the output values of an inertial sensor (acceleration sensor and angular velocity sensor). The two data are synthesized according to their respective reliabilities. Patent Document 2 states that this can compensate for the drawbacks of each other's measurement methods, such as the discontinuity of data by regression estimation and the accumulation of integration errors by integrating the output values.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0009] As described above, in a method in which output values of a plurality of sensors having different characteristics are used for synthesis, parameters such as which data to approximate the final measurement value to are set. However, in the above-described method, since the integration error in the inertia-derived data based on the inertial sensor is not removed, there is a concern that the accuracy may deteriorate due to long-term measurement.

[0010] Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of resetting the integration error in the inertia-derived data.

Means for Solving the Problems

[0011] In order to solve the above problems, according to an aspect of the present invention, an optical data processing unit that estimates optical-derived data indicating the position or orientation of a moving object based on data obtained from an optical sensor, and data obtained from an inertial sensor An inertial data processing unit that estimates inertial-derived data indicating the position or orientation of the moving object based on the above, and a combining unit that combines the optical-derived data and the inertial-derived data based on the reliability of the optical-derived data or the inertial-derived data, and the inertial data An initialization processing unit that repeatedly executes an initialization process for setting an initial value of the position or orientation of the moving object used for estimation of the inertial-derived data in the processing unit, and the reliability of the inertial-derived data decreases as time elapses from the initialization process, and a new initialization process is performed. A reliability calculation unit that calculates the reliability so as to increase by execution is provided.

[0012] The data processing apparatus may further include an initialization management unit that determines the initial value and passes it to the initialization processing unit when the initialization process has not been executed and when the reliability of the inertial-derived data is below a threshold value.

[0013] The initialization management unit may determine the initial value using the optical-derived data.

[0014] When the initialization process has not been executed and when the initialization process is in progress, the reliability calculation unit may set the reliability of the inertia-derived data to a predetermined value, and the synthesis unit may continue to synthesize the optical-derived data and the inertia-derived data based on the reliability of the inertia-derived data set to the predetermined value.

[0015] The inertial data processing unit, the reliability calculation unit, the initialization processing unit, and the initialization management unit may execute processing for each of a plurality of parts constituting the moving body, and the synthesis unit may synthesize the optical-derived data and the inertia-derived data for the same part.

[0016] The initialization management unit may determine the initial value of a certain part of the moving body using the optical-derived data for that part.

[0017] Also, according to another aspect of the present invention to solve the above problems, estimating optical-derived data indicating the position or posture of a moving body based on data obtained from an optical sensor, estimating inertia-derived data indicating the position or posture of the moving body based on data obtained from an inertial sensor, synthesizing the optical-derived data and the inertia-derived data based on the reliability of the optical-derived data or the inertia-derived data, repeatedly executing an initialization process for setting an initial value of the position or posture of the moving body used for the estimation of the inertia-derived data, and calculating the reliability of the inertia-derived data so as to decrease with the passage of time from the initialization process and increase with the execution of a new initialization process. A method executed by a computer is provided.

[0018] Also, according to another aspect of the present invention for solving the above problems, a computer is provided with an optical data processing unit that estimates optical data indicating the position or posture of a moving object based on data obtained from an optical sensor, an inertial data processing unit that estimates inertial data indicating the position or posture of the moving object based on data obtained from an inertial sensor, a synthesizing unit that synthesizes the optical data and the inertial data based on the reliability of the optical data or the inertial data, an initialization processing unit that repeatedly executes an initialization process for setting an initial value of the position or posture of the moving object used for the estimation of the inertial data in the inertial data processing unit, and a reliability calculation unit that calculates the reliability of the inertial data so as to decrease with the passage of time from the initialization process and increase with the execution of a new initialization process. A program is provided for causing the computer to function as described above.

Effects of the Invention

[0019] According to the present invention described above, it is possible to reset the integration error in the inertial data.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0022] One embodiment of the present invention relates to a data processing system, and particularly to a motion measurement system that measures the position and posture of a moving object over time. Hereinafter, after explaining the outline of the data processing system according to one embodiment of the present invention, the detailed configuration and operation will be sequentially explained. In the following, an example in which the moving object is mainly a human will be described, but one embodiment of the present invention is also applicable to other moving objects such as animals and robots.

[0023] <Outline of the Motion Measurement System> FIG. 1 is an explanatory diagram showing the configuration of a motion measurement system according to one embodiment of the present invention. As shown in FIG. 1, it has an optical sensor 11, a plurality of inertial sensors 12, and a motion measurement device 20.

[0024] (Optical Sensor 11) The optical sensor 11 optically senses the person to be measured. The type of the optical sensor 11 is not particularly limited. For example, the optical sensor 11 may be an RGB camera, an infrared camera, or a Depth sensor. The optical sensor 11 outputs the data obtained by sensing to the motion measurement device 20. In FIG. 1, an example in which the optical sensor 11 and the motion measurement device 20 are connected by wire is shown, but the optical sensor 11 may wirelessly transmit the data obtained by sensing to the motion measurement device 20. Also, a plurality of optical sensors 11 may be provided in the motion measurement system according to one embodiment of the present invention.

[0025] (Inertial Sensor 12) The inertial sensor 12 senses the acceleration or angular velocity of the person to be measured. The type of the inertial sensor 12 is not particularly limited, and the inertial sensor 12 may be a three-axis gyro sensor, a three-axis acceleration sensor, or a combination of a three-axis gyro sensor and a three-axis acceleration sensor. In the configuration of only a three-axis gyro sensor, only the three-axis attitude can be calculated, and the three-axis position cannot be calculated. Also, in the configuration of only a three-axis acceleration sensor, the three-axis position and the two-axis attitude can be calculated. By combining the above two sensors, the six-axis position and attitude can be calculated. Hereinafter, an example in which the inertial sensor 12 is a combination of the above two sensors and can calculate the six-axis position and attitude will be mainly described.

[0026] As shown in FIG. 1, each inertial sensor 12 is attached to a different part of the person to be measured. For example, the inertial sensor 12 may be attached to a joint such as the wrist, elbow, shoulder, neck, knee, or ankle. Each inertial sensor 12 senses the acceleration or angular velocity of each part and transmits the data obtained by the sensing to the motion measurement device 20. The inertial sensor 12 and the motion measurement device 20 may be connected by a wireless method such as Bluetooth (registered trademark) or WiFi (registered trademark), or may be connected by wire.

[0027] (Motion measurement device 20) The motion measurement device 20 is an example of a data processing device, which receives data from the optical sensor 11 and each inertial sensor 12, and estimates the positions and postures of the joints of the measurement target person based on the received data. Specifically, the motion measurement device 20 estimates optical origin data indicating the positions and postures of the joints based on the data received from the optical sensor 11, and estimates inertial origin data indicating the positions and postures of the joints based on the data received from the inertial sensor 12. Then, the motion measurement device 20 obtains measurement data for the joint by synthesizing the optical origin data and the inertial origin data for the same joint. The measurement data for one joint obtained at a certain point in time indicates the position and posture of the joint at that point in time, and the measurement data for one joint obtained at a plurality of points in time indicates the motion, which is the change in the position and posture of the joint.

[0028] (Comparative Example) Here, the measurement method according to the comparative example will be described. In the measurement method according to the comparative example, the optical origin data and the inertial origin data are synthesized according to the parameter α.

[0029] FIG. 2 is an explanatory diagram showing an example of processing by the measurement method according to the comparative example. In FIG. 2, the time change of the error of the optical origin data, the time change of the error of the inertial origin data, and the time change of the error of the synthesized data obtained by synthesizing the optical origin data and the inertial origin data according to the parameter α are shown. In the example shown in FIG. 2, the parameter α is 0.2, which is a constant.

[0030] As shown in FIG. 2, it can be seen that an integration error that increases with time occurs in the inertial origin data, and such an integration error does not occur in the optical origin data. However, although the error of the optical origin data is shown as being constant in FIG. 2, this means that no integration error occurs and there is no time-dependent error. In reality, in the optical origin data, the accuracy can change according to environmental changes, such as when occlusion occurs.

[0031] The data synthesis method as shown in FIG. 2 is expressed by the following weighted calculation formula. p D = αp C + (1-α) p I (1)

[0032] In the above formula (1), p D represents synthetic data, p C represents optically-derived data, and p I represents inertia-derived data. p D , p C , p I respectively represent the position and posture of the joints of the measurement subject.

[0033] From the above formula (1), when the parameter α is a constant, the influence of each of the optically-derived data p C and the inertia-derived data p I on the synthetic data p D is constant. Therefore, by synthesizing the optically-derived data p C without integration error, it is possible to calculate the synthetic data p I with the integration error of the inertia-derived data p D reduced. However, the integration error continues to accumulate in the inertia-derived data p I itself, and the accuracy deteriorates. Therefore, when synthesizing continuously for a long time, as shown in the error characteristics of the synthetic data in Figure 2, the inertia-derived data p I with deteriorated accuracy will be synthesized, and there is a concern that the synthetic data p D will also be affected by the accumulation of integration error. As a countermeasure against the influence of such accumulation of integration error, it is conceivable to make the parameter α a variable.

[0034] Figure 3 is an explanatory diagram showing an example of processing when the variable of the parameter α is applied. Here, based on the property that the accuracy of the inertia-derived data deteriorates as the measurement time becomes longer, an example is considered in which a higher value is applied to the parameter α as the measurement time becomes longer. In the example shown in Figure 3, the parameter α increases from 0.2 as time elapses.

[0035] In this case, as time elapses, the inertia-derived data p D applied to the synthetic data pI It is possible to reduce the influence. That is, as the measurement time becomes longer, the parameter α approaches 1 indefinitely, and the influence of the integration error accumulated in the inertia-derived data p I can be reduced. As a result, as shown in FIG. 3, ultimately, it is possible to obtain synthetic data in which the influence of the integration error is completely suppressed.

[0036] Here, when the parameter α approaches 1 indefinitely, the influence of the optically-derived data on the synthetic data becomes infinitely large, meaning that the synthetic data and the optically-derived data become almost indistinguishable (synthetic data p D ≈ optically-derived data p C ). That is, in the synthetic data, as the measurement time becomes longer, the characteristics of the optically-derived data p C will be more prominently manifested.

[0037] FIG. 4 is an explanatory diagram showing a specific example of the synthetic data obtained by the process shown in FIG. 3. In FIG. 4, in section 1 where the parameter α = 0.2, even if occlusion occurs due to the synthesis of the inertia-derived data p I , the deterioration of the accuracy of the synthetic data is suppressed, and it is less likely to become discontinuous data.

[0038] However, in section 2 where the parameter α approaches 1 indefinitely and almost no inertia-derived data p I is synthesized, the influence of the deterioration of accuracy at the time of occlusion occurrence appears prominently in the synthetic data, and the synthetic data becomes discontinuous data.

[0039] As described above, in the measurement method according to the comparative example, there is a problem that the synthetic data is affected by the accumulation of integration errors due to long-time measurement, or the influence of the deterioration of accuracy at the time of occlusion occurrence appears prominently in the synthetic data.

[0040] The inventor of the present invention has created an embodiment of the present invention from a certain perspective in view of the above circumstances. The motion measurement device 20 according to an embodiment of the present invention autonomously and repeatedly performs an initialization process that is a reset of an initial value used for estimating data derived from inertia. With such a configuration, it is possible to reduce the influence of the integration error received by the combined data while continuing the combination of the inertia-derived data p I Hereinafter, the configuration and operation of the motion measurement device 20 according to such an embodiment of the present invention will be sequentially described in detail.

[0041] <Configuration of the motion measurement device 20> FIG. 5 is an explanatory diagram showing the configuration of the motion measurement device 20 according to an embodiment of the present invention. As shown in FIG. 5, the motion measurement device 20 according to an embodiment of the present invention includes an optical motion estimation unit 210, an inertial motion estimation unit 220, a reliability calculation unit 230, an initialization management unit 240, and a data synthesis unit 250.

[0042] (Optical motion estimation unit 210) The optical motion estimation unit 210 estimates the position and posture of each joint of the measurement subject based on the data received from the optical sensor 11. As shown in FIG. 5, the optical motion estimation unit 210 has functions as an optical sensor connection unit 212 and an optical data processing unit 216.

[0043] The optical sensor connection unit 212 is an interface connected to the optical sensor 11. The optical sensor connection unit 212 receives the data obtained by the optical sensor 11 from the optical sensor 11 and passes the data to the optical data processing unit 216. The optical sensor connection unit 212 may be an interface connected to the optical sensor 11 by wire, such as USB, or may be an interface connected to the optical sensor 11 wirelessly.

[0044] The optical data processing unit 216 estimates the motion of the person to be measured from the data received from the optical sensor connection unit 212. Specifically, the optical data processing unit 216 estimates optical origin data indicating the position and posture of each joint of the person to be measured. Various methods can be applied to the method for estimating the optical origin data, and the method for estimating the optical origin data is not particularly limited.

[0045] (Inertial motion estimation unit 220) The inertial motion estimation unit 220 estimates the position and posture of each joint of the person to be measured based on the data received from the inertial sensor 12. As shown in FIG. 5, the inertial motion estimation unit 220 has functions as an inertial sensor connection unit 222, an initialization processing unit 224, and an inertial data processing unit 226.

[0046] The inertial sensor connection unit 222 is an interface connected to the inertial sensor 12. The inertial sensor connection unit 222 receives the data obtained by the inertial sensor 12 from the inertial sensor 12 and passes the data to the inertial data processing unit 226. Since it is assumed that the inertial sensor 12 is attached to the person to be measured, the inertial sensor connection unit 222 is preferably an interface connected to the inertial sensor 12 by a wireless method such as Bluetooth (registered trademark) or WiFi (registered trademark), but may also be an interface connected to the inertial sensor 12 by wire.

[0047] The initialization processing unit 224 executes an initialization process for setting initial values of the position and posture of the joint, which are used for estimating the position and posture of the joint in the inertial data processing unit 226. The initialization processing unit 224 according to an embodiment of the present invention executes the initialization process in response to control from the initialization management unit 240 not only at the start of measurement but also during the execution of measurement.

[0048] The inertial data processing unit 226 estimates the positions and postures of each joint of the measurement subject based on the data received from the inertial sensor connection unit 222. Specifically, the inertial data processing unit 226 acquires the initial values of the joint positions and postures from the initialization processing unit 224, and adds the integration result of the data received from the inertial sensor connection unit 222 to the initial values to estimate the inertial-derived data indicating the joint positions and postures. Note that the inertial data processing unit 226 may add a process of correcting the posture calculated from the gyro sensor alone based on the output of the acceleration sensor or the geomagnetic sensor.

[0049] (Reliability calculation unit 230) The reliability calculation unit 230 calculates the reliability of the optically-derived data and the reliability of the inertial-derived data for each joint. The reliability calculation unit 230 adds the calculated reliability to the optically-derived data and the inertial-derived data of each joint, and passes the optically-derived data and the inertial-derived data with added reliability to the initialization management unit 240 and the data synthesis unit 250.

[0050] The reliability calculation unit 230 may apply different methods to calculate the reliability of the optically-derived data and the reliability of the inertial-derived data. Hereinafter, specific examples of methods for calculating the reliability of the optically-derived data and specific examples of methods for calculating the reliability of the inertial-derived data will be described.

[0051] - Reliability of optically-derived data As an example of a method for calculating the reliability of the optically-derived data, a method of calculating a parameter indicating the reliability using the amplitude (details will be described later) acquired as time-series data can be considered. In motion capture technologies using a single sensor or camera developed in recent years, machine learning technologies are often used. Therefore, even when there are parts that cannot be measured due to occlusion or the like from a single sensor or camera, it is possible to estimate the data of the joints that cannot be measured by inputting the data of other joints into the learning model.

[0052] However, if the learning model does not consider time-series information, the estimated data may be discontinuous data, which may be problematic for use, such as feeling uncomfortable as human behavior. Therefore, a method of representing the reliability of data can be considered using the amplitude, which is an index representing the magnitude of vibration. The parameter α indicating the reliability of each joint i is obtained by sampling the position of each joint at an arbitrary sampling rate, and the amplitude A i calculated from the sampling result can be expressed as follows. α i = 1 - G c A i (2)

[0053] In the above formula (2), G c is an arbitrary constant, which can control the influence of the amplitude of the data on the parameter α i . Also, G c has the role of scaling the amplitude A i to 0 to 1 by multiplying it. Therefore, it is desirable that G i is determined from the unit of the amplitude A c . i

[0054] The reliability calculation unit 230 can calculate the parameter α indicating the reliability of the optically-derived data according to the above formula (2). According to such a configuration, the larger the amplitude A i , the smaller the reliability parameter α i . That is, the more the accuracy of the optically-derived data deteriorates, the smaller the parameter α i can be made. i

[0055] Here, the reliability calculation unit 230 may calculate the amplitude A i by any calculation method. For example, the reliability calculation unit 230 calculates the amplitude A as the absolute value of the difference between the maximum value and the minimum value of the sampling result i ​​It may also be calculated as. Further, the reliability calculation unit 230 can also calculate the amplitude A by frequency analysis such as FFT (Fast Fourier Transform) analysis i is also possible.

[0056] -Reliability of inertia-derived data As described above, integration errors accumulate in the inertia-derived data as time elapses from the initialization process. Therefore, the reliability calculation unit 230 calculates the reliability of the inertia-derived data so that it decreases as time elapses from the initialization process and increases when a new initialization process is executed. Such reliability β of the inertia-derived data i is the time elapsed T from the initialization process i is expressed as follows using as a parameter. β i = 1 - G i T i (3)

[0057] In the above formula (3), G i is an arbitrary parameter, and it is desirable that it be determined using a threshold value described later used in the initialization process. According to formula (3), as the time elapsed T i increases, the reliability β of the inertia-derived data i is calculated to be smaller, so the influence of the inertia-derived data with accumulated integration errors on the composite data can be reduced.

[0058] However, regarding the method for calculating the reliability of the inertia-derived data, the reliability calculation unit 230 checks whether the initialization process has not been executed and whether the initialization process is being executed. If the initialization process has not been executed or if the initialization process is being executed, the reliability of the inertia-derived data may be set to a predetermined value (for example, 0 which is the minimum value). This setting is expressed as follows by adding a condition to the above formula (3).

[0059]

Equation

[0060] In the above formula (4), init is a variable that takes either true or false. This variable init is managed by the initialization management unit 240 and is changed in the initialization management flow described later. Since the initialization management flow operates in parallel with the overall processing flow, the reliability calculation unit 230 calculates the reliability of the inertia-derived data while referring to the variable init that is updated at any time in the initialization management flow.

[0061] Note that the reliability calculation unit 230 may more simply treat the elapsed time T i from the initialization process as the reliability. In this case, a smaller elapsed time T i indicates a higher reliability, and a larger elapsed time T i indicates a lower reliability.

[0062] - Supplementary As described above, an example has been explained in which the reliability calculation unit 230 calculates the reliability of the optically-derived data and the reliability of the inertia-derived data as two parameters that do not depend on each other. However, the reliability calculation unit 230 may calculate only one of the parameters. For example, when the data synthesis unit 250 synthesizes the optically-derived data and the inertia-derived data based on the above formula (1), the reliability calculation unit 230 may calculate only the parameter α in the formula (1) as the reliability of the optically-derived data. Since the reliability of the inertia-derived data in the formula (1) is represented by (1 - α), if the reliability calculation unit 230 calculates one parameter α, it is possible to synthesize the optically-derived data and the inertia-derived data. Regarding the synthesis of the attitude data, SLERP in the quaternion format may be applied, and also in SLERP, only one parameter is required to obtain the interpolation value of the two attitude data.

[0063] (Initialization Management Unit 240) The initialization management unit 240 manages the initialization process performed by the initialization processing unit 224. For example, the initialization management unit 240 determines for each joint whether the initialization process by the initialization processing unit 224 is necessary. If it is determined that the initialization process is necessary, the initialization management unit 240 determines the initial values of the position and orientation and passes the determined initial values to the initialization processing unit 224. The initialization management unit 240 may determine that the initialization process is necessary when the initialization process has not been executed and when the reliability of the inertia-derived data is below the threshold. Further, the initialization management unit 240 may determine the initial values of the position and orientation based on the optically-derived data. The functions of such an initialization management unit 240 will be described in detail later with reference to FIG. 7.

[0064] (Data synthesis unit 250) The data synthesis unit 250 is a synthesis unit that synthesizes the optically-derived data and the inertia-derived data for the same joint based on one or both of the reliability of the optically-derived data and the reliability of the inertia-derived data calculated by the reliability calculation unit 230. When the inertial sensor 12 is attached to a joint (for example, the right elbow), the data of the right elbow passed to the data synthesis unit 250 includes two types: optically-derived data and inertia-derived data. The data synthesis unit 250 detects the data of the joint having both the optically-derived data and the inertia-derived data as described above, and synthesizes them using the reliability calculated by the reliability calculation unit 230.

[0065] Here, the data synthesis method used by the data synthesis unit 250 may be any method. For example, the data synthesis unit 250 may synthesize the optically-derived data and the inertia-derived data by the method shown in the above mathematical formula (1).

[0066] In addition, the data synthesis unit 250 may synthesize the optically-derived data and the inertia-derived data using a Kalman filter or the like that synthesizes data by setting an error covariance matrix, or may synthesize the optically-derived data and the inertia-derived data by other methods. However, when the reliability of one of the data becomes 0, it is desirable to perform a process in which the data is not synthesized (a process in which the data with a reliability other than 0 is output as it is). For example, in the process shown in the above formula (1), when the parameter α, which is the reliability of the optically-derived data, becomes 0, the optically-derived data p C is not synthesized, and it can be seen that the inertia-derived data p I becomes the synthesized data as it is. In the initialization management flow of FIG. 7 described later, when it is determined that the initialization process has not been executed or the initialization process is being executed, as shown in formula (4), the reliability of the inertia-derived data is calculated as 0. As a result, when the initialization process has not been executed or the initialization process is being executed, the inertia-derived data is not synthesized, and the optically-derived data remains as the measurement data, and the acquisition of the measurement data continues. That is, it is possible to realize the initialization process by the initialization management unit 240 and the initialization processing unit 224 without stopping the acquisition of the measurement data.

[0067] In the method shown in the above formula (1), only the parameter α, which is the reliability of the optically-derived data, is used for synthesis, but there is also a method that uses both the reliability of the optically-derived data and the reliability of the inertia-derived data. For example, the data synthesis unit 250 may synthesize the optically-derived data and the inertia-derived data according to the following formula.

[0068]

Equation

[0069] In the above formula (5), the parameter β represents the reliability of the inertia-derived data. In formula (1), when the reliability of the optically-derived data decreases, a synthesis equivalent to an unconditional increase in the reliability of the inertia-derived data is performed. In contrast, in formula (5), weighting is performed from the perspective of which reliability is relatively higher.

[0070] Also, when the format of the joint posture data is a quaternion, SLERP or the like that calculates the interpolation values of the optically-derived data and the inertia-derived data based on parameters may be used.

[0071] As described above, there are various methods for the data synthesis method based on reliability, such as a method using a single parameter and a method using a plurality of parameters. Therefore, it is desirable that the processing in the data synthesis unit 250 be determined according to the synthesis method.

[0072] <Operation> The configuration of the motion measurement device 20 according to an embodiment of the present invention has been described above. Subsequently, with reference to FIGS. 6 and 7, the operation of the motion measurement device 20 according to an embodiment of the present invention will be organized.

[0073] FIG. 6 is an explanatory diagram showing the overall flow of the motion measurement device 20. FIG. 7 is an explanatory diagram showing the flow of initialization management. The overall flow shown in FIG. 6 and the flow of initialization management shown in FIG. 7 are executed in parallel. Thereby, the motion measurement device 20 can continue to measure the measurement target person even when performing the initialization process. That is, the motion measurement device 20 can execute the initialization process for each joint and reset the integration error of the inertia-derived data even during the execution of the measurement of the measurement target person.

[0074] (Overall flow) In the overall flow, as shown in FIG. 6, the optical motion estimation unit 210 estimates the optically-derived data of each joint based on the data received from the optical sensor 11 (S304), and the inertial motion estimation unit 220 estimates the inertia-derived data of each joint based on the data received from the inertial sensor 12 (S308).

[0075] Then, the reliability calculation unit 230 calculates the reliability of the optically-derived data and the reliability of the inertia-derived data for each joint (S312).

[0076] After that, the data synthesis unit 250 receives the optically-derived data and the inertia-derived data with the reliability added from the initialization management unit 240 for each joint, and determines whether there is a joint with both optically-derived data and inertia-derived data (S316). Basically, it is assumed that there is optically-derived data for all joints of the whole body and inertia-derived data for the joints to which the inertial sensor 12 is attached. For this reason, it is considered that there is both optically-derived data and inertia-derived data for the joints to which the inertial sensor 12 is attached. In S316, the data synthesis unit 250 compares each optically-derived data and each inertia-derived data, and determines the joints with both optically-derived data and inertia-derived data, thereby detecting the optically-derived data and the inertia-derived data to be synthesized.

[0077] Then, when the data synthesis unit 250 determines that there is a joint with both optically-derived data and inertia-derived data (S316 / Yes), based on the reliability of the optically-derived data and the reliability of the inertia-derived data of the corresponding joint, the optically-derived data and the inertia-derived data are synthesized, and the synthesized data as the synthesis result is used as the measurement data (S320).

[0078] On the other hand, when the data synthesis unit 250 determines that there is no joint with both optically-derived data and inertia-derived data (S316 / No), since it is considered that the inertial sensor 12 is not used, the optically-derived data is used as the measurement data (S324). After the processes of S320 and S324, the processes from S304 and S308 are repeated.

[0079] (Flow of Initialization Management) Next, the initialization management flow that operates in parallel with the overall flow shown in FIG. 6 will be described. This initialization management is periodically executed by the initialization management unit 240. Note that the processes of S404 to S428 shown in FIG. 7 are performed for each joint.

[0080] As shown in FIG. 7, the initialization management unit 240 receives the optically-derived data and the inertia-derived data to which the reliability has been added from the reliability calculation unit 230, and determines whether the initialization process has been performed (S404). Specifically, the initialization management unit 240 determines whether the variable init is true. If the initialization process has not been performed or is in the process of being executed (S404 / No), the process proceeds to S416. At the start of measurement, since the variable init of each joint is set to false, it is always determined that the initialization process has not been performed in the branch of S404, and the process proceeds to S416.

[0081] If the initialization management unit 240 determines that the initialization process has been performed (S404 / Yes), the initialization management unit 240 determines whether the reliability of the inertia-derived data is equal to or greater than the threshold value (S408). If the reliability of the inertia-derived data is equal to or greater than the threshold value (S408 / Yes), the initialization management unit 240 changes the target joint to another joint and repeats the process from S404. That is, if the reliability of the inertia-derived data is equal to or greater than the threshold value (S408 / Yes), the re-execution of the initialization process is not performed.

[0082] On the other hand, if the reliability of the inertia-derived data is less than the threshold value (S408 / No), the initialization management unit 240 sets the variable init to false (S412). That is, it is determined that the re-execution of the initialization process is necessary, and the variable init is returned to a state equivalent to the state where the initialization process has not been performed.

[0083] Here, in S312 of the overall flow executed in parallel, the reliability calculation unit 230 calculates the reliability of the inertia-derived data with reference to the variable init. Therefore, by setting the variable init to false in S412, the reliability of the inertia-derived data calculated in S312 becomes 0. That is, when the variable init is set to false again, until the re-execution of the initialization process in the initialization management flow is completed, the inertia-derived data does not affect the synthetic data, and the optical-derived data is directly calculated as the synthetic data. Note that there are as many variables init as there are inertia sensors 12 attached to the measurement subject, and the initialization management flow is performed for each joint attached to the measurement subject.

[0084] After S412, or after it is determined in S404 that the variable init is not true, the initialization management unit 240 determines the initial values of the position and orientation based on the optically-derived data given the reliability obtained from the reliability calculation unit 230 (S416). The calculation of the inertia-derived data in the inertial motion estimation unit 220 is performed by integrating the translational acceleration obtained from the inertial sensor 12 and adding the integration result to the initial values of the position and orientation. That is, S416 is a process of determining the initial values of the position and orientation that serve as a reference when calculating the inertia-derived data based on the optically-derived data.

[0085] When the integration error accumulates in the inertia-derived data of each joint and the reliability of the inertia-derived data decreases, the process of S416 is performed again to determine new initial values of the position and orientation. After the new initial values are applied, since the subsequent integration results are added to the new initial values of the position and orientation, it becomes possible to reset the integration error.

[0086] Note that the method for determining the initial values of the position and orientation may be any method. For example, the initialization management unit 240 may refer to the reliability of the optically-derived data and determine the position and orientation indicated by the optically-derived data whose reliability is equal to or higher than the reference as the initial values of the position and orientation.

[0087] In addition, the initialization management unit 240 may sample the optically-derived data at an arbitrary number of frames, and determine the average value of the sampled frames as the initial value. Note that while the initialization management unit 240 is sampling a plurality of frames in this way and calculating the average value, since the variable init is set to false, in S312 of the overall flow that operates in parallel, based on the above formula (4), the reliability of the inertia-derived data is calculated as 0. Therefore, in S320 where the data is synthesized, the component of the inertia-derived data is not synthesized into the synthesized data. For example, when the number of sampling frames is 10, the inertia-derived data is not synthesized for 10 frames until the sampling is completed and the re-execution of the initialization process is completed. In this way, by executing the initialization management flow in parallel with the overall flow, it is possible to dynamically re-execute the initialization process during the measurement of the movement of the measurement target person and reset the integration error.

[0088] The initialization management unit 240 passes the determined new initial value to the initialization processing unit 224 (S420), and the initialization processing unit 224 executes an initialization process of updating the initial values of the position and orientation with the new initial value (S424).

[0089] After that, the initialization management unit 240 returns the variable init to true. That is, the variable init is set to a state indicating that the initialization process has been executed (S428). When the variable init is returned to true again, in S312 of the overall flow executed in parallel, the reliability of the inertia-derived data is calculated as in the example of the formula shown in the upper part of formula (4).

[0090] Note that as the integration error is reset by the execution of the initialization process, the initialization management unit 240 also resets the elapsed time T shown in the upper part of formula (4) to 0. However, formula (4) is an example of a method for calculating the reliability, and even when another formula including parameters that change with the passage of time from the initialization process is used, the initialization management unit 240 initializes the above parameters included in the other formula with the execution of the initialization process. i

[0091] ​ <Function and Effect> According to an embodiment of the present invention described above, various functions and effects can be obtained. For example, the initialization processing unit 224 repeatedly executes an initialization process for setting initial values of the position and orientation. Further, the reliability calculation unit 230 calculates the reliability of the inertia-derived data so as to decrease as time elapses from the initialization process and increase by executing a new initialization process, and the initialization management unit 240 determines an initial value and passes it to the initialization processing unit 224 when the reliability of the inertia-derived data falls below a threshold. According to such a configuration, since the reset of the integration error in the inertia-derived data is repeated, it is possible to suppress the error of the inertia-derived data and the error of the synthetic data (measurement data) calculated using the inertia-derived data.

[0092] FIG. 8 is an explanatory diagram showing a processing example according to an embodiment of the present invention. As shown in FIG. 8, according to an embodiment of the present invention, since the initialization process is repeatedly executed at times t1, t2, t3, t4, etc., the error of the inertia-derived data is reset at times t1, t2, t3, t4, etc. For this reason, the error in the synthetic data also becomes low at times t1, t2, t3, t4, etc., so that the error in the synthetic data does not continue to increase as shown in FIG. 2, and it is possible to suppress the error in the synthetic data.

[0093] Note that when the data synthesis unit 250 synthesizes the inertia-derived data and the optically-derived data using the above formula (1), the parameter α may be a variable. For example, the data synthesis unit 250 may increase the parameter α from a predetermined value as time elapses and return it to the predetermined value at the timing of resetting the integration error in the inertia-derived data. In the example shown in FIG. 8, the data synthesis unit 250 may increase the parameter α over time t1, return the parameter α to the initial value (for example, 0.2) at time t1, and increase the parameter α again over time t2. According to such a configuration, it becomes possible to synthesize the optically-derived data and the inertia-derived data with a weight more suitable for the reliability of the inertia-derived data.

[0094] Also, in one embodiment of the present invention, the initialization management unit 240 determines an initial value using optically derived data. For example, the initialization management unit 240 refers to the reliability of the optically derived data, and determines the position and orientation indicated by the optically derived data with a reliability equal to or higher than a reference as the initial values of the position and orientation. According to such a configuration, it is possible to improve the accuracy of the inertia-derived data by setting the initial value to an appropriate value.

[0095] Also, in one embodiment of the present invention, when the initialization process has not been executed and when the initialization process is being executed, the reliability calculation unit 230 sets the reliability of the inertia-derived data to a predetermined value, and the data synthesis unit 250 continues to synthesize the optically derived data and the inertia-derived data based on the reliability of the inertia-derived data set to the predetermined value. That is, it is possible to execute the initialization process without interrupting the measurement of the motion of the measurement target person.

[0096] Also, in one embodiment of the present invention, since the calculation of inertia-derived data, the calculation of reliability, the initialization process, the data synthesis, etc. are performed for each joint of the measurement target person, it is possible to improve the accuracy of the measurement data indicating the position and orientation of each joint.

[0097] <Hardware Configuration> As described above, one embodiment of the present invention has been described. The information processing such as the calculation of the reliability and the determination of the initial value described above is realized by the cooperation of software and hardware. Hereinafter, a hardware configuration example applicable to the motion measurement device 20 will be described.

[0098] FIG. 9 is a block diagram showing an example of a hardware configuration 90. The hardware configuration 90 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904. The hardware configuration 90 also includes a bridge 905, an external bus 906, an interface 907, an input device 908, a display device 909, an audio output device 910, a storage device (HDD) 911, a drive 912, and a network interface 915.

[0099] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operation according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs, arithmetic parameters, etc. used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus or the like. Through the cooperation of these CPU 901, ROM 902, and RAM 903 with software, functions such as an optical data processing unit 216, an initialization processing unit 224, an inertial data processing unit 226, a reliability calculation unit 230, an initialization management unit 240, and a data synthesis unit 250 can be realized.

[0100] The host bus 904 is connected to an external bus 906 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 905. Note that it is not necessarily required to separately configure the host bus 904, the bridge 905, and the external bus 906, and these functions may be implemented on a single bus.

[0101] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, sensors, switches, and levers for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating the input device 908, the user can input various data or instruct processing operations.

[0102] The display device 909 includes, for example, display devices such as a liquid crystal display (LCD) device, a projector device, an organic light emitting diode (OLED) device, and a lamp. The audio output device 910 includes audio output devices such as speakers and headphones.

[0103] The storage device 911 is a data storage device configured as an example of the storage unit according to this embodiment. The storage device 911 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. The storage device 911 is composed of, for example, a hard disk drive (HDD) or a solid state drive (SSD), or a memory having an equivalent function. This storage device 911 drives the storage and stores the programs and various data executed by the CPU 901.

[0104] The drive 912 is a reader / writer for a storage medium, and is built in or externally attached to the hardware configuration 90. The drive 912 reads the information recorded on the removable storage medium 84 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory that is mounted, and outputs it to the RAM 903 or the storage device 911. The drive 912 can also write information to the removable storage medium 84.

[0105] The network interface 915 is a communication interface composed of, for example, a communication device for connecting to a network or the like. Further, the network interface 915 may be a wireless LAN (Local Area Network)-compatible communication device or a wire communication device that performs wired communication.

[0106] <Supplementary Note> As described above, the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modification examples or correction examples within the scope of the technical idea described in the claims, and it is naturally understood that these also belong to the technical scope of the present invention.

[0107] For example, each function shown in FIG. 5 may be implemented in one device (computer) or may be distributed and implemented in a plurality of devices. For example, the optical motion estimation unit 210 may be implemented in one device, the inertial motion estimation unit 220 may be executed in another device, and the reliability calculation unit 230, the initialization management unit 240, and the data synthesis unit 250 may be further executed in other devices. In this case, the communication between the devices may be performed wirelessly or wiredly, and the communication protocol is not limited. When each function is implemented in a plurality of devices in this way, it is possible to distribute the processing load.

[0108] Also, each step in the processing of the motion measurement device 20 in this specification does not necessarily need to be processed in time series in the order described as a flowchart. For example, each step in the processing of the motion measurement device 20 may be processed in an order different from the order described as a flowchart or may be processed in parallel.

[0109] In addition, a computer program for causing hardware such as a CPU, a ROM, and a RAM incorporated in the motion measurement device 20 to exhibit functions equivalent to those of the respective components of the motion measurement device 20 described above can also be created. Also provided is a non-transitory storage medium storing the computer program.

Explanation of Signs

[0110] 11 Optical sensor 12 Inertial sensor 20 Motion measurement device 210 Optical motion estimation unit 212 Optical sensor connection unit 216 Optical data processing unit 220 Inertial motion estimation unit 222 Inertial sensor connection unit 224 Initialization processing unit 226 Inertial data processing unit 230 Reliability calculation unit 240 Initialization management unit 250 Data synthesis unit

Claims

1. An optical data processing unit that estimates optical origin data indicating the position or orientation of a moving object based on data obtained from an optical sensor; An inertial data processing unit that estimates inertial origin data indicating the position or orientation of the moving object based on data obtained from an inertial sensor; A combining unit that combines the optical origin data and the inertial origin data based on the reliability of the optical origin data or the inertial origin data; An initialization processing unit that repeatedly executes an initialization process for setting an initial value of the position or orientation of the moving object used for the estimation of the inertial origin data in the inertial data processing unit; A reliability calculation unit that calculates the reliability of the inertial origin data so as to decrease as time elapses from the initialization process and increase when a new initialization process is executed; A data processing apparatus comprising the above.

2. The data processing apparatus further comprises: An initialization management unit that determines the initial value and passes it to the initialization processing unit when the initialization process has not been executed and when the reliability of the inertial origin data is below a threshold value. The data processing apparatus according to claim 1.

3. The initialization management unit determines the initial value using the optical origin data. The data processing apparatus according to claim 2.

4. When the initialization process has not been executed and when the initialization process is in progress, The reliability calculation unit sets the reliability of the inertial origin data to a predetermined value, The combining unit continues to combine the optical origin data and the inertial origin data based on the reliability of the inertial origin data set to the predetermined value. The data processing apparatus according to claim 3.

5. The inertial data processing unit, the reliability calculation unit, the initialization processing unit, and the initialization management unit execute processing for each of a plurality of parts constituting the moving object, The combining unit combines the optical origin data and the inertial origin data for the same part. The data processing apparatus according to claim 4.

6. The initialization management unit determines the initial value of a certain part of the moving object using the optical origin data for that part. The data processing apparatus according to claim 5.

7. Estimating optical origin data indicating the position or orientation of a moving object based on data obtained from an optical sensor; Estimating inertial origin data indicating the position or orientation of the moving object based on data obtained from an inertial sensor; Based on the reliability of the optically-derived data or the inertia-derived data, synthesizing the optically-derived data and the inertia-derived data; Repeatedly executing an initialization process for setting an initial value of the position or posture of the moving object used for the estimation of the inertia-derived data; Calculating the reliability of the inertia-derived data so as to decrease with the passage of time from the initialization process and increase with the execution of a new initialization process; A method executed by a computer, including the above.

8. A computer, An optical data processing unit that estimates optically-derived data indicating the position or posture of a moving object based on data obtained from an optical sensor; An inertia data processing unit that estimates inertia-derived data indicating the position or posture of the moving object based on data obtained from an inertial sensor; A synthesis unit that synthesizes the optically-derived data and the inertia-derived data based on the reliability of the optically-derived data or the inertia-derived data; An initialization processing unit that repeatedly executes an initialization process for setting an initial value of the position or posture of the moving object used for the estimation of the inertia-derived data in the inertia data processing unit; A reliability calculation unit that calculates the reliability of the inertia-derived data so as to decrease with the passage of time from the initialization process and increase with the execution of a new initialization process; A program for causing the computer to function as the above.

Images