Data processing device, data processing method and program

Abstract

This data processing device (1) includes a filtering unit (14) which performs a filtering process on second data at a first time included in time-series second data obtained by performing a predetermined operation on time series first data, by using a time difference that is a difference between the first time and a second time, and a feature amount difference that is a difference between a feature amount obtained from the first data at the first time and the feature amount obtained from the first data at the second time. This filtering process is a process for calculating the weighted average by weighting the second data at each of a plurality of different second times such that the larger the time difference, the smaller the weight, and the larger the feature amount difference, the smaller the weight.

Description

[Technical Field]

[0001] The present invention relates to a data processing device, a data processing method, and a program. This application claims priority based on Japanese Patent Application No. 2020-78396, filed on April 27, 2020, the contents of which are incorporated herein by reference. [Background technology]

[0002] Motion sensors, which measure human movement, have been widely used as one of the key elemental technologies for realizing future user interfaces that allow anyone to intuitively use computers and other machines by utilizing natural human hand and body movements and gestures, and as a technology for making characters move realistically in computer animation, etc. For this reason, sensors based on a variety of principles have been proposed.

[0003] A motion sensor uses the sensor's time-series measurement values ​​as raw data and outputs motion data by performing some kind of calculation on the raw data. Motion data is time-series data that represents movement. Regardless of the principle it is based on, this motion data generally contains a lot of noise. There are two types of noise: impulse noise and Gaussian noise. Impulse noise is noise that appears independently and randomly with a certain frequency. Gaussian noise is noise whose values ​​follow a Gaussian distribution.

[0004] To remove this noise, the motion data is subjected to filtering. For example, the Kalman filter is one of the filtering methods most commonly used for time-series data. Specifically, the Kalman filter is a filtering method that corrects predicted values ​​using the error obtained by comparing values ​​predicted from past states with actual measured values ​​by taking into account the correlation between data frames (see, for example, Non-Patent Document 1). There is also a filtering method that removes noise using a neural network (see, for example, Non-Patent Document 2).

[0005] Meanwhile, in the field of image processing, several nonlinear spatial filters have been proposed that calculate weighted averages by exploiting the correlation between pixels. These methods are particularly effective at removing noise from images. For example, Gaussian filters calculate weights based on the distance between pixels. However, Gaussian filters have the drawback of blurring edges in exchange for noise removal. Bilateral filters address this drawback. Bilateral filters consider both pixel values ​​and distances (see, for example, Non-Patent Document 3). Therefore, bilateral filters are less likely to destroy image structure while removing noise. Furthermore, the Non-local Means (NLM) method uses small patches centered on a pixel of interest as structures and compares patches across the entire image to calculate a weighted sum (see, for example, Non-Patent Document 4). This allows the NLM method to achieve even more effective filtering. There is also technology that applies bilateral filters to motion data (see, for example, Non-Patent Document 5). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] RE Kalman, "A New Approach to Linear Filtering and Prediction Problems", Transactions of the ASME-Journal of Basic Engineering, Vol. 82, Series D, p. 35-45, 1960. [Non-patent document 2] D. Holden, J. Saito, T. Komura and T. Joyce, "Learning motion manifolds with convolutional autoencoders", In SIGGRAPH Asia 2015 Technical Briefs, No. 18, p.1-4, 2015. [Non-patent document 3] C. Tomasi and R. Manduchi, "Bilateral filtering for gray and color images", In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), p.839-846, Jan 1998, doi: 10.1109 / ICCV.1998.710815 [Non-patent document 4] A. Buades,B. Coll and J.-M. Morel,"A non-local algorithm for image denoising",In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) vol. 2,p. 60-65,No.02,June 2005,doi:10.1109 / CVPR.2005.38 [Non-Patent Document 5] N. Courty, "Bilateral human motion filtering", 16th European Signal Processing Conference (EUSIPCO 2008), 2008. Summary of the Invention [Problem to be solved by the invention]

[0007] The technology of Non-Patent Document 1 above assumes that the motion of the subject is linear and that the noise is Gaussian. Therefore, it is difficult to apply it to motion data of random motion, such as human movement. Furthermore, the technology of Non-Patent Document 2 requires neural network training. Therefore, there is room for improvement in noise removal accuracy. The technologies of Non-Patent Documents 3 and 4 remove noise using the image data itself to be filtered. Therefore, they cannot be applied directly to filtering of motion data. Furthermore, the technology of Non-Patent Document 5 is expected to have reduced accuracy when the original data contains large noise, such as missing data.

[0008] In view of the above circumstances, an object of the present invention is to provide a data processing device, a data processing method, and a program that can accurately remove noise from data obtained by performing operations on time-series raw data. [Means for solving the problem]

[0009] One aspect of the present invention is a data processing device that includes a filtering unit that performs a filtering process on second data at a first time included in second data of a time series obtained by performing a predetermined operation on first data of a time series, using a time difference that is the difference between the first time and a second time, and a feature difference that is the difference between a feature obtained from the first data at the first time and a feature obtained from the first data at the second time, and the filtering process is a process that weights the second data for each of a plurality of different second times so that the larger the time difference, the smaller the weight, and so that the larger the feature difference, the smaller the weight, and calculates a weighted average.

[0010] One aspect of the present invention is the above-mentioned data processing device, wherein the predetermined operation is a non-linear process.

[0011] One aspect of the present invention is the above-mentioned data processing device, wherein the first data is measurement value data of a magnetic sensor, measurement value data of an inertial sensor, or image data, and the second data is motion data representing movement.

[0012] One aspect of the present invention is a data processing method including a filtering step of filtering second data at a first time included in second data of a time series obtained by performing a predetermined operation on first data of a time series, using a time difference that is the difference between the first time and a second time, and a feature difference that is the difference between a feature obtained from the first data at the first time and a feature obtained from the first data at the second time, wherein the filtering step is a process of weighting the second data at each of a plurality of different second times such that the larger the time difference, the smaller the weight, and the larger the feature difference, the smaller the weight, to calculate a weighted average.

[0013] One aspect of the present invention is a program for causing a computer to function as any one of the above-described data processing devices. [Effects of the Invention]

[0014] According to the present invention, it is possible to accurately remove noise from data obtained by performing calculations on time-series raw data. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a configuration of a data processing device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a flowchart showing processing of the data processing device according to the embodiment. [Figure 3] FIG. 1 shows an outline of the first experiment. [Figure 4] FIG. 1 shows the results of a first experiment. [Figure 5] FIG. 1 shows the results of a first experiment. [Figure 6A] FIG. 10 shows an outline of the second experiment. [Figure 6B] FIG. 10 shows an outline of the second experiment. [Figure 6C] FIG. 10 shows an outline of the second experiment. [Figure 7] FIG. 10 shows the results of a second experiment. [Figure 8A] FIG. 10 shows the results of a second experiment. [Figure 8B] FIG. 10 shows the results of a second experiment. [Figure 8C] FIG. 10 shows the results of a second experiment. [Figure 9A] FIG. 10 shows the results of a second experiment. [Figure 9B] FIG. 10 shows the results of a second experiment. [Figure 9C] FIG. 10 shows the results of a second experiment. [Figure 10A] FIG. 10 shows the results of a second experiment. [Figure 10B] FIG. 10 shows the results of a second experiment. [Figure 10C] FIG. 10 shows the results of a second experiment. [Figure 11A] FIG. 10 shows the results of a second experiment. [Figure 11B] FIG. 10 shows the results of a second experiment. [Figure 11C] FIG. 10 shows the results of a second experiment. [Figure 12A] FIG. 10 shows the results of a second experiment. [Figure 12B] FIG. 10 shows the results of a second experiment. [Figure 12C] FIG. 10 shows the results of a second experiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, a case will be described in which second time-series data obtained by performing a predetermined calculation on first time-series data is motion data obtained by performing a calculation such as nonlinear processing on raw data indicating measurement values ​​from a sensor.

[0017] In this embodiment, a nonlinear filter that applies the concept of spatial filtering for images to time-series motion data is used. This nonlinear filter is referred to as a Structure-Aware Temporal Bilateral Filter (SATBF). Motion data consists of a time-series of frames indicating position, velocity, and the like at each time. Each frame included in the motion data (hereinafter referred to as a motion data frame) is calculated based on time-series measurements by a sensor or the like. That is, to generate the motion data, raw data indicating the time-series measurements by a sensor or the like is stored in a storage device. Many previous filtering techniques have focused on removing noise using only the motion data. In contrast, the most important feature of the SATBF of this embodiment is that it calculates weights for each motion data frame using the raw data used to calculate the motion data, and then uses the calculated weights to calculate a weighted average. These weights are calculated taking into account the similarity between frames (hereinafter referred to as raw data frames) indicating sensor measurements at different times included in the raw data. For example, even if a certain raw data frame contains significant noise, the raw data frames before and after that raw data frame contain similar features. Therefore, SATBF simultaneously uses two different types of information, the original data frames and time, to filter the motion data frames, thereby removing noise without destroying the fine, essential structure of the motion.

[0018] 1 is a diagram showing the configuration of a data processing device 1 according to one embodiment of the present invention. The data processing device 1 has an original data storage unit 11, a motion data generation unit 12, a motion data storage unit 13, and a filtering unit 14.

[0019] The raw data storage unit 11 stores raw data including a time-series raw data frame. The raw data frame is data including a time and measurement data at that time. The measurement data is data indicating measurement values ​​obtained by a sensor or the like measuring a measurement target.

[0020] The motion data generation unit 12 generates motion data by performing a predetermined calculation on the raw data stored in the raw data storage unit 11. The motion data consists of a time series of motion data frames. A motion data frame is data indicating a time and the position of the measurement target at that time. A motion data frame may further include data representing the movement of the measurement target, such as the direction, magnitude, and speed of the movement. The motion data storage unit 13 stores the motion data generated by the motion data generation unit 12.

[0021] The filtering unit 14 performs a filtering process using SATBF on each motion data frame of the motion data stored in the motion data storage unit 13. The SATBF is based on the original data frame group of the original data stored in the original data storage unit 11.

[0022] The SATBF is designed based on the Bilateral Filter, which calculates the weighted average value using multiple elements. n The i-th (i=1, 2, ...) motion data frame r i The result of filtering by SATBF is C i is calculated by the following formula (1):

[0023]

number

[0024] In equation (1), d(i,n) is the i-th original data frame I(t i ) and the nth original data frame I(t n ) and the difference between the features obtained from t i is the i-th motion data frame r i is the time t n is the nth motion data frame r nThe constants α and β are adjustable coefficients. The constants ρ and σ are coefficients of a Gaussian filter obtained by noise analysis. For example, the values ​​of the constants ρ and σ are calculated based on the noise present in the motion data obtained when the measurement target is stationary.

[0025] As described above, the filtering unit 14 filters the time series motion data frame (second data) obtained by performing a predetermined calculation on the time series original data frame (first data) at the time t i Motion data frame r (first time) i (Second data) at time t i (first time) and time t n (second time) i -t n , and time t i (First time) original data frame I(t i ) (first data) and time t n (Second time) original data frame I(t n ) (first data) and the feature difference d(i,n) between them. This filtering process by SATBF is performed at different times t n (Second time) Each motion data frame r n , the time difference t i -t n The larger the difference in feature quantity d(i,n) of the original data frame, the smaller the weight. The weighting is performed so that the larger the difference in feature quantity d(i,n) of the original data frame, the smaller the weight. The weighted motion data frame r n This is a process of calculating a weighted average of

[0026] The measurement data used in the raw data, the feature values ​​obtained from the raw data frame, the type of data used as motion data, and the calculations for generating the motion data can be any. For example, the raw data can include data representing measurements from magnetic sensors, such as time-series current measurements, voltage measurements, and acceleration measurements; data representing measurements from inertial sensors, such as angular velocity sensors and acceleration sensors; and image data. A camera capturing image data is a type of optical sensor. When the raw data includes measurements from current, voltage, acceleration, and angular velocity sensors, these measurements can be used as feature values. Furthermore, when the raw data includes measurements from multiple sensors, some or all of these measurements can be used as feature values. For example, when the raw data includes current and voltage measurements, the current measurements, voltage measurements, or both current and voltage measurements can be used as feature values. When the raw data is image data, the pixel values ​​of each pixel can be used as feature values. One example of a calculation for generating motion data from raw data is nonlinear processing, such as a neural network.

[0027] 2 is a flow diagram showing the processing of the data processing device 1. The raw data storage unit 11 stores raw data I that the data processing device 1 receives from a sensor or another information processing device (step S1). For example, the data processing device 1 acquires raw data I every hour from equipment such as a sensor array, and stores the acquired raw data I in the raw data storage unit 11. The raw data I includes raw data frames I(t1), I(t2), ...

[0028] The motion data generating unit 12 performs some processing on the original data I stored in the original data storage unit 11 to generate motion data R consisting of time-series motion data frames r1, r2, ... (step S2). For example, the motion data generating unit 12 performs processing such as nonlinear transformation on the original data frame I(t i ) to time t i A motion data frame r(ti ) is calculated from the original data frame I(t i -k1) ~ original data frame I(t i +k2) (k1 ≥ 0, k2 ≥ 0), time t i The motion data frame r(t i ) may be generated. The motion data storage unit 13 stores the motion data R generated by the motion data generation unit 12 (step S3).

[0029] The filtering unit 14 filters the motion data frame r(t i ), the original data frame I(t i -k3)~I(t i +k4) (k3 ≥ 0, k4 ≥ 0), and the filtering result C(t i ) is generated (step S4). Filtering by SATBF is i-2 ), r(t i-1 ), r(t i ), r(t i+1 ), r(t i+2 )... and calculate the average value. The weights are based on two factors: the similarity between the original data frames I(t) at each time t, and the difference between the times t. The weighted average value using weights based on these factors is the output of the SATBF filtering. The filtering result C(t i ) is C in Eq. (1) i The filtering unit 14 outputs a processed data group C including the filtering results C(t1), C(t2), . . .

[0030] The data processing device 1 does not need to have the motion data generation unit 12. In this case, the data processing device 1 does not perform the process of step S2, but acquires raw data and motion data from another device or system that generates motion data. In step S1, the data processing device 1 writes the acquired raw data to the raw data storage unit 11, and in step S3, writes the acquired motion data to the motion data storage unit 13. This allows the data processing device 1 to, for example, use SATBF to remove noise from the motion data that is the raw output from the motion sensor.

[0031] Furthermore, when the original data is an image, and the original data includes multiple objects for which motion data is to be generated (for example, a head and elbows), the processing of FIG. 2 is performed for each of the objects for which motion data is to be generated.

[0032] The data processing device 1 is realized by one or more computers. Examples of the computers include, but are not limited to, personal computers, server computers, wearable computers, tablet terminals, and smartphones. When the data processing device 1 is realized by multiple computers connected via a network, it is possible to arbitrarily determine which computer implements which functional unit. For example, a first computer may have the original data storage unit 11, and a second computer may have the motion data generation unit 12, the motion data storage unit 13, and the filtering unit 14. Alternatively, the first computer may have the original data storage unit 11, the motion data generation unit 12, and the motion data storage unit 13, and the second computer may have the filtering unit 14. Furthermore, one functional unit may be realized by multiple computers. For example, the first computer may have the original data storage unit 11, the motion data generation unit 12, and the motion data storage unit 13, and the second computer may have the original data storage unit 11, the motion data storage unit 13, and the filtering unit 14. In this case, the original data storage unit 11 and the motion data storage unit 13 of the second computer device store the original data and the motion data transmitted from the first computer device.

[0033] The results of experiments confirming the effectiveness of this embodiment are described below. As described above, SATBF can more effectively remove noise than other filtering methods without destroying the detailed, essential structure of the motion represented by the motion data. Therefore, we conducted several experiments using IM3D+, a magnetic 3D motion capture system, to confirm this effect. IM3D+ is a system that inputs current and voltage data measured by multiple pickup coils into a neural network and outputs motion data. The markers whose 3D positions are measured by the pickup coils are LC coils, and the 3D positions of the markers over time can be obtained from the motion data.

[0034] (First experiment) Figure 3 shows an overview of the first experiment. Marker M is the target for 3D position measurement using IM3D+. Marker M was fixed at a certain position within the measurement area and rotated 360 degrees in a vertical plane around a certain horizontal axis. As described above, IM3D+ generates motion data using current and voltage data detected by multiple pickup coils P. However, due to the magnetic measurement principle of IM3D+, position measurement becomes impossible when marker M enters a certain angular range. Within this angular range, measurement accuracy is significantly reduced due to noise. Therefore, we compared the motion data before and after filtering using SATBF.

[0035] 4 and 5 show the results of the first experiment. Fig. 4 shows the ratio of the error before SATBF processing to the error after SATBF processing. The error before SATBF processing is the percentage deviation between the 3D position indicated by the motion data (raw data) before SATBF filtering and the actual 3D position of the marker. The error before SATBF processing is the percentage deviation between the 3D position indicated by the motion data after SATBF filtering and the actual 3D position of the marker. Fig. 4 shows the average error ratio for nine measurement points within the measurement region. When the orientation of marker M was near 90 degrees and 270 degrees, the measurement error before SATBF processing was significantly large. However, it can be seen that SATBF processing significantly improved these errors.

[0036] Figure 5 shows a list of the results for each of the nine measurement points. ID is the identification of the measurement point. xyz indicates the position of the measurement point. EP is the error before SATBF processing, and ER is the ratio of the error before SATBF processing to the error after SATBF processing. LRP is the proportion of raw data before SATBF processing that contains unmeasurable data. LRO is the proportion of motion data after SATBF processing that contains unmeasurable data. As shown by LRO, all data was restored after SATBF processing.

[0037] (Second experiment) 6A to 6C show an overview of the second experiment. In the second experiment, an IM3D+ marker M was fixed to the tip of a high-precision robot arm, and the robot arm was moved along three types of motion trajectories. The types of motion trajectories were pendulum motion, linear motion, and circular motion. FIG. 6A shows a robot arm performing pendulum motion, FIG. 6B shows a robot arm performing linear motion, and FIG. 6C shows a robot arm performing circular motion. The upper diagrams in each of FIGS. 6A to 6C show the positional relationship between the robot arm and the pickup coil, and the lower diagrams in each of FIGS. 6A to 6C show the marker M attached to the robot arm shown in the rectangle in the upper diagram and the motion trajectory of that marker M.

[0038] When marker M moves along the three types of trajectories described above, it passes through postures that make it impossible to measure, resulting in significant noise in the raw data measured. Therefore, we compared the motion data (raw) obtained using this raw data with the motion data filtered by SATBF and the motion data filtered by conventional filters. Four types of widely used filters, namely, UKF (Unscented Kalman Filter), BF-conf-dis, NLM (Non Local Means), and TCF (Temporal Convolutional Filter), were used as conventional filters.

[0039] UKF is described in, for example, Reference 1, "EA Wan and R. Van Der Merwe, 'The unscented Kalman filter for nonlinear estimation,' In Adaptive Systems for Signal Processing, Communications, and Control Symposium 2000. AS-SPCC. The IEEE 2000, pp. 153-158, February 2000." BF-conf-dis is a bilateral filter that uses a distance function based on the configuration of markers. NLM is described in, for example, Non-Patent Document 4, and TCF is described in, for example, Non-Patent Document 2.

[0040] Figures 7, 8A to 8C, 9A to 9C, 10A to 10C, 11A to 11C, and 12A to 12C show the results of the second experiment. Figure 7 shows a graph of the error rate (error rate) of the motion data after filtering by each filter to the error in the motion data (raw) before filtering. The results of filtering using the SATBF of this embodiment show that more effective noise removal is possible compared to filtering using other conventional filters, and that this method is superior to all of the conventional techniques.

[0041] 8A to 8C, 9A to 9C, 10A to 10C, 11A to 11C, and 12A to 12C show the three-dimensional position information of the marker M indicated by the motion data before and after filtering processing using each filter for pendulum motion, linear motion, and circular motion, respectively. Figs. 8A to 8C show the results when SATBF of this embodiment is used, Figs. 9A to 9C show UKF, Figs. 10A to 10C show BF-conf-dis, Figs. 11A to 11C show NLM, and Figs. 12A to 12C show TCF for filtering processing. 8A, 9A, 10A, 11A, and 12A show the case of pendulum motion, 8B, 9B, 10B, 11B, and 12B show the case of linear motion, and 8C, 9C, 10C, 11C, and 12C show the case of circular motion. In these figures, circles represent the three-dimensional positions indicated by the motion data (raw data) before filtering, and triangles represent the three-dimensional positions indicated by the motion data after filtering. These figures also show that the SATBF of this embodiment accurately reproduces the original motion trajectory even when the raw data contains significant noise.

[0042] The data processing device 1 described above includes a processor such as a central processing unit (CPU), a memory, an auxiliary storage device, and the like, all connected via a bus, and the processor executes a program to realize the above functions. Note that all or part of the functions of the data processing device 1 may be realized using hardware such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). The program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as a flexible disk, a magneto-optical disk, a read-only memory (ROM), a compact disc read-only memory (CD-ROM), a universal serial bus (USB) memory, and a solid-state drive (SSD), as well as storage devices such as a hard disk built into a computer system. The program may be transmitted via a telecommunications line.

[0043] According to the above-described embodiment, the data processing device includes a filtering unit. The filtering unit performs a filtering process on second data at a first time included in time-series second data obtained by performing a predetermined calculation on time-series first data, using a time difference between the first time and the second time, and a feature difference between a feature obtained from the first data at the first time and a feature obtained from the first data at the second time. This filtering process is a process of calculating a weighted average by weighting the second data at each of a plurality of different second times such that the larger the time difference is, the smaller the weight is, and the larger the feature difference is.

[0044] For example, the predetermined calculation for obtaining the second time-series data is nonlinear processing. Also, for example, the first data is data of measurement values ​​of a magnetic sensor, data of measurement values ​​of an inertial sensor, or image data, and the second data is motion data representing movement.

[0045] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0046] 1... data processing device, 11... original data storage unit, 12... motion data generation unit, 13... motion data storage unit, 14... filtering unit

Claims

1. a filtering unit that performs a filtering process on second data at a first time among second data in time series obtained by performing a predetermined operation on first data in time series, using a time difference that is a difference between the first time and a second time, and a feature difference that is a difference between a feature obtained from the first data at the first time and a feature obtained from the first data at the second time, the filtering process is a process of calculating a weighted average by weighting the second data of each of the plurality of different second time points such that the weight decreases as the time difference between the first time point and the second time point of the second data increases and the feature amount difference increases, the first data is data of a measurement value obtained by a sensor, The second data is motion data representing a movement. Data processing device.

2. The predetermined operation is a nonlinear process.

2. The data processing device according to claim 1.

3. The first data is data of a measurement value of a magnetic sensor, data of a measurement value of an inertial sensor, or image data.

3. The data processing device according to claim 1.

4. a filtering step of performing a filtering process on second data at a first time among second data in time series obtained by performing a predetermined operation on first data in time series, using a time difference that is the difference between the first time and a second time, and a feature difference that is the difference between a feature obtained from the first data at the first time and a feature obtained from the first data at the second time; and the filtering process is a process of calculating a weighted average by weighting the second data of each of the plurality of different second time points such that the weight decreases as the time difference between the first time point and the second time point of the second data increases and the feature amount difference increases, the first data is data of a measurement value obtained by a sensor, The second data is motion data representing a movement. Data processing methods.

5. Computer, A program for causing the data processing device according to any one of claims 1 to 3 to function.

Images