Motion classification method using millimeter waves, millimeter wave radar sensor device, and program

Abstract

A method for classifying motion using millimeter waves is provided, which can achieve highly accurate motion classification. The method for classifying actions using millimeter waves includes a fractal dimension calculation step for calculating a fractal dimension Df(t) from an input signal x(t) from a millimeter wave radar sensor; a mean value μ(t) and a variance σ 2 A statistical feature calculation process calculates statistical features of (t) and the moving average difference Δμ(t) and combines them with the fractal dimension Df(t) to generate a feature vector f(t); and a Gaussian process regression interpolation is applied to the feature vector f(t) to generate a predictive distribution f GPR (t) is a Gaussian process regression imputation process that generates a predictive distribution f GPR A motion classification method including: a dimensionality reduction step of applying principal component analysis and an autoencoder to (t) to reduce the dimension and generate a low-dimensional feature vector y(t); a vector parallel body geometry visualization step of generating a geometric boundary Pc from the low-dimensional feature vector y(t); and a generative model inference step of applying a generative model using the low-dimensional feature vector y(t) and the geometric boundary Pc as conditions and outputting a motion classification result.

Description

[Technical Field]

[0001] The present invention relates to a motion classification technique using a non-contact sensor, and more particularly to a motion classification method using millimeter waves, a millimeter wave radar sensor device, and a program. [Background technology]

[0002] A millimeter-wave radar sensor transmits electromagnetic waves (millimeter waves) in the frequency band of 30 GHz to 300 GHz, receives the reflected waves that hit the sensing target and are reflected back, and uses the difference between the transmitted and received waves to detect the presence of an object and measure the distance to the object, its relative speed, and its positional relationship (direction).

[0003] Millimeter-wave radar sensors have traditionally been installed in vehicles such as automobiles and motorcycles to ensure vehicle safety and enable autonomous driving, and millimeter-wave radar sensors have been installed on roads and intersections to alleviate traffic congestion.

[0004] Furthermore, because millimeter-wave radar sensors are non-contact, highly accurate, and highly resistant to environmental conditions, they are expected to be used in a variety of applications, such as for watching over the elderly, monitoring infants, and crime prevention. For example, Patent Document 1 (JP 2024-540672 A) discloses a non-contact electrocardiogram monitoring method using millimeter-wave radar. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2024-540672 Summary of the Invention [Problem to be solved by the invention]

[0006] The present inventors have considered conventional millimeter wave radar sensor technology and have found that the following problems exist. (1) Threshold judgment cannot detect minute movements, and false positives or oversights may occur. (2) To determine behavior, deep learning methods such as LSTM require a large amount of labeled training data, which is expensive to collect. (3) Classification accuracy may become unstable due to environmental noise or missing data. (4) The feature space has poor interpretability, making it difficult to explain the classification basis.

[0007] The present inventors have completed this invention with the idea of ​​solving at least one of the above problems, and in one embodiment, it is an object of the present invention to provide an action classification method using millimeter waves that can achieve highly accurate action classification. In another embodiment, it is an object of the present invention to provide a millimeter-wave radar sensor device and a program for implementing such an action classification method. [Means for solving the problem]

[0008] As a result of intensive research, the inventors have found that highly accurate motion classification can be achieved even for minute motions or under noise by integrating the application of fractal dimension calculation, statistical feature calculation, probabilistic completion (Gaussian process regression completion), dimensionality reduction, vector parallelepiped geometric visualization, and generative model inference to time series signals acquired from a millimeter-wave radar sensor. The present invention was completed based on the above findings, and is exemplified below.

[0009] [Aspect 1] A method for classifying motion using millimeter waves, comprising: a fractal dimension calculation step of calculating a fractal dimension Df(t) from an input signal x(t) from a millimeter-wave radar sensor; Mean value μ(t), variance σ 2 a statistical feature calculation step of calculating statistical features of (t) and the moving average difference Δμ(t) and combining them with the fractal dimension Df(t) to generate a feature vector f(t); Apply Gaussian process regression imputation to the feature vector f(t) to obtain the predictive distribution f GPR Gaussian process regression imputation step to generate (t); Predictive distribution f GPR A dimension reduction process in which (t) is subjected to principal component analysis and an autoencoder to reduce the dimension and generate a low-dimensional feature vector y(t); A vector parallelepiped geometry visualization process that generates a geometric boundary Pc from the low-dimensional feature vector y(t); a generative model inference process, which applies a generative model to the low-dimensional feature vector y(t) and the geometric boundary Pc as conditions and outputs an action classification result; A method for classifying behavior, including: [Aspect 2] 2. The motion classification method of claim 1, wherein the generative model is a diffusion model. [Aspect 3] A millimeter wave radar sensor device including a sensor unit, an information processing unit, and a communication unit, The millimeter-wave radar sensor device, wherein the information processing unit is configured to execute the motion classification method according to aspect 1 or 2. [Aspect 4] A program configured to execute the action classification method according to aspect 1 or 2 when executed by a computer. [Aspect 5] A computer-readable storage medium having the program according to embodiment 4 recorded thereon. [Effects of the Invention]

[0010] According to one embodiment of the present invention, it is possible to provide a motion classification method using millimeter waves that can achieve highly accurate motion classification. In another embodiment, the present invention can provide an apparatus and a program for implementing such a motion classification method. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a flowchart of a method for motion classification using millimeter waves in accordance with an embodiment of the present invention. [Figure 2] FIG. 1 is a hardware configuration diagram of a millimeter-wave radar sensor according to an embodiment of the present invention. [Figure 3]10 shows the time variation of generated data Df(t) in the fractal dimension calculation step in an embodiment of the present invention. [Figure 4] 10 shows a time change of generated data μ in a statistical feature calculation step in an embodiment of the present invention. [Figure 5] 10 shows a time change of generated data σ2 in the statistical feature amount calculation step in an embodiment of the present invention. [Figure 6] 10 shows a time change of generated data Δμ(t) in a statistical feature calculation step in an embodiment of the present invention. [Figure 7] 10 is a graph showing the effect of smoothing by a Gaussian process regression interpolation process in an embodiment of the present invention. [Figure 8] FIG. 2 is a scatter plot of a reduced-dimensional feature vector y(t) obtained by a dimensionality reduction process in an embodiment of the present invention. [Figure 9] 10 shows the ellipse boundaries for each operation class of the vector parallel body geometry visualization process in an embodiment of the present invention. [Figure 10] 10 shows the change over time in prediction probability by the generative model inference process in an embodiment of the present invention. [Figure 11] 10 shows a change over time in sensing resolution of a millimeter-wave radar sensor device in an embodiment of the present invention. [Figure 12] The results of analyzing received data by distance from the sensing target using the conventional method are shown below. [Figure 13] 10 shows the results of analyzing received data according to the distance of the sensing target using the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Next, an embodiment of the present invention will be described with reference to the drawings. It should be understood that the present invention is not limited to the following embodiment, and that appropriate design changes and improvements may be made based on the ordinary knowledge of those skilled in the art without departing from the spirit of the present invention.

[0013] (1. Motion classification method) In one aspect, the present invention provides a method for classifying motion using millimeter waves. Hereinafter, an embodiment of the method for classifying motion of a sensing target using a signal from a millimeter-wave radar sensor will be described.

[0014] The phase and amplitude information from the millimeter-wave radar sensor is expected to be intermittent. In particular, when miniaturizing and power-saving the sensor device, the amount of data that can be processed is limited, resulting in intermittent data. In one embodiment, the time difference between such data may be 10 milliseconds to 10 seconds.

[0015] In order to fill in the gaps in the intermittent data and approximate the real situation more closely, the action classification method of this embodiment sequentially executes the following steps.

[0016] (1-1. Fractal dimension calculation process) First, the fractal dimension is analyzed. That is, the fractal dimension Df(t) is calculated from the input signal x(t) from the millimeter-wave radar sensor, and sent to the next process, the statistical feature calculation process. This has the effect of quantifying the complexity and pattern characteristics of behavior in the target area. The box-counting method is used for the calculation, and the calculation formula is as follows:

number

[0017] Details of how to calculate fractal dimension are described in "Falconer, K., Fractal Geometry: Mathematical Foundations and Applications, Wiley, 2014.", the entire contents of which are incorporated herein by reference.

[0018] (1-2. Statistical feature calculation process) Next, the mean value μ(t), variance σ 2Statistical features such as (t) and moving average difference Δμ(t) are calculated and combined with the fractal dimension Df(t) to generate a feature vector f(t), which is sent to the next step, the Gaussian process regression (GPR) interpolation step. This has the effect of expressing the features of the time series signal as a multidimensional vector. The calculation formulas for each statistical feature are as follows:

number

number

number

number

[0019] The feature vector f(t) is a vector that bundles all of the above elements. Each element has its own unit. In other words, the feature vector f(t) is a collection of multifaceted information that is input to subsequent processing (GPR, PCA, classification).

[0020] The above mean value μ(t) and variance σ 2 In addition to the moving average difference Δμ(t), phase features can also be used. In this case, the statistical features of the phase average φμ(t) and phase standard deviation φstd(t) are included in the feature vector f(t) (for example, f(t) = [μ(t),Df(t),Δμ(t),φμ(t),φstd(t)]). The calculation formulas for the phase average φμ(t) and phase standard deviation φstd(t) are as follows:

number

number

number

[0021] Details of how statistical features are calculated are described in Bendat & Piersol (2010), Chatfield (2003), Box et al. (2015), and Oppenheim & Schafer (2009), all of which are incorporated herein by reference in their entirety.

[0022] (1-3. Gaussian process regression imputation process) Next, we apply Gaussian process regression imputation to the feature vector f(t) to obtain the predictive distribution f GPR This generates (t), which then performs missing value imputation and noise reduction, and is sent to the next step, dimension reduction. As a result, Gaussian process regression imputation smooths out noise and makes discontinuous signals continuous, improving the smoothness and reliability of the data.

[0023] The Gaussian process regression imputation process yields the following prior and predictive distributions:

number

number

[0024] Details of how to compute Gaussian process regression imputation are described in "Rasmussen, CE, Williams, CKI, Gaussian Processes for Machine Learning, MIT Press, 2006.", which is incorporated herein by reference in its entirety.

[0025] (1-4. Dimensional compression process) Next, the predictive distribution f GPR Principal component analysis (PCA) and an autoencoder are applied to (t) to reduce its dimension and generate a low-dimensional feature vector y(t). This reduces the feature dimension (e.g., to 2-3 dimensions) while retaining important information, improving the computational efficiency of subsequent processing. Similar actions are close to each other, while different actions are farther apart. y(t) is plotted as points on a scatter plot and may trace a trajectory over time. The low-dimensional feature vector y(t) is then sent to the next process: vector parallelepiped geometry visualization and generative model inference.

[0026] Details of how to compute dimensionality reduction can be found in Jolliffe (2002, 2016), Goodfellow et al. (2016), and Hinton & Salakhutdinov (2006), which are incorporated herein by reference in their entireties.

[0027] (1-5. Vector parallel body geometry visualization process) Next, a geometric boundary (vector parallelepiped) Pc is generated from the low-dimensional feature vector y(t) and provided to the next step as a classification boundary condition. This clarifies the boundaries between action classes in the feature space, improving classification accuracy. The calculation formula for the geometric boundary Pc is as follows:

number

[0028] Details of vector parallelepiped geometry visualization can be found in Grunbaum (2003), Ziegler (1995), Boyd & Vandenberghe (2004), Barber et al. (1996), and Edelsbrunner (1987), which are incorporated herein by reference in their entireties.

[0029] (1-6. Generative model inference process) In the generative model inference process, a generative model is applied using the low-dimensional feature vector y(t) and the geometric boundary Pc as conditions, and the motion classification results (label and probability value) are output. This has the effect of enabling highly accurate motion classification even for unknown data. Classification labels include, for example, human motion classifications such as "walking," "sitting," "standing," "falling," and "lying down," medical and monitoring applications such as "normal breathing," "absent breathing," "abnormal body movement," and "resting state," and general state detection such as "motion present," "no motion," "minor motion (arm and hand movement)," and "major motion (walking and running)." While the type of generative model is not particularly limited, a diffusion model is preferred. The calculation formula when using a diffusion model is as follows:

[0030] Forward diffusion "noise addition process"

number

number

[0031] Details of the diffusion model can be found in Ho et al. (2020), Song et al. (2021), Dhariwal & Nichol (2021), Goodfellow et al. (2016), and Song & Ermon (2019), which are incorporated herein by reference in their entireties.

[0032] (1-7. Flowchart explanation) Figure 1 shows a flowchart of a motion classification method using millimeter waves according to one embodiment of the present invention. In Step 0 (sensor input step), an input signal x(t) from a sensing target (target area) is input from a millimeter-wave radar sensor and provided to Step 1 (fractal dimension calculation step). Here, it is possible to acquire motion information of the sensing target in a non-contact and continuous (or intermittent) manner.

[0033] In Step 1 (fractal dimension calculation process), the input signal x(t) from the millimeter-wave radar sensor is input, and the aforementioned fractal dimension Df(t) is generated and provided to Step 2 (statistical feature calculation process). This quantifies the complexity and scale dependency of the target behavior, contributing to improved classification performance.

[0034] In Step 2 (statistical feature calculation process), the fractal dimension Df(t) and the signal components derived from the input signal x(t) from the millimeter-wave radar sensor are input, and the mean value μ(t) and variance σ 2 A feature vector f(t) is generated by combining (t) and the moving average difference Δμ(t), and is provided to Step 3 (Gaussian process regression interpolation process). Information representation that integrates time trends and fluctuation characteristics can improve noise resistance and reproducibility.

[0035] In Step 3 (Gaussian process regression interpolation process), the feature vector f(t) is input and the aforementioned predictive distribution f GPR (t) is generated and provided to Step 4 (dimensionality reduction process). By imputing missing data and reducing noise, the accuracy of the subsequent dimensionality reduction and classification process can be improved.

[0036] In Step 4 (dimensionality reduction process), the predictive distribution f GPR (t) is input, and a low-dimensional feature vector y(t) is generated and provided to Step 5 (vector parallel body geometry visualization process) and Step 6 (generative model inference process). This reduces the redundancy of high-dimensional features and improves computational efficiency.

[0037] In Step 5 (vector parallelosphere geometric visualization step), the low-dimensional feature vector y(t) is input, and the geometric boundary Pc is generated and provided to Step 6 (generative model inference step). By expressing the classification boundary geometrically, the generalization performance of the model can be improved.

[0038] In Step 6 (generative model inference process), the low-dimensional feature vector y(t) and the geometric boundary Pc are input, and the classification label and its probability value are output. When using a diffusion model, the conditional generative ability of the diffusion model can be utilized to achieve high-precision classification even with a small amount of data.

[0039] (2. Millimeter-wave radar sensor device) In another aspect, the present invention provides a millimeter-wave radar sensor device for implementing the above-described motion classification method. The millimeter-wave radar sensor device used in one embodiment of the present invention may have the same basic hardware configuration as that of the prior art. Figure 2 shows an example of a hardware configuration diagram of a millimeter-wave radar sensor device.

[0040] The millimeter-wave radar sensor device comprises a sensor unit, an information processing unit, and a communication unit, and the sensor unit comprises one or more antennas for transmission (Tx) and reception (Rx) and an analog-to-digital converter (ADC). In one embodiment of the present invention, the millimeter-wave radar sensor can be, for example, Infineon's BGT60TR13C.

[0041] The information processing unit can identify information about the sensing target from phase and amplitude information in the signal from the sensor unit. The signal processing can be performed offline or in real time using a general-purpose processor. The information processing unit can be realized by a microcontroller unit (MCU). The information processing unit is configured to execute the above-mentioned motion classification method. The detailed steps of the motion classification method have been described above, so a detailed description thereof will be omitted here.

[0042] The communication unit displays information about the sensing target identified from the phase and amplitude information on a user interface or outputs the information for further processing.

[0043] Other configurations of the millimeter wave radar sensor device can be made using known techniques, and detailed description thereof will be omitted here.

[0044] (3. Program) In another aspect, the present invention provides a program configured to execute the action classification method of the present invention when executed by a computer. Since the detailed steps of the action classification method have been described above, a detailed description thereof will be omitted here. The present invention also provides a computer-readable storage medium having such a program recorded thereon. [Example]

[0045] Examples of the present invention are given below, but these examples are provided for a better understanding of the present invention and its advantages, and are not intended to limit the invention.

[0046] A real-time processing system was constructed using an Infineon BGT60TR13C millimeter-wave radar sensor as the sensor unit and a Teensy or Raspberry Pi for information processing, with the classification results and their probabilities used for monitoring and alert generation. The input signal x(t) from the millimeter-wave radar sensor was processed sequentially through the steps (1-1) to (1-6) above, and the classification results and their probabilities were output and used for monitoring and alert generation.

[0047] Figure 3 shows the change over time in the data Df(t) generated by the fractal dimension calculation process. This quantifies the complexity of the signal over time. As mentioned above, the calculation method is to count N(ε, t) using the box counting method and calculate the fractal dimension Df(t). The value of the fractal dimension Df(t) changes rapidly with changes in behavior (for example, hand or body movement), and the typical range is about 1.0 to 2.0 (depending on the implementation and window length). To read the graph, the fractal dimension Df(t) will be low in sections where the sensing object is nearly stationary, and high in sections where there is a lot of movement or noise. For this reason, this graph is effective for detecting change points.

[0048] Next, a feature vector f(t) is generated. The feature vector f(t) indicates the stability and fluctuation of the signal, and its value fluctuates significantly when an abnormality or movement occurs. Figure 4 shows the time evolution of the generated data μ generated by the statistical feature calculation process. The average value μ(t) is calculated from the μ data in the observation window. Its unit corresponds to the input unit and is the reference level of the signal. The average value μ(t) responds to slow trends and posture changes.

[0049] Figure 5 shows the generated data σ 2 It shows the time change of (t). Its unit is the square of the input unit. This shows the strength of the fluctuation, and in sections with a lot of movement or noise, the variance σ 2 The value of (t) becomes higher and becomes lower in the stable zone.

[0050] Figure 6 shows the time change of the generated data Δμ(t) in the statistical feature calculation process. The moving average difference Δμ(t) represents the time difference (rate of change) such as μ(t)-μ(t-1), and is expressed in input units. When there is a sudden movement, the moving average difference Δμ(t) rises and a spike waveform is observed, making it sensitive to the moment of change.

[0051] The feature vector f(t) is a vector that bundles all of the above elements. Each element has its own unit. The feature vector f(t) is a collection of multifaceted information that is input to subsequent processing (GPR, PCA, classification).

[0052] 7A and 7B are graphs showing the effect of smoothing by the Gaussian process regression interpolation step in an embodiment of the present invention. FIG. 7A shows the “original signal” (f raw (t)), and Fig. 7(B) shows the smoothed and imputed estimates (f GPR This smooths out any defects or noise and outputs a continuous signal.

[0053] Figure 8 is a scatter plot of the low-dimensional feature vector y(t) obtained by the dimensionality reduction process in an embodiment of the present invention. Here, multidimensional data is reduced to two-dimensional data and plotted using PC1 (principal component 1) / PC2 (principal component 2). Similar behaviors are close to each other, and different behaviors are far apart, so separability can be evaluated by the cluster shape and distance.

[0054] Figure 9 shows the elliptical boundaries of each action class in the vector parallel body geometry visualization process in an embodiment of the present invention. This clarifies the identification range. Note that the boundaries may be elliptical, rectangular, hexagonal, or other shapes.

[0055] 10 shows the time evolution of the prediction probability by the generative model inference process. The prediction probability is for the labels “no movement of the sensing target (person)” and “movement of the sensing target (person).”

[0056] Figure 11 shows the change over time in the sensing resolution of a millimeter-wave radar sensor device. The X-axis (horizontal axis) represents time, and indicates the period during which sensing data is acquired (e.g., 0 to 10 seconds). The Y-axis (vertical axis) represents the sensing resolution, indicating the minimum position change that can be detected in a single sensing operation. The smaller the resolution, the higher the detection capability (e.g., 0.02 m = 2 cm).

[0057] The dashed line shows the sensing resolution when using the conventional method, which uses the sensor input as is without using the motion classification method of the present invention, resulting in high noise and unstable resolution. The solid line shows the case where the motion classification method of the present invention is applied, resulting in stable resolution and high accuracy. Specifically, the sensing resolution improved from 0.06 m to 0.02 m or less. This enables non-contact, highly sensitive sensing, such as detecting falls, breathing, and subtle body movements.

[0058] Figure 12 shows the results of analyzing received data according to the distance of the sensing target using the conventional method. The horizontal axis shows the distance (cm) from the sensing target (person), and the vertical axis shows the strength of the received data (no units). In this test, measurements were taken at a distance of approximately 70 cm between the millimeter-wave radar sensor and the sensing target (person).

[0059] Figure 13 shows the data results for the method of the present invention, with and without all-step processing, depending on the distance to the sensing target. The horizontal axis shows the distance (cm) from the sensing target (person), and the vertical axis shows the received data strength (unitless). In this test, measurements were taken when the distance between the millimeter-wave radar sensor and the sensing target (person) was approximately 70 cm.

[0060] Comparing Figures 12 and 13, the peak near 70 cm appears more prominently in Figure 5, demonstrating that highly accurate detection is possible.

[0061] From the above, it can be understood that various embodiments of the present invention can achieve at least one of the following effects. (1) Stable detection of minute movements (shaking or signs of abnormalities). (2) Achieve high-precision classification even in a small data environment. (3) To provide a classifier that is robust to noise and missing data. (4) Visualize the basis of classification results to improve interpretability. [Industrial Applicability]

[0062] The present invention is widely applicable to fields requiring high-precision non-contact sensing, such as elderly care, infant monitoring, crime prevention, pest control, smart home control, in-vehicle sensing, and drone-mounted monitoring.

Claims

1. A method for classifying motion using millimeter waves, comprising: a fractal dimension calculation step of calculating a fractal dimension Df(t) from an input signal x(t) from the millimeter wave radar sensor; For the input signal x(t), the average value μ(t) and the variance σ 2 a statistical feature calculation step of calculating statistical features of the moving average difference Δμ(t) and the fractal dimension Df(t) to generate a feature vector f(t); Apply Gaussian process regression imputation to the feature vector f(t) to obtain the predicted distribution f GPR a Gaussian process regression imputation step to generate (t); Predictive distribution f GPR a dimension reduction step of applying principal component analysis and an autoencoder to (t) to reduce the dimension and generate a low-dimensional feature vector y(t); A vector parallelosphere geometry visualization process that generates a geometric boundary Pc from the low-dimensional feature vector y(t); a generative model inference step in which a generative model is applied to the low-dimensional feature vector y(t) and the geometric boundary Pc as conditions, and an action classification result is output; A method for classifying behavior, including:

2. The method of claim 1 , wherein the generative model is a diffusion model.

3. A millimeter wave radar sensor device including a sensor unit, an information processing unit, and a communication unit, 3. A millimeter wave radar sensor device, wherein the information processing unit is configured to execute the motion classification method according to claim 1 or 2.

4. A program configured to, when executed by a computer, carry out the action classification method according to claim 1 or 2.

5. A computer-readable storage medium on which the program according to claim 4 is recorded.

Images