Multi-dimensional radar + X human body three-dimensional sensing method and system

Abstract

The invention discloses a multi-dimensional radar + X human body three-dimensional sensing method and system. Comprising the steps that signals are collected, and multi-mode original data with timestamps are output based on a multi-mode millimeter wave radar, a visual mode and X kinds of medical gold standard modes; performing labeling and alignment based on the multi-modal original data; preprocessing the radar echo, and outputting an original feature tensor of a distance-speed-time dimension; performing space-time alignment, mutual information / correlation evaluation and weight learning on the multi-modal features, training a detection / identification / regression model, and feeding the model back to signal processing parameters; and generating a detection result. According to the method, the problems of original data missing and mode singleness are solved, and a reproducible and comparable reference environment is provided for subsequent algorithm development, clinical verification and open data set construction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional perception in general. More specifically, the present application relates to a multi-dimensional radar + X human three-dimensional perception method and system. BACKGROUND

[0002] There are several practical bottlenecks in the current millimeter wave radar human perception field, and the urgent need for higher quality, multi-dimensional data in scientific research and clinical application. First, existing public data sets and systems often do not provide original I / Q or bin-level data, only giving dimensionality reduction or pre-processed representations, which makes it impossible for researchers to develop method innovations or reproduce experimental results at the signal level; second, most work focuses on a single frequency band or a single array configuration, which cannot systematically evaluate the impact of different frequency bands, antenna apertures, and array topologies on spatial resolution, micro-motion detection capability, and physiological signal extraction; third, auxiliary labeling often stops at 2D posture or coarse-grained action labels, lacking frame-by-frame three-dimensional voxels, surface meshes, or labeling aligned with medical gold standard timelines, which greatly limits the feasibility of radar in fine-grained physiological state monitoring and clinical transformation. In addition to this, the problems of cross-modal synchronization and spatial calibration, and the contradiction between device interfaces and data formats, also hinder the systematic progress of the field. Therefore, how to establish a modular, scalable, and process-transparent multi-modal platform through engineering and process means, so that multi-frequency multi-aperture radar, vision and motion capture labeling, and medical gold standard modalities can achieve high-precision synchronous acquisition and joint labeling in the same system, thus solving the problems of missing original data and single modality, and providing a reproducible and comparable benchmark environment for subsequent algorithm development, clinical verification, and open dataset construction, has become a technical problem to be solved. SUMMARY

[0003] In order to at least solve the above-mentioned background art that the prior art only uses a single radar or only provides processed data, the present application emphasizes preserving the underlying original signal, supporting multiple radar parameter combinations, and providing high-quality three-dimensional labeling and medical verification data, thereby providing a reliable data foundation for fine-grained human function and physiological research. In view of this, the present application provides solutions in the following aspects. The first aspect of the application provides a multi-dimensional radar + X human three-dimensional perception method, comprising: collecting signals, respectively based on a multi-modal millimeter wave radar, a visual modal, and X medical gold standard modal, outputting time-stamped multi-modal raw data; based on the multi-modal raw data, labeling and aligning to output three-dimensional labels and medical gold standard reference values on the same time axis and the same spatial coordinates of the radar; performing down-conversion, filtering, ADC sampling, window function processing, pulse compression / distance FFT, slow-time Doppler FFT preprocessing steps on the radar echo, outputting original feature tensors in the distance-velocity-time dimension; performing spatio-temporal alignment, mutual information / correlation evaluation and weight learning on the multi-modal features, training a detection / recognition / regression model, and feeding the model back to signal processing parameters, the processing parameters including amplitude threshold, window length, synthetic aperture step; generating a final human detection, fall / abnormal alarm, physiological parameter estimation and archived multi-modal data package.

[0004] In one embodiment, the multi-modal millimeter wave radar signal includes output signals under different frequency bands, different apertures, and multi-view arrangements; the visual modal signal includes output signals of an RGB-D camera and a Mocap motion capture system; the X medical gold standard modal includes signals output by an ECG, a respiratory belt, and an IMU device.

[0005] In one embodiment, the labeling and alignment based on the multi-modal raw data includes: visual labeling: 3D joint detection by RGB-D and depth back-projection (camera frame); high-precision 3D joint given by Mocap (mocap frame); both are solved by external parameter T camera←mocap or calibration board / common view mark; transformation matrix: external parameter is a 4x4 rigid transformation T=[Rt;01], R and t are solved by SolvePnP / Procrustes / ICP on known corresponding point sets; time synchronization: prefer hard trigger / PPS cascade trigger; without hardware, use high-precision timestamp and event cross-correlation correction offset in post-processing; thereby mapping all modal joints / meshes to the world frame first, then projecting or mapping to the radar range-azimuth grid to output the aligned labels.

[0006] In one embodiment, the output raw feature tensor of distance-velocity-time dimension includes: down conversion and demodulation: decompose RF signal into baseband I / Q (de-carrier), preserve phase; fast time processing: apply window function to each radar chirp or frame's sample vector, then do N-point FFT to get range profile, range resolution formula: AR = c / (2B); pulse compression: filter echo with matched filter, improve range resolution / sidelobe suppression, implement in time domain or in frequency domain with multiplication; slow time processing: take consecutive chirps or frames of the same range-bin along time series, window and do Doppler-FFT to get velocity spectrum Doppler, Doppler and radial velocity relationship still use fd = 2·v·f0 / c; thus organize each range-bin's energy and phase history after fast / slow time processing into R-V-T three-dimensional tensor; save each bin's complex phase sequence for phase tracking.

[0007] In one embodiment, the spatio-temporal alignment, mutual information / correlation evaluation and weight learning of multi-modal features include: STFT or continuous short window FFT on phase / amplitude time series to get time-frequency map on selected range-bin, or wavelet decomposition to extract multi-scale spectrum, respiration in low frequency band (≈0.1-0.7Hz), heartbeat in higher micro-amplitude interval, micro-Doppler in time-frequency map with slow periodic component; CFAR integration: estimate neighborhood noise power (guard cell + background window) on Doppler or Range-Doppler plane with sliding window, set threshold to stabilize false alarm rate, detect spectral peaks as target candidates; feature vector items: include features' vector: distance, radial velocity, amplitude peak value, peak energy ratio, respiration / heart rate candidate, phase statistics, time-frequency energy distribution features, angle estimation, micro-Doppler curve shape description, combined as vector input to classification / regression model; alignment: time alignment according to timestamp or hard trigger; space alignment use extrinsic parameter matrix to map each modality to the same coordinate system, generate training pairs; evaluation / weight learning: calculate correlation between features or use model weight learning on validation set, train different modal inputs with weights and update weights with validation error.

[0008] In one embodiment, the training detection / recognition / regression model comprises: using CNN+RNN, Transformer, or hybrid Bayesian graph model as the uncertainty estimation model; outputting human three-dimensional perception classification labels and regression values, the classification labels including human body, non-human body, fall, normal, and the regression values including respiratory rate, heart rate, and displacement amplitude; the training method comprises supervised training, data augmentation, cross-validation, early stopping, and hyperparameter search; and MC-Dropout or Bayesian posterior is used for uncertainty modeling.

[0009] In one embodiment, the model is fed back to the signal processing parameters, comprising: using grid search or Bayesian optimization method to optimize the signal processing parameters by using the performance index on the validation set, and comparing F1 / RMSE under different window lengths / CFAR thresholds / integration lengths to select the optimal parameters and deploy them to the processing flow.

[0010] In one embodiment, the generation of the final human body detection, fall / abnormal alarm, physiological parameter estimation, and archived multi-modal data package comprises: human body detection / positioning: model output classification + corresponding range / azimuth / height unit; combining the time sequence smoothing of continuous frames to obtain a stable human body trajectory; abnormality / fall determination: using a classifier to identify a fall pattern or using abnormality detection, training a density model of normal behavior, and determining a low-probability event as an abnormality; combining an acceleration threshold or a trajectory height / angle mutation to trigger an alarm; physiological parameter extraction: extracting a phase time sequence φ(t) in a target range-bin, phase transformation into displacement Δd≈(λ / (4π))·Δφ; band-pass filtering the displacement signal, estimating the frequency by using FFT or spectrum peak detection, and outputting the respiratory rate / heart rate; and performing quality control on the amplitude and outputting the confidence.

[0011] The second aspect of the application provides a multi-dimensional radar+X human three-dimensional perception system using any of the above-mentioned human three-dimensional perception methods.

[0012] The existing human perception inventions and datasets based on millimeter wave radar often only collect and publish point cloud data containing spatial three-dimensional information, but ignore the original signals rich in more detailed information (such as Doppler information). And when collecting, the frequency band and array size of the radar are fixed, and these inventions are difficult to further develop to scientifically explore the imaging performance of the radar. The present invention proposes to build a modular, reproducible and scalable "multi-dimensional radar + X invention" data collection and annotation platform to achieve: 1. Support for synchronous collection of radars with multiple frequency bands, multiple apertures and multiple perspectives; 2. Retain and publish original intermediate frequency / baseband echoes (containing phase information) to facilitate secondary analysis of Doppler / micro-Doppler; 3. Establish a clear data flow and closed loop (collection -> visual / capture annotation -> multi-modal fusion -> model training -> parameter backwriting) from input to output, improve reproducibility and provide verifiable gold standard alignment data for downstream medical applications. Specifically, by realizing the synchronous collection of radars with different frequency bands and apertures and exploring external factors such as radar position and placement angle, the human perception performance of millimeter wave radar is fully explored; and in order to meet the needs of high degree of freedom of radar array and explore the performance limit of radar imaging, the present invention also provides a large size high precision guide rail; in order to provide accurate human annotation, the present invention is equipped with an RGB-D camera to obtain human 2D annotation, and a motion capture invention to provide human 3D annotation, and real mesh information matching the SMPL series of human parameterized models can be generated through an inverse kinematics based method. In addition, the invention can also synchronously collect "X" kinds of medical gold standard modalities, and has an expandable interface to adapt to various downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the exemplary embodiments of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which several embodiments of the present invention are shown by way of example, and wherein like or corresponding elements show like or corresponding parts, by referring to which; and in which: Figure 1 is a human perception method according to an embodiment of the present invention; Figure 2 is a system correlation diagram according to an embodiment of the present invention; Figure 3 is a high-precision stepper motor guide rail according to an embodiment of the present invention; Figure 4 is a radar human perception task 2D annotation diagram according to an embodiment of the present invention; Figure 5 is a real joint position collection schematic diagram according to an embodiment of the present invention; Figure 6 is a posture parameter generation for the right elbow of a human body according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the scope of protection of the present application. It should be understood that the terms “first”, “second”, “third”, and “fourth” and the like in the claims, the specification, and the drawings of the present application are used to distinguish different objects, and are not used to describe a particular order. The terms “include” and “contain” used in the specification and claims of the present application indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof. It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms “a”, “an”, and “the” are intended to include the plural forms, unless the context clearly indicates otherwise. It should be further understood that the term “and / or” used in the specification and claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. As used in the specification and claims, the term “if’ can be interpreted as “when” or “upon” or “in response to a determination” or “in response to detecting” depending on the context. Similarly, the phrases “if determined” or “if detected [the described condition or event]” can be interpreted to mean “upon being determined” or “in response to being determined” or “upon detecting [the described condition or event]” or “in response to detecting [the described condition or event]” depending on the context. The core subject of the present application is "multi-dimensional radar + X human three-dimensional perception", and its goal is to establish a multi-modal, scalable data acquisition and processing for human three-dimensional perception and real-time monitoring of physiological state. In simple terms, the system is like a "brain of multi-source sensors" that can simultaneously acquire raw data from different types of millimeter wave radar (across frequency bands, across apertures), multi-view RGB-D cameras, motion capture devices, and several medical gold standard devices (i.e. X modalities, such as lung function measuring instruments, electrocardiograms, sleep monitoring devices, etc.) in the same world coordinate system, and accurately synchronize and jointly calibrate these seemingly unrelated data, and finally output a complete set of reproducible data chain including raw radar samples, common intermediate representations (such as range-doppler plots), frame-by-frame three-dimensional voxels and grid annotations, and physiological parameters aligned with medical gold standards. The present application is suitable for home health monitoring (such as fall detection for the elderly, long-term monitoring of respiration and heart rate, sleep quality assessment), clinical research (such as dynamic lung function verification), rehabilitation assessment and basic research benchmarking, and other scenarios. Unlike traditional solutions that only use a single radar or only provide processed data, the present application emphasizes preserving the underlying raw signals, supporting multiple radar parameter combinations, and providing high-quality three-dimensional annotations and medical verification data, thereby providing a reliable data foundation for fine-grained human function and physiological research.

[0015] The first aspect of the present application provides a multi-dimensional radar + X human three-dimensional perception method. As shown in Figure 1 The human three-dimensional perception method of the present application is shown. It includes steps 100-500, including: Step 100, collect signals, respectively based on multi-modal millimeter wave radar, visual modal, X medical gold standard modal, output time-stamped multi-modal raw data; Step 200, based on the multi-modal raw data, perform annotation and alignment to output three-dimensional annotations and medical gold standard reference values on the same time axis and spatial coordinates as the radar; Step 300, perform down-conversion, filtering, ADC sampling, window function processing, pulse compression / distance FFT, slow-time Doppler FFT preprocessing steps on radar echoes, output raw feature tensors in distance-velocity-time dimensions; Step 400, perform spatio-temporal alignment, mutual information / correlation evaluation and weight learning on multi-modal features, train detection / recognition / regression models, and feed the models back to signal processing parameters, including amplitude threshold, window length, synthetic aperture step; Step 500, generate a final multi-modal data package for human detection, fall / alarm, physiological parameter estimation and archiving.

[0016] In the present application, the signal acquisition step gives the original data and the synchronization timestamp, which is the input for everything that follows. The annotation and alignment step projects the visual / capture / medical data into the radar coordinate system, generates supervised labels, and ensures the spatiotemporal consistency of the training data. The signal processing step converts the original samples into structured intermediate representations (range / Doppler / phase) for feature extraction and visualization. The feature extraction output vector serves as both the input of the training sample and the real-time rule judgment (such as preliminary moving / static determination). The data fusion / model step trains the detection / regression model using the aligned multi-modal features and labels; the trained model proposes optimization suggestions or directly triggers parameter adjustment when reasoning. The output step converts the model decision into an alarm / report and archives the original / intermediate representation and metadata, forming a reproducible data chain. Specifically, the system correlation diagram is shown in Figure 2 .

[0017] It can be understood that the medical gold standard device refers to a sensing instrument that is considered "close to the true value / reference" in clinical or experimental tests. Common ones include: electrocardiogram (ECG) to record heart waveforms, lung function meter or respiratory flow meter to measure respiratory capacity / speed, chest strap or respiratory membrane to measure respiratory displacement / frequency, finger clip blood oxygen / pulse meter, workstation-level IMU (for joint angular velocity / acceleration), etc. The role is to provide a verifiable "sensing" reference value for the non-contact estimation of the radar - to calibrate and evaluate the estimation accuracy and stability of the radar on physiological parameters such as respiration, heart rate, motion amplitude, and lung function. In short: the radar gives an "estimate", the medical gold standard gives a "true value", and the comparison between the two is used to verify and quantify the error. Common devices and I / O: ECG: input is electrode voltage waveform, output is heart rate, ECG waveform, R-R interval, etc. Respiratory belt / flow meter / lung function meter: input airflow or chest strap displacement, output respiratory frequency, tidal volume, instantaneous flow, lung volume curve. Finger clip pulse oximeter: input photoelectric volume curve, output SpO2, pulse rate. High-precision IMU / accelerometer: input three-axis acceleration / angle speed, output joint angular speed, gait parameters.

[0018] Use: Provide time-aligned "true values" for radar physiological and motion estimation, for labeling training, error quantification (RMSE, bias, correlation coefficient), and clinical usability verification.

[0019] Multiband radar: provides echo and phase information at different frequency bands, apertures, and perspectives, which is conducive to extracting distance, radial velocity, and micro-Doppler (respiration / heartbeat) features. RGB-D camera: provides visual 2D / depth information and fast skeleton estimation, which is the main visual basis for generating and verifying three-dimensional labels. Motion capture (Mocap): provides high-precision three-dimensional joint trajectories and reference points, which are used to generate high-credibility frame-by-frame 3D labels and external parameter calibration. Medical gold standard equipment: provides clinical-level "true value" of physiological parameters to verify radar physiological estimation. These modules can complement each other: radar is good at perspective / transparency and phase sensitivity (it is the main research modality), vision / capturing provides geometric accuracy, and medical equipment provides physiological true value. None of them is dispensable, because a single modality cannot meet the three requirements of "spatial accuracy, time synchronization, and physiological true value verification" at the same time.

[0020] Calibration board: a known-size checkerboard or AprilTag board for visual calibration, used to estimate camera intrinsic parameters and relative poses. Reference points (Mocap reference points): Mocap markers (passive reflective or active LEDs) at known relative skeleton positions, serving as high-precision references for "Mocap→world". Known geometric landmarks: known coordinate objects (corner reflector, calibration sphere, scale ruler) placed in space and observable by radar / camera / guideway, facilitating the establishment of a common viewpoint among multiple modalities. Joint optimization: first calibrate single modalities (camera intrinsic parameters, mocap calibration), and then solve the external parameter matrix (for example, solve the rigid transform T that minimizes∑‖T·p_mocap p_camera‖) by minimizing the re-projection error of observation points under different modalities; solve it using nonlinear least squares (with a robust term); add timestamp constraints to multiple landmarks to perform bundle-adjustment-style joint optimization to improve global consistency. Mapping method: after obtaining the external parameters, the point cloud or joint point p_mod of any modality can be projected to the world coordinate system through the 4×4 transformation matrix T_mod→world: p_world=T_mod→world·p_mod. This world coordinate is then mapped to the radar range / azimuth / bin grid through the coordinate transformation of the radar, completing the spatial alignment.

[0021] In one specific embodiment, the window function, pulse compression / Range-FFT, and Doppler-FFT specific implementation steps and formulas are as follows: 1. Down-conversion and demodulation: decompose the RF signal into baseband I / Q (remove the carrier), and retain the phase.

[0022] 2. Fast time processing: Apply window function (e.g. Hann) to each chirp's sample vector, then do N-point FFT to get range profile (Range-FFT). Range resolution formula: AR = c / (2B).

[0023] 3. Pulse compression (if linear frequency modulation is used): Filter the echo with matched filter (to improve range resolution / sidelobe suppression), either in time domain or in frequency domain with multiplication.

[0024] 4. Slow time processing: Take the windowed FFT along time series (Doppler-FFT) of consecutive chirps (or frames) in the same range-bin to get velocity profile (Doppler). The relation between Doppler and radial velocity is still fd = 2-v-f0 / c.

[0025] 5. Output: Organize the energy and phase history of each range cell after slow / fast time processing into R-V-T 3D tensor; save the complex phase sequence of each cell for phase tracking.

[0026] In one embodiment, the specific steps of extracting micro-Doppler / time-frequency, combining CFAR, and getting the feature vector are as follows: 1. Time-frequency extraction: Do STFT (or consecutive short window FFT) on the phase / amplitude time series in the selected range-bin to get time-frequency map, or use wavelet decomposition to extract multi-scale spectrum. Respiration is usually in low frequency band (≈0.1-0.7 Hz), heartbeat is in higher micro-amplitude interval, and micro-Doppler is represented by slow periodic component in the time-frequency map.

[0027] 2. CFAR integration: Estimate the neighborhood noise power (guard cell + background window) in Doppler or Range-Doppler plane with sliding window, set threshold to stabilize false alarm rate, and detect spectral peaks as target candidates.

[0028] 3. Feature vector items: Vectors representing some features. For example: range, radial velocity, amplitude peak value, peak energy ratio, dominant frequency (respiration / heart rate candidate), phase statistics (mean, variance), time-frequency energy distribution features (spectrum centroid, spectrum bandwidth), angle estimation (if there is an array), micro-Doppler curve shape descriptor (period, amplitude modulation rate). These are combined into a vector input to the classification / regression model.

[0029] In one embodiment, the steps of generating annotations based on RGB-D and gold standard, and aligning and synchronizing are as follows: 1. Visual annotation: RGB-D does skeleton detection / depth back-projection to get 3D joints (camera frame); Mocap gives high-precision 3D joints (mocap frame); both are solved by extrinsic T_camera←mocap or calibration board / co-view marker.

[0030] 2. Transformation matrix: extrinsic is 4x4 rigid transformation T=[R t;01], solve least squares solution on known correspondence point set by SolvePnP / Procrustes / ICP to get R, t.

[0031] 3. Time synchronization: prefer hard trigger / PPS cascade trigger; without hardware time use high-precision timestamp and in post-processing use event cross-correlation (e.g. motion peak, synchronous light flashing event) to correct offset.

[0032] Result: map all modal joints / meshes to world frame first, then project or map to radar range-azimuth grid to output aligned labels.

[0033] In one specific embodiment, the steps of alignment / evaluation / learning and back-feeding are as follows: 1. Alignment: time-wise align by timestamp or hard trigger; space-wise map each modality to the same coordinate system by extrinsic matrix, generate training pairs.

[0034] 2. Evaluation / weight learning: calculate correlation between features (mutual information / Pearson) or use model weight learning on validation set (train different modal inputs with weights and update weights with validation error).

[0035] 3. Model type and input / output examples: input = stitched multi-modal feature vector or time-series tensor; model = CNN+RNN (time-series), Transformer (long-time dependency), or hybrid Bayesian graph model for uncertainty estimation; output = classification label (human / non-human, fall / normal), regression value (respiration rate, heart rate, displacement amplitude).

[0036] 4. Training method: supervised training (cross-entropy / MSE loss), data augmentation, cross-validation, early stopping, and hyperparameter search; uncertainty modeling can use MC-Dropout or Bayesian posterior.

[0037] 5. Back-feeding (parameter optimization): use performance indicators on the validation set to drive signal processing parameter tuning (automated methods such as grid search or Bayesian optimization), for example, compare F1 / RMSE under different window length / CFAR threshold / integration length, select the optimal parameters and deploy to the processing flow.

[0038] In one embodiment, the steps of obtaining human detection value, anomaly detection, obtaining physiological parameters are as follows: 1. Human detection / positioning: model output classification + corresponding range / azimuth / height unit; combined with time sequence smoothing of consecutive frames (sliding window voting) to obtain stable human trajectory.

[0039] 2. Abnormality / fall determination: use a classifier to identify a fall pattern or use anomaly detection (train a density model of normal behavior, and determine low probability events as abnormal); combined with acceleration threshold (if there is an IMU) or trajectory height / angle mutation to trigger an alarm.

[0040] 3. Physiological parameter extraction (workflow): extract phase time series φ(t) in the target range-bin, and phase transform into displacement Δd≈(λ / (4π))·Δφ; band-pass filter the displacement signal (respiration bandwidth ~0.1-0.7Hz, heart rate bandwidth is set according to the experiment), estimate the frequency using FFT or spectral peak detection, and output the respiration rate / heart rate; perform quality control on the amplitude (SNR, spectral peak significance) and output the confidence.

[0041] 4. Verification: after aligning these values with the gold standard time, calculate the error indicators (RMSE, bias, correlation coefficient) and classification indicators (sensitivity / specificity), and report the reliability under different SNR / distance / angle conditions.

[0042] In one embodiment, the radar used by the present application includes AWR1843BOOST of Texas Instruments and VBLU development kit of Vayyar company.

[0043] AWR1843BOOST is a commercial portable millimeter wave radar produced by TI company (8.3cm×6.4cm, 30g). The millimeter wave device contains 3 transmitting antennas and 4 receiving antennas, and the working frequency band is 76-81GHz. It has a built-in digital signal processor, microcontroller and hardware accelerator, which can complete a lot of computing work on a single chip. Different radar parameter settings will get different radar hardware performance. We should get the required radar hardware performance according to the purpose of different experiments, and then deduce the parameters that the radar should set, for example, according to the requirements, the maximum sensing distance of the radar is about 11m, the distance resolution is about 4.3cm, the maximum sensing speed is about 4.5m / s, and the speed resolution is about 7.1cm / s. According to the formula of FMCW radar principle, the frame rate of AWR1843 should be 10, the number of chirp should be 128, and the number of sampling points should be 256.

[0044] To realize the comparable test and performance limit analysis, the system supports the parameter set management according to the experimental design, such as: working waveband, transmitting power, chirp frame structure (chirp number, sampling point number), frame rate, bandwidth B, etc. The following gives the definition and formula of the commonly used key performance indicators for reference of parameter selection and result analysis: ① Distance resolution AR=c / 2B, wherein c is the light speed, and B is the transmitting signal bandwidth. The formula shows that the increase of bandwidth can directly improve the distance resolution (for example: the bandwidth of 1GHz corresponds to the resolution of 15cm level or better, which is calculated according to the actual bandwidth). Therefore, when designing the test, we can control AR by changing the bandwidth and sampling strategy of chirp to verify the influence of the human body detail capture ability under different bandwidths. ② Velocity (Doppler) resolution and maximum alias-free velocity: determined by the pulse repetition frequency (PRF) / frame rate and the slow-time FFT point number, and inversely proportional to the sampling time window length (the longer the window, the higher the velocity resolution, but the slower the time domain response).

[0045] On the data processing chain, the present application adopts the following specific algorithm steps to separate the motion information and physiological information: ① Raw echo preprocessing: The DC removal, band-pass filtering and window function (such as Hann window) processing are performed on each frame of sampling data to reduce the window leakage and side lobe effect; then the distance FFT (Range-FFT) is performed to obtain the complex echo history S(r, t) of each distance unit. ② Doppler (motion) detection-first FFT: The Doppler-FFT is performed on the fixed distance unit along the slow time dimension (frame sequence) to obtain the frequency spectrum X(fd). The maximum spectral peak Xmax is detected in the specified frequency domain range (for example, ±1.6 kHz corresponding to a given speed range) and compared with the threshold XTH. If it exceeds the threshold, it is judged that there is motion information (C1=1), otherwise not. ③ Physiological (respiration) detection-second accumulation and FFT: The low-frequency component (or the first spectral point X0) of the first FFT of each distance unit is accumulated in time to form a sequence Y(t) (N consecutive frames), and the normalized STFT / FFT is performed on Y(t) to detect the spectral peak Ymax in the range of 0.2-0.5 Hz and compare it with the threshold YTH. If it exceeds, it is judged that there is a respiratory signal (C2=1). In order to improve the sensitivity, the subspace method (such as MUSIC) or the spectrum estimation method (such as Welch or multiple window average) can be used to improve the robustness of low-frequency spectrum estimation. ④ Phase micro-motion extraction: For small displacements such as heartbeats / respirations, the regression estimation is directly performed on the small changes of echo phase φ(t). There is an approximate relationship between the phase and the displacement, which can be used for more precise physiological motion amplitude estimation. ⑤ Signal-to-noise ratio evaluation (SNR): The power ratio is used to define SNR=Psignal / Pnoise (expressed in dB as 10\log10), and SNR is recorded at each processing stage to evaluate the detection performance under different parameters and adjust the threshold strategy. At the same time, in order to reduce false alarms and motion / non-human clutter interference, the system can use the following improvements: CFAR (constant false alarm rate) instead of fixed threshold, coherent integration to improve SNR, adaptive noise estimation and baseline drift compensation, etc.

[0046] In one embodiment, the DC removal is to subtract the mean value from each frame (x[n]<-x[n]-mean(x)), and the band-pass filtering is to keep the target frequency band (e.g. remove DC and RF noise) with a digital filter. The window function processing is to multiply each fast-time sample vector with a window w[n] (commonly Hann), and then do N-point DFT: X[k]=sum_{n=0}^{N-1}x[n]·exp(-j2πkn / N). This X[k] is the complex value of the range profile for the k-th range-bin, which can be approximately mapped to the real distance as R_k=k·ΔR, ΔR=c / (2B). Arranging the complex spectrum of each range-bin in each frame in time sequence, we get S(r,t) (r represents the range-bin, t represents the frame time), i.e. the complex echo history, on which we can do phase tracking or energy statistics later.

[0047] The “fixed range cell” is a range-bin (a distance cell) after the Range-FFT, and we take the complex sample sequence s_t=S(r,t) of this cell in different frame times from S(r,t). We do M-point FFT (Doppler-FFT) on this sequence along the slow-time dimension: D[m]=sum_{t=0}^{M-1}s_t·exp(-j2πmt / M). The amplitude spectrum of D[m] gives the energy distribution of this distance cell at different Doppler shifts. We map the Doppler index m to the Doppler frequency f_d, and then calculate the radial velocity v with f_d=2vf0 / c (f0 is the radar center frequency, c is the speed of light).

[0048] The “each range cell” refers to each range-bin after the Range-FFT of each frame. The “low-frequency component of the first FFT” usually refers to the low-frequency term near DC in the Doppler spectrum (reflecting slow motion or micro-Doppler). The accumulation method is to take the complex or amplitude of this low-frequency component of each frame to form a time sequence y[t], and then do normalization (commonly subtract the mean and divide by the standard deviation) to remove DC drift and amplitude scale difference. Doing STFT or long-window FFT (selecting the length to cover the breathing period, typically 30-120s or converted according to the number of frames) on y[t] can get the power spectrum, and search for the spectral peak in the range of 0.2-0.5Hz. The subspace method (such as MUSIC) is to estimate the covariance matrix, separate the signal subspace and noise subspace, and find the position that leads to the minimum / peak projection on the frequency axis, which is suitable for narrowband weak signal detection; the spectral estimation method (such as Welch) is to segment the sequence, window, FFT, and then average to reduce variance and improve the robustness of low-frequency spectral estimation.

[0049] Echo is the complex echo sample s(t) = A(t) · exp(jφ(t)) in a range-bin. Phase φ(t) is obtained from the angle of complex sample (after unwrapping and detrending). Micro-motion estimation can fit φ(t) as a narrowband sinusoid or directly measure amplitude after bandpass, the approximate relationship between displacement Δd and phase change is Δd ≈ λ · Δφ / (4π) (single station single round trip phase relationship, λ is wavelength). The actual practice is to filter φ(t) (keep breathing / heartbeat bandwidth), use least squares fitting or FFT to find frequency and amplitude, and the amplitude multiplied by λ / (4π) is the displacement amplitude estimate, and the period, peak-to-peak value and confidence can be output.

[0050] Signal power can be calculated on a specific range-bin and Doppler cell (e.g. P_signal = mean |s(t)|^2 or peak energy), and noise power is obtained by averaging the energy of adjacent blank cells or using CFAR background estimation. "Each processing stage" refers to the SNR measurement at different processing intermediate points such as raw I / Q, after Range-FFT, after Doppler integration, after phase tracking, etc. SNR-based adjustment strategy: when SNR is low, increase coherent integration length or use long window spectrum estimation, reduce detection threshold but cooperate with CFAR to control false alarm; when SNR is high, shorten the window length to improve time domain resolution and improve real-time performance. Threshold setting is based on target detection probability and false alarm rate, which can be determined by empirical verification or statistical method (ROC curve).

[0051] In one specific embodiment, the perception and imaging effect of the radar on the human body will be affected by multiple factors, including the arrangement angle, number, position and aperture size of the radar, which will affect the integrity and accuracy of the data. For example, the angle at which the radar is placed determines the distance and viewing angle of each part of the human body. Different angles may result in different reflection signal strengths of each part of the human body, thereby affecting the perception effect. The use of multiple radars can improve the coverage and accuracy of imaging, and through the cross-analysis of multiple viewing angles, the information loss caused by shielding or single-angle viewing limitations can be reduced. In addition, the position of the radar is also crucial, radars close to the human body can provide more detailed information, while radars at a distance can be helpful for large-scale monitoring. The comprehensive use can provide more optimized perception data in different situations. Based on this, the research plan is to further explore the influencing factors of radar perception based on existing hardware, to ensure that data acquisition in future downstream tasks can more effectively capture human features. Specifically, we will analyze the signal characteristics of radars under different layouts through systematic experiments, optimize the sensor position and angle, and strive to obtain the best human perception effect, providing high-quality basic data support for subsequent application scenarios such as fall detection and physiological signal recognition.

[0052] Among them, for the study of the impact of aperture size, the system is equipped with high-precision stepper motor guide rails, as shown in Figure 3 The radar can collect echo signals along the specified path step by step by controlling the precise movement of the guide rails. The combination of such movement trajectories is equivalent to a larger "virtual aperture", which realizes a larger observation range than a single fixed radar. The echo signals collected at each position are equivalent to observing the target from different angles, and then these signals are coherently synthesized to improve the resolution and enhance the imaging details. This study can freely analyze how the change of aperture size affects the human perception accuracy and imaging effect of millimeter wave radar by changing the radar aperture size and observation angle. In addition, by adjusting the aperture range, it can also verify the limit of detail capture of millimeter wave radar in human imaging, which provides support for evaluating the human perception effect of millimeter wave radar in different application scenarios.

[0053] To implement the concept of "synthetic aperture" into a reproducible engineering procedure, the system adopts the following implementation procedure: ① Trajectory planning (track gauge and step length): The synthetic path length Lsynth is selected on the guide rail, and the radar is sampled at equal intervals with a step length Δx (the step length determines the phase sampling density, and the phase continuity needs to be met to ensure coherent superposition). The step length and sampling timing are guaranteed by high-precision stepping motor and encoder (position accuracy << wavelength). ② Phase calibration and reference echo: The accurate position information xi and time stamp ti are recorded for each position sampling, and the echo at each sampling position is phase-corrected (to compensate for the phase term caused by the difference in transmission distance) to meet the coherent superposition condition. The phase compensation term adopted can be estimated by a calibration target (corner reflector) or by directly measuring the phase history. ③ Coherent superposition (imaging algorithm): The phase is accumulated along the synthetic aperture direction for each range cell and processed for azimuth focusing, and one of the common SAR imaging algorithms, back projection (Backprojection) or Range-Doppler (RD) algorithm, is used to realize coherent synthesis and imaging focusing. The algorithm procedure is: sub-aperture segmentation → each segment matching filtering (Range compression) → calculating azimuth Doppler and doing azimuth compression → synthesizing the final image. Here we record each step parameter to ensure reproducibility. ④ Resolution estimation (theory): The synthetic aperture will improve the azimuth (lateral) resolution, and its theoretical relationship is related to the synthetic aperture length Lsynth, target distance R and wavelength λ. The following approximate expression can be used to estimate the azimuth resolution: δaz≈Rλ / 2Lsynth. The formula shows that the longer the synthetic aperture (the larger Lsynth), the smaller (the better) the azimuth resolution can be obtained at the same distance R. ⑤ Experimental variables and measurable indicators: In the guide rail test, we will gradually change Lsynth, Δx, imaging mode (static multi-point / spotlight), radar operating frequency band and angle, and record and compare the following indicators: azimuth resolution (determined by the smallest lateral spacing that can be distinguished by the target point), SNR gain (peak enhancement ratio after coherent superposition), imaging contrast and detail resolution (for example, the minimum value of the chest wall micro-displacement that can be detected). Experimental data will be archived and publicly disclosed in the form of quantitative tables and example images.

[0054] In one specific embodiment, the visual modalities employed in the present invention include a visual aid device comprising an Azure Kinect RGB-D camera and a Mocap motion capture system.

[0055] Azure Kinect is a high-performance spatial computing camera developed by Microsoft, which integrates multiple sensors. The RGB camera has a resolution of 3840x2160, and the ToF depth sensor has a resolution of 1024x1024. In low-light environments, Azure Kinec can capture clear infrared images. In addition, it also has a built-in array of 7 microphones optimized for voice capture and spatial audio processing. Azure Kinect also comes with an IMU (Inertial Measurement Unit) that can provide acceleration and angular velocity data, enabling motion tracking and spatial awareness. With the SDK provided by Microsoft, Azure Kinect can achieve powerful real-time skeletal tracking, capturing 32 skeletal joints of multiple people, and providing 2D annotations required for radar human perception tasks, such as Figure 4

[0056] Motive is a high-precision motion capture system developed by OptiTrack, which consists of infrared cameras, markers, and motion capture suits. Infrared cameras use high-speed image capture technology to record motion data at hundreds of frames per second. These cameras can work in various lighting conditions, especially in low-light environments, making them suitable for capturing complex and fast movements. Markers are usually small spherical shapes covered with highly reflective material. When infrared cameras shine on these markers, they reflect light, and the system calculates the precise position of the markers in three-dimensional space by capturing these reflected signals. The layout and number of markers can affect the accuracy and completeness of the capture. Motion capture suits are specially designed clothing for motion capture, usually equipped with multiple markers at key joint locations, ensuring that markers remain stable during movement.

[0057] In the study of radar-based human perception tasks, the marker placement of the motion capture suit refers to the position of the SMPL model joint, and the average position of the markers on both sides of the joint is taken as the real joint position. The collection diagram is shown in Figure 5

[0058] ​​In one specific embodiment, after collecting the motion capture raw data, a method based on inverse kinematics can be used to generate the 3D annotations required for radar human perception tasks. After preprocessing (denoising, interpolation, outlier rejection) of the collected mocap raw marker points, the physical markers are first mapped to the skeleton reference points (marker→joint). Then inverse kinematics (IK) is used to solve for each frame: taking joint angles as unknowns, the weighted least squares method is used to minimize the residual error between the model reference points and the observed markers (solved by pseudo-inverse of Jacobian or Levenberg-Marquardt), and the weight is set according to the marker confidence. To suppress frame-by-frame jitter, a time smoothing term is introduced or a joint optimization of several frames is performed, and when there is occlusion, the weight of the missing marker is reduced or a robust loss is used instead of the quadratic term. If the SMPL mesh, the joint angles obtained by IK are used as initialization, and the shape and pose parameters are further optimized to obtain consistent mesh output. Finally, the 3D joint coordinates, joint angles, SMPL parameters and confidence of each frame are output, with time stamp and coordinate transformation matrix, which is convenient for time / spatial alignment with radar data for training and evaluation. The obtained pose parameters of the right elbow of the human body are generated, as shown in Figure 6

[0059] The mapping is obtained by a pre-mapping table of "marker→joint" and static calibration. The steps are: record the position of each marker under the calibration pose (such as T-pose), and assign several markers as the reference set of a certain joint according to human anatomy or pre-agreement; use the average position of these markers under the calibration pose and the model joint position to calculate the fixed offset vector (or use the optimal rigid transformation of the marker set to the skeleton reference point to solve); in actual operation, the initial position of the joint is given by the weighted average of the corresponding marker (if multiple markers correspond to the same joint), or the real-time marker position is transformed to obtain the joint position by using the offset obtained by calibration. In short: first determine the mapping / offset table by calibration, and then use weighted average or rigid transformation to map marker coordinates to joint coordinates in operation.

[0060] The mathematical form of the weighted least squares is to minimize E(theta)=∑iwi||fc(i)(theta)-mi||^2+lambda||theta-thetaprior||^2, where fc(i)(theta) is the position of the joint angle theta mapped by forward kinematics to the c(i)th reference point, and mi is the i th observed marker. The solving steps are: take the initial theta (which can be the result of the previous frame), calculate the residual vector r=f(theta)-m and the Jacobian matrix J= r / ​θ; construct linear equation (J^TWJ+λI)Δθ=-J^TWr; solve Δθ and update θ←θ+Δθ, iterate to convergence. Levenberg–Marquardt adds a damping μ to the diagonal in the above equation (i.e. J^TWJ+(θ+μ)I), while the pseudo-inverse of Jacobian is approximated by a single update of J^+Wr in real-time scenarios. The weights wi come from the confidence and visibility of markers.

[0061] Marker weights come from mocap system's visibility / confidence output and residual adaptive strategy. Implementation: initial weights are set to visibility values (0 or 0~1); weights are adjusted by current residuals at each iteration step, e.g. wi←wi0 / (1+α·||ri||^2) (α is a scaling constant), and wi is set to zero if residual is abnormal or marker is lost. Robust loss (e.g. Huber or Tukey) can also be used directly, replacing the quadratic term in the optimization with small residual square penalty and linear or upper bound penalty for large residuals; numerically this is equivalent to multiplying each residual by a weight factor given by rho'(r) / r, thus automatically reducing the impact of abnormal markers. In practice, it is recommended to use visibility for static weights first, and then use residual-driven dynamic decay or robust loss to improve stability.

[0062] SMPL is a widely used parametric mesh model for human body, outputting a set of vertices V(β,θ), where β represents shape parameters (body type) and θ represents pose (joint angles). Source can be either the public SMPL model library or an internally licensed model. After using IK to get initial joint angles as initial values for θ, an optimization is performed to minimize E(β,θ)=sumjwj||Rj(V(β,θ))-pj||^2+regularizer(β,θ), where Rj is a selection mapping mesh or regress joint to marker points (or using joint regress matrix), and pj is the corresponding point of mocap / marker. The optimization updates both β and θ (e.g. using LM or gradient method), and adds shape prior and joint amplitude constraints to prevent overfitting. The optimization result is the mesh and joint parameters that best fit the mocap markers, which can be directly output as SMPL parameters and mesh vertices for each frame for radar alignment.

[0063] In one specific embodiment, the medical gold standard modalities of the present application are extended to enable the system to further explore the application of radar in the medical and health fields such as fall monitoring for the elderly at home, sleep disorder disease prediction, and dynamic human lung function measurement. The present application leaves an extensible interface that can add the synchronous measurement of various medical gold standard modalities, such as IMU for measuring the motion state of each part of the human body, electrodes for measuring the electrocardiogram signal, and a band for measuring the respiratory state of the human body, etc. Since most medical gold standard devices are inductive, i.e. need to be in contact with the human body for measurement, in order to make the motion of the subject more natural, the present application intends to use medical gold standard devices that support Bluetooth data transmission.

[0064] Modal synchronization method: The present application uses a method of recording timestamps to achieve multi-modal synchronization. In order to ensure that the radar echo is consistent with the annotation of vision / capturing, the present application uses the following steps to achieve space-time alignment: ① Time synchronization: All devices (radar, RGB-D camera, Mocap, medical gold standard) record local timestamps ti when data is collected. The host computer uses NTP or local hardware clock as the absolute time reference. If multiple devices require multiple acquisition machines, clock alignment can be achieved through network cable (PPS or NTP) / hardware trigger line or time offset correction in the post-processing stage based on known synchronization events (such as calibration pulses). For Bluetooth or low-bandwidth medical devices, record the transmission delay and correct the timestamp based on the measured delay in the post-processing stage.

[0065] ② Space coordinate calibration: Use a calibration target (such as an angle reflector + precisely positioned point) to estimate the external parameters of radar-camera-motion capture, calculate the transformation matrix, and map the 3D joint nodes of vision / capturing to the radar coordinate system, so as to project the vision annotation onto the radar distance unit or pixel for label alignment.

[0066] ③ Modal time window cutting: Record the start time of each modal acquisition and cut the overlapping time period common to all modes for subsequent multi-modal alignment and training. If multiple computers are used for simultaneous acquisition, the machine time is aligned through the network cable. If there is frame loss, record the frame index and eliminate / complete in the post-processing.

[0067] ④ Alignment verification: After alignment, use a calibration moving target (such as the subject performing a predetermined action) to produce obvious features in both radar time sequence and vision time sequence. Use cross-correlation or mutual information methods to quantify the alignment effect. If the alignment error exceeds the preset threshold, return to the calibration step for re-calibration.

[0068] Specifically, the specific steps of the calibration target / transform matrix and projection are as follows: 1. Calculate the external parameters (transform matrix T) step (concise and reproducible): Collect N sets of corresponding 3D point pairs (pi, qi) visible in both modalities, pi in modality A coordinate system, qi in modality B. Corresponding points can come from corner reflectors, calibration spheres or mocap markers measured at known positions.

[0069] 2. Compute centroid and centerize: p'i = pi mean(p), q'i = qi mean(q).

[0070] Construct matrix H = ∑p'iq'i^T, do SVD(H) = U∑V^T, then best rotation R = VU^T; translation t = mean(q) R·mean(p).

[0071] 3. Final 4x4 homogeneous transform T = [R t; 01]. This method is the classic Procrustes, robust to noise and small amount of outliers; when there are outliers, RANSAC can be used to filter them out.

[0072] Specifically, the step of projecting to the radar grid is as follows: 1. world coordinates pworld = Tmod→world·pmod.

[0073] 2. Calculate in radar coordinate system: range = sqrt(x^2 + y^2 + z^2), azimuth = atan2(y, x), elevation = asin(z / range).

[0074] 3. Grid index: rangeidx = round(range / ΔR), azidx = floor((azimuth + FOV / 2) / azres). Project joint to corresponding range / azimuth cell to locate on radar pixel.

[0075] The above flow is executed for each labeled point, thereby accurately mapping visual / mocap joints to radar pixels / cells, supporting subsequent consistency checking and training label generation.

[0076] Specifically, the common overlapping time period is intercepted, and the steps are as follows: 1. Record mode: record the start time t_start and end time t_end (in master clock or local timestamp) of each modality, and send / receive delay estimation of the modality record containing network delay.

[0077] 2. Common time window calculation: t_common_start = max_i(t_start_i), t_common_end = min_i(t_end_i). If t_common_end <= t_common_start, no overlap; otherwise, take [t_common_start, t_common_end].

[0078] 3. Handle lost frames / delays: correct timestamps for Bluetooth / wireless devices with measured round-trip delay or ping measurement; for lost frames, record frame index and fill in or remove incomplete segments with interpolation / removal strategy in post-processing.

[0079] 4. Output: construct aligned frames for each time point within the common time window; save frame pair index table for direct reading of corresponding sample pairs in training / verification.

[0080] Specifically, use cross-correlation or mutual information to quantify alignment effect and determine whether aligned as follows: 1. Cross-correlation method (simple and reliable): A. Select representative signal sequences x(t), y(t) of two modalities on the same spatial target (e.g., energy of visual bone point trajectory speed mode, amplitude envelope of radar range-bin).

[0081] B. Calculate cross-correlation r(τ) = Σ_tx(t)·y(t+τ). Best delay τ = argmax_τr(τ). If |τ | is less than a preset threshold (e.g., 1 frame or a few ms if device precision is high), consider it well aligned; otherwise, apply time offset correction or recalibration.

[0082] 2. Mutual information method (nonlinear, not sensitive to amplitude scale): A. Estimate joint histogram p(x,y) and calculate I(X;Y) = Σ_p(x,y)log(p(x,y) / (p(x)p(y))). Calculate I for multiple time windows, higher mutual information indicates highly shared time characteristics (indicating alignment and correlation).

[0083] B. Use the delay corresponding to the peak position of mutual information as the alignment estimate; or compare the amount of I value improvement before and after alignment as an alignment quality indicator.

[0084] 3. Determination method: First, find τ and correct; second, calculate the mutual information or correlation coefficient of the corrected window. If the correlation coefficient > ρthreshold (e.g., 0.7) or the mutual information > Ithreshold, consider it successfully aligned; otherwise, return to the external parameter / time calibration step.

[0085] The second aspect of the present application provides a multi-dimensional radar + X human three-dimensional perception system, which applies the human three-dimensional perception method.

[0086] In addition to "preserving the original signal of the bottom layer", the present application actually solves and realizes the key capabilities: Systematic parameter scanning capability: parameter scanning tables of cross-bandwidth, different chirp / frame / sampling configurations can be executed on the same platform according to the experimental design, and the influence of bandwidth, frame rate, chirp number, etc. on the distance / speed / phase accuracy is quantitatively evaluated.

[0087] High-precision space-time synchronization and calibration chain: provides hardware trigger / PPS / precise timestamp strategy and external parameter calibration process, ensures that the radar-RGB-D-Mocap-medical modalities are aligned at the microsecond / millimeter level, and solves the problem of different modalities or missing external parameters.

[0088] Controllable synthetic aperture acquisition capability: equipped with high-precision guide rails and realizes phase calibration and coherent superposition process, and inventively studies the influence of synthetic aperture on the limit of azimuth resolution and detail capture (which cannot be achieved by fixed array public data sets).

[0089] Unified data packet and metadata standard: save original I / Q, distance / Doppler / phase history, etc. intermediate representation, frame-by-frame 3D annotation (joints / grid / SMPL), device parameters, external parameters, timestamps and guide rail position information, so as to ensure experimental reproducibility and fair comparison of subsequent algorithms.

[0090] Medical gold standard synchronous verification: synchronously collect ECG, respiratory band, lung function and other clinical gold standards and do time / space alignment, so that the physiological parameter estimation of the radar can be clinically evaluated for error instead of only relying on visual comparison.

[0091] Open interface and scalability: provide API and hardware abstraction layer to allow third parties to load new radars or add new modalities, support community replication and horizontal comparison.

[0092] Although the present application has been shown and described with respect to several embodiments thereof, it will be apparent to those skilled in the art that many modifications, changes and substitutions can be made without departing from the spirit and scope of the present application. It should be understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application. The appended claims are intended to cover all such alternatives as falling within the scope of the present application.

Claims

1. A multi-dimensional radar + X human body three-dimensional perception method, characterized in that, Comprise: Collect signals, respectively based on multi-modal millimeter wave radar, visual modal, X kind of medical gold standard modal, output time-stamped multi-modal original data; Based on the multi-modal original data, labeling and alignment are performed to output three-dimensional labels and medical gold standard reference values on the same time axis and the same spatial coordinates of the radar; The radar echo is subjected to down-conversion, filtering, ADC sampling, window function processing, pulse compression / distance FFT, slow time Doppler FFT preprocessing steps, and the original feature tensor in the distance-velocity-time dimension is output; The multi-modal features are spatio-temporally aligned, mutual information / correlation is evaluated and weight learning is performed, a detection / recognition / regression model is trained, and the model is fed back to the signal processing parameters, including amplitude threshold, window length, synthetic aperture step; Generate the final human body detection, fall / abnormal alarm, physiological parameter estimation and archived multi-modal data package.

2. The human three-dimensional perception method according to claim 1, characterized in that, The multi-modal millimeter wave radar signal includes output signals under different frequency bands, different apertures and multi-view arrangements; the visual modal signal includes output signals of RGB-D cameras and Mocap motion capture systems; the X medical gold standard modal includes signals output by ECG, respiratory belt and IMU devices.

3. The human three-dimensional perception method according to claim 1, wherein, The labeling and alignment based on the multi-modal original data include: Visual labeling: 3D joint nodes (camera frame) are obtained by skeleton detection / deep back projection of RGB-D; high-precision 3D joint nodes (mocap frame) are obtained by Mocap; both are solved by external parameter T_camera←mocap or calibration board / commensurate mark; Transformation matrix: the external parameter is a 4×4 rigid transformation T=[Rt;01], and R and t are obtained by solving the least squares solution of SolvePnP / Procrustes / ICP on the known corresponding point set; Time synchronization: prefer hard trigger / PPS cascade trigger; without hardware time, use high-precision timestamp and event cross-correlation correction offset in post-processing; Thus, the joint nodes / grids of all modalities are first mapped to the world frame, and then projected or mapped to the range-azimuth grid of the radar to output the aligned labels.

4. The human three-dimensional perception method according to claim 1, wherein The output of the original feature tensor in the distance-velocity-time dimension includes: Down-conversion and demodulation: decompose the RF signal into baseband I / Q (remove the carrier), and retain the phase; Fast time processing: apply a window function to the sampling vector of each radar chirp or frame, and then perform N-point FFT to obtain the range spectrum, with the range resolution formula: ΔR=c / (2B); Pulse compression: filter the echo with matched filtering to improve range resolution / sidelobe suppression, implemented in time domain or in frequency domain using multiplication; Slow time processing: take the windowed time series of consecutive chirps or frames for the same range bin, and perform Doppler-FFT to obtain the velocity spectrum Doppler, with the relationship between Doppler and radial velocity still using fd=2·v·f0 / c; Thus the energy and phase history of each range cell after slow / fast time processing is organized into a R-V-T 3D tensor; the complex phase sequence of each cell is also saved for phase tracking.

5. The method of claim 1, wherein The spatio-temporal alignment, mutual information / correlation evaluation and weight learning of the multi-modal features Time-frequency extraction: STFT or continuous short window FFT on the phase / amplitude time series in selected range-bins to get time-frequency map, or wavelet decomposition to extract multi-scale spectrum, respiration in low frequency band (≈0.1-0.7Hz), heartbeat in higher micro-amplitude interval, micro-Doppler in time-frequency map as slow periodic component; CFAR integration: sliding window estimation of neighborhood noise power (guard cell + background window) in Doppler or Range-Doppler plane, set threshold to stabilize false alarm rate, detect spectral peaks as target candidates; Feature vector items: include features' vector: distance, radial velocity, amplitude peak value, peak energy ratio, respiration / heart rate candidate, phase statistics, time-frequency energy distribution features, angle estimation, micro-Doppler curve shape description, combined as vector input to classification / regression model; Alignment: time alignment by timestamp or hard trigger; spatial alignment by extrinsic matrix to map each modality to the same coordinate system, generate training pairs; Evaluation / weight learning: calculate correlation between features or use model weight learning on validation set, train different modality input weighting and update weight with validation error.

6. The human three-dimensional perception method according to claim 5, wherein, The training detection / recognition / regression model includes: Use CNN+RNN, Transformer, or hybrid Bayesian graph model as uncertainty estimation model; output human three-dimensional perception classification label and regression value, the classification label includes human, non-human, fall, normal, the regression value includes respiratory rate, heart rate, displacement amplitude; The training method includes supervised training, data augmentation, cross-validation, early stopping and hyperparameter search; use MC-Dropout or Bayesian posterior for uncertainty modeling.

7. The human three-dimensional perception method according to claim 6, wherein, The model back to signal processing parameters includes: Use grid search or Bayesian optimization method to optimize signal processing parameters, including comparing F1 / RMSE under different window length / CFAR threshold / integration length, to select the optimal parameters and deploy to the processing flow.

8. The human three-dimensional perception method according to claim 1, wherein, The generation of the final human detection, fall / abnormal alarm, physiological parameter estimation and archived multi-modal data package includes: Human detection / positioning: model output classification + corresponding range / azimuth / height cell; combine time series smoothing of consecutive frames to get stable human trajectory; Abnormal / fall determination: use classifier to identify fall patterns or use anomaly detection, train normal behavior density model, low probability event is determined as abnormal; combine acceleration threshold or trajectory height / angle mutation to trigger alarm; Physiological parameter extraction: Extract phase time series φ(t) in the range-bin of the target, phase transform to displacement Δd≈(λ / (4π))·Δφ; do band-pass filter to displacement signal, estimate frequency by FFT or spectrum peak detection and output respiration rate / heart rate; do quality control to amplitude and output confidence.

9. A multi-dimensional radar+X human three-dimensional perception system for operating the human three-dimensional perception method according to any one of claims 1-9.

Images