Omnidirectional human body activity recognition method and device based on HDMR, computer and medium

Abstract

The invention specifically discloses an omni-directional human body activity recognition method and device based on an HDMR, a computer and a medium, and relates to the technical field of radar signal processing. According to the method, an HDMR-based synthetic data generation algorithm is utilized, data expansion can be carried out on collected samples in a radar zero-degree observation angle direction and a small number of samples in other non-zero-degree observation angle directions, and high-quality training data containing all observation angle directions are generated; the objective of the invention is to solve the problem of angle sensitivity of a monostatic radar in omni-directional human body activity recognition. In addition, according to the invention, the dynamic time warping distance DTWD is adopted to measure the similarity between the synthetic sample and the real sample, so that the quality of the synthetic sample is evaluated. And finally, inputting the synthesized sample data in different observation angle directions into a CNN classifier based on ResNet50 for training so as to realize omnidirectional human body activity recognition.

Description

Technical Field

[0001] The present invention relates to the field of radar signal processing technology, in particular to an omnidirectional human activity recognition method, device, computer and medium based on HDMR. Background Art

[0002] With the rapid development of millimeter-wave radar sensor technology, its application value in human activity recognition (HAR) is becoming increasingly prominent. With its all-weather operation, non-contact detection characteristics, long detection range, and strong penetration capabilities, millimeter-wave radar shows broad application prospects in medical health monitoring, intelligent security, assisted living systems, and smart home control. This technology captures the micro-Doppler (mD) effect caused by human motion and can extract radial velocity characteristics of targets in dynamic motion. Traditional signal processing methods, such as short-time Fourier transform (STFT), can convert the raw radar signal into a time-frequency spectrum, thereby extracting a discriminative micro-Doppler time-frequency spectrum, providing an important basis for human activity classification.

[0003] Existing human activity recognition methods based on micro-Doppler time-frequency spectrum usually assume that human activities occur in the direction of the radar's zero-degree observation angle, where the observation angle is the angle between the radar's line of sight (LoS) direction and the target's walking direction. However, in actual application scenarios, human activity trajectories are usually free to change, making the HAR task more complicated. This is mainly because the Doppler frequency shift caused by radial velocity is affected by the target's observation angle. That is, when the target performs the same human activity under different observation angles, the amplitude and characteristics of its micro-Doppler time-frequency spectrum will vary significantly with the change of the observation angle. This "angle sensitivity" significantly affects the accuracy of omnidirectional human activity recognition (Omnidirectional HAR).

[0004] To mitigate the aforementioned angular sensitivity issue, sufficient radar data from different observation angles must be collected to train the neural network. Ideally, multiple radar stations could be deployed to obtain multi-angle training data, but manual data collection and annotation is time-consuming and difficult to implement. Therefore, data augmentation has become an effective means of addressing the data shortage issue. Common data augmentation methods currently include simulation based on motion capture (MOCAP) databases and synthetic data generation using generative adversarial networks (GANs). The former can generate diverse micro-Doppler signatures using different human body model parameters, but typically requires complex analytical modeling and optimization for specific scenarios. The latter can generate synthetic data similar to real data by learning from the distribution of existing data, but its complex neural network structure results in long training times and limited data generation efficiency. Summary of the Invention

[0005] The purpose of the present invention is to provide an omnidirectional human activity recognition method, device, computer and medium based on HDMR to solve the angular sensitivity problem of monostatic radar in omnidirectional human activity recognition.

[0006] To achieve the above objectives, the present invention proposes an omnidirectional human activity recognition method based on HDMR. The omnidirectional human activity recognition is not limited to the zero-degree radar observation angle direction, but includes human activity recognition in all observation angle directions. The specific steps are as follows:

[0007] Step S1: using a single frequency modulated continuous wave (FMCW) radar sensor to collect sample data of different human activities at the radar's zero-degree observation angle and other non-zero-degree observation angles, where the observation angle is the angle between the radar's line of sight (LoS) direction and the human walking direction;

[0008] The FMCW radar sensor uses time division multiplexing to transmit FMCW radar signals with a carrier frequency of 77GHz through two transmitting antennas. After being reflected by the target, four receiving signals are obtained from four receiving antennas. The received signals are mixed with the corresponding transmitted signals to obtain the beat signals corresponding to the eight virtual array elements. Wherein, the kth column represents the kth chirp signal, the dth row represents the dth sampling point in each chirp signal, u is the uth transmitting antenna, v is the vth receiving antenna, and u and v are positive integers;

[0009] Step S2: Eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], successively b [k, d] performs fast-time fast Fourier transform FFT and short-time Fourier transform STFT to obtain the target range R[k] and radial micro-Doppler time-frequency spectrum S[l, n], where l is the time index and n is the frequency index;

[0010] Step S3, perform envelope detection on the micro-Doppler time-frequency spectrum S[l,n] of the collected samples at different observation angles, and convert the two-dimensional time-frequency spectrum into the envelope of one-dimensional data; by presetting different percentage thresholds Ra f Extract multiple envelopes reflecting the micro-Doppler characteristics of human activity in the time spectrum; when the observation angle ∈ is equal to 0°, the corresponding envelope matrix is ​​A∈R E ×L , the envelope matrix corresponding to the observation angle ∈ not equal to 0° is B∈R E×L , where E is the total number of envelopes and L is the maximum time index;

[0011] Step S4, using the high-dimensional model representation HDMR technology, respectively solve the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°; based on the mapping function of samples between different angles, generate synthetic samples of other observation angle directions from the samples when the observation angle ∈ is equal to 0° to construct a high-quality omnidirectional training set;

[0012] Step S5: Evaluate the quality of the synthesized sample based on the dynamic time warping distance (DTWD), i.e., use the DTWD value between the envelopes to measure the similarity between the synthesized sample and the real sample;

[0013] Step S6: Input high-quality synthetic sample data of different observation angles into a classifier based on the ResNet50 convolutional neural network (CNN) for training to achieve omnidirectional human activity recognition covering all observation angles.

[0014] Preferably, in step S2, the specific steps of obtaining the target distance and radial micro-Doppler time-frequency spectrum are as follows:

[0015] Step S21: enhance the beat signal S b [k, d] is arranged into a signal matrix according to fast time and slow time, where k is the slow time index and d is the fast time index; first, a fast time FFT operation is performed on each chirp signal to obtain the range gate of the target changing with slow time, and the intermediate frequency related to the target distance is obtained from the range gate changing with slow time, and the target distance R[k] is obtained based on the intermediate frequency;

[0016] Step S22: Extract the range gate corresponding to the target from each column of the enhanced beat signal after fast-time FFT, obtain a row of beat signals that vary with slow time, and perform an STFT operation on the row of beat signals that vary with slow time to obtain a radial micro-Doppler time-frequency spectrum related to the dynamic radial velocity.

[0017] Preferably, in step S3, the steps of envelope extraction are as follows:

[0018] Step S31: Preset different percentage thresholds Ra f , each percentage threshold Ra f Used to extract an envelope;

[0019] Step S32: Search each column of the micro-Doppler time spectrum S[l,n] for the closest Ra f R a (l,n), and extract R a (l,n) corresponds to the frequency index n; where R a (l,n) represents the frequency index n to the maximum frequency index N of each column mThe ratio of the cumulative energy of to the total energy of all frequency indices at each time index l;

[0020] Step S33: for each Ra f The corresponding frequency index n that varies with time is extracted, and the frequency component that varies with time can be extracted according to the frequency index. The frequency component that varies with time is the extracted envelope.

[0021] Preferably, in step S4, the steps of solving the mapping function are as follows:

[0022] Step S41: construct a mean square error between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°. The formula is as follows:

[0023]

[0024] in, is the Doppler frequency of all envelopes with time index l in the envelope matrix A, is the Doppler frequency corresponding to the lth frequency index in the Eth envelope, is the Doppler frequency of the e-th envelope at time index l in the envelope matrix B, where e is the index of the envelope, e = 1, 2, ..., E, f(·) is the mapping function, and T is the transpose of the matrix;

[0025] Step S42: Using HDMR expansion, convert f(a l ) is decomposed into a linear combination of low-order functions, thereby obtaining the corresponding decomposition function:

[0026]

[0027] Among them, f0 represents the average response of the sample, through Perform calculations; is the Doppler frequency corresponding to the lth frequency index in the eth envelope in the envelope matrix A, α em for peace The coefficients of the mth orthogonal basis functions decomposed by HDMR are: is an orthogonal basis function, β e =[β 11 …α 1M …α E1 …α EM ] T , is the column vector of all relevant orthogonal basis functions, m=1,2,...,M, M is the number of orthogonal basis functions;

[0028] Step S43: Optimize the mean square error. The formula is as follows:

[0029]

[0030] Get the α in the corresponding mapping function e , calculate all α of the mapping function between each envelope of envelope matrix B and envelope matrix A respectively e , thus obtaining the final mapping function between the two envelope matrices, where λ h is the regularization parameter.

[0031] Preferably, in step S5, the DTWD value between envelopes is calculated as follows:

[0032] Randomly select p real samples and p synthetic samples for each human activity, where the d1th real sample and the d2th synthetic sample are and

[0033]

[0034] in, for and DTWD value between f is the final DTWD value between p pairs of selected samples.

[0035] The present invention also provides an omnidirectional human activity recognition device based on HDMR, comprising:

[0036] Mixing module: uses millimeter wave radar sensor, which includes two transmitting antennas and four receiving antennas arranged in sequence in the horizontal direction. The horizontal distance between two adjacent transmitting antennas is 2λ, and the horizontal distance between two adjacent receiving antennas is λ / 2, where λ is the distance; the two transmitting antennas use time division multiplexing to transmit carrier frequency f respectively. c The 77GHz FMCW radar signal is reflected by the target and four receiving signals are obtained from the four receiving antennas. Each receiving antenna performs mixing processing according to the corresponding transmitting signal of the received signal to obtain the beat signals corresponding to the eight virtual array elements.

[0037] Human activity radial micro-Doppler time-spectrum map acquisition module: used for eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], successively b [k, d] performs fast time fast Fourier transform FFT and short time Fourier transform STFT to obtain the target range R[k] and radial micro-Doppler time spectrum S[l, n];

[0038] Human activity time spectrum envelope acquisition module: used to detect the envelope of the micro-Doppler time spectrum S[l,n] of human activity and convert the two-dimensional time spectrum into the envelope of one-dimensional data; by presetting different percentage thresholds Ra f Extract multiple envelopes reflecting the micro-Doppler characteristics of human activity in the time spectrum;

[0039] Human activity synthetic data generation module: This module uses a high-dimensional model to represent the HDMR technology and solves the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°. Based on the mapping function of samples between different angles, synthetic samples of other observation angle directions are generated from samples with an observation angle ∈ equal to 0° to construct a high-quality omnidirectional training set.

[0040] Human activity synthetic sample quality assessment module: This module is used to evaluate the quality of synthetic samples based on the dynamic time warping distance (DTWD). This module uses the DTWD value between envelopes to measure the similarity between synthetic samples and real samples.

[0041] Omnidirectional human activity recognition module: used to input a sufficient number of high-quality synthetic sample data from different observation angles into a classifier based on the ResNet50 convolutional neural network (CNN) for training, thereby realizing omnidirectional human activity recognition covering all observation angles.

[0042] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions to implement the above-mentioned HDMR-based omnidirectional human activity recognition method.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned HDMR-based omnidirectional human activity recognition method.

[0044] Therefore, the present invention proposes an omnidirectional human activity recognition method, device, computer and medium based on HDMR, which have the following beneficial effects:

[0045] The present invention can generate a synthetic data algorithm based on a high-dimensional model representation, and can perform data expansion on the collected samples in the zero-degree observation angle direction of the radar and a small number of samples in other non-zero-degree observation angle directions to generate high-quality training sample data containing all observation angle directions, thereby realizing omnidirectional human activity recognition.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1Schematic diagram of a flow chart of an omnidirectional human activity recognition method based on high-dimensional model representation (HDMR) of the present invention;

[0048] Figure 2 The array model of the TDM-MIMO radar used in the embodiment of the present invention, wherein: Figure 2 (a) in the figure is the physical array model. Figure 2 (b) in the figure is a virtual array model;

[0049] Figure 3 This is a schematic diagram of the human activity "walking" obtained in an embodiment of the present invention, wherein: Figure 3 A in the figure is the micro-Doppler time spectrum. Figure 3 B in the figure is the envelope diagram extracted from the time-frequency spectrum diagram;

[0050] Figure 4 A schematic diagram of a mapping model based on high-dimensional model representation (HDMR) adopted in an embodiment of the present invention;

[0051] Figure 5 Schematic diagram of the real sample envelope and the synthesized sample envelope obtained by the embodiment of the present invention, wherein: Figure 5 Where A is the true sample envelope with an observation angle of 0° for “walking” and the synthetic sample envelope with an observation angle of [30°:30°:180°]; Figure 5 Where B is the true sample envelope with an observation angle of 0° for “kicking” and the synthetic sample envelope with an observation angle of [30°:30°:180°]; Figure 5 C is the real sample envelope with an observation angle of 0° for “picking up” and the synthetic sample envelope with an observation angle of [30°:30°:180°]; Figure 5 Where D is the real sample envelope with an observation angle of 0° about the “jumping jack” and the synthetic sample envelope with an observation angle of [30°:30°:180°]; Figure 5 Where E is the true sample envelope with an observation angle of 0° for “standing up” and the synthetic sample envelope with an observation angle of [30°:30°:180°]; Figure 5 Where F is the true sample envelope with an observation angle of 0° for “no activity” and the synthetic sample envelope with an observation angle of [30°:30°:180°];

[0052] Figure 6 A schematic diagram of multiple observation angle directions concerned with omnidirectional human activity recognition in an embodiment of the present invention;

[0053] Figure 7 Schematic diagram of a synthetic sample when the observation angle is 30° in an embodiment of the present invention, wherein: Figure 7 A in the figure is a synthetic sample generated by the MOCAP-based method; Figure 7 B in the figure is a synthetic sample generated by the CGANs-based method;

[0054] Figure 8 Schematic diagram of synthetic samples generated by different methods at different observation angles in an embodiment of the present invention, wherein: Figure 8 A in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 30°; Figure 8 B in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 60°; Figure 8 C in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 90°; Figure 8 D in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 120°; Figure 8 E in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 150°; Figure 8 F in FIG is a schematic diagram of synthetic samples generated by different methods when the observation angle is 180°;

[0055] Figure 9 Schematic diagram of confusion matrix for identifying and classifying six activities at different observation angles in an embodiment of the present invention, where: Figure 9 A in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 0°; Figure 9 Figure B is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 30°; Figure 9 C in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 60°; Figure 9 D in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 90°; Figure 9 E in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 120°; Figure 9 F in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 150°; Figure 9 G in the figure is a diagram of the confusion matrix for identifying and classifying six activities when the observation angle is 180°;

[0056] Figure 10 A schematic structural diagram of an omnidirectional human activity recognition device based on high-dimensional model representation (HDMR) provided by an embodiment of the present invention;

[0057] Figure 11 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention.

[0058] Reference numerals:

[0059] 1100: Omnidirectional human activity recognition device; 1101: Frequency mixing module; 1102: Human activity radial micro-Doppler time-frequency spectrum acquisition module; 1103: Human activity time-frequency spectrum envelope acquisition module; 1104: Human activity synthetic data generation module; 1105: Human activity synthetic sample quality assessment module; 1106: Omnidirectional human activity recognition module; 1200: Computer device; 1201: Processor; 1202: Memory; 1203: Communication component; 1204: Bus. DETAILED DESCRIPTION

[0060] To make the technical solutions, advantages, and objectives of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are part of the embodiments of the present invention, not all of them. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0062] First, the relevant concepts or nouns involved in the present invention are explained:

[0063] HDMR (high-dimensional model representation): a mathematical model used to decompose complex high-dimensional mappings into low-dimensional functions, thereby effectively capturing the nonlinear relationship between input-output systems.

[0064] Omnidirectional HAR (Omnidirectional human activity recognition): Human activity recognition in all observation angles, where the observation angle is the angle between the radar line of sight and the target's walking direction.

[0065] DTWD (dynamic time wrapping distance): is a method for measuring the similarity between two time series. It uses dynamic programming to calculate the minimum matching distance required to nonlinearly align two time series.

[0066] FMCW (Frequency Modulated Continuous Wave): FMCW technology is a waveform modulation method used in radar systems. It continuously transmits a frequency-modulated signal (usually a linear frequency-modulated sine wave) and receives the signal reflected from the target object. The frequency deviation between the transmitted and received signals is proportional to the distance to the target.

[0067] Example 1

[0068] This method requires only a single 77GHz millimeter-wave radar sensor to achieve omnidirectional human activity recognition. It can be used in auxiliary systems in multiple fields. For example, in intelligent transportation and advanced driver assistance systems (ADAS), it can be used for driver status monitoring, in-vehicle passenger behavior analysis, and pedestrian detection; in the field of smart healthcare, it can achieve fall detection, gait analysis, and contactless vital sign monitoring; in smart home and security monitoring, it can provide privacy-friendly abnormal behavior detection and behavior pattern analysis. At the same time, this method is described in detail below with reference to the accompanying drawings to describe the vehicle following control method provided by the embodiment of the present invention.

[0069] like Figure 2 As shown in FIG. , an array arrangement diagram of antennas in a millimeter-wave radar used in an embodiment of the present invention is shown. The millimeter-wave radar used in an embodiment of the present invention includes two transmitting antennas and four receiving antennas, wherein: Figure 2 (a) is the array layout of the transmitting antenna in the millimeter wave radar. Figure 2 (b) is the array layout of the receiving antenna in the millimeter wave radar, S T1 and S T2 They are the first transmitting antenna and the second transmitting antenna arranged in sequence in the horizontal direction, S R1 、S R2 、S R3 and S R4 The millimeter-wave radar comprises two transmitting antennas and four receiving antennas arranged in sequence in the horizontal direction. The horizontal distance between two adjacent transmitting antennas is 2λ, and the horizontal distance between two adjacent receiving antennas is λ / 2, where λ is the wavelength.

[0070] The radar sensor used in this embodiment adopts Figure 2 As shown in the array arrangement, the two transmitting antennas of the millimeter wave radar transmit FMCW signals in a time division multiplexing (TDM) manner, and the carrier frequency of the transmitted signal is f c The receiving antenna receives the echo signal reflected from the target, and then mixes the received signal with the corresponding transmitted signal to obtain the beat signal corresponding to the eight virtual array elements;

[0071] like Figure 1 FIG. 1 is a flow chart of a method for omnidirectional human activity recognition based on high-dimensional model representation (HDMR) provided by the present invention. The method can perform data expansion on samples collected at the zero-degree observation angle direction of the radar and a small number of samples at other non-zero-degree observation angle directions based on the synthetic data generation algorithm of HDMR to generate high-quality training sample data covering all observation angle directions, thereby realizing omnidirectional human activity recognition. The specific steps are as follows:

[0072] S110, two transmitting antennas use time division multiplexing to transmit carrier frequency f c The 77GHz FMCW signal is reflected by the target and four receiving signals are obtained from the four receiving antennas. The mixed signal is processed according to the corresponding transmitting signal of the receiving signal to obtain the beat signal corresponding to the eight virtual array elements. Wherein, the kth column represents the kth chirp signal, the dth row represents the dth sampling point in each chirp signal, u is the uth transmitting antenna, v is the vth receiving antenna, and u and v are positive integers;

[0073] Specifically, the processing method of step 110 includes:

[0074] The pulse repetition period of the frequency modulated continuous wave signal transmitted by the transmitting antenna is T, the signal bandwidth is B, each pulse repetition period signal contains D sampling points, and the sampling interval is T s , let the kth pulse repetition period signal of the uth transmitting antenna be , calculated using the following formula:

[0075]

[0076] where t = kT + dT s , d=1,2,...,D, j represents the imaginary part of the complex number, f c is the carrier frequency, μ=B / T represents the frequency modulation slope. Without considering the propagation loss, when the transmitted signal is irradiated to a radial velocity of V r The received signal corresponding to the kth pulse repetition period of the uth transmitting antenna and the vth transmitting antenna can be expressed as:

[0077]

[0078] Among them, the time delay It is expressed as the time delay caused by the distance between the target and the uth transmitting antenna and the vth receiving antenna, R uv [k] represents the target distance relative to the uth transmitting antenna and the vth receiving antenna within the kth pulse repetition period, It is represented as the initial distance of the target relative to the uth transmitting antenna and the vth receiving antenna, and c represents the speed of light. The received signal is then mixed with the conjugate of the transmitted signal to generate a low-frequency beat signal, which is expressed as:

[0079]

[0080] S120, for eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], and then successively b [k,d] performs fast time fast Fourier transform (FFT) and short time Fourier transform (STFT) to obtain the target range R[k] and radial micro-Doppler time spectrum S[l,n];

[0081] Specifically, the Fourier beamforming technology can be used to process eight-way beat signals. Perform phase compensation to make the eight-way receiving signals have the same phase, thereby achieving the purpose of enhancing the receiving signal;

[0082] From step S110, it can be seen that the transmission is performed through the first transmitting antenna, and the corresponding four-way beat signals are:

[0083]

[0084] where τ 1v =[τ 1v τ 1v +τ0 τ 1v +2τ0 τ 1v +3τ0], is the delay time caused by the arrangement of two adjacent receiving antennas, d s =λ / 2 is the baseline length between the two receiving antennas, and θ is the target azimuth. Similarly, the four-way beat signal corresponding to the second transmitting antenna is:

[0085]

[0086] where τ 2v =[τ 1v +4τ0τ 1v +5τ0τ 1v +6τ0τ 1v +7τ0], so the eight beat signals only have phase differences caused by integer multiples of τ0. When Fourier beamforming is performed to obtain the target angle θ, the solution is Thus, the phases of the eight received signals are aligned and superimposed to obtain an enhanced beat signal S b [k,d].

[0087] To enhance the beat signal Sb [k, d] performs fast time FFT to extract the range gate associated with the target, thereby obtaining the target's distance R[k]. At the same time, since the human body is a distributed target, it may occupy multiple range gates in each pulse. Assuming that the target occupies Q range gates, the beat signal of the qth range gate changing with slow time is x q [k], where q = 1, 2, ..., Q. The final micro-Doppler time-spectrum diagram can be obtained by separately analyzing each x q [k] performs a short-time Fourier transform and then adds the corresponding time-frequency spectrograms to obtain it. From a mathematical point of view, it can be expressed as:

[0088]

[0089] Where ω[·] is the sliding window, l is the time index, n is the frequency index, N is the length of the sliding window, and S[l,n] represents the power of the spectrogram.

[0090] S130, perform envelope detection on the micro-Doppler time-frequency spectrum S[l,n] of samples of different observation angle directions, thereby converting the two-dimensional time-frequency spectrum into the envelope of one-dimensional data. Preset different percentage thresholds Ra f Extract multiple envelopes reflecting the micro-Doppler characteristics of human activity in the time spectrum graph. The specific steps are as follows:

[0091] S131. Preset different percentage thresholds Ra in advance f , where each Ra f It can be used to extract a different envelope;

[0092] S132, search each column of the micro-Doppler time spectrum S[l,n] for the closest Ra f R a (l,n), and extract R a The corresponding frequency index n in (l,n), where R a (l,n) represents the frequency index n to the maximum frequency index N of each column m The ratio of the cumulative energy of to the total energy of all frequency indices at each time index l;

[0093] S133、For each Ra f The corresponding frequency index n that changes with time can be extracted. The frequency component that changes with time can be extracted according to the frequency index. The frequency component that changes with time is the extracted envelope. Based on experience, the present invention extracts ten corresponding envelopes for the micro-Doppler time spectrum of each sample. Taking the human activity "walking" as an example, the corresponding micro-Doppler time spectrum and the extracted envelope are as follows: Figure 3 A and Figure 3 As shown in B;

[0094] S134. When the observation angle ∈ is equal to 0°, the corresponding envelope matrix is ​​recorded as A∈R E×L , the envelope matrix corresponding to the observation angle ∈ not equal to 0° is recorded as B∈R E×L , where E is the total number of envelopes and L is the maximum time index;

[0095] S140. Using a high-dimensional model representation (HDMR) technique, respectively solve the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°. Then, based on the mapping function of samples between different angles, generate a sufficient number of synthetic samples of other observation angle directions from a large number of samples when the observation angle ∈ is equal to 0°, thereby constructing a high-quality omnidirectional training set. The specific steps are as follows:

[0096] S141, such as Figure 4 As shown, by constructing the mapping function between the envelope matrices A and B through HDMR, HDMR can decompose the complex high-dimensional mapping into low-dimensional functions, thereby effectively capturing the nonlinear relationship between the input and output systems. represents the Doppler frequency of all envelopes at time index l; the e-th row vector in B corresponds to the e-th envelope, where is the Doppler frequency of the e-th envelope in the envelope matrix B at time index l, l = 1, 2, ..., L, e = 1, 2, ..., E, and f(·) is the mapping function;

[0097] S142, a l and The mapping function f(·) between can be solved by minimizing the mean square error, and its specific expression is as follows:

[0098]

[0099] For the input vector a in formula (8) l , which can be decomposed into a linear combination of low-order functions using HDMR expansion, resulting in the following function:

[0100]

[0101] Where f0 represents the average response of the sample, which can be obtained by Perform simple calculations to obtain; α e =[α 11 …α 1M …α E1 …α EM ] T , the second term of formula (9) is a linear combination of orthogonal basis functions, It is represented as an orthogonal basis function. The present invention selects the orthogonal basis function as Legendre polynomials (LPs), M is the number of orthogonal basis functions, α e represents the column vector containing all relevant LPs coefficients;

[0102] S143. By substituting formula (9) into formula (8) and introducing the regularization term, we can obtain the following function:

[0103]

[0104] Among them, λ h represents the regularization parameter; in formula (8) can be rewritten in matrix form,

[0105]

[0106] Where, Q = [Φ(a 1 ),Φ(a 2 ),…,φ(a L )] T ∈R L×EM ; Solve by solving the derivative of the objective function with respect to each coefficient and setting it to zero The optimization problem is as follows:

[0107]

[0108] Once all the α values ​​of the mapping function between each envelope of the envelope matrix B and the envelope matrix A are calculated, e , the final mapping function between the two envelope matrices can be obtained according to formula (9).

[0109] S144. Based on the solved mapping function of human activity samples between different angles, generate a sufficient number of synthetic samples in other observation angle directions from a large number of samples with an observation angle ∈ equal to 0°, thereby constructing a high-quality omnidirectional training set;

[0110] S150, evaluating the quality of the synthesized sample based on the dynamic time warping distance (DTWD), that is, using the DTWD value between the envelopes to measure the similarity between the synthesized sample and the real sample;

[0111] Specifically, 10 real samples and 10 synthetic samples of each human activity are randomly selected, where the d1th real sample and the d2th synthetic sample are represented as and and The DTWD value between is defined as The final DTWD value of 10 pairs of samples is recorded as DTWDf , its specific mathematical expression is as follows:

[0112]

[0113] Among them E 1(e) and E 2(e) Respectively and The e-th envelope of , DTW(·) is a function that calculates the DTWD value between two time series signals. The smaller the DTWD value, the higher the matching degree between the synthetic sample and the real sample, and the higher the quality of the synthetic sample.

[0114] S160. Input a sufficient number of high-quality synthetic sample data with different observation angles into a classifier based on the ResNet50 convolutional neural network (CNN) for training, thereby achieving omnidirectional human activity recognition including all observation angles.

[0115] Specifically, when the observation angle is 0°, 150 real samples of each human activity are divided into training and test sets in a 4:1 ratio. When the observation angle is not 0°, 120 synthetic samples of each human activity are used as the training set, and 30 real samples are used as the test set. The envelope of the training samples is input into a ResNet50 CNN-based classifier for training. The initial learning rate is set to 0.01, 16 samples are sampled for each training session, and two training iterations are performed before each test.

[0116] This embodiment uses the AWR1843 radar platform for experiments, which is a 77GHz system-on-chip (SoC) FMCW radar. The parameter list of the transmitted signal in the experiment is shown in Table 1. This embodiment considers the recognition and classification of six human activities, namely "walking", "kicking", "picking up", "jumping jack", "standing up" and "no activity". For omnidirectional human activity recognition, it is usually necessary to consider all observation angle directions from 0° to 360°. However, when the observation angle is symmetrical about 180°, the spectrum diagram will show symmetrical Doppler characteristics in micro-Doppler. Therefore, in order to reduce data acquisition time, the present invention only focuses on the observation angles between 0° and 180°, with an interval of 30°, such as Figure 6As shown in Figure 2. When the observation angle is 0°, five participants (three males and two females) collected 30 samples for each human activity, for a total of 150 samples per activity. When the observation angle is not equal to 0°, the proposed algorithm only requires a small number of samples for each activity at different observation angles. For each non-zero observation angle, five participants collected 6 samples for each activity, for a total of 30 samples.

[0117] Parameter name Parameter value <![CDATA[Center frequency (f c )]]> 77GHz Signal bandwidth (B) 3.602GHz Pulse repetition interval (PRI) 120μs <![CDATA[Sampling frequency (f s )]]> 10.24MHz Number of samples collected per pulse (M) 256 Number of pulses per frame signal (K) 256 <![CDATA[Number of frames of the signal (N f )]]> 64 FM slope (MHz / μs) 36.017

[0118] First, five pairs of samples are randomly selected from the samples collected when the observation angle is equal to 0° and the observation angle is not equal to 0°, and the envelopes extracted from each pair of samples are respectively input into the proposed synthetic data generation algorithm. At the same time, in order to improve the generalization ability of the mapping model, the final mapping function is obtained by averaging the mapping functions solved for the five pairs of samples. Subsequently, 120 samples with an observation angle equal to 0° are used to generate synthetic samples under different observation angles. That is, 120 (samples / activity / observation angle) × 6 (activity) × 6 (observation angle) = 4320 synthetic samples are generated. The envelopes of the real samples and synthetic samples of the six activities are as follows Figure 5 A- Figure 5 As shown in F.

[0119] In order to compare the quality of synthetic samples, a simulation method based on MOCAP and an image translation model based on conditional adversarial generative networks (CGANs) were selected as comparison methods. The micro-Doppler time-spectrograms and envelopes corresponding to the synthetic samples of the two benchmark methods are shown in Figure 2. Figure 7 A and Figure 7 As shown in B. Figure 7 Only the results when the observation angle is equal to 30° are shown. DTWD of synthetic samples of six activities generated by different methods for different observation angles f Value Figure 8 A- Figure 8 Shown in F. Smaller DTWD f It indicates that the match between the synthetic sample and the real sample is high. Figure 8 It is clearly shown that the synthetic samples generated by the proposed method achieve the lowest DTWD compared with the other two methods. f .like Figure 8 As shown in B, compared with the MOCAP-based method, the proposed method achieves the greatest quality improvement in the "jumping jack" activity, corresponding to the DTWD f The value decreased by 3001.8. Figure 8 The “no activity” in E in the proposed method is superior to the CGANs-based method, DTWD fThe value is reduced by 4195.7. Therefore, compared with the methods based on MOCAP and CGANs, the synthetic samples generated by the proposed method have better quality.

[0120] To evaluate sample generation time, we measured the time required to generate 70 synthetic samples using different methods. The experiments were conducted on a computer with 16GB of memory, an Intel(R) Core(TM) i7-6700H 3.40GHz CPU, and an Intel(R) HD Graphics 530 GPU. The results, reported in minutes, are shown in Table 2. Table 2 clearly shows that the MOCAP-based method is the fastest, with sample generation times ranging from 1.5 to 2.2 minutes per activity. The CGANs-based method requires significantly longer runtimes, with an average sample generation time of approximately 71.3 minutes per activity. The algorithm in our method demonstrates significant improvement over the CGANs-based method, with an average sample generation time ranging from 7.5 to 10.2 minutes per activity. While the MOCAP-based method generates samples faster, it is typically designed for specific human activities in the MOCAP dataset and lacks generalizability. Therefore, compared to the two commonly used synthetic data generation algorithms, our method reduces sample generation time while maintaining high quality.

[0121] When the observation angle is equal to 0°, 150 real samples of each human activity are divided into training set and test set in a ratio of 4:1. When the observation angle is not equal to 0°, 120 synthetic samples of each human activity are used as training set and 30 real samples are used as test set. The envelope of the training samples is input into the classifier based on ResNet50 CNN for training. The initial learning rate is set to 0.01, 16 samples are extracted for each training, and 2 iterations of training are performed before each test. The confusion matrix of the six human activity recognition classifications is shown in Figure 2. Figure 9 A- Figure 9 As shown in Figure 1, the average HAR accuracy rates at different observation angles are 94.23%, 92.18%, 91.13%, 87.58%, 90.96%, 92.55%, and 93.45%, respectively. When the observation angle is close to 90°, the smaller radial micro-Doppler feature leads to a lower average HAR accuracy rate. On the contrary, when the observation angle is 0° or 180°, the radial micro-Doppler feature becomes significant, resulting in a higher average HAR accuracy rate. The average HAR accuracy rate at all observation angles exceeds 91%, verifying the effectiveness of the method in the present invention in omnidirectional human activity recognition.

[0122] Table 2 Time required for different methods to generate 70 synthetic samples

[0123] walk Pick up Kick jumping jacks Get up No activity MOCAP 2.1 1.5 1.8 1.7 2.2 1.9 CGANs 73.3 68.6 71.3 70.8 69.3 74.9 HDMR 8.3 10.2 8.4 9.6 9.1 7.5

[0124] In order to evaluate the performance of the method in the present invention in omnidirectional human activity recognition, the present invention compares the synthetic samples generated by the MOCAP-based and CGANs-based methods with the training set generated by the method in the present invention. The number of training samples generated by the two methods is consistent with the method in the present invention. The results of omnidirectional human activity recognition by different methods are shown in Table 3. The average HAR accuracy of the method in the present invention is always better than that of the CGANs-based method at all observation angles, with an improvement of 3.95%. Compared with the MOCAP-based method, the proposed algorithm achieves a HAR accuracy that is close to or slightly higher at most angles. Therefore, compared with the MOCAP-based and CGANs-based methods, the method in the present invention has better omnidirectional human activity recognition performance.

[0125] Table 3 Results of omnidirectional human activity recognition using different methods

[0126] MOCAP CGANs HDMR ∈＝0° 93.61％ 91.11％ 94.23％ ∈＝30° 92.78％ 89.97％ 92.18％ ∈＝60° 90.07％ 86.77％ 91.13％ ∈＝90° 85.69％ 82.33％ 87.58％ ∈＝120° 91.39％ 85.76％ 90.96％ ∈＝150° 91.93％ 88.23％ 92.55％ ∈＝180° 92.56％ 90.24％ 93.45％

[0127] In the embodiments of the present invention, a computer device or a main control device can be divided into functional modules according to the above-mentioned method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiments of the present invention is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0128] Example 2

[0129] like Figure 10 As shown in FIG, a schematic diagram of the structure of an omnidirectional human activity recognition device based on a high-dimensional model representation (HDMR) according to an embodiment of the present application is provided. The millimeter-wave radar sensor used includes two transmitting antennas and four receiving antennas arranged in sequence in the horizontal direction. The horizontal distance between two adjacent transmitting antennas is 2λ, and the horizontal distance between two adjacent receiving antennas is λ / 2. Figure 10 As shown, the omnidirectional human activity recognition device 1100 includes:

[0130] Mixing module S1101, two transmitting antennas use time division multiplexing to transmit carrier frequency f c The 77GHz FMCW radar signal is reflected by the target, and four receiving signals are obtained from the four receiving antennas. The mixed signal is processed according to the corresponding transmitting signal of the receiving signal to obtain the beat signal corresponding to the eight virtual array elements. Wherein, the kth column represents the kth chirp signal, the dth row represents the dth sampling point in each chirp signal, u is the uth transmitting antenna, v is the vth receiving antenna, and u and v are positive integers;

[0131] Human activity radial micro-Doppler time spectrum acquisition module S1102 is used for eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], and then successively b [k,d] performs fast-time fast Fourier transform (FFT) and short-time Fourier transform (STFT) to obtain the target range R[k] and radial micro-Doppler time-frequency spectrum S[l,n].

[0132] The human activity time spectrum envelope acquisition module S1103 is used to perform envelope detection on the micro-Doppler time spectrum S[l,n] of human activity, thereby converting the two-dimensional time spectrum into the envelope of one-dimensional data. f Extract multiple envelopes in the spectrogram that reflect the micro-Doppler characteristics of human activity.

[0133] The human activity synthetic data generation module S1104 is used to solve the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0° for the high-dimensional model representation (HDMR) technology; then, based on the mapping function of samples between different angles, a sufficient number of synthetic samples in other observation angle directions are generated from a large number of samples when the observation angle ∈ is equal to 0°, thereby constructing a high-quality omnidirectional training set.

[0134] The human activity synthetic sample quality assessment module S1105 is used to evaluate the quality of the synthetic samples based on the dynamic time warping distance (DTWD), that is, to measure the similarity between the synthetic samples and the real samples using the DTWD value between the envelopes.

[0135] The omnidirectional human activity recognition module S1106 is used to input a sufficient number of high-quality synthetic sample data from different observation angles into a classifier based on the ResNet50 convolutional neural network (CNN) for training, thereby realizing omnidirectional human activity recognition including all observation angles.

[0136] Example 3

[0137] like Figure 111 is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device 1200 includes at least one processor 1201 and a memory 1202. The computer device 1200 also includes a communication component 1203. The processor 1201, the memory 1202, and the communication component 1203 are connected via a bus 1204.

[0138] During the specific implementation process, at least one processor 1201 executes the computer execution instructions stored in the memory 1202, so that at least one processor 1201 executes a target tracking method based on 5D millimeter wave radar as executed by the computer device side above.

[0139] The specific implementation process of the processor 1201 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0140] In the above embodiments, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0141] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0142] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0143] The solutions provided by the embodiments of the present invention are introduced with respect to the functions implemented by the computer device and the main control device. It is understandable that, in order to implement the above functions, the computer device or the main control device includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of the various examples described in the embodiments disclosed in the embodiments of the present invention, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solutions of the embodiments of the present invention.

[0144] The present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the above-mentioned omnidirectional human activity recognition method based on high-dimensional model representation (HDMR) is implemented.

[0145] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0146] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a computer device or a main control device.

[0147] The present invention also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of a computer device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the computer device executes the solution provided by any of the above embodiments.

[0148] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0149] Therefore, the present invention provides an omnidirectional human activity recognition method, device, computer and medium based on HDMR. The synthetic data generation algorithm based on HDMR can expand the data of the collected samples in the zero-degree observation angle direction of the radar and a small number of samples in other non-zero-degree observation angle directions to generate high-quality training sample data containing all observation angle directions, thereby realizing omnidirectional human activity recognition.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. The omnidirectional human activity recognition method based on HDMR is characterized by: The omnidirectional human activity recognition is not limited to the zero-degree observation angle direction of the radar, but includes human activity recognition in all observation angle directions. The specific steps are as follows: Step S1: using a single frequency modulated continuous wave (FMCW) radar sensor to collect sample data of different human activities at the radar's zero-degree observation angle and other non-zero-degree observation angles, where the observation angle is the angle between the radar's line of sight (LoS) direction and the human walking direction; The FMCW radar sensor uses time division multiplexing to transmit FMCW radar signals with a carrier frequency of 77GHz through two transmitting antennas. After being reflected by the target, four receiving signals are obtained from four receiving antennas. The received signals are mixed with the corresponding transmitted signals to obtain the beat signals corresponding to the eight virtual array elements. Wherein, the kth column represents the kth chirp signal, the dth row represents the dth sampling point in each chirp signal, u is the uth transmitting antenna, v is the vth receiving antenna, and u and v are positive integers; Step S2: Eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], successively b [k, d] performs fast-time fast Fourier transform FFT and short-time Fourier transform STFT to obtain the target range R[k] and radial micro-Doppler time-frequency spectrum S[l, n], where l is the time index and n is the frequency index; Step S3, perform envelope detection on the micro-Doppler time-frequency spectrum S[l,n] of the collected samples at different observation angles, and convert the two-dimensional time-frequency spectrum into the envelope of one-dimensional data; by presetting different percentage thresholds Ra f Extract multiple envelopes reflecting the micro-Doppler characteristics of human activity in the time spectrum; when the observation angle ∈ is equal to 0°, the corresponding envelope matrix is ​​A∈R E×L , the envelope matrix corresponding to the observation angle ∈ not equal to 0° is B∈R E×L , where E is the total number of envelopes and L is the maximum time index; Step S4, using the high-dimensional model representation HDMR technology, respectively solve the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°; based on the mapping function of samples between different angles, generate synthetic samples of other observation angle directions from the samples when the observation angle ∈ is equal to 0° to construct a high-quality omnidirectional training set; Step S5: Evaluate the quality of the synthesized sample based on the dynamic time warping distance (DTWD), i.e., use the DTWD value between the envelopes to measure the similarity between the synthesized sample and the real sample; Step S6: Input high-quality synthetic sample data of different observation angles into a classifier based on the ResNet50 convolutional neural network (CNN) for training to achieve omnidirectional human activity recognition covering all observation angles.

2. The omnidirectional human activity recognition method based on HDMR according to claim 1, characterized in that: In step S2, the specific steps for obtaining the target range and radial micro-Doppler time-frequency spectrum are as follows: Step S21: enhance the beat signal S b [k, d] is arranged into a signal matrix according to fast time and slow time, where k is the slow time index and d is the fast time index; first, a fast time FFT operation is performed on each chirp signal to obtain the range gate of the target changing with slow time, and the intermediate frequency related to the target distance is obtained from the range gate changing with slow time, and the target distance R[k] is obtained based on the intermediate frequency; Step S22: Extract the range gate corresponding to the target from each column of the enhanced beat signal after fast-time FFT, obtain a row of beat signals that vary with slow time, and perform an STFT operation on the row of beat signals that vary with slow time to obtain a radial micro-Doppler time-frequency spectrum related to the dynamic radial velocity.

3. The omnidirectional human activity recognition method based on HDMR according to claim 2, characterized in that: In step S3, the steps of envelope extraction are as follows: Step S31: Preset different percentage thresholds Ra f , each percentage threshold Ra f Used to extract an envelope; Step S32: Search each column of the micro-Doppler time spectrum S[l,n] for the closest Ra f R a (l,n), and extract R a (l,n) corresponds to the frequency index n; where R a (l,n) represents the frequency index n to the maximum frequency index N of each column m The ratio of the cumulative energy of to the total energy of all frequency indices at each time index l; Step S33: for each Ra f The corresponding frequency index n that varies with time is extracted, and the frequency component that varies with time can be extracted according to the frequency index. The frequency component that varies with time is the extracted envelope.

4. The omnidirectional human activity recognition method based on HDMR according to claim 3, characterized in that: In step S4, the steps for solving the mapping function are as follows: Step S41: construct a mean square error between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°. The formula is as follows: in, is the Doppler frequency of all envelopes with time index l in the envelope matrix A, is the Doppler frequency corresponding to the lth frequency index in the Eth envelope, is the Doppler frequency of the e-th envelope at time index l in the envelope matrix B, where e is the index of the envelope, e = 1, 2, ..., E, f(·) is the mapping function, and T is the transpose of the matrix; Step S42: Using HDMR expansion, convert f(a l ) is decomposed into a linear combination of low-order functions, thereby obtaining the corresponding decomposition function: Among them, f0 represents the average response of the sample, through Perform calculations; is the Doppler frequency corresponding to the lth frequency index in the eth envelope in the envelope matrix A, α em for peace The coefficients of the mth orthogonal basis functions decomposed by HDMR are: is an orthogonal basis function, α e =[α 11 …α 1M …α E1 …α EM ] T , is the column vector of all relevant orthogonal basis functions, m=1,2,...,M, M is the number of orthogonal basis functions; Step S43: Optimize the mean square error. The formula is as follows: Get the α in the corresponding mapping function e , calculate all α of the mapping function between each envelope of envelope matrix B and envelope matrix A respectively e , thus obtaining the final mapping function between the two envelope matrices, where λ h is the regularization parameter.

5. The omnidirectional human activity recognition method based on HDMR according to claim 4, characterized in that: In step S5, the DTWD value between envelopes is calculated using the following formula: Randomly select p real samples and p synthetic samples for each human activity, where the d1th real sample and the d2th synthetic sample are and in, for and DTWD value between f is the final DTWD value between p pairs of selected samples.

6. An omnidirectional human activity recognition device based on HDMR, characterized in that: include: Mixing module: uses millimeter wave radar sensor, which includes two transmitting antennas and four receiving antennas arranged in sequence in the horizontal direction. The horizontal distance between two adjacent transmitting antennas is 2λ, and the horizontal distance between two adjacent receiving antennas is λ / 2, where λ is the wavelength; the two transmitting antennas use time division multiplexing to transmit carrier frequencies f respectively. c The 77GHz FMCW radar signal is reflected by the target and four receiving signals are obtained from the four receiving antennas. Each receiving antenna performs mixing processing according to the corresponding transmitting signal of the received signal to obtain the beat signals corresponding to the eight virtual array elements. Human activity radial micro-Doppler time-spectrum map acquisition module: used for eight-way beat signal Perform Fourier beamforming technology to obtain an enhanced beat signal S b [k,d], successively b [k, d] performs fast time fast Fourier transform FFT and short time Fourier transform STFT to obtain the target range R[k] and radial micro-Doppler time spectrum S[l, n]; Human activity time spectrum envelope acquisition module: used to detect the envelope of the micro-Doppler time spectrum S[l,n] of human activity and convert the two-dimensional time spectrum into the envelope of one-dimensional data; by presetting different percentage thresholds Ra f Extract multiple envelopes reflecting the micro-Doppler characteristics of human activity in the time spectrum; Human activity synthetic data generation module: This module is used to represent the HDMR technology in a high-dimensional model. It solves the mapping function between the envelope matrix A when the observation angle ∈ is equal to 0° and the envelope matrix B when the observation angle ∈ is not equal to 0°. Based on the mapping function of samples between different angles, it generates synthetic samples in other observation angle directions from samples with an observation angle ∈ equal to 0°, thus constructing a high-quality omnidirectional training set. Human activity synthetic sample quality assessment module: This module is used to evaluate the quality of synthetic samples based on the dynamic time warping distance (DTWD). This module uses the DTWD value between envelopes to measure the similarity between synthetic samples and real samples. Omnidirectional human activity recognition module: used to input a sufficient number of high-quality synthetic sample data from different observation angles into a classifier based on the ResNet50 convolutional neural network for training, thereby realizing omnidirectional human activity recognition covering all observation angles.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions to implement the omnidirectional human activity recognition method based on HDMR as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the omnidirectional human activity recognition method based on HDMR as claimed in any one of claims 1 to 5 is implemented.

Images