Gait recognition method and system based on multi-time sequence ambiguity features of millimeter wave radar

Abstract

The application relates to a foot gesture recognition method and system based on a millimeter wave radar multi-time sequence fuzzy feature, which comprises the following steps: obtaining a radar echo signal of a to-be-recognized foot gesture action; extracting a multi-time sequence fuzzy feature of the to-be-recognized foot gesture action from the radar echo signal of the to-be-recognized foot gesture action; inputting the multi-time sequence fuzzy feature of the to-be-recognized foot gesture action into a pre-trained foot gesture action recognition model to obtain the category of the to-be-recognized foot gesture action. The foot gesture recognition method and system based on the millimeter wave radar multi-time sequence fuzzy feature can extract the multi-time sequence fuzzy feature from the echo signal of the to-be-recognized foot gesture action obtained by the millimeter wave radar, and predict the category of the to-be-recognized foot gesture action based on the pre-trained foot gesture action recognition model, so that the accuracy is high, and the false negative and false positive of the foot gesture action can be reduced.

Description

Technical Field

[0001] This invention relates to the field of millimeter-wave radar technology, and more specifically to a foot gesture recognition method and system based on multi-temporal ambiguity features of millimeter-wave radar. Background Technology

[0002] Today, the automotive technology field is increasingly moving towards automation and intelligence. As the main area for temporarily storing users' belongings, the trunk of a car is frequently opened and closed by users during daily use. Therefore, in response to user needs, the opening and closing control of the tailgate has gradually transitioned from the initial manual trigger spring mechanical opening and closing to the current motor-driven tailgate opening and closing, improving user convenience.

[0003] However, current electric motor-driven systems still require manual activation of physical buttons on the vehicle body or key to control the motor's opening and closing. This doesn't meet common usage scenarios, such as when users' hands are occupied while carrying items and they still need to operate the trunk opening and closing buttons. Furthermore, non-contact control methods have emerged in electric vehicle tailgate opening and closing systems that combine various sensors to detect user foot movements. Commonly used sensors include infrared sensors, ultrasonic sensors, and millimeter-wave radar sensors. Among these, millimeter-wave radar sensors are more suitable for practical applications in tailgate opening scenarios due to their insensitivity to environmental factors, strong anti-interference capabilities, and rich sensor data dimensions.

[0004] However, existing methods that use millimeter-wave radar to identify foot gestures and thus control the opening and closing of a car's tailgate have the following problems:

[0005] 1) Radar point clouds can accurately depict moving targets within the radar field of view in dimensions of distance, velocity, and angle, and can obtain a group of target points representing target information at different times. However, in extremely close-range scenarios, the characteristics of multiple target points representing the same individual target differ significantly, presenting an overall dispersed state and making it difficult to reflect a relatively unified spatial feature. Using this as a basis to describe foot movements cannot generalize and summarize the foot movements of different users, nor can it be expanded. For example, foot stroking or stepping movements of different users in different directions and with different amplitudes; at the same time, a single foot movement cannot meet the actual usage needs, and it is necessary to fully consider the user's expansion of door opening actions and the restrictions on actions by the scene to ensure the diversity of opening actions.

[0006] 2) Common methods of image recognition using graph features involve large amounts of data and complex data operations. On current low-power industrial-grade chips, signal processing and image processing are performed simultaneously, and complex logical decisions are also required. It is difficult to guarantee the efficiency and real-time performance of foot gesture judgment, and it cannot meet the needs of scenarios where the tailgate is opened and closed by foot gestures.

[0007] 3) The algorithm logic based on threshold parameter judgment can only be set empirically for specific actions and scenarios. It is difficult to adapt different logics to different users and different actions in different tailgate scenarios. The algorithm has poor maintainability and does not take into account the continuity of foot movements. Even when judging the order of parameters, which is close to the concept of time sequence, it does not use the correlation between the current frame data and the frames before and after it. In general, it does not fully explore the characteristics of the action, has poor generalization, and is prone to missing effective actions.

[0008] 4) In addition, the length of the feature sequences generated when users perform different foot movements at different speeds varies in time. Fixed-length features, whether using threshold decision or traditional model decision, cannot take into account the random variation of the length of the movement. They can only adapt to some compromise situations and are prone to missing valid movements. Summary of the Invention

[0009] This invention provides a foot gesture recognition method and system based on multi-temporal ambiguity features of millimeter-wave radar. It extracts multi-temporal ambiguity features from the echo signal of the foot gesture to be recognized obtained from millimeter-wave radar, and predicts the category of the foot gesture to be recognized based on a pre-trained foot gesture recognition model. It has high accuracy and can reduce the false alarms and missed alarms of foot gestures.

[0010] To achieve the above objectives, the present invention provides a foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar, comprising:

[0011] Acquire radar echo signals of the foot movements to be identified;

[0012] Extract the multi-temporal fuzzy features of the foot gesture to be identified from the radar echo signal of the foot gesture to be identified;

[0013] The multi-temporal fuzzy features of the foot gesture to be identified are input into a pre-trained foot gesture recognition model to obtain the category of the foot gesture to be identified.

[0014] Furthermore, multi-temporal fuzzy features of the foot gesture to be identified are extracted from the radar echo signal of the foot gesture to be identified, specifically including:

[0015] The range Doppler energy spectrum of the foot gesture to be identified is obtained based on the radar echo signal of the foot gesture to be identified.

[0016] The weighted average value of the unit distance energy is determined based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit.

[0017] The weighted average value of the unit Doppler energy is determined based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot posture to be identified and the Doppler index of each spectral unit.

[0018] Select each spectral unit whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot gesture to be identified, and use them as threshold-crossing units, and determine the number of threshold-crossing units.

[0019] Angle measurements are performed on each threshold unit to obtain the horizontal and pitch angles of each threshold unit;

[0020] The horizontal angle energy weighted average value is determined based on the horizontal angle and energy of each threshold unit, and the pitch angle energy weighted average value is determined based on the pitch angle and energy of each threshold unit.

[0021] The unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average are arranged in time sequence to obtain the time sequence of unit distance energy weighted average, the time sequence of unit Doppler energy weighted average, the time sequence of the number of threshold-crossing units, the time sequence of horizontal angle energy weighted average, and the time sequence of pitch angle energy weighted average. Each time sequence is used as a multi-temporal fuzzy feature of the foot gesture to be identified.

[0022] Further, the range Doppler energy spectrum of the foot gesture is obtained based on the radar echo signal of the foot gesture to be identified, specifically including:

[0023] Before performing the foot gesture to be identified, multiple frames of data within a preset time after the radar is powered on are acquired as multiple environmental frames.

[0024] The range-Doppler energy spectrum of each environmental frame is obtained, and the average range-Doppler energy spectrum of all environmental frames is obtained by averaging the range-Doppler energy spectra of all environmental frames.

[0025] Based on the radar installation height and radar range resolution, several range cells representing the distance to the vehicle underside are determined in the average range Doppler energy spectrum of the environmental frame, which serve as the environmental noise cells under the vehicle.

[0026] The average energy of all spectral units on each environmental noise unit is obtained as the environmental noise floor value;

[0027] The original range-Doppler energy spectrum of the foot gesture to be identified is obtained based on the radar echo signal of the foot gesture to be identified. The energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified is subtracted from the ambient noise floor value to obtain the range-Doppler energy spectrum of the foot gesture to be identified.

[0028] Further, based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot movement to be identified and the distance index of each spectral unit, a weighted average value of the unit distance energy is determined, specifically including:

[0029] For each spectral cell of the distance-Doppler energy spectrum of the foot gesture to be identified, the distance index of the spectral cell is multiplied by the energy of the spectral cell to obtain the distance energy of the spectral cell;

[0030] The total distance energy is obtained by summing the distance energies of all spectral units.

[0031] Add up the energies of all spectral units to get the total energy;

[0032] Divide the total distance energy by the total energy to obtain the weighted average value of the unit distance energy.

[0033] Further, based on the energy of each spectral unit of the distance Doppler energy spectrum of the footwork to be identified and the Doppler index of each spectral unit, the weighted average value of the unit Doppler energy is determined, specifically including:

[0034] For each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, the Doppler index of the spectral unit is multiplied by the energy of the spectral unit to obtain the Doppler energy of the spectral unit;

[0035] Add up the Doppler energies of all spectral units to obtain the total Doppler energy;

[0036] Add up the energies of all spectral units to get the total energy;

[0037] Divide the total Doppler energy by the total energy to obtain the weighted average value of the unit Doppler energy.

[0038] Furthermore, a weighted average of horizontal angle energy is determined based on the horizontal angle and energy of each threshold-crossing unit, and a weighted average of pitch angle energy is determined based on the pitch angle and energy of each threshold-crossing unit, specifically including:

[0039] The horizontal angle, pitch angle and energy of each threshold unit are used as three-dimensional data to perform three-dimensional density clustering, resulting in multiple threshold unit clusters.

[0040] Select the cluster containing the most threshold cells from multiple threshold cell clusters as the largest cluster;

[0041] For each threshold cell in the largest cluster, the horizontal angle of the threshold cell is multiplied by its energy to obtain the horizontal angle energy of the threshold cell; the pitch angle of the threshold cell is multiplied by its energy to obtain the pitch angle energy of the threshold cell.

[0042] The horizontal angle energy of each threshold-crossing unit in the largest cluster is added together to obtain the total horizontal angle energy, and the pitch angle energy of each threshold-crossing unit in the largest cluster is added together to obtain the total pitch angle energy.

[0043] The total energy of the maximum cluster is obtained by adding the energies of each threshold unit in the maximum cluster.

[0044] Divide the total horizontal angle energy by the total energy of the largest cluster to obtain the weighted average horizontal angle energy, and divide the total pitch angle energy by the total energy of the largest cluster to obtain the weighted average pitch angle energy.

[0045] Further, the unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average are arranged in time sequence, specifically including:

[0046] Determine the actual duration of the footwork motion to be identified;

[0047] The length of the radar frame window to be captured is determined based on the actual duration of the foot gesture to be identified.

[0048] Using the radar frame sequence as the time axis, the range-weighted average values ​​of the unit range energy of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the unit range energy weighted average value. The Doppler energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the unit Doppler energy weighted average value. The number of threshold-crossing units of each radar frame within the window length to be captured is arranged in time sequence to obtain the time sequence of the number of threshold-crossing units. The horizontal angle energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the horizontal angle energy weighted average value. The elevation angle energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the elevation angle energy weighted average value.

[0049] Furthermore, the foot gesture recognition model includes a feature batch normalization layer, a multi-feature fusion mapping layer, a multi-scale feature extraction layer, a temporal feature convolutional layer, and an action mapping layer. The feature batch normalization layer normalizes different temporal sequences to the same dimension; the multi-scale feature extraction layer fuses and remaps different temporal sequences together; the multi-scale feature extraction layer extracts features from temporal sequences of different lengths; the temporal feature convolutional layer includes two cascaded gated recurrent units; and the action mapping layer includes a fully connected layer.

[0050] Another aspect of the present invention provides a foot gesture recognition system based on multi-temporal ambiguity features of millimeter-wave radar, comprising:

[0051] The first acquisition module is configured to acquire radar echo signals of the foot movements to be identified;

[0052] The extraction module is configured to extract multi-temporal fuzzy features of the foot gesture to be identified from the radar echo signal of the foot gesture to be identified;

[0053] The recognition module is configured to input the multi-temporal fuzzy features of the foot gesture to be recognized into a pre-trained foot gesture recognition model to obtain the category of the foot gesture to be recognized.

[0054] Furthermore, the extraction module includes:

[0055] The second acquisition module is configured to acquire the range Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified.

[0056] The first determining module is configured to determine the weighted average value of the unit distance energy based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit;

[0057] The second determining module is configured to determine the weighted average value of the Doppler energy of the unit based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot movement to be identified and the Doppler index of each spectral unit.

[0058] The selection module is configured to select each spectral unit whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot gesture to be identified, as each threshold unit, and determine the number of threshold units.

[0059] The angle measurement module is set to measure the angles of each threshold unit to obtain the horizontal and pitch angles of each threshold unit.

[0060] The third determining module is configured to determine the horizontal angle energy weighted average value based on the horizontal angle and energy of each threshold unit, and to determine the pitch angle energy weighted average value based on the pitch angle and energy of each threshold unit.

[0061] The temporal arrangement module is configured to arrange the unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average in time sequence to obtain the temporal sequence of the unit distance energy weighted average, the temporal sequence of the unit Doppler energy weighted average, the temporal sequence of the number of threshold-crossing units, the temporal sequence of the horizontal angle energy weighted average, and the temporal sequence of the pitch angle energy weighted average. Each temporal sequence is used as a multi-temporal fuzzy feature of the foot gesture to be identified.

[0062] The second acquisition module specifically includes:

[0063] The first acquisition unit is configured to acquire multiple frames of data within a preset time after the radar is powered on, before performing the foot gesture to be identified, as multiple environmental frames.

[0064] The second acquisition unit is configured to acquire the range-Doppler energy spectrum of each environmental frame, and average the range-Doppler energy spectrum of all environmental frames to obtain the average range-Doppler energy spectrum of the environmental frames.

[0065] The first determining unit is set to determine several distance units representing the distance to the vehicle bottom in the average range Doppler energy spectrum of the environmental frame based on the radar installation height and the radar range resolution, and serve as the environmental noise units under the vehicle.

[0066] The third acquisition unit is configured to acquire the average energy of all spectral units on each environmental noise unit as the environmental noise baseline value;

[0067] The subtraction unit is configured to obtain the original range-Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified, and subtract the ambient noise floor value from the energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified to obtain the range-Doppler energy spectrum of the foot gesture to be identified.

[0068] The first determining module specifically includes:

[0069] The first multiplication unit is configured to multiply the distance index of the spectral unit with the energy of the spectral unit for each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, so as to obtain the distance energy of the spectral unit.

[0070] The first summing unit is set to sum the distance energies of all spectral units to obtain the total distance energy;

[0071] The second summing unit is set to add the energies of all spectral units to obtain the total energy;

[0072] The first division unit is configured to divide the total distance energy by the total energy to obtain a weighted average of the unit distance energy.

[0073] The second determining module specifically includes:

[0074] The second multiplication unit is configured to multiply the Doppler index of the spectral unit with the energy of the spectral unit for each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, so as to obtain the Doppler energy of the spectral unit.

[0075] The third summing unit is set to add the Doppler energies of all spectral units to obtain the total Doppler energy;

[0076] The fourth summation unit is set to add the energies of all spectral units to obtain the total energy;

[0077] The second phase division unit is configured to divide the total Doppler energy by the total energy to obtain a weighted average value of the unit Doppler energy.

[0078] The third determining module specifically includes:

[0079] The clustering unit is set to use the horizontal angle, pitch angle and energy of each threshold unit as three-dimensional data to perform three-dimensional density clustering, resulting in multiple threshold unit clusters;

[0080] The selection cell is set to choose the maximum cluster from multiple threshold cell clusters, which contains the largest number of threshold cells.

[0081] The third multiplication unit is configured to, for each threshold unit in the largest cluster, multiply the horizontal angle of the threshold unit by its energy to obtain the horizontal angle energy of the threshold unit; and multiply the pitch angle of the threshold unit by its energy to obtain the pitch angle energy of the threshold unit.

[0082] The fifth summing unit is configured to sum the horizontal angle energy of each threshold unit in the largest cluster to obtain the total horizontal angle energy, and to sum the pitch angle energy of each threshold unit in the largest cluster to obtain the total pitch angle energy.

[0083] The sixth summing unit is configured to sum the energies of each threshold unit in the maximum cluster to obtain the total energy of the maximum cluster;

[0084] The third division unit is configured to divide the total horizontal angle energy by the total energy of the largest cluster to obtain a weighted average horizontal angle energy, and to divide the total pitch angle energy by the total energy of the largest cluster to obtain a weighted average pitch angle energy.

[0085] The timing arrangement module specifically includes:

[0086] The second determining unit is configured to determine the actual duration of the foot movement to be identified;

[0087] The third determining unit is configured to determine the length of the radar frame window to be captured based on the actual duration of the foot gesture to be identified.

[0088] The arrangement unit is set with the radar frame sequence as the time axis. The unit range-energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of unit range-energy weighted average. The unit Doppler energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of unit Doppler energy weighted average. The number of threshold-crossing units of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of the number of threshold-crossing units. The horizontal angle energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of horizontal angle energy weighted average. The elevation angle energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of elevation angle energy weighted average. Each time sequence is used as a sample.

[0089] The foot gesture recognition model includes a feature batch normalization layer, a multi-feature fusion mapping layer, a multi-scale feature extraction layer, a temporal feature convolutional layer, and an action mapping layer. The feature batch normalization layer normalizes different temporal sequences to the same dimension; the multi-scale feature extraction layer fuses and remaps different temporal sequences together; the multi-scale feature extraction layer extracts temporal sequences of different lengths; the temporal feature convolutional layer includes two cascaded gated recurrent units; and the action mapping layer includes a fully connected layer.

[0090] The foot gesture recognition method and system based on millimeter-wave radar multi-temporal fuzzy features of this invention only measures the angles of each threshold unit, reducing the computational load of angle measurement; the horizontal and pitch angles of each threshold unit are clustered to remove the feature differences and dispersion of multiple target points in near-range foot gestures, unifying the spatial features of the threshold units; the multi-temporal fuzzy features include only five temporal sequence features, greatly reducing the computational and storage pressure on the lower-level machine; the pre-trained foot gesture recognition model can recognize a variety of foot gestures, has good scalability, and only requires updating the model parameters during updates and maintenance, making it highly maintainable; the foot gesture recognition model is a neural network model, which on the one hand can correlate and mine temporal features of a continuous foot gesture, improving the accuracy of foot gesture classification; on the other hand, it takes into account the different speeds of foot gestures of different users, solving the problem of different lengths of foot gestures in real-world scenarios, and reducing the false alarms and missed alarms in foot gesture recognition. Attached Figure Description

[0091] Figure 1 A flowchart of a foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to an embodiment of the present invention;

[0092] Figure 2 This is a schematic diagram of radar installation according to an embodiment of the present invention;

[0093] Figure 3 This is a schematic diagram of the single-cycle transmission and reception waveforms of a radar according to an embodiment of the present invention;

[0094] Figure 4 This is a schematic diagram of the structure of a foot gesture recognition model according to an embodiment of the present invention;

[0095] Figure 5 This is a structural block diagram of a foot gesture recognition system based on multi-temporal ambiguity features of millimeter-wave radar according to another embodiment of the present invention. Detailed Implementation

[0096] The preferred embodiments of the present invention are given below with reference to the accompanying drawings and described in detail.

[0097] like Figure 1 As shown, this embodiment of the invention provides a foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar, including the following steps:

[0098] S100: Acquire radar echo signals of the footwork motion to be identified.

[0099] like Figure 2 The radar 101 is typically mounted on the lower edge of the rear bumper 102 of a car, facing the ground. The angle and height of the radar facing the ground, as well as its distance from the outermost part of the rear tailgate, together determine the effective range of the radar's foot gesture detection. Therefore, the angle of the radar facing the ground can be adjusted according to the actual height of the rear bumper and the distance from the outermost part of the tailgate to make the core area of ​​the radar's field of view (FOV) the effective area for foot gestures and reduce the background interference introduced.

[0100] In this embodiment, the radar used is a frequency-modulated continuous wave radar with elevation and horizontal angle measurement capabilities, a range resolution of not less than 0.1m, an angle measurement accuracy of not less than 5°, and a sampling frame rate of not less than 8 frames per second after radar signal processing.

[0101] In an exemplary embodiment, a schematic diagram of a single-cycle (chirp) transmit and receive waveform of the radar is shown below. Figure 3 As shown, where f0 is the starting frequency of the transmitted signal, f sw T is the frequency sweep bandwidth for the transmitted signal. chirpLet τ be the duration of the transmitted signal, and τ be the time delay of the echo signal caused by the target distance, which results in a frequency change f caused by the echo signal shifting along the time axis. R Target speed v r The frequency change caused by the translation of the echo signal along the frequency axis is f D The difference frequency f is obtained by mixing the received target echo signal with the transmitted signal. B It can be seen that, for a stationary target, the difference frequency f B It is only affected by the target distance R, while for moving targets, the difference frequency f B The frequency shift includes not only the frequency shift caused by the target distance, but also the frequency change caused by the target velocity, as shown in the following formula:

[0102]

[0103] In the formula, R is the distance between the target and the radar, and v represents the distance between the target and the radar. r If negative, it is close to the radar v r It is positive.

[0104] S200: Extract multi-temporal fuzzy features of the footwork to be identified from the radar echo signal of the footwork to be identified.

[0105] In some embodiments, step S200 specifically includes the following steps S210-S270:

[0106] S210: Obtain the range Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified.

[0107] For each chirp, the received echo signal is sampled to obtain N sampling points. Performing an FFT (Fourier Transform) along these sampling points yields the range dimension information. Performing an FFT on the same range cell across M chirps yields the velocity dimension information, thus forming a Range-Doppler (RD) energy spectrum. The method for obtaining the corresponding range-Doppler energy spectrum from radar echo signals is well-known in the field and will not be elaborated upon here.

[0108] Because the road surface medium varies depending on the environment in which the vehicle is parked, the ambient noise in the radar illumination area also varies. To eliminate the influence of ambient noise, it can be eliminated first to obtain a denoised range-Doppler energy spectrum, and then the denoised range-Doppler energy spectrum can be used for subsequent steps. Specifically, step S210 may include the following steps S211-S215:

[0109] S211: Before performing the foot gesture to be identified, acquire multiple frames of data within a preset time (e.g., 1 second) after the radar is powered on, and use them as multiple environmental frames;

[0110] S212: Obtain the range-Doppler energy spectrum of each environmental frame, and average the range-Doppler images of all environmental frames to obtain the average range-Doppler energy spectrum of the environmental frames.

[0111] The range-Doppler energy spectrum of each environmental frame can be obtained from the radar echo signal of that environmental frame. The energy values ​​of all corresponding spectral units in the range-Doppler energy spectrum of each environmental frame are added together and then divided by the number of environmental frames to obtain the average energy value of each spectral unit. These spectral units constitute the average range-Doppler energy spectrum of the environmental frame.

[0112] S213: Based on the radar installation height and radar range resolution, determine several range units in the average range Doppler energy spectrum of the environmental frame that represent the distance to the vehicle bottom, and use them as environmental noise units under the vehicle.

[0113] The ratio of radar installation height to radar range resolution is the number of range cells representing the distance to the vehicle bottom in the average range Doppler energy spectrum of the environmental frame. Through the environmental noise cell, the influence of different environmental noise caused by environmental differences on motion feature detection and recognition can be eliminated.

[0114] S214: Obtain the average energy of all spectral units on each environmental noise unit under the vehicle as the environmental noise floor value;

[0115] The energy of all spectral units on each environmental noise unit within the vehicle undercarriage distance range is summed, and then divided by the total number of spectral units within the vehicle undercarriage distance range to obtain the environmental noise floor value within the vehicle undercarriage distance range.

[0116] S215: Obtain the original range-Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified. Subtract the ambient noise floor value from the energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified to obtain the range-Doppler energy spectrum of the foot gesture to be identified, i.e., the range-Doppler energy spectrum after noise removal.

[0117] By using the environmental noise baseline value, the problem of environmental noise differences caused by vehicles in different environments can be eliminated.

[0118] S220: Determine the weighted average value of the unit distance energy based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit.

[0119] The distance-Doppler energy spectrum of the footwork to be identified includes multiple spectral units. For example, if the distance-Doppler energy spectrum includes R... N If there are 1 distance unit and D velocity units, then it includes R N*D spectral units.

[0120] Step S220 specifically includes the following steps S221-S224:

[0121] S221: For each spectral cell, multiply the distance index of the spectral cell by the energy of the spectral cell to obtain the distance energy of the spectral cell;

[0122] S222: Sum the distance energies of all spectral units to obtain the total distance energy;

[0123] S223: Add the energies of all spectral units to obtain the total energy;

[0124] S224: Divide the total distance energy by the total energy to obtain the weighted average value of the unit distance energy.

[0125] The weighted average energy of a cell distance satisfies the following relationship:

[0126]

[0127] In the formula, R E E represents the energy weighted average of the unit distance. i R represents the energy of the i-th spectral unit, I represents the number of all spectral units on the distance-Doppler energy spectrum, and R represents the energy of the i-th spectral unit. i This represents the distance index corresponding to the i-th spectral unit.

[0128] S230: Determine the weighted average value of the unit Doppler energy based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot movement to be identified and the Doppler index of each spectral unit.

[0129] Step S230 specifically includes the following steps S231-S234:

[0130] S231: For each spectral unit, multiply the Doppler index of the spectral unit by the energy of the spectral unit to obtain the Doppler energy of the spectral unit;

[0131] S232: Add the Doppler energies of all spectral units to obtain the total Doppler energy;

[0132] S233: Add the energies of all spectral units to obtain the total energy;

[0133] S234: Divide the total Doppler energy by the total energy to obtain the weighted average value of the unit Doppler energy.

[0134] Unit Doppler energy weighted average D E The following relationship must be satisfied:

[0135]

[0136] In the formula, E i Let I represent the energy of the i-th spectral unit, and let D represent the number of all spectral units on the distance-Doppler energy spectrum. i This represents the velocity index corresponding to the i-th spectral unit.

[0137] S240: Select each spectral unit whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot posture to be identified, and use them as threshold units, and determine the number of threshold units.

[0138] The preset threshold can be selected as needed, for example, based on experience, existing algorithms, or any other suitable method. The preset threshold represents the set limit. Among the spectral units in the Doppler energy spectrum, those with energies exceeding the preset threshold are considered threshold-crossing units, and the number of spectral units with energies exceeding the preset threshold is the number of threshold-crossing units.

[0139] S250: Measure the angles of each threshold unit to obtain the horizontal and pitch angles of each threshold unit.

[0140] In some embodiments, for each threshold unit, multi-channel data corresponding to that threshold unit can be acquired. Specifically, the angle measurement can be performed using the DBF (Digital Beam Forming) method. The DBF method is a mature angle measurement method in this field, and its principle and specific steps will not be described here.

[0141] S260: Determine the horizontal angle energy weighted average value based on the horizontal angle and energy of each threshold unit, and determine the pitch angle energy weighted average value based on the pitch angle and energy of each threshold unit.

[0142] In some embodiments, the horizontal energy weighted average is the ratio of the sum of the products of the horizontal angle and energy of each threshold unit to the sum of the energy of each threshold unit, and the pitch angle energy weighted average is the ratio of the sum of the products of the pitch angle and energy of each threshold unit to the sum of the energy of each threshold unit.

[0143] Because radar performance, angle measurement methods, and three-dimensional footwork have multidirectional effects on radar wave reflection, the obtained angles may contain false angles and angles from other surfaces where the foot is not directly facing the radar surface. To ensure accuracy, in some embodiments, false angles can be removed first through three-dimensional density clustering. Specifically, step S260 includes the following steps S261-S266:

[0144] S261: Use the horizontal angle, pitch angle and energy of each threshold cell as three-dimensional data to perform three-dimensional density clustering to obtain multiple threshold cell clusters;

[0145] The cluster radius and minimum number of points in 3D density clustering can be adjusted according to actual needs. 3D density clustering is an existing data processing technique, and its principles and specific steps will not be elaborated here.

[0146] S262: Select the cluster of over-threshold cells with the largest number of over-threshold cells from multiple over-threshold cell clusters, and use it as the largest cluster;

[0147] S263: For each threshold cell in the largest cluster, multiply the horizontal angle of the threshold cell by its energy to obtain the horizontal angle energy of the threshold cell; multiply the pitch angle of the threshold cell by its energy to obtain the pitch angle energy of the threshold cell.

[0148] S264: Add the horizontal angle energies of all threshold cells in the largest cluster to obtain the total horizontal angle energy, and add the pitch angle energies of all threshold cells in the largest cluster to obtain the total pitch angle energy;

[0149] S265: Add the energies of all threshold cells in the maximum cluster to obtain the total energy of the maximum cluster;

[0150] S266: Divide the total horizontal angle energy by the total energy of the largest cluster to obtain the weighted average horizontal angle energy; divide the total pitch angle energy by the total energy of the largest cluster to obtain the weighted average pitch angle energy.

[0151] Horizontal angular energy weighted average Azi E The following relationship must be satisfied:

[0152]

[0153] Pitch angle energy weighted average Elv E The following relationship must be satisfied:

[0154]

[0155] In the formula, J represents the number of threshold cells contained in the largest cluster, and E j Let Azi represent the energy of the j-th threshold cell. j Elv represents the horizontal angle corresponding to the j-th threshold cell within the largest cluster, where j represents the j-th threshold cell in the largest cluster. j This represents the pitch angle corresponding to the j-th point within the largest cluster.

[0156] S270: Arrange the weighted average energy of unit distance, the weighted average energy of unit Doppler, the number of threshold units, the weighted average energy of horizontal angle, and the weighted average energy of pitch angle in time sequence to obtain the time sequence of the weighted average energy of unit distance, the time sequence of the weighted average energy of unit Doppler, the time sequence of the number of threshold units, the time sequence of the weighted average energy of horizontal angle, and the time sequence of the weighted average energy of pitch angle. Use each time sequence as a multi-temporal fuzzy feature of the foot gesture to be identified.

[0157] Specifically, step S270 includes:

[0158] S271: Determine the actual duration of the footwork motion to be identified;

[0159] S272: Determine the length of the radar frame window to be captured based on the actual duration of the foot gesture to be identified;

[0160] S273: Using the radar frame sequence as the time axis, arrange the unit range energy weighted average of each radar frame within the length of the radar frame window to be captured in time sequence to obtain the time sequence sequence of the unit range energy weighted average; arrange the unit Doppler energy weighted average of each radar frame within the length of the radar frame window to be captured in time sequence to obtain the time sequence sequence of the unit Doppler energy weighted average; arrange the number of threshold-crossing units of each radar frame within the length of the radar frame window to be captured in time sequence to obtain the time sequence sequence of the number of threshold-crossing units; arrange the horizontal angle energy weighted average of each radar frame within the length of the radar frame window to be captured in time sequence to obtain the time sequence sequence of the horizontal angle energy weighted average; arrange the elevation angle energy weighted average of each radar frame within the length of the radar frame window to be captured in time sequence to obtain the time sequence sequence of the elevation angle energy weighted average.

[0161] When a user performs different footwork actions at different speeds, the length of the resulting feature sequences varies in time. Even for the same type of footwork action, the temporal fuzzy feature sequence is shorter when the action speed is faster, and longer when the action speed is slower. Therefore, it is necessary to determine the actual duration of the footwork action to extract a suitable radar frame window length. The extracted radar frame window length is the length of the window containing a complete footwork action. By using the radar frame sequence as the time axis, and arranging the five features—the weighted average of cell range energy, the weighted average of cell Doppler energy, the number of threshold cells, the weighted average of horizontal angle energy, and the weighted average of pitch angle energy—in time, the temporal sequence of each feature can be obtained, thus obtaining the coherent features within a complete footwork action. For example, if the length of the radar frame window to be extracted is W, then the final feature can be 5*W in dimension. Inputting this into a pre-trained footwork action recognition model will yield the category of the footwork action to be recognized.

[0162] S300: Input the multi-temporal fuzzy features of the footwork to be identified into the pre-trained footwork recognition model to obtain the category of the footwork to be identified.

[0163] In some embodiments, such as Figure 4As shown, the foot gesture recognition model includes a feature batch normalization layer 1000, a multi-feature fusion mapping layer 2000, a multi-scale feature extraction layer 3000, a temporal feature convolutional layer 4000, and a motion mapping layer 5000. The feature batch normalization layer 1000 normalizes five different temporal sequence features to the same dimension, reducing the influence of dimensions on the model weight parameters. It consists of one BatchNormalization layer. The multi-scale feature extraction layer 3000 fuses and remaps five different temporal sequence features together. It consists of two 3×1 1D convolutional neural networks (1D CNNs), with a stride of 1 for the first 1D CNN and a stride of 2 for the second 1D CNN. The multi-scale feature extraction layer 3000 extracts temporal sequences of different lengths and includes four parallel 1D convolutional layers. The CNN and a concatenate layer, with kernel sizes of 1×1, 1×2, 1×3, and 1×5 respectively, extract temporal sequence features of different lengths, and then the concatenate layer splices and fuses the extracted features. The temporal feature convolutional layer 4000 consists of two GRU (Gate Recurrent Unit) networks with 10 hidden units connected in series. It can extract temporal axis feature information based on the time sequence between radar frames and learn to extract deep features from the multi-scale feature information obtained from the previous layer in the time dimension. The action mapping layer 5000 consists of a Dense layer (i.e., a fully connected layer) with a softmax activation function, which is used to map the processed features to classification categories in the form of probabilities.

[0164] In some embodiments, an Activation layer is provided between the multi-feature fusion mapping layer 2000 and the multi-scale feature extraction layer 3000; an Activation layer and a Max Pooling layer are connected in series between the multi-scale feature extraction layer 3000 and the temporal feature convolutional layer 4000; and a flattening layer and an Avg Pooling layer are connected in series between the temporal feature convolutional layer 4000 and the action mapping layer 5000.

[0165] In some embodiments, the footwork recognition model can be trained using the following methods:

[0166] The user performs footwork actions of a preset category within the radar's field of view (FOV). The radar then collects echo signals of different categories of footwork actions. These preset categories may include kicking in different directions, sliding from left to right, sliding from right to left, lifting and stomping, and stepping up and down with the sole of the foot in place, etc. Then, the methods in steps S210-S270 are used to extract multi-temporal fuzzy features of different footwork actions. These multi-temporal fuzzy features are used as samples (i.e., input), and the corresponding footwork category is used as the sample label (i.e., output). This is used to train the footwork recognition model, resulting in a trained footwork recognition model. To improve the model's generalization ability, multiple footwork actions can be collected for each category, and the duration of footwork actions within the same category can be the same or different.

[0167] The foot gesture recognition method based on multi-temporal fuzzy features of millimeter-wave radar in this invention only measures the angles of each threshold unit, reducing the computational load of angle measurement. Clustering is performed on the horizontal and elevation angles of each threshold unit to remove feature differences and dispersion of multiple target points in near-range foot gestures, unifying the spatial features of the threshold units. The multi-temporal fuzzy features include only five temporal sequence features, greatly reducing the computational and storage pressure on the lower-level machine. The pre-trained foot gesture recognition model can recognize multiple foot gestures, has good scalability, and only requires updating the model parameters during updates and maintenance, making it highly maintainable. The foot gesture recognition model is a neural network model, which, on the one hand, can correlate and mine temporal features of a continuous foot gesture, improving the accuracy of foot gesture classification; on the other hand, it takes into account foot gestures of different users at different speeds, solving the problem of varying lengths of foot gestures in real-world scenarios, and reducing false positives and false negatives in foot gesture recognition.

[0168] like Figure 5 As shown, another embodiment of the present invention provides a foot gesture recognition system based on multi-temporal ambiguity features of millimeter-wave radar, including: a first acquisition module 10, an extraction module 20 and a recognition module 30.

[0169] The first acquisition module 10 is used to acquire the radar echo signal of the foot posture to be identified.

[0170] The extraction module 20 is used to obtain the range Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified.

[0171] Extraction module 20 specifically includes:

[0172] The second acquisition module is used to acquire the range Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified.

[0173] The first determining module is used to determine the weighted average value of the unit distance energy based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit;

[0174] The second determining module is used to determine the weighted average value of the Doppler energy of the unit based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot posture to be identified and the Doppler index of each spectral unit.

[0175] The selection module is used to select spectral units whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot posture to be identified, as each threshold unit, and to determine the number of threshold units.

[0176] The angle measurement module is used to measure the angles of each threshold unit to obtain the horizontal and pitch angles of each threshold unit.

[0177] The third determining module is used to determine the horizontal angle energy weighted average value based on the horizontal angle and energy of each threshold unit, and to determine the pitch angle energy weighted average value based on the pitch angle and energy of each threshold unit.

[0178] The temporal arrangement module is configured to arrange the unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold units, the horizontal angle energy weighted average, and the pitch angle energy weighted average in time sequence to obtain the temporal sequence of the unit distance energy weighted average, the temporal sequence of the unit Doppler energy weighted average, the temporal sequence of the number of threshold units, the temporal sequence of the horizontal angle energy weighted average, and the temporal sequence of the pitch angle energy weighted average. Each temporal sequence is used as a multi-temporal fuzzy feature of the foot gesture to be identified.

[0179] The second acquisition module specifically includes:

[0180] The first acquisition unit is used to acquire multiple frames of data within a preset time (e.g., 1 second) after the radar is turned on before performing the foot gesture to be identified, as multiple environmental frames.

[0181] The second acquisition unit is used to acquire the range-Doppler energy spectrum of each environmental frame, and average the range-Doppler energy spectrum of all environmental frames to obtain the average range-Doppler energy spectrum of the environmental frames.

[0182] The first determining unit is used to determine several distance units representing the distance to the vehicle bottom in the average range Doppler energy spectrum of the environmental frame based on the radar installation height and the radar range resolution, as each environmental noise unit under the vehicle.

[0183] The third acquisition unit is used to acquire the average energy of all spectral units on each environmental noise unit, as the environmental noise baseline value;

[0184] The subtraction unit is used to obtain the original range-Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified. The energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified is subtracted from the ambient noise floor value to obtain the range-Doppler energy spectrum of the foot gesture to be identified, that is, the range-Doppler energy spectrum after removing noise.

[0185] The first determination module specifically includes:

[0186] The first multiplication unit is used to multiply the distance index of each spectral unit by the energy of that spectral unit to obtain the distance energy of that spectral unit.

[0187] The first summing unit is used to sum the distance energies of all spectral units to obtain the total distance energy;

[0188] The second summing unit is used to add the energies of all spectral units to obtain the total energy;

[0189] The first dividing unit is used to divide the total distance energy by the total energy to obtain the weighted average of the unit distance energy.

[0190] The second determination module specifically includes:

[0191] The second multiplication unit is used to multiply the Doppler index of each spectral unit by the energy of that spectral unit to obtain the Doppler energy of that spectral unit.

[0192] The third summing unit is used to add the Doppler energies of all spectral units to obtain the total Doppler energy;

[0193] The fourth summation unit is used to add the energies of all spectral units to obtain the total energy;

[0194] The second dividing unit is used to divide the total Doppler energy by the total energy to obtain the weighted average of the unit Doppler energies.

[0195] The third determination module specifically includes:

[0196] Clustering units are used to perform three-dimensional density clustering of the horizontal angle, pitch angle and energy of each threshold unit as three-dimensional data to obtain multiple threshold unit clusters.

[0197] The selection unit is used to select the threshold cell cluster with the largest number of threshold cells from multiple threshold cell clusters, and use it as the largest cluster;

[0198] The third multiplication unit is used to multiply the horizontal angle of each threshold unit in the largest cluster by its energy to obtain the horizontal angle energy of the threshold unit; and to multiply the pitch angle of the threshold unit by its energy to obtain the pitch angle energy of the threshold unit.

[0199] The fifth summing unit is used to sum the horizontal angle energy of each threshold unit in the largest cluster to obtain the total horizontal angle energy, and to sum the pitch angle energy of each threshold unit in the largest cluster to obtain the total pitch angle energy.

[0200] The sixth summing unit is used to add the energies of each threshold unit in the maximum cluster to obtain the total energy of the maximum cluster;

[0201] The third division unit is used to divide the total horizontal angle energy by the total energy of the largest cluster to obtain the weighted average horizontal angle energy, and to divide the total pitch angle energy by the total energy of the largest cluster to obtain the weighted average pitch angle energy.

[0202] The timing arrangement module specifically includes:

[0203] The second determining unit is used to determine the actual duration of the foot movement to be identified;

[0204] The third determining unit is used to determine the length of the radar frame window to be captured based on the actual duration of the foot gesture to be identified.

[0205] The arrangement unit is used to arrange the unit range energy weighted average of each radar frame within the radar frame window length to be captured in time sequence, with the radar frame sequence as the time axis, to obtain a time sequence sequence of unit range energy weighted average; arrange the unit Doppler energy weighted average of each radar frame within the radar frame window length to be captured in time sequence, to obtain a time sequence sequence of unit Doppler energy weighted average; arrange the number of threshold-crossing units of each radar frame within the radar frame window length to be captured in time sequence, to obtain a time sequence sequence of the number of threshold-crossing units; arrange the horizontal angle energy weighted average of each radar frame within the radar frame window length to be captured in time sequence, to obtain a time sequence sequence of horizontal angle energy weighted average; arrange the pitch angle energy weighted average of each radar frame within the radar frame window length to be captured in time sequence, to obtain a time sequence sequence of pitch angle energy weighted average; and use each time sequence sequence as a multi-temporal fuzzy feature of the foot gesture to be identified.

[0206] The recognition module 30 is used to input the multi-temporal fuzzy features of the foot posture to be recognized into the pre-trained foot posture recognition model to obtain the category of the foot posture to be recognized.

[0207] The structure of the footwork recognition model is the same as that of the model in the method of the previous embodiment, and will not be described again here.

[0208] The foot gesture recognition system based on millimeter-wave radar multi-temporal fuzzy features in this invention only measures the angles of each threshold unit, reducing the computational load of angle measurement. It performs clustering processing on the horizontal and elevation angles of each threshold unit, removing feature differences and dispersion among multiple target points in near-range foot gestures, and unifying the spatial features of the threshold units. The multi-temporal fuzzy features include only five temporal sequence features, greatly reducing the computational and storage pressure on the lower-level machine. The pre-trained foot gesture recognition model can recognize multiple foot gestures, has good scalability, and only requires updating model parameters during updates and maintenance, making it highly maintainable. The foot gesture recognition model is a neural network model, which, on the one hand, can correlate and mine temporal features of a continuous foot gesture, improving the accuracy of foot gesture classification; on the other hand, it takes into account foot gestures of different users at different speeds, solving the problem of varying lengths of foot gestures in real-world scenarios, and reducing false positives and false negatives in foot gesture recognition.

[0209] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0210] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0216] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0217] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0218] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0219] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0220] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0221] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0222] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. That is, all simple and equivalent changes and modifications made based on the claims and description of this invention fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar, characterized in that, include: Acquire radar echo signals of the foot movements to be identified; Extract the multi-temporal fuzzy features of the foot gesture to be identified from the radar echo signal of the foot gesture to be identified; The multi-temporal fuzzy features of the foot gesture to be identified are input into a pre-trained foot gesture recognition model to obtain the category of the foot gesture to be identified. Extracting multi-temporal fuzzy features of the footwork to be identified from the radar echo signal specifically includes: The range-Doppler energy spectrum of the foot gesture to be identified is obtained based on the radar echo signal of the foot gesture to be identified. The weighted average value of the unit distance energy is determined based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit. The weighted average value of the unit Doppler energy is determined based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot movement to be identified and the Doppler index of each spectral unit. Select each spectral unit whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot gesture to be identified, and use them as threshold-crossing units, and determine the number of threshold-crossing units. Angle measurements are performed on each threshold unit to obtain the horizontal and pitch angles of each threshold unit; The horizontal angle energy weighted average value is determined based on the horizontal angle and energy of each threshold unit, and the pitch angle energy weighted average value is determined based on the pitch angle and energy of each threshold unit. The unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average are arranged in time sequence to obtain the time sequence of unit distance energy weighted average, the time sequence of unit Doppler energy weighted average, the time sequence of the number of threshold-crossing units, the time sequence of horizontal angle energy weighted average, and the time sequence of pitch angle energy weighted average. Each time sequence is used as a multi-temporal fuzzy feature of the foot gesture to be identified. The horizontal angle energy weighted average is determined based on the horizontal angle and energy of each threshold-crossing unit, and the pitch angle energy weighted average is determined based on the pitch angle and energy of each threshold-crossing unit. Specifically, this includes: The horizontal angle, pitch angle and energy of each threshold unit are used as three-dimensional data to perform three-dimensional density clustering, resulting in multiple threshold unit clusters. Select the cluster containing the most threshold cells from multiple threshold cell clusters as the largest cluster; For each threshold cell in the largest cluster, the horizontal angle of the threshold cell is multiplied by its energy to obtain the horizontal angle energy of the threshold cell; the pitch angle of the threshold cell is multiplied by its energy to obtain the pitch angle energy of the threshold cell. The horizontal angle energy of each threshold-crossing unit in the largest cluster is added together to obtain the total horizontal angle energy, and the pitch angle energy of each threshold-crossing unit in the largest cluster is added together to obtain the total pitch angle energy. The total energy of the maximum cluster is obtained by adding the energies of each threshold unit in the maximum cluster. Divide the total horizontal angle energy by the total energy of the largest cluster to obtain the weighted average horizontal angle energy, and divide the total pitch angle energy by the total energy of the largest cluster to obtain the weighted average pitch angle energy.

2. The foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to claim 1, characterized in that, The range Doppler energy spectrum of the foot gesture is obtained based on the radar echo signal of the foot gesture to be identified, specifically including: Before performing the foot gesture to be identified, multiple frames of data within a preset time after the radar is powered on are acquired as multiple environmental frames. The range-Doppler energy spectrum of each environmental frame is obtained, and the average range-Doppler energy spectrum of all environmental frames is obtained by averaging the range-Doppler energy spectra of all environmental frames. Based on the radar installation height and radar range resolution, several range cells representing the distance to the vehicle underside are determined in the average range Doppler energy spectrum of the environmental frame, which serve as the environmental noise cells under the vehicle. The average energy of all spectral units on each environmental noise unit is obtained as the environmental noise floor value; The original range-Doppler energy spectrum of the foot gesture to be identified is obtained based on the radar echo signal of the foot gesture to be identified. The energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified is subtracted from the ambient noise floor value to obtain the range-Doppler energy spectrum of the foot gesture to be identified.

3. The foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to claim 1, characterized in that, The weighted average value of the unit distance energy is determined based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit. Specifically, this includes: For each spectral cell of the distance-Doppler energy spectrum of the foot gesture to be identified, the distance index of the spectral cell is multiplied by the energy of the spectral cell to obtain the distance energy of the spectral cell; The total distance energy is obtained by summing the distance energies of all spectral units. Add up the energies of all spectral units to get the total energy; Divide the total distance energy by the total energy to obtain the weighted average value of the unit distance energy.

4. The foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to claim 1, characterized in that, The weighted average value of the unit Doppler energy is determined based on the energy of each spectral unit in the distance Doppler energy spectrum of the foot movement to be identified and the Doppler index of each spectral unit. Specifically, this includes: For each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, the Doppler index of the spectral unit is multiplied by the energy of the spectral unit to obtain the Doppler energy of the spectral unit; Add up the Doppler energies of all spectral units to obtain the total Doppler energy; Add up the energies of all spectral units to get the total energy; Divide the total Doppler energy by the total energy to obtain the weighted average value of the unit Doppler energy.

5. The foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to claim 1, characterized in that, The unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average are arranged in time sequence, specifically including: Determine the actual duration of the footwork motion to be identified; The length of the radar frame window to be captured is determined based on the actual duration of the foot gesture to be identified. Using the radar frame sequence as the time axis, the range-weighted average values ​​of the unit range energy of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the unit range energy weighted average value. The Doppler energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the unit Doppler energy weighted average value. The number of threshold-crossing units of each radar frame within the window length to be captured is arranged in time sequence to obtain the time sequence of the number of threshold-crossing units. The horizontal angle energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the horizontal angle energy weighted average value. The elevation angle energy weighted average values ​​of each radar frame within the window length to be captured are arranged in time sequence to obtain the time sequence of the elevation angle energy weighted average value.

6. The foot gesture recognition method based on multi-temporal ambiguity features of millimeter-wave radar according to claim 1, characterized in that, The foot gesture recognition model includes a feature batch normalization layer, a multi-feature fusion mapping layer, a multi-scale feature extraction layer, a temporal feature convolutional layer, and an action mapping layer. The feature batch normalization layer normalizes different temporal sequences to the same dimension; the multi-scale feature extraction layer fuses and remaps different temporal sequences together; the multi-scale feature extraction layer extracts features from temporal sequences of different lengths; the temporal feature convolutional layer includes two cascaded gated recurrent units; and the action mapping layer includes a fully connected layer.

7. A foot gesture recognition system based on multi-temporal ambiguity features of millimeter-wave radar, characterized in that, include: The first acquisition module is configured to acquire radar echo signals of the foot movements to be identified; The extraction module is configured to extract multi-temporal fuzzy features of the foot gesture to be identified from the radar echo signal of the foot gesture to be identified; The recognition module is configured to input the multi-temporal fuzzy features of the foot gesture to be recognized into a pre-trained foot gesture recognition model to obtain the category of the foot gesture to be recognized. The second acquisition module is configured to acquire the range Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified. The first determining module is configured to determine the weighted average value of the unit distance energy based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot posture to be identified and the distance index of each spectral unit; The second determining module is configured to determine the weighted average value of the Doppler energy of the unit based on the energy of each spectral unit of the distance Doppler energy spectrum of the foot movement to be identified and the Doppler index of each spectral unit. The selection module is configured to select each spectral unit whose energy exceeds a preset threshold from the distance Doppler energy spectrum of the foot gesture to be identified, as each threshold unit, and determine the number of threshold units. The angle measurement module is set to measure the angles of each threshold unit to obtain the horizontal and pitch angles of each threshold unit. The third determining module is configured to determine the horizontal angle energy weighted average value based on the horizontal angle and energy of each threshold unit, and to determine the pitch angle energy weighted average value based on the pitch angle and energy of each threshold unit. The temporal arrangement module is configured to arrange the unit distance energy weighted average, the unit Doppler energy weighted average, the number of threshold-crossing units, the horizontal angle energy weighted average, and the pitch angle energy weighted average in time sequence to obtain the temporal sequence of the unit distance energy weighted average, the temporal sequence of the unit Doppler energy weighted average, the temporal sequence of the number of threshold-crossing units, the temporal sequence of the horizontal angle energy weighted average, and the temporal sequence of the pitch angle energy weighted average. Each temporal sequence is used as a multi-temporal fuzzy feature of the foot gesture to be identified. The third determining module specifically includes: The clustering unit is set to use the horizontal angle, pitch angle and energy of each threshold unit as three-dimensional data to perform three-dimensional density clustering, resulting in multiple threshold unit clusters; The selection cell is set to choose the maximum cluster from multiple threshold cell clusters, which contains the largest number of threshold cells. The third multiplication unit is configured to, for each threshold unit in the largest cluster, multiply the horizontal angle of the threshold unit by its energy to obtain the horizontal angle energy of the threshold unit; and multiply the pitch angle of the threshold unit by its energy to obtain the pitch angle energy of the threshold unit. The fifth summing unit is configured to sum the horizontal angle energy of each threshold unit in the largest cluster to obtain the total horizontal angle energy, and to sum the pitch angle energy of each threshold unit in the largest cluster to obtain the total pitch angle energy. The sixth summing unit is configured to sum the energies of each threshold unit in the maximum cluster to obtain the total energy of the maximum cluster; The third division unit is configured to divide the total horizontal angle energy by the total energy of the largest cluster to obtain a weighted average horizontal angle energy, and to divide the total pitch angle energy by the total energy of the largest cluster to obtain a weighted average pitch angle energy.

8. The foot gesture recognition system based on multi-temporal fuzzy features of millimeter-wave radar according to claim 7, characterized in that, The second acquisition module specifically includes: The first acquisition unit is configured to acquire multiple frames of data within a preset time after the radar is powered on, before performing the foot gesture to be identified, as multiple environmental frames. The second acquisition unit is configured to acquire the range-Doppler energy spectrum of each environmental frame, and average the range-Doppler energy spectrum of all environmental frames to obtain the average range-Doppler energy spectrum of the environmental frames. The first determining unit is set to determine several distance units representing the distance to the vehicle bottom in the average range Doppler energy spectrum of the environmental frame based on the radar installation height and the radar range resolution, and serve as the environmental noise units under the vehicle. The third acquisition unit is configured to acquire the average energy of all spectral units on each environmental noise unit as the environmental noise baseline value; The subtraction unit is configured to obtain the original range-Doppler energy spectrum of the foot gesture to be identified based on the radar echo signal of the foot gesture to be identified, and subtract the ambient noise floor value from the energy of each spectral unit on the original range-Doppler energy spectrum of the foot gesture to be identified to obtain the range-Doppler energy spectrum of the foot gesture to be identified. The first determining module specifically includes: The first multiplication unit is configured to multiply the distance index of the spectral unit with the energy of the spectral unit for each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, so as to obtain the distance energy of the spectral unit. The first summing unit is set to sum the distance energies of all spectral units to obtain the total distance energy; The second summing unit is set to add the energies of all spectral units to obtain the total energy; The first division unit is configured to divide the total distance energy by the total energy to obtain a weighted average of the unit distance energy. The second determining module specifically includes: The second multiplication unit is configured to multiply the Doppler index of the spectral unit with the energy of the spectral unit for each spectral unit of the distance Doppler energy spectrum of the foot gesture to be identified, so as to obtain the Doppler energy of the spectral unit. The third summing unit is set to add the Doppler energies of all spectral units to obtain the total Doppler energy; The fourth summation unit is set to add the energies of all spectral units to obtain the total energy; The second phase division unit is configured to divide the total Doppler energy by the total energy to obtain a weighted average value of the unit Doppler energy. The timing arrangement module specifically includes: The second determining unit is configured to determine the actual duration of the foot movement to be identified; The third determining unit is configured to determine the length of the radar frame window to be captured based on the actual duration of the foot gesture to be identified. The arrangement unit is set with the radar frame sequence as the time axis. The unit range-energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of unit range-energy weighted average. The unit Doppler energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of unit Doppler energy weighted average. The number of threshold-crossing units of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of the number of threshold-crossing units. The horizontal angle energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of horizontal angle energy weighted average. The elevation angle energy weighted average of each radar frame within the window length to be captured is arranged in time sequence to obtain a time sequence of elevation angle energy weighted average. Each time sequence is used as a sample. The foot gesture recognition model includes a feature batch normalization layer, a multi-feature fusion mapping layer, a multi-scale feature extraction layer, a temporal feature convolutional layer, and an action mapping layer. The feature batch normalization layer normalizes different temporal sequences to the same dimension; the multi-scale feature extraction layer fuses and remaps different temporal sequences together; the multi-scale feature extraction layer extracts temporal sequences of different lengths; the temporal feature convolutional layer includes two cascaded gated recurrent units; and the action mapping layer includes a fully connected layer.

Images