1t1r-based millimeter wave gesture recognition method and system

Abstract

The application discloses a 1T1R-based millimeter wave gesture recognition method and system, in the 1T1R-based millimeter wave gesture recognition method, radar original data of each frame is processed to obtain a target point, the target point at least reflects distance and speed information; a statistical quantity of the target point information is calculated as a target feature; the target feature is input into a machine learning model for gesture recognition and classification. Thus, gesture recognition under the condition of 1T1R antenna millimeter wave radar hardware is realized, thereby reducing the cost, effectively realizing fast response, and reducing the power consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a 1T1R-based millimeter wave gesture recognition method and system. BACKGROUND

[0002] In the existing gesture recognition, the distance, speed, angle and other information of a target are measured by using a multi-transmitting and multi-receiving antenna radar, and the information change time sequence of the target is obtained by sliding window, the time sequence is taken as the extracted feature, and input into a machine learning model to realize gesture recognition.

[0003] It can be seen that the prior art has specific requirements for hardware, such as high requirements for the number of antennas and hardware computing resources, which is not suitable for the demand of low-cost, fast-response and low-power practical scenarios. SUMMARY

[0004] In view of the above problems, the present application provides a 1T1R-based millimeter wave gesture recognition method and system to reduce the cost and achieve the purpose of fast response and low power consumption.

[0005] According to a first aspect of the present application, a 1T1R-based millimeter wave gesture recognition method is disclosed, comprising:

[0006] Processing the radar raw data of each frame to obtain target points, the target points at least reflecting distance and speed information;

[0007] Calculating the statistical quantity of the target point information as a target feature;

[0008] Inputting the target feature into a machine learning model to classify and recognize gestures.

[0009] Optionally, the step of processing the radar raw data of each frame to obtain target points comprises:

[0010] After two-dimensional Fourier transform is performed on the radar raw data of each frame, a range-Doppler map is obtained, and points in the range-Doppler map with an amplitude greater than a set value, a speed greater than a set value and a maximum value are selected as target points.

[0011] Optionally, if there are multiple target points in a frame, the weighted values of all target points in the frame are calculated, wherein the weight is represented as the information value of a target point multiplied by its corresponding energy value in the range-Doppler map and accumulated with each other.

[0012] Optionally, in the step of calculating the statistical quantity of the target information as a target feature, frames without target points are regarded as invalid frames and are not included in feature calculation, and frames with target point information are regarded as valid frames; the length and position of the valid frames of a gesture can meet the completeness of the detected gesture data.

[0013] Optionally, the target features include: a larger value of a proportion of positive / negative velocity frames to all frames, an absolute value of a mean velocity of all the valid frames, a distance variance of all the valid frames, a variance of velocity of all the valid frames, a mean velocity of a first half of all the valid frames, and a mean velocity of all the valid frames.

[0014] Optionally, in the step of inputting the target features into a machine learning model to perform gesture recognition classification, two stacked support vector machine models are used for classification.

[0015] Optionally, the input features of the larger value of the proportion of positive / negative velocity frames to all frames, the absolute value of the mean velocity of all the valid frames, and the distance variance of all the valid frames are input into a support vector machine, and a push-pull gesture is taken as a positive class to realize push-pull gesture recognition.

[0016] The input features of the variance of velocity of all the valid frames, the mean velocity of a first half of all the valid frames, and the mean velocity of all the valid frames are input into another support vector machine to realize swipe and circle gesture classification.

[0017] According to a second aspect of the present application, a gesture recognition system is provided, comprising:

[0018] a radar module, which is a millimeter wave radar with only a 1T1R antenna;

[0019] a raw data processing module, configured to process each frame of raw data of the radar module to obtain a target point reflecting at least distance and velocity information;

[0020] a target feature extraction module, configured to obtain target features according to information of the target point; and

[0021] a gesture recognition module, configured to perform gesture recognition classification according to the target features.

[0022] According to a third aspect of the present application, an electronic device is provided, comprising a processor, a storage medium, and a bus, the storage medium stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine readable instructions to perform the method of the first aspect.

[0023] According to a fourth aspect of the present application, a storage medium is provided, the storage medium stores a computer program, and the computer program is executed by a processor to perform the method of the first aspect.

[0024] Compared with the prior art, in the 1T1R-based millimeter wave gesture recognition method of the present application, the radar raw data of each frame is processed to obtain target points, which at least reflect distance and speed information; the statistics of the target point information are calculated as target features; and the target features are input into a machine learning model for gesture recognition and classification. Thus, gesture recognition is realized under the condition of 1T1R (1 transmitting and 1 receiving) antenna of the millimeter wave radar hardware, thereby greatly reducing the cost, effectively realizing fast response, and reducing the power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of a 1T1R-based millimeter wave gesture recognition method in an embodiment of the present application.

[0026] Figure 2 is a diagram showing the segmentation of the model on the training set in the case where push and pull are defined as positive classes in an embodiment of the present application;

[0027] Figure 3 is a diagram showing the segmentation of the model on the test set in the case where push and pull are defined as positive classes in an embodiment of the present application;

[0028] Figure 4 is a diagram showing the segmentation of the model on the training set in the case where circle drawing is defined as a positive class among the remaining two classes of gestures in an embodiment of the present application;

[0029] Figure 5 is a diagram showing the segmentation of the model on the test set in the case where circle drawing is defined as a positive class among the remaining two classes of gestures in an embodiment of the present application;

[0030] Figure 6 is a diagram showing the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The 1T1R-based millimeter wave gesture recognition method and method of the present application will be described in more detail below with reference to the accompanying drawings, which show preferred embodiments of the present application, it being understood that those skilled in the art can modify the present application described herein while still achieving the advantageous effects of the present application. Therefore, the following description should be understood as a broad knowledge for those skilled in the art, and not as a limitation on the present application.

[0032] The present application is described in more detail in the following paragraphs with reference to the accompanying drawings. The advantages and features of the present application will be more apparent from the following description and claims. It should be noted that the drawings are very simplified and use non-precise proportions, only to facilitate, clarify the purpose of assisting the description of the embodiments of the present application.

[0033] The inventor studies the prior art and finds that, in order to obtain the angle information of the target, currently, the hardware must support multiple antennas. In addition, the process of calculating the angle information and directly taking the time sequence as the feature in the prior art will bring additional calculation load and power consumption overhead.

[0034] Based on this, the embodiment of the present application provides a new mode, which realizes the recognition of at least three gestures of pushing, drawing and circling under the condition of the 1T1R millimeter wave radar hardware.

[0035] Reference Figure 1 As shown in the drawings, the embodiment of the present application discloses a 1T1R-based millimeter wave gesture recognition method, comprising:

[0036] S1, processing the radar raw data of each frame to obtain a target point, the target point reflecting at least distance and speed information;

[0037] S2, calculating the statistical quantity of the target point information as a target feature; and

[0038] S3, inputting the target feature into a machine learning model to recognize and classify the gesture.

[0039] Based on the above method, the present application can realize the recognition of at least three gestures of pushing, drawing and circling under the condition of the 1T1R millimeter wave radar.

[0040] In one implementation, for S1, the step of processing the radar raw data of each frame to obtain a target point comprises:

[0041] After performing two-dimensional Fourier transform on each frame of received radar raw data, a range Doppler map (RDM) is obtained, and a point with an amplitude greater than a set value, a speed greater than a set value and a maximum value in the range Doppler map is selected as the target point.

[0042] Wherein, the received radar echo signal can be converted into digital data through ADC (Analogue-to-Digital Conversion), and then the data is converted into a range Doppler map through two-dimensional FFT (Fast Fourier Transform).

[0043] In this process, static targets can be removed, and the interference of environmental factors can be excluded.

[0044] Further, if there are multiple target points in a frame, the weighted values of all target points in the frame are calculated, wherein the weight is the product of the information value of a target point and the corresponding energy value in the range-Doppler map and the mutual accumulation. The point obtained after the weighted processing can be considered as the center of gravity of the target point, which reflects the distance, velocity and other information of the target.

[0045] In a specific implementation, for S2, in the step of calculating the statistical quantity of target information as the target feature, the frame without target point is regarded as invalid frame and is not included in the feature calculation, and the frame with target point information is regarded as valid frame; the length and position of the valid frame of a gesture can meet the integrity of the detected gesture data.

[0046] Specifically, the target feature includes: a larger value of a proportion of positive / negative velocity frames to all frames, an absolute value of a mean velocity of all the valid frames, a distance variance of all the valid frames, a variance of the velocity of all the valid frames, a mean velocity of the first half of the velocity of all the valid frames, and a mean velocity of the velocity of all the valid frames.

[0047] In the embodiment of the present application, the velocity close to the radar is positive, and the velocity away from the radar is negative. The larger proportion refers to: positive proportion = number of positive velocity frames / number of valid frames, negative proportion = number of negative velocity frames / number of valid frames, and the larger one of the two is the larger proportion, wherein the number of valid frames = number of positive velocity frames + number of negative velocity frames.

[0048] Based on the above setting, in one embodiment of the present application, in S3, two stacked support vector machines (SVM) models are used for classification respectively.

[0049] Specifically, the following classification mode can be used:

[0050] The input features, the larger value of the proportion of positive / negative velocity frames to all frames, the absolute value of the mean velocity of all the valid frames, and the distance variance of all the valid frames, are input into a support vector machine, the push-pull gesture is taken as a positive class, and the recognition of the push-pull gesture is realized.

[0051] The input features, the variance of the velocity of all the valid frames, the mean velocity of the first half of the velocity of all the valid frames, and the mean velocity of the velocity of all the valid frames, are input into another support vector machine, and the classification of the swipe and circle gestures is realized.

[0052] Reference can be made to Figures 2-3For example, push-pull is defined as the positive class, which is represented as point A in the figure, and the other two classes of gestures are defined as the negative class, which is represented as point B in the figure, and the plane C represents a dividing plane of an SVM. The three-dimensional coordinates represent the features introduced in the embodiment of the application respectively: (a) a larger value of the proportion of positive / negative velocity frames to all frames; (b) an absolute value of the mean of the speeds of all valid frames; and (c) a distance variance of all valid frames. Figure 2 The division of the model on the training set is shown in FIG. 6, wherein the circled part D is a support vector. Figure 3 The division of the model on the test set is shown in FIG. 7, wherein the circled part E is a misclassified sample, and it can be seen that the number of misclassified samples is very small and can be ignored.

[0053] It can be seen that in this implementation, the classification of push-pull and the other two classes of gestures is realized.

[0054] Please refer to Figures 4-5 For the remaining two classes of gestures, circle drawing is defined as the positive class, which is represented as point P in the figure, and swiping is defined as the negative class, which is represented as point Q in the figure, and the plane C represents a dividing plane of another SVM. The three-dimensional coordinates represent the features introduced in the application respectively: (d) a variance of the speeds of all valid frames; (e) a mean of the speeds of the first half of all valid frames; and (f) a mean of the speeds of all valid frames. Figure 4 The division of the model on the training set is shown in FIG. 6, wherein the circled part D is a support vector, Figure 5 The division of the model on the test set is shown in FIG. 7, wherein the circled part E is a misclassified sample, and it can be seen that the number of misclassified samples is very small and can be ignored.

[0055] It can be seen that in this implementation, the classification of circle drawing and swiping is realized.

[0056] Therefore, the application realizes the recognition of three kinds of gestures, i.e., push-pull, swiping and circle drawing.

[0057] Based on Figures 2-5 It can be seen that in the embodiment of the application, two linear SVM models only need 8 parameters, which embodies the super-lightweight characteristic.

[0058] In another embodiment of the application, a gesture recognition system is also provided, which comprises:

[0059] A radar module, which is a millimeter wave radar with only a 1T1R antenna;

[0060] An original data processing module, which is configured to process each frame of original data of the radar module to obtain a target point reflecting at least distance and speed information;

[0061] A target feature extraction module, which is configured to obtain a target feature according to the information of the target point; and

[0062] a gesture recognition module for recognizing and classifying gestures according to the target features.

[0063] Optionally, the raw data processing module can be configured to perform two-dimensional Fourier transform on the received radar raw data of each frame to obtain a range Doppler map (RDM), and select points in the range Doppler map with an amplitude greater than a set value, a speed greater than a set value and a maximum value as target points.

[0064] Optionally, the raw data processing module can be configured to perform two-dimensional Fourier transform on the received radar raw data of each frame to obtain a range Doppler map (RDM), and select points in the range Doppler map with an amplitude greater than a set value, a speed greater than a set value and a maximum value as target points.

[0065] In this process, static targets can be removed, i.e. invalid frames are removed, to reduce the workload.

[0066] Optionally, the gesture raw data processing module can be further configured to: if there are multiple target points in a frame, calculate the weighted values of all target points in the frame, wherein the weight is represented as the information value of a target point multiplied by its corresponding energy value in the range Doppler map and accumulated with each other. The point obtained after the weighted processing can be considered as the center of gravity of the target point, which reflects the distance, speed and other information of the target.

[0067] Optionally, the target feature extraction module can regard frames without target points as invalid frames and not include them in feature calculation, and regard frames with target point information as valid frames; the length and position of valid frames of a gesture can meet the completeness of the detected gesture data.

[0068] Specifically, the target feature extraction module can extract target features including: a relatively large value of the proportion of positive / negative speed frames to all frames, an absolute value of the mean speed of all valid frames, a distance variance of all valid frames, a speed variance of all valid frames, a mean speed of the first half of all valid frames, and a mean speed of all valid frames.

[0069] Optionally, the gesture recognition module uses two stacked support vector machine (SVM) models for classification.

[0070] Specifically, the gesture recognition module can use the following classification mode:

[0071] The input features, a large value of a proportion of the positive / negative velocity frames in all frames, an absolute value of a mean velocity of all the valid frames, and a distance variance of all the valid frames are input into a support vector machine, a push-pull gesture is taken as a positive class, and a push-pull gesture is recognized;

[0072] The input features, a variance of a velocity of all the valid frames, a mean velocity of a first half of all the valid frames, and a mean velocity of all the valid frames are input into another support vector machine, and a swipe and a circle gesture are classified.

[0073] It should be understood that the above-described embodiments are merely illustrative, and the disclosed system and method of the embodiments can also be implemented in other manners. For example, the division of the modules is merely a logical function division, and the actual implementation can be in another manner. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the modules can be indirect coupling or communication connection through some communication interfaces, devices or modules, and can be electrical, mechanical or in other forms. In addition, each function unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in a unit.

[0074] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a user terminal or a driver terminal to execute all or part of the steps of the method described in the various embodiments of the present application.

[0075] That is, those skilled in the art should understand that the embodiments of the present application can be implemented in the form of any one of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0076] Based on this, the embodiments of the present application also provide a program product, which can be a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and the like storage medium. The storage medium can store a computer program, and the computer program is executed by a processor to perform the steps of the 1T1R-based millimeter wave gesture recognition method as described in the foregoing method embodiments. The specific implementation and technical effects are similar, and will not be repeated here.

[0077] Optionally, embodiments of the present invention also provide an electronic device, which may be a server, computer, or other similar device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0078] like Figure 6 As shown, the electronic device may include a processor 601, a storage medium 602, and a bus 603. The storage medium 602 stores machine-readable instructions executable by the processor 601. When the target detection device is running, the processor 601 and the storage medium 602 communicate via the bus 603. The processor 601 executes the machine-readable instructions to perform the steps of the millimeter-wave gesture recognition method based on 1T1R as described in the previous embodiment. The specific implementation and technical effects are similar and will not be repeated here.

[0079] For ease of explanation, only one processor is described in the target detection device described above. However, it should be noted that in some embodiments, the target detection device of the present invention may also include multiple processors, and therefore the steps performed by one processor described in the present invention may also be performed jointly by multiple processors or individually.

[0080] In some embodiments, the processor may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, the processor may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction-set Processor (ASIP), a Graphics Processing Unit (GPU), a Physics Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computing (RISC) computer, or a microprocessor, or any combination thereof.

[0081] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the appended claims are intended to encompass all variations and modifications that fall within the scope of the present application. Various alterations, modifications, and improvements will occur to those skilled in the art upon the reading of the foregoing description. It is intended that all such alterations, modifications, and improvements fall within the scope of the present application as claimed in the appended claims.

Claims

1. A 1T1R-based millimeter wave gesture recognition method, characterized in that, The method comprises: processing radar raw data of each frame to obtain target points, wherein the target points at least reflect distance and speed information, the raw data is collected by a 1T1R antenna, and no angle information is obtained; calculating statistical quantities of the target point information as target features, wherein the target features comprise a maximum value of a proportion of positive / negative speed frames to all frames, an absolute value of a speed mean value of valid frames, a distance variance, a speed variance, a speed mean value of a first half, and a speed mean value; inputting the target features into two stacked support vector machine models to recognize and classify gestures, wherein a first support vector machine model recognizes push-pull gestures, and a second support vector machine model distinguishes between swiping and drawing circle gestures. 2.The 1T1R-based millimeter wave gesture recognition method of claim 1, wherein, The step of processing radar raw data of each frame to obtain target points comprises: obtaining a range-Doppler map after two-dimensional Fourier transform is performed on radar raw data of each frame, and selecting a point in the range-Doppler map as a target point, wherein the point has an amplitude greater than a set value, a speed greater than a set value, and is a maximum value. 3.The 1T1R-based millimeter wave gesture recognition method of claim 2, wherein, If multiple target points exist in a frame, a weighted value of all target points in the frame is calculated, wherein the weight is a product of an information value of a target point and an energy value corresponding to the target point in the range-Doppler map. 4.The 1T1R-based millimeter wave gesture recognition method of claim 1, wherein, In the step of calculating statistical quantities of the target information as target features, frames without target points are regarded as invalid frames and are not included in feature calculation, and frames with target point information are regarded as valid frames; the length and position of valid frames of a gesture can meet the completeness of detected gesture data. 5.The 1T1R-based millimeter wave gesture recognition method of claim 4, wherein, The target features comprise a maximum value of a proportion of positive / negative speed frames to all frames, an absolute value of a speed mean value of all valid frames, a distance variance of all valid frames, a speed variance of all valid frames, a speed mean value of a first half of all valid frames, and a speed mean value of all valid frames.

6. The 1T1R-based millimeter wave gesture recognition method according to claim 5, wherein a support vector machine is inputted with the maximum value of the proportion of positive / negative speed frames to all frames, the absolute value of the speed mean value of all valid frames, and the distance variance of all valid frames, and the push-pull gesture is regarded as a positive class to realize recognition of the push-pull gesture; another support vector machine is inputted with the speed variance of all valid frames, the speed mean value of a first half of all valid frames, and the speed mean value of all valid frames to realize classification of swiping and drawing circle gestures.

7. A gesture recognition system, characterized by The method comprises: a radar module, wherein the radar module is a millimeter wave radar with only a 1T1R antenna; a raw data processing module, configured to process raw data of each frame of the radar module to obtain target points reflecting at least distance and speed information; a target feature extraction module, configured to obtain target features according to information of the target points; and a gesture recognition module, configured to recognize and classify gestures according to the target features.

8. An electronic device, comprising: The method comprises: A processor, a storage medium storing machine readable instructions executable by the processor, and a bus for communication between the processor and the storage medium when the electronic device is running, the processor executing the machine readable instructions to perform the method of any one of claims 1-6.

9. A storage medium, characterized by A computer program stored on the storage medium, the computer program performing the method of any one of claims 1-6 when executed by a processor.

Images