Gesture and identity combined recognition method based on radar multi-dimensional features

Through the gesture and identity recognition method of radar multi-dimensional features, the problems of senseless detection and privacy leakage in the existing technology are solved, and gestures and identity recognition are achieved with high precision and stability, which are suitable for smart homes and practical applications.

CN118942155APending Publication Date: 2024-11-12FUZHOU UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411006061.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art has problems such as difficulty in sensorless detection, privacy leakage and poor stability in gestures and identity recognition. Especially in the applications of portable sensors, cameras and WiFi devices, it is difficult to achieve high-precision and long-term detection of wireless perception. .

Method used

Using the joint gesture and identity recognition method of radar multi-dimensional features, the gesture action data of the device user is collected through the radar, the distance-time matrix, Doppler-time matrix and angle-time matrix are generated, the feature map enhancement algorithm is applied and the three inputs are designed. - The two output networks perform deep feature mapping, outputting gestures and identity recognition results.

Benefits of technology

It realizes high-precision gestures and identity recognition without the need for users to carry additional devices, has good privacy protection, is not affected by occlusion, and is highly stable, suitable for smart homes and practical application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942155A_ABST
    Figure CN118942155A_ABST
Patent Text Reader

Abstract

The invention relates to a gesture and identity combined recognition method based on radar multi-dimensional features, and belongs to the technical field of Internet of Things application and wireless detection. The method comprises the following steps: firstly, acquiring original data of corresponding gesture actions of an equipment user through a radar, and generating a distance-time matrix, a Doppler-time matrix and an angle-time matrix corresponding to hand actions through a series of preprocessing operations; carrying out information enhancement processing on the distance-time matrix and the angle-time matrix of which the feature information is not obvious enough by using a feature map enhancement algorithm; and then designing a three-input two-output network structure for receiving three feature maps of a distance-time map RTM, a Doppler-time map DTM and an angle-time map ATM and performing deep feature mapping, and finally respectively outputting gesture and identity recognition results through two classification heads. According to the method, the instruction affiliation and the identity affiliation of the gesture action of the user in the equipment control process can be effectively recognized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things applications and wireless detection technology, and in particular relates to a gesture and identity joint recognition method based on radar multi-dimensional features. Background Art

[0002] Radar technology is a technology that uses electromagnetic waves to detect information such as the spatial position of unknown targets. Radar transmits a signal waveform through a transmitting antenna. The signal wave will be reflected after contacting the target, so that the reflected signal of the target object is received by the reflecting antenna to capture the original signal containing information such as the spatial position, speed, angle, etc. of the target. Radar has the characteristics of high frequency, short wavelength, high precision, and is not affected by light. It can be used day and night, and has strong anti-interference to bad weather (rain, fog, snow) and smoke. It is widely used in smart home, vehicle control, human perception and other fields.

[0003] Gesture recognition is a type of human behavior perception technology. Gesture recognition technology is to use specific equipment or means to determine the hand movements of the human body. Among them, active recognition technology based on portable sensors has the advantages of being resistant to environmental interference, low cost, and sensitive to changes in user movements. Most of these technologies use wearable devices or smartphones, both of which require the person being measured to carry a designated sensor to obtain their corresponding hand movement information. Compared with solutions based on portable sensors, passive perception methods based on non-portable devices are more convenient to use and have more common application scenarios. Therefore, passive recognition technology based on non-portable devices has a wider application prospect. For example, camera-based gesture recognition technology uses a camera to capture an image sequence of hand movements and pre-processes it, including a series of operations such as background elimination, image filtering, and hand area detection, and then uses machine learning or deep learning methods to extract features and classify them. In addition, some wireless sensing devices are also used in gesture recognition tasks, such as WiFi, infrared sensors, radar systems, etc.

[0004] For identity recognition tasks, video data or image information captured by a camera is usually used to recognize faces to achieve identity authentication.

[0005] The disadvantages of the prior art are as follows:

[0006] Disadvantage 1: Although wearable sensors can accurately capture hand movement trajectories to identify specific gestures, they cannot achieve seamless detection and recognition, which will cause discomfort to users and does not meet the comfort requirements of smart homes. In addition, most of these devices require batteries to be installed and cannot achieve long-term detection.

[0007] Disadvantage 2: Although camera-based wireless sensing technology has been developed to a very mature stage, it is easy to cause privacy leakage, which is contrary to the concept of privacy that people attach importance to today. It is also unable to identify obscured targets and cannot fully play its role in indoor environments. Although WiFi and infrared sensors avoid the above problems, they are easily affected by changes in WiFi signal strength and ambient temperature, and their stability cannot be guaranteed, making it difficult to meet the requirements of commercialization and practical applications.

[0008] Disadvantage 3: In actual human-computer interaction scenarios, the present invention considers a solution for the joint implementation of device control and identity confirmation. Although facial images or videos captured by the camera can provide identity information, they are not suitable as body parts for sending instructions when controlling devices. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and, in combination with the advantages of radar such as non-contact, not easily affected by lighting conditions, and high precision, provide a method for joint recognition of gestures and identities based on multi-dimensional features of radar, so as to solve the problems existing in the joint recognition of gestures and identities by wireless sensing devices such as portable sensors, cameras, and WiFi.

[0010] To achieve the above object, the technical solution of the present invention is: a gesture and identity joint recognition method based on radar multi-dimensional features, comprising:

[0011] The radar raw data corresponding to the gesture action of the device user is collected through the radar, and the distance-time matrix, Doppler-time matrix and angle-time matrix corresponding to the hand action are generated through a series of preprocessing operations;

[0012] Apply feature map enhancement algorithm to enhance the information of distance-time matrix and angle-time matrix;

[0013] A three-input and two-output network is designed for the three feature maps generated by the three matrices, and deep feature mapping is performed to output the recognition results of gesture and identity respectively through two classification heads.

[0014] In one embodiment of the present invention, the method comprises the following steps:

[0015] Step 1: Collect the radar raw data about the user's gesture and transmit the data to the PC host computer for ADC sampling and preprocessing operations;

[0016] Step 2: Filter the background clutter of the data, perform FFT on the distance dimension, velocity dimension, and angle dimension respectively, and project them to the corresponding dimensions to generate the corresponding distance-time matrix, Doppler-time matrix, and angle-time matrix, and visualize them as corresponding feature maps, namely, the distance-time map RTM, the Doppler-time map DTM, and the angle-time map ATM;

[0017] Step 3, performing information enhancement processing on the distance-time matrix and the angle-time matrix;

[0018] Step 4: Design a three-input-two-output network for multi-stream information input and multi-task recognition;

[0019] Step 5: Use a three-input and two-output network to perform multiple iterations of training on the feature map to learn effective information for classification, and finally output the recognition results of gesture and identity respectively.

[0020] In one embodiment of the present invention, step 1 is specifically implemented as follows:

[0021] FMCW radar uses linear frequency modulated continuous wave as the transmission signal, that is, the frequency of the modulated signal changes linearly with time. The transmission signal is expressed as:

[0022]

[0023] Among them, A t is the transmitted signal amplitude, is the initial phase, f0 is the initial frequency when the signal is transmitted, B is the sweep bandwidth, T c Represents the sweep period of a chirp signal;

[0024] After the signal emitted by the radar reaches the human hand, it forms an echo signal and is transmitted back to the receiving antenna. Assuming that this process takes time τ, the echo signal obtained by reflection from the hand is expressed as:

[0025]

[0026] Among them, A r is the received signal amplitude;

[0027] Then, the transmission signal is mixed with the echo signal to obtain an intermediate frequency signal, which is expressed as:

[0028]

[0029] Among them, A if is the intermediate frequency signal amplitude, j is the imaginary unit;

[0030] The ADC samples the intermediate frequency signal to obtain the corresponding digital signal, and reorganizes the digital signal into a three-dimensional data matrix in the format of Samples_num×Chirps_num×Antennas_num according to the order of the transmitting and receiving antennas, that is, the original radarcube, called R.

[0031] In one embodiment of the present invention, in step 2, background clutter is filtered out of the data, and FFT is performed on the distance dimension, velocity dimension, and angle dimension respectively, and projected to the corresponding dimensions to generate the corresponding distance-time matrix, Doppler-time matrix, and angle-time matrix. The specific implementation method is as follows:

[0032] The clutter in R is filtered out using the average cancellation algorithm, and a three-dimensional matrix is ​​obtained, called R1. RTM, DTM, and ATM are used to present the changes of distance, speed, and angle of arrival over time during hand movement. The process of constructing the original feature matrix from R1 is as follows:

[0033] Add a Blackman window to the distance dimension of R1 to reduce spectral leakage, and perform N r The FFT of the point is used to obtain the distance information of the hand target. The whole process is expressed as:

[0034]

[0035] R r =FFT range (R′,N r )

[0036] Among them, w B () represents the Blackman window, R r Represents the target matrix after 1D-FFT;

[0037] The target matrix R r Take the average in the antenna dimension and project it to the speed dimension to get the distance information in one frame. Superimpose each frame of data along the time dimension to get the distance-time matrix I represents the frame number;

[0038] For the target matrix R r Add a Hamming window on the velocity dimension to suppress side lobes, and perform N d The FFT of the point is used to obtain the velocity information of the hand target. The whole process is expressed as:

[0039]

[0040] R r,d =FFT doppler (R″,N d )

[0041] Among them, w H() represents the Hamming window, R r,d Represents the target matrix after 2D-FFT;

[0042] R r,d The Doppler-time matrix is ​​obtained by averaging the speed information in the antenna dimension and projecting it to the distance dimension to obtain the speed information in one frame.

[0043] For the target matrix R r,d Perform N on the angle dimension a FFT of the points to get the angle information of the hand movement:

[0044] R r,d,a =FFT angle (R r,d ,N a )

[0045] Among them, R r,d,a Represents the target matrix after 3D-FFT; R r,d,a The average is calculated in the distance dimension, and the angle information in a frame is obtained by projecting it to the Doppler dimension. Each frame of data is superimposed along the time dimension to obtain the angle-time matrix.

[0046] In one embodiment of the present invention, in step 3, an RTM information enhancement method and an ATM information enhancement method are used to perform information enhancement processing on the distance-time graph RTM and the angle-time graph ATM respectively.

[0047] In one embodiment of the present invention, the specific implementation method of using the RTM information enhancement method to perform information enhancement processing on the distance-time matrix is ​​as follows:

[0048] Design a target information extractor and use the 2DCFAR algorithm on the distance-time matrix to find all possible hand target points in it. Take the average distance value corresponding to all detected target points to get d ges , d ges Represents the distance value corresponding to the center point of the entire swing range of the hand relative to the radar direction during the swing process, so the hand target points are basically distributed in d ges Nearby; The key step of the target information extractor is to introduce the target extraction window, d ges will be set to the distance value corresponding to the line in the window, d s It is half of the maximum hand movement distance of all experimental subjects relative to the radar direction when making gestures, increasing or decreasing d up and down along the midline s Thus, a window is formed, and an extractor capable of automatically locating the hand target information is designed;

[0049] Perform highlighting operation on the target information and obtain the new data matrix through the extractor Using the spline interpolation technique, it is enlarged to the same size as the original distance-time matrix to obtain Then, M r "Perform Gaussian filtering and frame-by-frame normalization to obtain the final target matrix

[0050] In one embodiment of the present invention, the specific implementation of using the ATM information enhancement method to perform information enhancement processing on the angle-time matrix is ​​as follows:

[0051] In the original ATM, the target information points of the hand fluctuate around a certain angle value, which is regarded as the central angle of the entire range of motion when performing the gesture; find the corresponding angle value α of this central angle in the figure ges First, search for the maximum amplitude point in each frame and average their corresponding angles to get α ges Next, the target information extractor is used to extract the hand target information. The target information extractor contains a target information extraction window. ges Set it as the angle value corresponding to the middle line of the window, design the adaptive target information extraction window of corresponding size for different positions, and calculate the closest distance d between the hand and the radar during the movement process. ges Considered as the position of the hand, combined with the unilateral movement distance d of the hand horizontally to the left along the center angle h , use the arc length formula to estimate the maximum angle α of the unilateral deviation s :

[0052]

[0053] Increase or decrease α along the midline s Thus, a window is formed, and the adaptive target information extraction window size is obtained in the original angle-time matrix at different distances and arbitrary angle positions; after forming the adaptive target information extraction window that can automatically locate the target information, the intercepted new matrix Use spline interpolation to enlarge to the size of the original angle-time matrix, and then perform Gaussian filtering and frame-by-frame normalization operations in sequence to highlight the target information to obtain the final target matrix

[0054] In one embodiment of the present invention, step 4 is specifically implemented as follows:

[0055] A three-input and two-output network named MS-STC is designed, which contains three input branches, which are used to process the three feature maps of RTM, DTM and ATM respectively. Each branch is designed as a different network module. The processed feature tensor YR ,Y D ,Y A will be multiplied by different weights w1, w2, w3 respectively. The weights are a set of learnable parameters that will be continuously updated during the network training iteration. R ,Y D ,Y A It will be spliced ​​into fusion feature CB in the channel dimension; the whole process is shown in the following formula:

[0056]

[0057] in represents the Hadamard product;

[0058] The CB is subjected to batch normalization and multi-layer perceptron (MLP) operations to reduce the risk of overfitting and better fuse feature parameters. The final recognition is achieved by two classification heads, namely the gesture classification head and the identity classification head, which are composed of different linear layers.

[0059] For the network module processing the RTM branch, SepConvLSTM is used as the feature extraction module of this branch;

[0060] For the network module that processes DTM branches, a sliding window-based Transformer module is selected to process DTM;

[0061] For the network module that processes ATM branches, a CNN module based on the CBAM attention mechanism is adopted.

[0062] The present invention also provides a gesture and identity joint recognition system based on radar multi-dimensional features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0063] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention uses radar as a sensor for recognition tasks, which does not require users to carry additional equipment. Compared with camera-based recognition solutions, it has technical advantages such as good privacy protection, high distance resolution and measurement accuracy, and no fear of occlusion.

[0066] In addition, the method proposed in the present invention comprehensively uses the multi-dimensional features of radar, including distance, speed and angle features, and proposes an innovative feature enhancement method to improve the problem of insufficient information prominence in feature maps. According to the characteristics of different feature maps, the present invention also designs a network structure specifically for multi-feature input to improve the recognition effect, and finally outputs the corresponding gesture and identity recognition results after feature fusion through two classification heads.

[0067] Different gestures map out different features, and these differences can be used to achieve gesture recognition. In addition, according to the differences in gesture habits between different human targets, it can be extended to research such as identity recognition. It can be seen that the research on gesture and identity recognition based on multiple features has important practical significance. At present, there are few related studies on the joint recognition of gesture information and identity information. It is proposed to apply a special neural network to the joint recognition of gestures and identities, thereby solving the problem of the difficulty in achieving joint recognition of user gestures and identity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The figure is a flow chart of the method of the present invention.

[0069] Figure 2 This is a simulation scenario for a gesture and identity joint recognition example according to an example of the present invention.

[0070] Figure 3 This is an example RTM information enhancement method of the present invention.

[0071] Figure 4 This is an example of an ATM information enhancement method of the present invention.

[0072] Figure 5 This is a schematic diagram of the same person performing the same gesture at different locations on the radar in an example of the present invention.

[0073] Figure 6 This is the overall structure of the MS-STC example of the present invention.

[0074] Figure 7 This is the SepConvLSTM network of the present invention.

[0075] Figure 8 This is a Transformer module based on a sliding window in an example of the present invention.

[0076] Fig. 9 This is a CNN module of the CBAM attention mechanism used as an example in the present invention. DETAILED DESCRIPTION

[0077] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0078] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0079] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0080] The present invention provides a method for joint recognition of gestures and identities based on radar multi-dimensional features, comprising:

[0081] The radar raw data corresponding to the gesture action of the device user is collected through the radar, and the distance-time matrix, Doppler-time matrix and angle-time matrix corresponding to the hand action are generated through a series of preprocessing operations;

[0082] Apply feature map enhancement algorithm to enhance the information of distance-time matrix and angle-time matrix;

[0083] A three-input and two-output network is designed for the three feature maps generated by the three matrices, and deep feature mapping is performed to output the recognition results of gesture and identity respectively through two classification heads.

[0084] The following is the specific implementation process of the present invention.

[0085] like Figure 1 As shown, this embodiment provides a gesture and identity joint recognition method based on radar multi-dimensional features, including:

[0086] Step 1: Collect the radar raw data signal about the user's gesture and transfer the data file to the PC host computer for subsequent processing, including ADC sampling and preprocessing operations;

[0087] Step 2: Filter the background clutter of the data signal, perform FFT on the distance dimension, velocity dimension, and angle dimension respectively, and project them to the corresponding dimensions to generate the corresponding distance-time matrix, Doppler-time matrix, and angle-time matrix, and visualize them as the corresponding feature graph;

[0088] Step 3, performing information enhancement processing on the distance-time matrix and the angle-time matrix;

[0089] Step 4: Design a three-input-two-output network for multi-stream information input and multi-task recognition;

[0090] Step 5: Use the network to perform multiple iterations of training on the feature map to learn effective information for classification, and finally output the recognition results of gesture and identity respectively.

[0091] like Figure 2 As shown, it is a simulation scenario for an example of joint recognition of gesture and identity in the present invention.

[0092] 1. Data processing part

[0093] In the method proposed in the present invention, the feature map composed of feature information extracted from the original data collected by the millimeter wave radar will serve as the basis for subsequent network recognition and classification, so its generation process is a key step in the entire solution.

[0094] FMCW (Frequency Modulated Continuous Wave) radar uses linear frequency modulated continuous wave as the transmission signal, that is, the frequency of the modulated signal changes linearly with time. The transmission signal can be expressed as:

[0095]

[0096] Among them, A t is the transmitted signal amplitude, is the initial phase, f0 is the initial frequency when the signal is transmitted, B is the sweep bandwidth, T c Indicates the sweep period of a chirp signal.

[0097] After the signal emitted by the radar reaches the human hand, it forms an echo signal and is transmitted back to the receiving antenna. Assuming that this process takes time τ, the echo signal obtained by reflection from the hand can be expressed as:

[0098]

[0099] Among them, A r is the amplitude of the received signal. Then, the transmitted signal is mixed with the echo signal to obtain the intermediate frequency signal. The intermediate frequency signal can be expressed as:

[0100]

[0101] Among them, A if is the intermediate frequency signal amplitude.

[0102] The intermediate frequency signal contains the information about the distance, speed and angle of the hand movement. In order to obtain this information and visualize it as a feature map, the present invention first performs ADC sampling on the intermediate frequency signal to obtain the corresponding digital signal. However, the storage of the signal is not arranged in the order of equivalent receiving antennas at this time. Therefore, the present invention needs to reorganize the digital signal into a three-dimensional data matrix in the format of Samples_num×Chirps_num×Antennas_num according to the order of the transmitting and receiving antennas, that is, the original radarcube, referred to as R. In a real environment, there are some static clutters generated by reflections from objects such as walls and tables. In order to improve the quality of the generated feature map, the present invention uses an average cancellation algorithm to filter out these clutters. The average cancellation algorithm is similar to the moving target indication (MTI) technology in traditional radar signal processing. It can maintain a high signal-to-noise ratio without weakening the target amplitude. The present invention refers to the three-dimensional matrix after clutter elimination as R1.

[0103] The purpose of generating radar feature maps is to visualize the motion information of hand targets in different domains. The present invention uses RTM, DTM and ATM to present the changes of distance, speed and arrival angle over time when the hand moves. The process of constructing the original feature matrix from R1 is as follows.

[0104] First, the present invention adds a Blackman window on the distance dimension of R1 to reduce spectrum leakage, and performs N r The FFT of the point is used to obtain the distance information of the hand target. The above process can be expressed as:

[0105]

[0106] R r =FFT range (R′,N r )

[0107] Among them, w B () represents the Blackman window, R r represents the target matrix after 1D-FFT. Then, the present invention converts the target matrix R r Take the average in the antenna dimension and project it to the speed dimension to get the distance information in one frame. Finally, superimpose each frame of data along the time dimension to get the distance-time matrix Wherein, I represents the frame number.

[0108] The present invention targets the matrix R r Add a Hamming window on the velocity dimension to suppress side lobes, and perform N d The FFT of the point is used to obtain the velocity information of the hand target. The above process can be expressed as:

[0109]

[0110] R r,d =FFT doppler (R″,N d )

[0111] Among them, w H () represents the Hamming window, R r,d represents the target matrix after 2D-FFT. Then, the present invention converts R r,d The average is taken in the antenna dimension and projected to the distance dimension to obtain the velocity information in one frame. Finally, the Doppler-time matrix is ​​obtained by accumulating the velocity information in each frame frame by frame.

[0112] The present invention targets the matrix R r,d Perform N on the angle dimension a The FFT of the points is used to obtain the angle information of the hand movement:

[0113] R r,d,a =FFT angle (R r,d ,N a )

[0114] Among them, R r,d,a represents the target matrix after 3D-FFT. Then, the present invention converts R r,d,a The average is calculated in the distance dimension and projected to the Doppler dimension to obtain the angle information in one frame. Finally, each frame of data is superimposed along the time dimension to obtain the angle-time matrix

[0115] Figure 3 The RTM information enhancement method proposed in the present invention is described. Generally, when a person makes a gesture facing a radar, the distance between the hand and the radar is closer than the body trunk. Figure 3 It can also be observed that the amplitude of the hand target signal is more prominent than the micro-motion clutter of the body trunk. Therefore, it is feasible to locate and extract the hand target information from the distance-time matrix. In order to extract the effective information generated by the hand target from the original distance-time matrix, the present invention designs a target information extractor, such as Figure 3 As shown. First, the present invention uses the 2DCFAR algorithm on the original distance-time matrix to find all possible hand target points in it. The CFAR technology can dynamically adjust the detection threshold value while maintaining a certain false alarm rate, so as to accurately detect the target signal from the background clutter. Next, the present invention averages the distance values ​​corresponding to all detected target points to obtain d ges , d ges Represents the distance value corresponding to the center point of the entire swing range of the hand relative to the radar direction during the swing process, so the hand target points are basically distributed in dges The key step of the target information extractor is to introduce the target extraction window, d ges will be set to the distance value corresponding to the line in the window, d s It is half of the maximum hand movement distance of all experimental subjects relative to the radar direction when making gestures, increasing or decreasing d up and down along the midline s Thus forming a window. After the above steps, an extractor that can automatically locate the hand target information is designed. Figure 3 It can also be seen that although the target information can be intercepted, it is not very obvious. Therefore, the present invention will then highlight the target information, such as Figure 3 First, the present invention obtains the new data matrix through the extractor Using the spline interpolation technique, it is enlarged to the same size as the original distance-time matrix to obtain Then, M r "Perform Gaussian filtering and frame-by-frame normalization to obtain the final target matrix Gaussian filtering can blur some outlier clutter points in the matrix and remove some noise, while frame-by-frame normalization can ensure that each frame of data is within the same numerical range and will not be affected by points with excessively large or small amplitudes in a certain frame that affect the overall generated image.

[0116] The specific process of the ATM information enhancement method is as follows Figure 4 As shown. Since the problems existing in the original ATM and the original RTM are different, the present invention adopts different processing steps in the design of the target information extractor. Figure 4 As shown in , in the original ATM, the signal-to-noise ratio of the hand target is low, resulting in its feature information being less prominent. In addition, as the distance between the hand and the radar increases, the angular offset range of the same gesture target in the original ATM also decreases. This is because, Figure 5 As shown in the figure, when the same person is at two locations A and B at different distances from the radar and makes the same gesture, although the horizontal swing distance of the hand is x A =x B , but the horizontal angle θ corresponding to point B B Less than theta corresponding to point A A Therefore, the present invention can find that as the distance between the person and the radar increases, the horizontal angle variation range of the hand swing detected by the radar will decrease accordingly. In view of the above problem, the present invention adopts another solution to perform information enhancement processing on the original angle-time matrix.

[0117] In the original ATM, the target information points of the hand fluctuate around a certain angle value. The present invention regards this angle value as the center angle of the entire action range when performing the gesture. In order to find the corresponding angle value α of this center angle in the figureges The present invention first searches for the maximum amplitude point in each frame and averages their corresponding angles to obtain α ges Next, the present invention uses a target information extractor to extract the hand target information, and the extractor still includes a target information extraction window, such as Figure 4 As shown, the present invention will ges However, as the distance increases, the fluctuation range of the same gesture in the ATM becomes smaller and smaller. In order to accurately select all the information of the gesture target and exclude unnecessary clutter, the present invention will design an adaptive window of corresponding size for different positions. The present invention will use the closest distance d to the radar during the hand movement process obtained previously. ges Considered as the position of the hand, combined with the unilateral movement distance d of the hand horizontally to the left along the center angle h , use the arc length formula to estimate the maximum angle α of the unilateral deviation s :

[0118]

[0119] Next, the present invention increases and decreases α up and down along the midline. s Thus, a window is formed. Through this processing method, the present invention can obtain a suitable window size in the original angle-time matrix at different distances and arbitrary angle positions. After forming an adaptive window that can automatically locate the target information, the subsequent operation is the same as the last processing step in RTM information enhancement. First, the intercepted new matrix Use spline interpolation to enlarge to the size of the original angle-time matrix, and then perform Gaussian filtering and frame-by-frame normalization operations in sequence to highlight the target information to obtain the final target ATM matrix

[0120] 2. Deep learning part

[0121] In order to simultaneously utilize the information in RTM, DTM, and ATM for joint recognition of gestures and identities, the present invention designs a three-input and two-output network structure called MS-STC to cope with the challenges brought by complex gestures and multi-task recognition. The RTM, DTM, and ATM corresponding to the same digital gesture have different characteristics, but in previous multi-stream neural network research in the radar field, each branch is often designed as the same network module, such as a CNN module. This approach fails to fully consider the differences between each feature map. Therefore, in order to fully extract the useful information in the three feature maps, the present invention specifically designs different network branch modules. In addition, in response to the needs of multi-task recognition, the present invention also expands the original single classification head into multiple classification heads to output the processing results of different tasks.

[0122] Figure 6 The overall structure of MS-STC is shown. The network contains three input branches, which are used to process the three feature maps of RTM, DTM and ATM respectively. Each branch is designed as a different network module. The processed feature tensor Y R ,Y D ,Y A will be multiplied by different weights w1, w2, w3 respectively. The weights are a set of learnable parameters that will be continuously updated during the network training iteration. R ,Y D ,Y A It will be spliced ​​into fusion feature CB in the channel dimension. The above process is shown in the following formula:

[0123]

[0124] in Represents Hadamard product. Then, the present invention sequentially performs batch normalization and multi-layer perceptron (MLP) operations on CB to reduce the risk of overfitting and better fuse feature parameters. The final recognition is implemented by two classification heads, namely, gesture classification head and identity classification head, which are composed of different linear layers, and the final number of output channels is determined by the number of gesture actions and the number of experimenters. The specific input branch module will be introduced below.

[0125] The RTM after information enhancement contains a curve showing the relationship between distance and time. In order to capture the temporal change relationship contained in the RTM, the present invention transforms SepConvLSTM as the feature extraction module of this branch, such as Figure 7 As shown. SepConvLSTM is a variant network evolved from LSTM. LSTM is a network specifically used to process and predict time series data, but it is not good at processing spatial information in image data. In order to solve the problem of LSTM's insufficient ability to extract spatial features when processing image data, ConvLSTM was later proposed. By replacing the connection between the input and each gate in LSTM, and the transition between states from dot multiplication to convolution operation, it has the ability to process both time series and spatial features. Compared with the situation where LSTM needs to expand the input image into a one-dimensional vector before input, the input and output of ConvLSTM are both three-dimensional tensors, which better preserves the spatial information of the feature map. However, due to the introduction of the convolution operation, the number of parameters of ConvLSTM will be relatively large. In order to reduce the number of parameters and improve computational efficiency, the present invention replaces the convolution in ConvLSTM with a separation convolution module to form the final SepConvLSTM network.

[0126] DTM can not only accurately capture the speed changes of hand movements, but also contains richer distinguishing features because each person has different gesture habits and the execution trajectories of different gestures are also different. In addition, in the multi-task recognition of the present invention, gesture recognition pays more attention to global features, such as the duration of the entire action, the fluctuation range of hand speed, etc. Identity recognition pays more attention to local features, such as the color depth changes caused by the different amplitudes of each area in the DTM map. In order to take into account both global and local features to cope with the challenges brought by multi-task recognition, the present invention selects a sliding window-based Transformer module to process DTM.

[0127] The sliding window-based Transformer module adopts a hierarchical structure. As the network depth increases, the receptive field also increases. Figure 8 As shown in the figure, the feature map is first divided into multiple small blocks containing local information through the PatchPartition layer and processed into a sequence, which makes it easier for the network to capture the fine-grained features. The serialized features will be processed into deep features containing rich semantic information through four stages in turn. Among them, the LinearEmbedding layer in the first stage will perform a linear transformation on the input sequence and map it to a higher-dimensional representation space, which helps the module to better learn the relationship between the input data. The PatchMerging layer in the remaining stages will downsample the output of the previous layer, thereby reducing the resolution and adjusting the number of channels to form a hierarchical structure. SwinTransformerBlock is the core module of the entire network, which includes two parts. The first part consists of a multi-layer perceptron (MLP), a window-based multi-head attention mechanism (W-MSA) and layer normalization (LN), where both W-MSA and MLP use a residual connection structure. In the second part, the W-MSA in the first part is replaced with a sliding window-based attention mechanism (SW-MSA), while other places remain unchanged. The traditional self-attention module ignores the position information of the patches involved in the calculation, so the relative position information of the patches is introduced into the window of W-MSA and SW-MSA. The calculation of the self-attention module in W-MSA and SW-MSA can be given by the following formula:

[0128] Q=XW Q ,K=XW K ,V=XW V

[0129]

[0130] in, is the vector matrix of input patches, p is the number of patches, and d is the feature dimension. First, X will be multiplied by three parameter matrices To obtain queryQ, keyK and valueV. Then, Q and K will be used to calculate the attention map between patches, and matrix multiplication will be performed based on the attention map and V to achieve feature enhancement. B represents relative position encoding, which indicates the relative position information between patches in a single window. W-MSA divides the image into different windows for self-attention calculation, thereby reducing the complexity of calculation. SW-MSA solves the problem of information non-interaction between windows by extracting features across windows based on W-MSA. The specific operation of SwinTransformerBlock can be expressed as:

[0131]

[0132] After passing through four stages with stacking layers of 2, 2, 6, and 2 respectively, the feature information of each position in the entire image can be shared, increasing the flow of information in the image, thereby improving the local correlation and global perception between features.

[0133] The processed ATM generally contains a curve image that reflects the change of angle over time during hand movement. Therefore, ATM contains a large amount of local change information, such as local fluctuations on the curve, the concavity of the curve, etc. In order to allow the network to pay attention to these local details, the present invention designs a CNN module branch based on the CBAM attention mechanism, such as Fig. 9 As shown in Figure 2, CNN is a mainstream network structure specifically used to extract local features. Embedding the CBAM attention mechanism in it enables the branch module to dynamically adjust the feature representation in the channel and spatial dimensions, thereby enhancing the performance of the input branch.

[0134] like Fig. 9 As shown in the figure, the CNN branch contains five hidden layers, each of which consists of a convolutional layer, a ReLU activation function, and a maximum pooling layer. All convolutional layers use a fixed kernel size of 3×3, and the kernel size of the maximum pooling layer is 2×2. The number of filters in the first hidden layer is 64, and it increases in each consecutive hidden layer. The last layer uses 768 filters to dynamically adjust the number of output channels to facilitate subsequent feature fusion. The basic process of a hidden layer can be expressed as:

[0135] X out =Maxpool(ReLU(Conv(X in )))

[0136] In addition, the present invention adds a CBAM attention mechanism between the first and second hidden layers of the CNN branch to improve the model's attention to effective features. The CBAM attention mechanism consists of two parts: a channel attention module and a spatial attention module, which can adaptively learn the importance weights of each channel and spatial position, thereby guiding the model to focus on useful feature channels and key areas in the image.

[0137] The present invention also provides a gesture and identity joint recognition system based on radar multi-dimensional features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0138] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A gesture and identity joint recognition method based on radar multi-dimensional features, characterized in that: include: The radar raw data corresponding to the gesture action of the device user is collected through the radar, and the distance-time matrix, Doppler-time matrix and angle-time matrix corresponding to the hand action are generated through a series of preprocessing operations; Apply feature map enhancement algorithm to enhance the information of distance-time matrix and angle-time matrix; A three-input and two-output network is designed for the three feature maps generated by the three matrices, and deep feature mapping is performed to output the recognition results of gesture and identity respectively through two classification heads.

2. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 1, characterized in that: The method comprises the following steps: Step 1: Collect the radar raw data about the user's gesture and transmit the data to the PC host computer for ADC sampling and preprocessing operations; Step 2: Filter the background clutter of the data, perform FFT on the distance dimension, velocity dimension, and angle dimension respectively, and project them to the corresponding dimensions to generate the corresponding distance-time matrix, Doppler-time matrix, and angle-time matrix, and visualize them as corresponding feature maps, namely, the distance-time map RTM, the Doppler-time map DTM, and the angle-time map ATM; Step 3, performing information enhancement processing on the distance-time matrix and the angle-time matrix; Step 4: Design a three-input-two-output network for multi-stream information input and multi-task recognition; Step 5: Use a three-input and two-output network to perform multiple iterations of training on the feature map to learn effective information for classification, and finally output the recognition results of gesture and identity respectively.

3. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 2, characterized in that: Step 1 is implemented as follows: FMCW radar uses linear frequency modulated continuous wave as the transmission signal, that is, the frequency of the modulated signal changes linearly with time. The transmission signal is expressed as: Among them, A t is the transmitted signal amplitude, is the initial phase, f0 is the initial frequency when the signal is transmitted, B is the sweep bandwidth, T c Represents the sweep period of a chirp signal; After the signal emitted by the radar reaches the human hand, it forms an echo signal and is transmitted back to the receiving antenna. Assuming that this process takes time τ, the echo signal obtained by reflection from the hand is expressed as: Among them, A r is the received signal amplitude; Then, the transmission signal is mixed with the echo signal to obtain an intermediate frequency signal, which is expressed as: Among them, A if is the intermediate frequency signal amplitude, j is the imaginary unit; The ADC samples the intermediate frequency signal to obtain the corresponding digital signal, and reorganizes the digital signal into a three-dimensional data matrix in the format of Samples_num×Chirps_num×Antennas_num according to the order of the transmitting and receiving antennas, that is, the original radarcube, called R.

4. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 3 is characterized in that: In step 2, background clutter is filtered out of the data, and FFT is performed on the distance dimension, velocity dimension, and angle dimension respectively, and projected to the corresponding dimensions to generate the corresponding distance-time matrix, Doppler-time matrix, and angle-time matrix. The specific implementation method is as follows: The clutter in R is filtered out using the average cancellation algorithm, and a three-dimensional matrix is ​​obtained, called R1. RTM, DTM, and ATM are used to present the changes of distance, speed, and angle of arrival over time during hand movement. The process of constructing the original feature matrix from R1 is as follows: Add a Blackman window to the distance dimension of R1 to reduce spectral leakage, and perform N r The FFT of the point is used to obtain the distance information of the hand target. The whole process is expressed as: R r =FFTT range (R',N r ) Among them, w B ( ) represents the Blackman window, R r Represents the target matrix after 1D-FFT; The target matrix R r Take the average in the antenna dimension and project it to the speed dimension to get the distance information in one frame. Superimpose each frame of data along the time dimension to get the distance-time matrix I represents the frame number; For the target matrix R r Add a Hamming window on the velocity dimension to suppress side lobes, and perform N d The FFT of the point is used to obtain the velocity information of the hand target. The whole process is expressed as: R r,d =FFT doppler (R",N d ) Among them, w H ( ) represents the Hamming window, R r,d Represents the target matrix after 2D-FFT; R r,d The Doppler-time matrix is ​​obtained by averaging the speed information in the antenna dimension and projecting it to the distance dimension to obtain the speed information in one frame. For the target matrix R r,d Perform N on the angle dimension a FFT of the points to get the angle information of the hand movement: R r,d,a =FFT angte (R r,d ,N a ) Among them, R r,d,a Represents the target matrix after 3D-FFT; R r,d,a The average is calculated in the distance dimension, and the angle information in a frame is obtained by projecting it to the Doppler dimension. Each frame of data is superimposed along the time dimension to obtain the angle-time matrix.

5. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 4, characterized in that: In step 3, the RTM information enhancement method and the ATM information enhancement method are used to perform information enhancement processing on the distance-time graph RTM and the angle-time graph ATM respectively.

6. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 5, characterized in that: The specific implementation method of using the RTM information enhancement method to enhance the distance-time matrix is ​​as follows: Design a target information extractor and use the 2D CFAR algorithm on the distance-time matrix to find all possible hand target points in it. Take the average distance value corresponding to all detected target points to get d ges , d ges Represents the distance value corresponding to the center point of the entire swing range of the hand relative to the radar direction during the swing process, so the hand target points are basically distributed in d ges Nearby; The key step of the target information extractor is to introduce the target extraction window, d ges will be set to the distance value corresponding to the line in the window, d s It is half of the maximum hand movement distance of all experimental subjects relative to the radar direction when making gestures, increasing or decreasing d up and down along the midline s Thus, a window is formed, and an extractor capable of automatically locating the hand target information is designed; Perform highlighting operation on the target information and obtain the new data matrix through the extractor Using the spline interpolation technique, it is enlarged to the same size as the original distance-time matrix to obtain Then, M r "Perform Gaussian filtering and frame-by-frame normalization to obtain the final target matrix 7. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 6, characterized in that: The specific implementation method of using the ATM information enhancement method to perform information enhancement processing on the angle-time matrix is ​​as follows: In the original ATM, the target information points of the hand fluctuate around a certain angle value, which is regarded as the central angle of the entire range of motion when performing the gesture; find the corresponding angle value α of this central angle in the figure ges First, search for the maximum amplitude point in each frame and average their corresponding angles to get α ges Next, the target information extractor is used to extract the hand target information. The target information extractor contains a target information extraction window. ges Set it as the angle value corresponding to the middle line of the window, design the adaptive target information extraction window of corresponding size for different positions, and calculate the closest distance d between the hand and the radar during the movement process. ges Considered as the position of the hand, combined with the unilateral movement distance d of the hand horizontally to the left along the center angle h , use the arc length formula to estimate the maximum angle α of the unilateral deviation s : Increase or decrease α along the midline s Thus, a window is formed, and the adaptive target information extraction window size is obtained in the original angle-time matrix at different distances and arbitrary angle positions; after forming the adaptive target information extraction window that can automatically locate the target information, the intercepted new matrix Use spline interpolation to enlarge to the size of the original angle-time matrix, and then perform Gaussian filtering and frame-by-frame normalization operations in sequence to highlight the target information to obtain the final target matrix 8. The method for joint recognition of gesture and identity based on radar multi-dimensional features according to claim 2, characterized in that: Step 4 is implemented as follows: A three-input and two-output network named MS-STC is designed, which contains three input branches, which are used to process the three feature maps of RTM, DTM and ATM respectively. Each branch is designed as a different network module. The processed feature tensor Y R ,Y D ,Y A will be multiplied by different weights w1, w2, w3 respectively. The weights are a set of learnable parameters that will be continuously updated during the network training iteration. R ,Y D ,Y A It will be spliced ​​into fusion feature CB in the channel dimension; the whole process is shown in the following formula: in represents the Hadamard product; The CB is subjected to batch normalization and multi-layer perceptron (MLP) operations to reduce the risk of overfitting and better fuse feature parameters. The final recognition is achieved by two classification heads, namely the gesture classification head and the identity classification head, which are composed of different linear layers. For the network module processing the RTM branch, SepConvLSTM is used as the feature extraction module of this branch; For the network module that processes DTM branches, a sliding window-based Transformer module is selected to process DTM; For the network module that processes ATM branches, a CNN module based on the CBAM attention mechanism is adopted.

9. A gesture and identity joint recognition system based on radar multi-dimensional features, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Citation Information

Cited By

  • Metaaction-based few-sample wireless gesture recognition method

    CN120234618A

  • A Few-Shot Wireless Gesture Recognition Method Based on Meta-Actions

    CN120234618B

  • Identity determination method and device, storage medium and electronic device

    CN121971055A