Sign language recognition method based on millimeter wave radar 3D point cloud

Abstract

The invention relates to a sign language action recognition problem, and provides a sign language recognition method based on a millimeter wave radar 3D point cloud, sign language recognition is carried out in a sequence-to-sequence mode through a non-contact FMCW radar, and the sign language recognition method mainly comprises a first part, namely a continuous sign language action recognition method based on a millimeter wave radar; a second part: establishing an evaluation index STT Score of radar continuous sign language action recognition; according to the invention, a low-cost portable wearable acquisition device is adopted, so that a subject can use the device anytime and anywhere; meanwhile, millimeter-wave radar detection does not depend on light rays, the calculated amount is small, the privacy of a user is effectively protected due to the fact that signals collected by the millimeter-wave radar are radar radio frequency signals, and meanwhile the use feeling is good and the cost is low due to the non-contact collection mode of the millimeter-wave radar.

Description

TECHNICAL FIELD

[0001] The present application relates to a sign language recognition method, in particular to a sign language recognition method based on millimeter wave radar 3D point cloud. BACKGROUND

[0002] With the increasing demand for barrier-free communication of the global deaf and mute population and the in-depth application of human-computer interaction concept in intelligent device scenarios, the importance of short-range sign language recognition technology is increasingly prominent. Accurate recognition of continuous sign language gestures not only helps deaf and mute people to communicate seamlessly with normal hearing people, but also provides a more natural instruction input method for various application scenarios (such as public places, homes and office environments), thereby greatly improving the quality of life of disabled people.

[0003] Traditional wearable devices (such as wristbands, glove-type sensors) can collect hand movement signals, but users must wear them for a long time, and there are limitations such as battery life and wearing comfort, making it difficult to use them for a long time in a wide range of scenarios. In contrast, non-contact sensors have gradually become a research hotspot for sign language recognition due to their convenience and ease of use.

[0004] Compared with cameras, millimeter wave radar sensors have significant advantages. They only collect physical parameters such as distance, speed and angle of the hand and arm, do not directly expose user images, do not rely on external lighting conditions, and can penetrate thin barriers to capture subtle motion characteristics of gestures in real time. 3D point cloud can preserve the spatial shape and small motion of each part of the hand, and its ability to capture finger details is significantly better than that of two-dimensional radar spectrum.

[0005] In actual sign language communication, continuous gestures are switched quickly and coherently, and the duration of each gesture and the transition stage has high variability. Therefore, how to use the 3D point cloud data generated by the millimeter wave radar to accurately capture, segment and recognize sentence-level, phrase-level and finger-level continuous sign language in real time has become a key problem in current research. Only a system with the ability to model and classify different gesture passages in time sequence can achieve high robustness and high accuracy of real-time sign language translation and human-computer interaction without relying on camera devices. SUMMARY

[0006] The technical problem solved by the present application is the recognition of sign language actions. A sign language recognition method based on millimeter wave radar 3D point cloud is proposed, which recognizes sign language in a sequence-to-sequence manner using a non-contact FMCW radar. The main content includes two parts:

[0007] The first part: continuous sign language action recognition method based on millimeter wave radar, including: millimeter wave radar signal processing; millimeter wave radar collects continuous sign language action data set; millimeter wave radar collects continuous sign language action data set; the PKTNet network model of fusion space-time double branch attention mechanism is proposed, KNN neighborhood feature aggregation is carried out in local point cloud branch, and the frame static attention and interframe dynamic attention are combined, the sequence prediction result is output by processing continuous point cloud sequence data;

[0008] The millimeter wave radar signal processing includes: millimeter wave radar signal acquisition parameter configuration; millimeter wave radar point cloud signal pretreatment;

[0009] The spatial feature and time long short-term dependence information are extracted from the sequence data of continuous point clouds by processing the spatial feature information and time long short-term dependence information in the point cloud sequence generated by the FMCW radar;

[0010] The output sequence prediction result includes: the sign language action type prediction corresponding to each frame of data, which implies the duration information of the activity and pays special attention to the capture of the transition key frame of the activity;

[0011] The millimeter wave radar point cloud signal pretreatment process performs RangeFFT on the radar data of each frame along the fast time direction to obtain distance information:

[0012]

[0013] Where X(c,s) represents the frame data of the cth Chirp and the sth Sample, N S is the size of Sample, w(l) represents the Hamming window with length l; X r (c,r) represents the distance processing data of the cth Chirp and the rth distance bin, r∈[0,N S );

[0014] After completing the Range FFT, CFAR is used to eliminate static clutter, and then 3D-FFT is used along the angle estimation to obtain point cloud data, and then the three-dimensional coordinates of the point cloud are converted from polar coordinates to rectangular coordinates to obtain the three-dimensional coordinates of each point, and the specific calculation is as follows:

[0015]

[0016] Where: r i is the distance from the radar to the target, θ i is the pitch angle, is the azimuth angle;

[0017] The PKTNet network model integrating the spatiotemporal dual-branch attention mechanism consists of three parts: a local feature extraction module, a temporal feature extraction module, and a classification module.

[0018] The local feature extraction module, whose input consists of five-dimensional point cloud data of T frames. For the radar point cloud in each frame, the local feature extraction module first performs input transformation and feature transformation, and obtains point-by-point features through shared MLP;

[0019] The temporal feature extraction module obtains a T×512 feature sequence from the local feature extraction module. To model the time series in continuous human motion data, a spatiotemporal dual-branch attention mechanism is used to parallelize the temporal evolution. The spatiotemporal dual-branch attention mechanism is decomposed into two parts: intra-frame static attention and inter-frame dynamic attention. The intra-frame static attention branch uses small-kernel deep convolution, dilated deep convolution, and one-dimensional convolution for modeling. By calculating the attention weight for each position, the importance of different spatial regions in feature extraction is automatically adjusted.

[0020] Among them, the intra-frame static attention SA can be calculated by the formula, which is as follows:

[0021] SA=Conv 1×1 (Conv DW-D (Conv DW (H)))#(3)

[0022] In addition, in order to capture the changes in temporal features on the timeline, the inter-frame dynamic attention branch uses squeezing and excitation to learn the attention weights between different frames. First, through squeezing, the features of each frame are reduced in dimension using a global average pooling operation to obtain a global temporal feature vector. Then, the excitation step uses a fully connected layer network to calculate the weight of each time step. The purpose is to dynamically adjust the feature response of each time step so that important time points receive higher attention. The inter-frame dynamic attention DA can be obtained by calculation, and the calculation formula is as follows:

[0023] DA=FC(AvgPool(H))#(4)

[0024] Finally, the two attention branches are integrated, where H is the sequence input to the temporal feature extraction module and H' is the final weighted result. The formula is as follows:

[0025]

[0026] in represents the Kronecker product, ⊙ represents the Hadamard product;

[0027] The classification module directly splices the features learned by the local feature extraction module and the time sequence feature extraction module, and takes the features as the input of the classification module; after obtaining a Tx1024 feature, the feature dimension is reduced to nxC by using a plurality of shared MLPs, C refers to the number of action categories, and finally the frame-by-frame classification probability is output by Softmax, so as to complete the segmentation and recognition of continuous sign language actions;

[0028] Second part: establish the evaluation index STT Score of radar continuous sign language action recognition,

[0029] The evaluation index STT Score of radar continuous sign language action recognition allows a certain tolerance for the error of action recognition in a very short time interval, so as to more accurately reflect the continuity and transition ambiguity of human actions;

[0030] The prediction model is evaluated, and the segmented F1 score is used for further evaluation;

[0031] The segmented F1 score evaluates the performance by calculating the IOU of each predicted action and the real action;

[0032]

[0033] Wherein, P represents the predicted action time interval, and T represents the real action time interval;

[0034] As shown in the following formula, P represents the prediction result, and GT represents the real label; the segmentation method compares the IOU and the threshold K to determine the true positive (TP), false positive (FP) and false negative (FN);

[0035]

[0036] If the segmented prediction is correct, when the IOU is greater than or equal to the threshold K, it is marked as TP, and less than K is FP; if the segmented prediction is wrong, it is marked as FN; in addition, when there are multiple correct predictions within a single real action range, only one will be classified as TP, and the others will be classified as FP;

[0037] Then, the precision and recall of the sum of TP, FP and FN of all classes are calculated, and the segmented F1 score is calculated;

[0038]

[0039] On the basis of F1, the STT Score is proposed for radar continuous sign language action recognition, as shown in formula 10, len pframe length representing the prediction time, when calculating the statistics TP, FP and FN, if the comparison of the prediction result for the motion transition frame with the true value is within 5 frames, it is not included in the calculation of FN;

[0040]

[0041] After CFAR and coordinate conversion are completed, a density clustering algorithm is further used to suppress noise, the density clustering algorithm is a spatial clustering method based on data density, and two important parameters are included: a neighborhood radius eps and a minimum number MinPts of points in a neighborhood radius of a core point, the neighborhood radius refers to a circular or spherical region with a certain point as a center and an eps radius, the algorithm divides data points into three categories: a core point: the neighborhood radius of the point contains at least MinPts points (including itself); a boundary point: located in the eps neighborhood of the core point, but the number of points contained in the neighborhood radius of the boundary point is less than MinPts; a noise point: neither located in the eps neighborhood of the core point nor the number of points contained in the neighborhood radius of the noise point is less than MinPts; specific operation steps are as follows:

[0042] 1) initialization: given a neighborhood radius eps and a minimum number MinPts of points in a neighborhood radius of a core point;

[0043] 2) core point determination: starting from an arbitrary point p, check whether the number of points in the eps neighborhood of the current point is greater than or equal to MinPts; if yes, mark the point as a core point and create a new cluster C for p, if not, mark the point as a noise point, which may be reclassified as a boundary point later;

[0044] 3) cluster expansion: recursively determine each unvisited point q in the neighborhood, if the number of points in the neighborhood of point q is greater than or equal to MinPts, point q is a core point, and the neighborhood of point q is added to cluster C, if the number of points in the neighborhood of point q is less than MinPts, point q is a boundary point;

[0045] 4) termination condition: select the next object from the remaining objects and repeat process 3) until all points are core points, boundary points or noise points, and cluster division is completed.

[0046] The millimeter wave radar collects continuous sign language action data, and a sign language action data set is established, 400 sentences containing 108 sign language words generated by 10 volunteers; the sign language action data set includes: 10 volunteers: 7 males, 3 females, aged 21-25 years old, radar data and synchronous video data of Chinese commonly used sentences containing sign language words in daily life; collect sign language sentence samples, each sign language sentence sample contains 4 different sign language words, and each sign language sentence sample is collected 50 times repeatedly; collect sign language word samples, each sign language word sample contains 4 same sign language words, and each sign language word sample is collected 50 times repeatedly, a total of 400 sentences containing 108 sign language words; the subject naturally sits in front of the table, the distance between the two chest center points is kept at 40-60 cm from the radar chip, and the subject starts to make sign language actions after pressing the collection button.

[0047] In the MLP layer, each MLP extracts high-dimensional features from it through linear transformation, and the dimension is 128; then the KNN algorithm is used for local feature extraction, assuming that a frame of point cloud samples is X=[x1, x2, …, xn]∈Rm, wherein n is the number of point clouds, and m represents the feature dimension of the data, then the K nearest points of each point xi are obtained, a k×128 feature vector is constructed, and multiple shared MLP operations are performed on each query point to enhance the local feature representation of the point cloud space, and then a maximum pooling function is used to reduce the dimension to obtain a 1×512 local feature vector, after repeating n times, n×512 local features of all points are obtained, and then a 1×1 convolution is performed for dimension reduction to obtain a single frame 1×512 spatial feature, finally, a T×512 feature sequence extracted from T frame point cloud data is obtained, and is further input into a time sequence feature extraction module.

[0048] The advantages of the present application are:

[0049] 1. Low-cost portable wearable collection device, which can be used by the subject anytime and anywhere.

[0050] 2. Millimeter wave radar detection does not depend on light and has small calculation amount, since the signal collected by the millimeter wave radar is a radar radio frequency signal, the user privacy is effectively protected, and since the millimeter wave radar is a non-contact collection method, the use is good, and the cost is low. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is the architecture diagram of the sign language action segmentation and detection method using FMCW radar;

[0052] Figure 2 It is the architecture diagram of the continuous sign language action recognition method based on millimeter wave radar;

[0053] Figure 3is the parameter configuration diagram of the cfg file of the FMCW radar;

[0054] Figure 4 is a single-frame original point cloud diagram;

[0055] Figure 5 is a single-frame point cloud diagram after density clustering;

[0056] Figure 6 is a point cloud data storage format diagram;

[0057] Figure 7 is a type diagram of collected information;

[0058] Figure 8 is a network structure diagram of the PKTNet;

[0059] Figure 9 is an architecture diagram of a local feature extraction module;

[0060] Figure 10 is an architecture diagram of a timing feature extraction module;

[0061] Figure 11 is an architecture diagram of a classification module;

[0062] Figure 12 is a segmented evaluation example diagram when K = 0.5;

[0063] Figure 13 is a comparative experiment result diagram;

[0064] Figure 14 is a confusion matrix diagram of PKTNet action recognition. DETAILED DESCRIPTION

[0065] The application will be further described below in conjunction with the accompanying drawings.

[0066] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0067] Glossary:

[0068] Frequency Modulated Continuous Wave: FMCW

[0069] Dual DilatedModule: Dual Dilated Module

[0070] Attention Module: Attention Module

[0071] Predictio Refinement Module: Predictio Refinement Module

[0072] IOU: Intersection over Union

[0073] STT Score: Short Time Difference Tolerant Score

[0074] Acc: Frame-wise Accuracy

[0075] Range FFT: Range Fast Fourier Transform

[0076] Doppler FFT: Range Slow Fourier Transform

[0077] The technical problem solved by the present application is to provide a sign language recognition method based on millimeter wave radar 3D point cloud. First, a PKTNet network model with a fusion of space-time double-branch attention mechanism is designed, a local feature extraction module and a time sequence feature extraction module are designed, a KNN algorithm is added in the original point feature extraction, the importance of local information is further strengthened, and the time sequence features are captured through intra-frame static attention and inter-frame dynamic attention, and a multi-layer perception MLP and a softmax are used in the type classification. Each frame of data is processed in a sequence-to-sequence manner to realize activity segmentation and recognition; the present application also establishes a continuous sign language action dataset, which collects 400 sentences containing 108 sign language words produced by 10 volunteers. Finally, different network models are used to carry out comparative experiments using the dataset, and the PKTNet network is evaluated by using the existing evaluation scheme and the STT method proposed by the present application, which verifies the feasibility of the network in processing continuous sign language actions, and the F1 score and the STT score are 0.7213 and 0.8692, and the inference time and the training time are very excellent.

[0078] Reference Figure 1 The present application mainly includes a sign language recognition method based on millimeter wave radar 3D point cloud and an evaluation index STT Score for millimeter wave radar continuous sign language recognition.

[0079] First part: continuous sign language action recognition method based on millimeter wave radar, reference Figure 2 , including: millimeter wave radar signal processing; millimeter wave radar continuous sign language action dataset collection; millimeter wave radar continuous sign language action dataset collection; improved PKTNet network is proposed, KNN neighborhood feature aggregation is performed in the local point cloud branch, and intra-frame static attention and inter-frame dynamic attention are combined, continuous point cloud sequence data is processed, and sequence prediction result is output.

[0080] Further, the millimeter wave radar signal processing includes: millimeter wave radar signal acquisition parameter configuration; millimeter wave radar point cloud signal preprocessing.

[0081] Further, the extracting spatial features and temporal long / short-term dependency information from the processed sequence data of the continuous point cloud comprises: extracting three-dimensional coordinate, reflection intensity and other spatial feature information in the point cloud sequence generated by the FMCW radar, and temporal long / short-term dependency information such as duration of activity, action transition time, key frame point cloud change information and global space-time features. Among them, the spatial features reflect the position distribution and deformation trend of the sign language action target in the three-dimensional space (for example, the coordinates of each key point of the hand in the point cloud and its motion trajectory over time), and the temporal long / short-term dependency information helps the model to understand the time structure and dynamic evolution of the action, including the gesture detail transition in a short time and the overall sign language action mode in a long time, which together support the accurate recognition of sign language actions.

[0082] Further, the output sequence prediction result comprises: a sign language action type prediction corresponding to each frame of data, which implies the duration information of the activity and pays special attention to the capture of the activity transition key frame. These prediction results are given in the form of a sequence, maintaining the continuity and time sequence of the activity, and achieving high-accuracy prediction through the proposed PKTNet network.

[0083] Specifically, the mmWave radar signal acquisition cfg file parameter reference Figure 3 After connecting the IWR1443 with the computer through the USB, the cfg file is configured according to the MMWAVE SDK user manual provided by the TI company, instead of skipping the TI official host computer mmWave Studio. The cfg file is configured based on the radar bottom layer driver, including parameters such as channel number, frequency, sampling rate and frame period. Its distance resolution is 0.044 meters, radial velocity resolution is 0.24 meters / second, and azimuth resolution is 15 degrees.

[0084] Specifically, the mmWave radar point cloud signal preprocessing process performs Range FFT on each frame of radar data along the fast time direction to obtain distance information:

[0085]

[0086] where X(c,s) represents the frame data of the cth Chirp and the sth Sample, N S is the size of Sample, and w(l) represents a Hamming window with length l. X r (c,r) represents the distance processing data of the cth Chirp and the rth distance bin, r∈[0,N S ).

[0087] After the completion of Range FFT, CFAR is used to eliminate static clutter. Then 3D-FFT is used to obtain point cloud data along the angle estimation. Then the three-dimensional coordinates of the point cloud are converted from polar coordinates to rectangular coordinates to obtain the three-dimensional coordinates of each point, which is calculated as follows:

[0088]

[0089] Where: r i is the distance from the radar to the target, θ i is the pitch angle, is the azimuth angle.

[0090] After the completion of CFAR and coordinate conversion, although the CFAR algorithm removes most of the static background noise, there are still some noise points in the indoor space caused by multipath reflection or sensor errors. Therefore, this paper chooses to use the density clustering algorithm to suppress noise. The density clustering algorithm is a spatial clustering method based on data density, which includes two important parameters: neighborhood radius eps and minimum number of points MinPts in the neighborhood radius of the core point. The neighborhood radius refers to a circular or spherical region with a radius of eps centered on a certain point. The algorithm divides the data points into three categories: core points: the neighborhood radius of the point contains at least MinPts points (including itself); boundary points: located in the eps neighborhood of the core point, but the number of points contained in its own neighborhood radius is less than MinPts; noise points: neither in the eps neighborhood of the core point, nor the number of points contained in its own neighborhood radius is less than MinPts. The specific operation steps are as follows:

[0091] 1) Initialization: Given the neighborhood radius eps and the minimum number of points MinPts in the neighborhood radius of the core point.

[0092] 2) Core point determination: Start from any point p, check if the number of points in the eps neighborhood of the current point is greater than or equal to MinPts. If it is satisfied, mark it as a core point and create a new cluster C for p, if it is not satisfied, mark it as a noise point (may be reclassified as a boundary point later).

[0093] 3) Cluster expansion: recursively determine each unvisited point q in the neighborhood, if the number of points in the neighborhood of point q is greater than or equal to MinPts, then q is a core point, and its neighborhood is added to cluster C. If the number of points in the neighborhood of point q is less than MinPts, then q is a boundary point.

[0094] 4) Termination condition: Select the next object from the remaining objects and repeat the process of 3) until all points are core points, boundary points or noise points, and the cluster division is completed.

[0095] Figure 4 The original point cloud and the clustered point cloud of a certain frame of data are shown, whereFigure 5 The middle red box is the human action point cloud, and the rest of the green points are determined as noise points.

[0096] Although the acquired three-dimensional point cloud data can be used to distinguish human actions, other features are ignored, which has certain limitations on the recognition effect. Therefore, the potential of using point clouds with more features as data representation to solve human action recognition is considerable.

[0097] This paper uses high-dimensional point cloud data for continuous HAR research. The final generated point cloud information contains the x, y, z three-dimensional coordinates of the target, as well as the Doppler (velocity) and signal strength, and the specific format is as shown in Figure 6 .

[0098] Specifically, the continuous sign language action data is collected by the millimeter wave radar, and a sign language action dataset is established. 400 sentences containing 108 sign language words are generated by 10 volunteers, and the specific rest type is referred to Figure 7 . The duration and transition time of the above activities are not limited.

[0099] Specifically, the sign language action dataset includes: radar data and synchronous video data of 10 volunteers (7 males, 3 females, aged 21-25 years old) in daily life containing Chinese common sentences with sign language words. Collect sign language sentence samples, each sign language sentence sample contains 4 different sign language words, and each sign language sentence sample is collected 50 times repeatedly; collect sign language word samples, each sign language word sample contains 4 identical sign language words, and each sign language word sample is collected 50 times repeatedly, a total of 400 sentences containing 108 sign language words. The subject naturally sits in front of the table, keeps the distance between the two chest center points about 40-60 cm away from the radar chip, and starts to make sign language actions after pressing the collection button.

[0100] Further, a PKTNet network model fusing spatio-temporal double-branch attention mechanism is proposed, as shown in Figure 8 , which extracts spatial features and temporal long and short-term dependency information from continuous point cloud data and outputs sequence prediction results.

[0101] Specifically, the PKTNet network model fusing spatio-temporal double-branch attention mechanism, in the continuous sign language recognition task, PointNet still has some problems, mainly including lack of local feature capture, lack of time series modeling ability, etc. This paper introduces KNN algorithm based on PointNet to extract local features, extracts cross-frame time series information of point cloud through spatio-temporal double-branch attention mechanism, and then inputs the two to the classification module. Through multi-layer MLP for dimension reduction, finally output the continuous human action recognition result. Reference Figure 9As shown, the network is named PKTNet, which consists of three parts: a local feature extraction module, a temporal feature extraction module, and a classification module.

[0102] Specifically, the local feature extraction module, with reference to Figure 9 , inputs five-dimensional point cloud data composed of T frames. For each frame of radar point cloud, the local feature extraction module first performs input transformation and feature transformation, and obtains point-by-point features through shared MLP. In the MLP layer, each MLP extracts high-dimensional features through linear transformation, with a dimension of 128. Then, KNN algorithm is used for local feature extraction. Assuming that a frame of point cloud samples is X = [x1, x2, …, xn] ∈ Rm, where n is the number of point clouds, and m represents the feature dimension of the data. Then, the K nearest points of each point xi are obtained, a k × 128 feature vector is constructed, and multiple shared MLP operations are performed on each query point to enhance the local feature representation of the point cloud space. Then, a max-pooling function is used to reduce the dimension to obtain a 1 × 512 local feature vector. After repeating n times, n × 512 local features of all points are obtained. Then, a 1 × 1 convolution is performed for dimension reduction to obtain a single frame of 1 × 512 spatial features. Finally, a T × 512 feature sequence is obtained from T frames of point cloud data, which is further input to the temporal feature extraction module.

[0103] Specifically, the temporal feature extraction module, with reference to Figure 10 After obtaining the T × 512 feature sequence from the local feature extraction module, in order to model the time sequence in continuous human motion data, a space-time dual-branch attention mechanism is used to process the time evolution in parallel. Referring to Figure 11 As can be seen, the space-time dual-branch attention mechanism is divided into two parts: intra-frame static attention and inter-frame dynamic attention. In the intra-frame static attention branch, small kernel deep convolution, deep convolution with expansion, and one-dimensional convolution are used for modeling. By calculating the attention weight of each position, the importance of different spatial regions in feature extraction is automatically adjusted. This enables the model to dynamically select the spatial position to focus on according to the important region in the current frame, thereby enhancing the expression of spatial features.

[0104] wherein the intra-frame static attention SA can be calculated by the formula as follows:

[0105] SA = Conv 1×1 (Conv DW-D (Conv DW (H)))#(3)

[0106] In addition, in order to capture the timing feature changes on the timeline, the squeeze and excitation method is used to learn the attention weights between different frames on the inter-frame dynamic attention branch. First, the global average pooling operation is used to reduce the dimension of the features of each frame to obtain the global time feature vector, and then the excitation step is to use a fully connected layer network to calculate the weight of each time step, the purpose of which is to adjust the feature response of each time step dynamically, so that important time points get more attention. The inter-frame dynamic attention DA can be calculated as follows:

[0107] DA=FC(AvgPool(H))#(4)

[0108] Finally, the two attention branches are integrated, where H is the sequence input to the timing feature extraction module, and H' is the final weighted result. The formula is as follows:

[0109]

[0110] where denotes the Kronecker product, and denotes the Hadamard product.

[0111] This module not only enhances the local information of the current frame, but also emphasizes the influence of key frames in the time sequence through weight learning. In the continuous action recognition task, key frames will show strong action features at certain moments, but not so important at other moments. Dynamic weight learning can automatically adjust the importance of different time frames, so that the model can adaptively select which time segments need more attention. Enhance key frame features while suppressing features that have less impact on the current action.

[0112] Specifically, the classification module, referring to Figure 11 , directly splices the features learned by the local feature extraction module and the timing feature extraction module, and takes them as the input of the classification module. After obtaining a Tx1024 feature, multiple shared MLPs are used to reduce the feature dimension to nxC, where C refers to the number of action categories, and finally the Softmax output is used to output the classification probability of each frame, thereby completing the segmentation and recognition of continuous sign language actions.

[0113] Second part: an evaluation index STT Score for radar continuous sign language action recognition is established.

[0114] Specifically, the evaluation index STT Score for radar continuous sign language action recognition allows a certain tolerance for action recognition errors within a very short time interval, thereby more accurately reflecting the continuity and transition ambiguity of human actions.

[0115] In the field of continuous sign language action recognition, the traditional accuracy metric provides a simple evaluation of recognition accuracy while ignoring the problem of over-segmentation. In addition, it is biased towards the evaluation of longer actions, thus weakening the importance of shorter actions. To address these shortcomings, this study introduces the segmental F1 score as a complementary evaluation metric to better assess model performance.

[0116] To evaluate the prediction model, the frame-wise accuracy (Acc) is generally used:

[0117]

[0118] Here, Correct represents the number of correctly predicted frames, and Total is the number of data frames.

[0119] However, from the perspective of continuous sign language action recognition, Acc can only obtain the approximate recognition accuracy and cannot reflect the problem of over-segmentation; and long actions within the sample will have a greater impact on Acc than short actions. Therefore, this invention proposes the segmental F1 score for further evaluation.

[0120] The segmental F1 score is generally used for action segmentation and detection tasks. This method evaluates performance by calculating the IOU of each predicted action and the true action.

[0121]

[0122] Here, P represents the predicted action time interval, and T represents the true action time interval.

[0123] As shown in the following formula, P represents the prediction result, and GT represents the true label. The segmental method compares the IOU and threshold K to determine true positives (TP), false positives (FP), and false negatives (FN).

[0124]

[0125] Reference Figure 12 As shown, if the segmental prediction is correct, when the IOU is greater than or equal to the threshold K, it is marked as TP, and less than K as FP. If the segmental prediction is incorrect, it is marked as FN. In addition, when there are multiple correct predictions within the range of a single true action, only one will be classified as TP, and the others will be classified as FP.

[0126] Then, the precision and recall of the sum of all classes of TP, FP, and FN are calculated, and the segmental F1 score is calculated.

[0127]

[0128] Segmentation F1 evaluation can effectively penalize over-segmentation errors, and the score depends on the number of activities in the data, not the number of frames, which is beneficial to provide more detailed and accurate evaluation for continuous sign language action recognition.

[0129] Since it is a real scene collection, the subjects perform one word after another when executing a sentence, and these activities are continuous. This results in ambiguous transitions between actions, especially within a 0.2s time period, which cannot well characterize an action, and it is almost possible to define it as either the previous action or the next action.

[0130] Therefore, based on F1, we propose the STT Score for radar continuous sign language action recognition. See Equation 10, len p The frame length representing the prediction time, when calculating TP, FP and FN, if the prediction result of the action transition frame is within 5 frames compared with the true value, it is not included in the calculation of FN. This evaluation method can tolerate the error of activity transition recognition within a very short time (0.2s), and can more accurately reflect the continuity of sign language actions while evaluating the performance of action recognition.

[0131]

[0132] To verify the performance of the proposed model, we use traditional time series models and some new models combined with PointNet, LSTM, GRU, PointNet-LSTM, PointNet-GRU, Point Transformer

[64] and PKTNet six models, and list the detailed frame-by-frame accuracy (Acc), F1 score and STT score.

[0133] References Figure 13 The traditional recurrent neural network models LSTM and GRU do not perform well in this experiment. The frame-by-frame accuracy is 0.6287 and 0.6319 respectively, the F1 score is 0.4315 and 0.4676 respectively, and the STT score is also relatively low, which is 0.5154 and 0.5269 respectively. Although these two models can capture some features in time series, they have great limitations in processing point cloud data. Each data point of point cloud data is composed of three-dimensional coordinates, with clear spatial geometric relationship. However, when processing time series data, LSTM and GRU cannot effectively preserve the geometric information in point cloud data. On the other hand, when processing long-lasting point cloud data, it is difficult to effectively capture long-term dependent features. Therefore, the performance of traditional LSTM and GRU models is not ideal.

[0134] While the models combined with PointNet, such as PointNet-LSTM and PointNet-GRU, have certain improvement in recognition accuracy. The model introduces the point cloud feature extraction ability of PointNet to better capture spatial structure information while retaining temporal information. The frame-by-frame accuracy is 0.8514 and 0.8621, respectively, the F1 score is 0.6235 and 0.6331, respectively, and the STT score is 0.7313 and 0.7421, respectively. Compared with traditional LSTM and GRU models, these models combined with PointNet show better recognition performance, indicating that the consideration of spatial features plays an important role in improving human action recognition.

[0135] Point Transformer model through self-attention mechanism to model the space-time features more precisely. Through this mechanism, Point Transformer can accurately capture complex temporal dependencies and spatial feature relationships, further improving the effect of continuous action recognition. Although the model achieves 0.9011 in frame-by-frame accuracy and 0.6729 in F1 score, due to the complexity of the action data, Point Transformer over-segmentation problem occurs, so the STT score is only 0.8058.

[0136] The PKTNet model we designed performs best in the comparative experiment and achieves the best recognition effect. The frame-by-frame accuracy is 0.9504, the F1 score is 0.7213, and the STT score is 0.8692. PKTNet introduces KNN algorithm and space-time dual-branch attention mechanism to capture local spatial information, and further strengthens the model's extraction ability of local features and temporal features by using intra-frame static attention and inter-frame dynamic attention mechanism. The strengthening of local information enables PKTNet to pay more attention to action details when processing point cloud data, while the accurate capture of temporal features effectively improves the continuous action recognition ability of long sequences. Compared with other models, PKTNet performs well in all three indicators, proving the effectiveness of the network architecture we proposed in the task of continuous sign language action recognition based on point cloud.

[0137] As Figure 14As shown, according to the confusion matrix, it can be seen that the recognition of the PKTNet model on each action category is relatively excellent, especially on the actions of walking, standing still and falling, and the accuracy rate is all above 95%. And the error recognition rate of most other action categories is low, indicating that the PKTNet model can effectively distinguish different action key spatial features. However, there are still some misclassifications between a few actions, such as sitting down and falling, and standing up and standing up after falling. These confusions may be because these actions have certain similarities in actual scenarios, making it difficult for the model to completely distinguish them. From the experimental results, it can be seen that the PKTNet model performs well in the continuous human action recognition task, can efficiently distinguish most action categories, especially the special action of falling, and the recognition accuracy rate reaches 96.99%.

[0138] The above describes a better complete embodiment of the present application, which does not limit the protection scope of the present application. Any improvement or equivalent replacement of the technical solutions of the present application in principle, structure or method without departing from the core design principles of the present application should be considered as belonging to the protection scope covered by the claims of the present application.

Claims

1. A sign language recognition method based on millimeter wave radar 3D point cloud, characterized in that: It consists of two parts: Part I: A method for continuous sign language action recognition based on millimeter-wave radar, including: millimeter-wave radar signal processing; millimeter-wave radar acquisition of a continuous sign language action dataset; millimeter-wave radar acquisition of a continuous sign language action dataset; a PKTNet network model that integrates a spatiotemporal dual-branch attention mechanism, performs KNN neighborhood feature aggregation in the local point cloud branch, and combines intra-frame static attention with inter-frame dynamic attention to process continuous point cloud sequence data and output sequence prediction results; The millimeter wave radar signal processing includes: millimeter wave radar signal acquisition parameter configuration; millimeter wave radar point cloud signal preprocessing; The method of processing the sequence data of the continuous point cloud to extract the spatial features and the long-term and short-term time dependency information comprises: extracting the spatial feature information and the long-term and short-term time dependency information from the point cloud sequence generated by the FMCW radar; The output sequence prediction results include: prediction of the sign language action type corresponding to each frame of data, implicitly including the duration information of the activity, and paying special attention to the capture of the key frames of the activity transition; During the millimeter-wave radar point cloud signal preprocessing process, the radar data of each frame is subjected to Range FFT along the fast time direction to obtain the distance information: Where X(c,s) represents the frame data of the cth Chirp and the sth Samples, N S is the size of Sample, w(l) represents the Hamming window of length l; X r (c, r) represents the distance processing data of the cth Chirp and the rth distance bin, r∈[0,N S ); After completing the Range FFT, CFAR is used to eliminate static clutter. Then, 3D-FFT is performed along the angle estimation to obtain point cloud data. The 3D coordinates of the point cloud are then converted from polar coordinates to a rectangular coordinate system to obtain the 3D coordinates of each point. The specific calculation is as follows: Where: r i is the distance from the radar to the target, θ i is the pitch angle, is the azimuth; The PKTNet network model integrating the spatiotemporal dual-branch attention mechanism consists of three parts: a local feature extraction module, a temporal feature extraction module, and a classification module. The local feature extraction module, whose input consists of five-dimensional point cloud data of T frames. For the radar point cloud in each frame, the local feature extraction module first performs input transformation and feature transformation, and obtains point-by-point features through shared MLP; The temporal feature extraction module obtains the feature sequence from the local feature extraction module. In order to model the time series in continuous human motion data, a spatiotemporal dual-branch attention mechanism is used to parallelize the temporal evolution. The spatiotemporal dual-branch attention mechanism is decomposed into two parts: intra-frame static attention and inter-frame dynamic attention. The intra-frame static attention branch uses small kernel depthwise convolution, dilated depthwise convolution, and one-dimensional convolution for modeling. By calculating the attention weight of each position, the importance of different spatial regions in feature extraction is automatically adjusted. Among them, the intra-frame static attention SA can be calculated by the formula, which is as follows: SA=Conv 1×1 (Conv DW-D (Conv DW (H)))#(3) In addition, in order to capture the changes in temporal features on the timeline, the inter-frame dynamic attention branch uses squeezing and excitation to learn the attention weights between different frames. First, through squeezing, the features of each frame are reduced in dimension using a global average pooling operation to obtain a global temporal feature vector. Then, the excitation step uses a fully connected layer network to calculate the weight of each time step. The purpose is to dynamically adjust the feature response of each time step so that important time points receive higher attention. The inter-frame dynamic attention DA can be obtained by calculation, and the calculation formula is as follows: DA=FC(AvgPool(H))#(4) Finally, the two attention branches are integrated, where H is the sequence input to the temporal feature extraction module and H' is the final weighted result. The formula is as follows: in represents the Kronecker product, ⊙ represents the Hadamard product; The classification module directly concatenates the features learned by the local feature extraction module and the temporal feature extraction module and uses them as input to the classification module. After obtaining a T×1024 feature, it uses multiple shared MLPs to reduce the feature dimension to n×C, where C is the number of action categories. Finally, a softmax function is used to output the classification probability for each frame, thus completing the segmentation and recognition of continuous sign language actions. Part II: Establishing the evaluation index STT Score for radar continuous sign language action recognition, The STT Score, an evaluation indicator for the radar's continuous sign language motion recognition, allows for a certain tolerance of motion recognition errors within a very short time interval, thereby more accurately reflecting the continuity and transition ambiguity of human motions. The prediction model was evaluated and further evaluated using the segmented F1 score; The segmental F1 score evaluates the performance by calculating the IOU between each predicted action and the true action; Among them, P represents the predicted action time interval, and T represents the actual action time interval; As shown in the following formula, P represents the prediction result and GT represents the true label; the segmentation method compares the IOU with the threshold K to determine the true positive (TP), false positive (FP) and false negative (FN); If the segment prediction is correct, when the IOU is greater than or equal to the threshold K, it is marked as TP, and if it is less than K, it is marked as FP; if the segment prediction is wrong, it is marked as FN; in addition, when there are multiple correct predictions within the range of a single real action, only one will be classified as TP, and the others are FP; Then calculate the precision and recall of the sum of TP, FP and FN for all classes, and calculate the segmented F1 score; Based on F1, STT Score is proposed for radar continuous sign language action recognition, see formula 10, len p Indicates the frame length of the prediction time. When calculating TP, FP, and FN, if the difference between the prediction result and the true value for the action transition frame is within 5 frames, it will not be included in the calculation of FN.

2. The sign language recognition method based on millimeter wave radar 3D point cloud according to claim 1, characterized in that: After completing CFAR and coordinate transformation, the density clustering algorithm is further used to suppress noise. The density clustering algorithm is a spatial clustering method based on data density. It includes two important parameters: the neighborhood radius eps and the minimum number of points MinPts within the neighborhood radius of the core point. The neighborhood radius refers to a circular or spherical area with a radius of eps centered on a certain point. The algorithm divides data points into three categories: core point: the neighborhood radius of the point contains at least MinPts points (including itself); boundary point: located in the eps neighborhood of the core point, but the number of points contained in its own neighborhood radius is less than MinPts; noise point: neither located in the eps neighborhood of the core point, nor the number of points contained in its own neighborhood radius is less than MinPts; the specific operation steps are as follows: 1) Initialization: Given a neighborhood radius eps and the minimum number of core points within the neighborhood radius MinPts; 2) Core point determination: Starting from any point p, check whether the number of points in the eps neighborhood of the current point is greater than or equal to MinPts; if it is satisfied, mark it as a core point and create a new cluster C for p; if not, mark it as a noise point and may be reclassified as a boundary point later; 3) Cluster expansion: recursively judge each unvisited point q in the neighborhood. If the number of points in the neighborhood of point q is greater than or equal to MinPts, then q is a core point and its neighborhood is added to cluster C. If the number of points in the neighborhood of point q is less than MinPts, then q is a boundary point. 4) Termination condition: Select the next object from the remaining objects and repeat the process of 3) until all points are clustered by core points, boundary points or noise points.

3. The sign language recognition method based on millimeter wave radar 3D point cloud according to claim 1, characterized in that: Millimeter-wave radar collected continuous sign language motion data to establish a sign language motion dataset, which consists of 400 sentences containing 108 sign language words generated by 10 volunteers. The sign language motion dataset includes radar data and synchronized video data of common Chinese sentences containing sign language words in daily life from 10 volunteers (7 males and 3 females, aged 21-25 years old); sign sentence samples were collected, each containing 4 different sign language words, and each sign sentence sample was repeatedly collected 50 times; sign language word samples were collected, each containing 4 identical sign language words, and each sign language word sample was repeatedly collected 50 times, for a total of 400 sentences containing 108 sign language words; the subjects sat naturally at a table, with the center of their chests kept 40-60 cm away from the radar chip. The subjects began to perform sign language movements after pressing the collection button.

4. The sign language recognition method based on millimeter wave radar 3D point cloud according to claim 1, characterized in that: In the MLP layer, each MLP extracts high-dimensional features from it through linear transformation, and its dimension is 128; then the KNN algorithm is used for local feature extraction. Assume that the point cloud sample of a frame is X = [x1, x2,…, xn]∈Rm, where n is the number of point clouds and m represents the feature dimension of the data. Then, the K points closest to each point xi are obtained, a k×128 feature vector is constructed, and multiple shared MLP operations are performed on each query point to enhance the local feature performance of the point cloud space. Then, the maximum pooling function is used to reduce the dimension to obtain 1×512 local feature vectors. After repeating n times, the local features of all points n×512 are obtained, and then the dimension is reduced by 1×1 convolution to obtain the spatial features of a single frame 1×512. Finally, a T×512 feature sequence extracted from T frame point cloud data is obtained and further input into the temporal feature extraction module.

Images