Lightweight multi-modal fusion gesture recognition method based on radar and vision cloud

Abstract

This invention relates to the field of non-contact gesture recognition technology, and more specifically, to a lightweight multimodal fusion gesture recognition method based on Rayvision Cloud. The method includes: acquiring video data, micro-Doppler data, and point cloud sequences; calculating the normalized values ​​and effective acquisition time thresholds for the three modalities, and dividing the effective acquisition time into uniform periods; inputting the effective acquisition data of the three modalities into corresponding feature extraction networks to obtain modal features and then performing feature fusion; and inputting the acquired multimodal fusion features into a gesture classification network for training. This invention uses video data, micro-Doppler data, and point cloud sequences as modal data, reducing the impact of single-modal data defects on gesture recognition results, achieving complementary advantages between various modalities, and improving the accuracy of gesture recognition in complex application scenarios.

Description

Technical Field

[0001] This invention relates to the field of non-contact gesture recognition technology, and more specifically, to a lightweight multimodal fusion gesture recognition method based on Rayvision Cloud. Background Technology

[0002] With the development of neural networks and deep learning technologies, non-contact gesture recognition technology has been applied in fields such as human-computer interaction, smart homes, and medical health. Industry and academia have conducted numerous related studies on gesture recognition technology. Current non-contact gesture recognition systems include those based on radio frequency signals (RFS) and those based on millimeter-wave radar. RFS-based gesture recognition is susceptible to interference. While millimeter-wave radar can acquire 4D point cloud sequences of gestures, comprehensively reflecting information such as distance, angle, and speed, these sequences are easily affected by noise and exhibit sparse output. In complex backgrounds, they are also prone to radial divergence and poor uniformity, thus affecting the final gesture recognition performance.

[0003] Existing technology discloses a gesture recognition method and system based on micro-Doppler features. The method includes: observing command gestures obscured by obstacles using through-wall radar to obtain radar echo signals; the obstacles include walls; performing Doppler spectrum and short-time Fourier transform time-spectrum analysis on the radar echo signals to extract multidimensional micro-Doppler features caused by the obstruction of command gestures; establishing a training set composed of multidimensional micro-Doppler features extracted from historically observed radar echo signals and corresponding command gestures, and training a classifier; inputting the multidimensional micro-Doppler features caused by obstructed command gestures extracted in real time into the trained classifier for recognition to determine the obstructed command gesture. However, this method still cannot overcome the problem that radially moving actions in the micro-Doppler image are not easily distinguished, thus affecting the final gesture recognition effect. Summary of the Invention

[0004] The purpose of this invention is to disclose a lightweight multimodal fusion gesture recognition method based on Rayvision Cloud with better recognition performance.

[0005] To achieve the above objectives, this invention provides a lightweight multimodal fusion gesture recognition method based on Rayvision Cloud, the specific technical solution of which is as follows:

[0006] S1: Acquire video data, micro-Doppler data, and point cloud sequences;

[0007] S2: Calculate the normalized values ​​of video data, micro-Doppler data, and point cloud sequence respectively to obtain the normalized values ​​of video data, micro-Doppler data, and point cloud sequence.

[0008] S3: Based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, obtain the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence.

[0009] S4: Calculate a unified effective acquisition time based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence. Extract the video data, micro-Doppler data, and point cloud sequence according to the unified effective acquisition time to obtain effective video data, micro-Doppler acquisition data, and effective point cloud sequence data.

[0010] S5: Input the effective video data, effective micro-Doppler data, and effective point cloud sequence data into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, perform feature fusion on the video modal features, micro-Doppler modal features, and point cloud sequence modal features to obtain multimodal fusion features.

[0011] S6: Construct a gesture classification network, input multimodal fusion features into the gesture classification network for training, and obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud;

[0012] S7: Obtain gesture information, and obtain the recognition result based on the gesture information through the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, thus completing gesture recognition.

[0013] 2. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, in step S1, the video data, micro-Doppler data, and point cloud sequence include:

[0014] The video data is represented as follows:

[0015] VD = [VD1, VD2, ..., VD] n , ...,VD N ],

[0016]

[0017] Where VD represents video data, VD n Indicates the nth frame VD, This represents the coordinates of the m-th keypoint in the n-th frame of video data, where m = 1, 2, ..., M represents the keypoint index, and n = 1, 2, ..., N represents the frame index of the video data.

[0018] Micro-Doppler data are represented as follows:

[0019] DD = [DD1, DD2, ..., DD] s ,...,DDs ],

[0020]

[0021] Where DD represents micro-Doppler data, DD s This represents the S-th frame of micro-Doppler data, where s = 1, 2, ..., S represents the frame index of the micro-Doppler data. DD s The element value at index (rg, dp) is obtained by FFT. D (rg, dp) represents the two-dimensional frequency domain transform of the intermediate frequency signal, where rg and dp represent the two dimensions of the micro-Doppler data, and rg = 1, 2, ..., RG, dp = 1, 2, ..., DP. (FFT) R (rg,n c ) represents the one-dimensional frequency domain transform of the intermediate frequency signal, n c =1,2,...,N c Indicates the chirp signal index;

[0022] The point cloud sequence is represented as follows:

[0023] PD = [PD1, PD2, ..., PD] l ,..,PD L ],

[0024]

[0025] Where PD is the point cloud sequence, PD l This represents the point cloud sequence of the l-th frame. Let h represent the h-th coordinate point of the l-th frame point cloud sequence, where h = 1, 2, ..., H represents the index of the coordinate point, and l = 1, 2, ..., L represents the frame index of the point cloud sequence.

[0026] Further, in step S2, calculating the normalized value of the video data includes:

[0027] S2.1.1: Initialize m to 1;

[0028] S2.1.2: Calculation Distance between two adjacent frames

[0029]

[0030] in, This represents the dimension value of the m-th keypoint in the n-th frame of VD. This represents the dimension value of the m-th keypoint in the (n+1)-th frame of video data.

[0031] S2.1.3: Calculation of mean

[0032] S2.1.4: Calculate VD n superior normalized value

[0033] in, Represents all of the m-th key points The maximum value, Represents all of the m-th key points The minimum value;

[0034] S2.1.5: Calculate the normalized value of the video data:

[0035] Further, in step S2, calculating the normalized value of the micro-Doppler data includes:

[0036] S2.2.1: rg, dp, n c All are initialized to 1;

[0037] S2.2.2: Calculation of micro-Doppler data mean

[0038]

[0039] S2.2.3: Calculation normalized value

[0040]

[0041] in, express The maximum value, express The minimum value;

[0042] S2.2.4: Calculate the normalized value of the micro-Doppler data:

[0043] Further, in step S2, calculating the normalized value of the point cloud sequence includes:

[0044] S2.3.1: Initialize h to 1

[0045] S2.3.2: Calculation Distance between two adjacent frames

[0046]

[0047] in, This represents the dimension value of the h-th coordinate point of the PD in the l-th frame. This represents the dimension value of the h-th coordinate point in the (l+1)-th frame of the point cloud sequence;

[0048] S2.3.3: Calculation Mean distance

[0049] S2.4.4: Calculation normalized value

[0050] in, Represents all coordinates of the h-th point The maximum value, Represents all coordinates of the h-th point The minimum value;

[0051] S2.3.5: Calculate the normalized value of PD:

[0052] Further, in step S3, based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence are obtained, including:

[0053] The effective acquisition time threshold for video data, derived from the normalized values ​​of the video data, includes:

[0054] S3.1.1: Calculate the traversal threshold TH of the video data v Threshold traversal step size for video data v :

[0055] TH v =min(VS) n )

[0056] Where, ε v Indicates the number of times the video data has been traversed;

[0057] S3.1.2: Initialize VD n Less than TH v Frame count Nv0, VD n Greater than TH v The number of frames Nv1, and all VDs in the video data less than VTh. n All VDs greater than VTh in Sv0 and VD nSv1 and n:

[0058] Nv0=0,Sv0=0,Nv1=0,Sv1=0,n=1

[0059] S3.1.3:

[0060] S3.1.4: Determine whether n is less than or equal to N. If yes, return to step S313; otherwise, proceed to step S315.

[0061] S3.1.5: Let n = n + 1;

[0062] S3.1.6: Calculate the proportion of Nv0 (pv0), the proportion of Nv1 (pv1), and the mean of Sv0. Sv1 mean

[0063] S3.1.7: Calculation in TH v Under the condition, the inter-class variance of VD

[0064]

[0065] S3.1.8: Judgment Is it more than If yes, proceed to step S319; otherwise, proceed to step S3.1.4.

[0066] S3.1.9: Calculate the effective acquisition time threshold VTh = TH for video data. v

[0067] The effective acquisition time threshold for micro-Doppler data, derived from the normalized value of micro-Doppler data, includes:

[0068] S3.2.1: Set the effective acquisition time threshold variable TH for micro-Doppler data. d Calculate the effective acquisition time threshold DTh and the threshold traversal step size for micro-Doppler data. d :

[0069] DTh=TH d

[0070]

[0071] Where, ε d Indicates the number of times the microDoppler data has been traversed;

[0072] S3.2.2: Calculate DS s Less than TH dThe proportion of frames p0(TH) d ), DS s Greater than TH d Frame count

[0073]

[0074] The proportion p1(TH) d ):

[0075] Among them, DS s (ds) represents the micro-Doppler data that is numerically equal to ds, where ds represents the normalized value that varies according to the traversal step size.

[0076] S3.2.3: Calculate the values ​​of microDoppler data smaller than TH. d DS s The weighted average M g (TH d ), greater than TH in micro-Doppler data d DS s The weighted average M G (TH d ):

[0077]

[0078] S3.2.4: Calculate the values ​​of microDoppler data smaller than TH. d DS s The average value Avr0(TH) d ), greater than TH in micro-Doppler data d DS s The average value of Avr1(TH) d ):

[0079]

[0080] S3.2.5: Calculate the inter-class variance σ 2 :

[0081] S3.2.6: For σ 2 Seeking TH d The partial derivative of zp is used to calculate the local maximum and zero point zp.

[0082]

[0083] Micro-Doppler data DTh satisfies: DTh = zp;

[0084] The effective acquisition time threshold for the point cloud sequence obtained from the normalized value of the point cloud sequence includes:

[0085] S3.3.1: Calculate the traversal threshold TH of the point cloud sequence p Threshold traversal step size for point cloud sequences p :

[0086] Where, ε l Indicates the number of times the point cloud sequence has been traversed;

[0087] S3.3.2: Initialize PS s Less than TH p Frame rate Np0, PS s Greater than TH p The number of frames Np1 and PD is less than TH p All PS s The sum of Sp0 and PD is greater than TH p All PS s The sum of Sp1, and 1:

[0088] Np0=0,Sp0=0,Np1=0,Sp1=0,l=1

[0089] S3.3.3:

[0090]

[0091] S3.3.4: Determine if l is less than or equal to L. If yes, return to step S3.3.3; otherwise, proceed to step S3.3.5.

[0092] S3.3.5: Let l = l + 1;

[0093] S3.3.6: Calculate the proportions of Np0 (pp0), Np1 (pp1), and Sp0, and their mean values. Mean of Sp1

[0094] S.3.3.7: Calculation in TH p Under the given conditions, the inter-class variance of PD:

[0095] S3.3.8: Judgment Is it more than If yes, proceed to step S3.3.9; otherwise, proceed to step S3.3.4.

[0096] S3.3.9: Calculate the effective acquisition time threshold PTh = TH for the point cloud sequence. p

[0097] Further, in step S4, a unified effective acquisition time is calculated based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequences. The video data, micro-Doppler data, and point cloud sequences are then truncated according to the unified effective acquisition time to obtain the effective acquisition data for video, micro-Doppler, and point cloud sequences, including:

[0098] S4.1: Based on the effective acquisition time threshold of the acquired video data, divide the effective acquisition time of the video data into segments:

[0099] Among them, VT begin Indicates the valid start time of video data acquisition, VT end ΔT represents the effective end time of video data acquisition. v This indicates the time interval for transmitting video data frames.

[0100] Based on the effective acquisition time threshold of the acquired micro-Doppler data, the effective acquisition time of micro-Doppler data is divided as follows:

[0101] Among them, DT begin DT represents the effective start time of micro-Doppler data acquisition. end ΔT represents the effective acquisition end time of micro-Doppler data. d This indicates the time interval for transmitting micro-Doppler data frames.

[0102] Based on the effective acquisition time threshold of the acquired point cloud sequence, the effective acquisition time of the point cloud sequence is divided as follows:

[0103]

[0104] Among them, PT begin PT represents the valid start time of point cloud acquisition. end ΔT represents the effective acquisition end time of the point cloud sequence. p This indicates the time interval for transmitting point cloud sequence frames.

[0105] S4.2: According to VT begin VT end DT begin DT end PT begin PT end Unified effective sampling time for the three modes:

[0106]

[0107] Among them, T begin T represents the unified and valid start time of data collection. endT represents the unified valid data collection end time. new This indicates a unified valid data collection time interval;

[0108] S4.3: According to T new The VD, DD, and PD data are extracted, and finally the valid video data V is obtained. row Effective data D of micro-Doppler data row Valid data P of point cloud sequence row :

[0109]

[0110] Where k = 1, 2, ..., K is the group index. V represents row The Kth group of data, D represents row The Kth group of data, P represents row In the Kth group of data, V row For valid video data, D row For valid micro-Doppler data, P row This is valid data for the point cloud sequence.

[0111] Further, in step S5, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, the video modal features, micro-Doppler modal features, and point cloud sequence modal features are fused to obtain multimodal fusion features, including:

[0112] S5.1: Divide the valid video data, valid micro-Doppler data, and valid point cloud sequence data into K groups, with each group represented as follows:

[0113]

[0114] Initialize k and the total loss value ΔW: k = 1, ΔW = 0

[0115] S5.2: For each set of data, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into the corresponding feature extraction network, and the feature matrices of the effective video data, effective micro-Doppler data, and effective point cloud sequence data are obtained respectively.

[0116] S5.3: Calculate VM i×j and DM i×j The basic product between, VM i×j and PM i×j The basic product between, DMi×j and PM i×j The fundamental product between them, where VM i×j DM is the feature matrix of the effective video data. i×j The feature matrix of effective micro-Doppler data, PM i×j The feature matrix of the valid data for the point cloud sequence:

[0117] Hvd i，j =VM i，j ×DM i,j

[0118] Hvp i,j =VM i，j ×PM i，j

[0119] Hdp i，j =DM i,j ×PM i,j

[0120] Where i = 1, 2, ..., I represents the row index of the feature matrix, and j = 1, 2, ..., J represents the column index of the feature matrix;

[0121] S5.4: Calculate the combination matrix G i，j :G i，j =Hvd i，j +Hvp i，j +Hdp i，j

[0122] S5.5: Computing VM i×j tensor product Kv p,q DM i×j tensor product Kd p，q PM i×j tensor product Kp p，q G i×j tensor product KG p，q :

[0123]

[0124] in, Let VD represent the eigentranspose matrix. This represents the eigentranspose matrix of DD. This represents the eigentranspose matrix of PD. G represents i，j The transpose of the tensor product is given by p = 1, 2, ..., where P represents the row index of the tensor product and q = 1, 2, ..., where Q represents the column index of the tensor product.

[0125] S5.6: Calculate VM i×j With and G i，jThe loss values ​​between Wv and DM i×j With and G i，j The loss values ​​between Wd and PM i×j With and G i，j The loss value Wp between:

[0126]

[0127] S5.7: Calculate ΔW:

[0128]

[0129] Where δ represents the loss linearity factor, u = 1, 2, ..., U represents the number of categories, p(u) represents the predicted value of the u-th class, and y(u) represents the actual value of the u-th class;

[0130] S5.8: Determine if ΔW is less than the loss threshold W th If yes, proceed to step S5.9; otherwise, proceed to step S5.2.

[0131] S5.9: Input the combined matrix into the second feature network to extract dynamic temporal features;

[0132] S5.10: Determine if k is greater than K. If yes, proceed to step S5.12; otherwise, proceed to step S5.11.

[0133] S5.11: Let k = k + 1, and execute step S5.1;

[0134] S5.12: Input the dynamic temporal features obtained from all groups into the fully connected layer to obtain multimodal fusion features.

[0135] Further, in step S6, a gesture classification network is constructed, and the multimodal fusion features are input into the gesture classification network for training, resulting in a lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, including:

[0136] S6.1: Construct a gesture classification network to classify multimodal fusion features, where group C is used as training data. c Group Z is used as categorical data. z Set the initial preweight matrix WM for the gesture classification network. all Initialize the initial preweight index φ:

[0137]

[0138] WM all =[WM1,WM2,..,WM φ ,..,WM Φ-1 WM Φ ]

[0139] φ=1

[0140] in, The i×j-th data point of the c-th training data set. Let Wm represent the i×j-th data point in the z-th classification dataset, where c = 1, 2, ..., C represents the training data index, z = 1, 2, ..., Z represents the classification data index. φ This represents the φ-th initial preweight;

[0141] S6.2: Input the training data into the gesture classification network for one round of training;

[0142] S6.3: After this round of training is completed, determine whether φ is greater than Φ. If yes, execute S6.5; otherwise, execute S6.4.

[0143] S6.4: Let φ = φ + 1, then return to step S6.2;

[0144] S6.5: Obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud;

[0145] Further, in step S6.2, inputting the training data into the gesture classification network for one round of training includes:

[0146] S6.2.1: Initialize Train c The initial preweights Wm used in this round T Number of iterations (num):

[0147]

[0148] num=1

[0149] in, This represents the transpose of the φ-th initial preweight;

[0150] S6.2.2: Initialize c to 1, and initialize the statistics flag mark to 0;

[0151] S6.2.3: Initialize u to 1, and initialize the flag category to u:

[0152] S6.2.4: Calculate the discriminant function value r obtained in the num-th iteration for the u-th category. u (num), and the discriminant function value r of flag. flag (num):

[0153]

[0154] S6.2.5: Determine r flagIs (num) less than r? u If (num) is true, proceed to step S6.2.6; otherwise, proceed to step S6.2.7.

[0155] S6.2.6: Set mark = mark + 1, then execute S6.2.8;

[0156] S6.2.7: Calculate the weight matrix Wm for the next round of labeling categories. flag (num+1), the weight matrix Wm corresponding to the category in this round in the next round. u (num+1):

[0157] Wm flag (num+1)=Wm flag (num)+μTrain c

[0158] Wm u (num+1)=Wm u (num)-μTrain c

[0159] Among them, Wm flag (num) represents the weight matrix of the label categories in this round, Wm u (num) represents the weight matrix of class u in this round, μ represents the weight modification factor and μ > 0;

[0160] S6.2.8: Determine if u is less than U. If yes, proceed to step S6.2.9; otherwise, proceed to step S6.2.10.

[0161] S6.2.9: Let u = u + 1 and falg = u, then execute step S6.2.4;

[0162] S6.2.10: Determine if c is less than C. If yes, proceed to step S6.2.11; otherwise, proceed to step S6.2.12.

[0163] S6.2.11: Let c = c + 1, num = num + 1, mark = mark + 1, and execute step S6.2.3;

[0164] S6.2.12: Determine if mark is equal to C. If yes, proceed to step S6.2.14; otherwise, proceed to step S6.2.13.

[0165] S6.2.13: Let num = num + 1, and execute step S6.2.2;

[0166] S6.2.14: Obtain information on using Wm T The gesture classification hyperplane obtained under the given conditions

[0167] Complete the training.

[0168] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0169] This invention fuses the modal features of video data, micro-Doppler data, and point cloud sequences to obtain multimodal fusion features, reducing the impact of single-modal data defects on gesture recognition results and achieving complementary advantages between various modalities. This enables the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, obtained from the multimodal fusion features, to have higher accuracy in complex application scenarios. Attached Figure Description

[0170] Figure 1 This is a flowchart of the lightweight multimodal fusion gesture recognition method based on Rayvision Cloud as described in Example 1;

[0171] Figure 2 This is a flowchart of the algorithm for calculating the normalized value of video data as described in Example 2;

[0172] Figure 3 This is a schematic diagram illustrating the process of calculating the effective acquisition time thresholds for the three modes and dividing the unified effective acquisition time as described in Example 3;

[0173] Figure 4 This is a flowchart illustrating the feature extraction and feature fusion of three modalities as described in Example 3;

[0174] Figure 5 This is a flowchart illustrating the training and classification of the gesture recognition network described in Example 3;

[0175] Figure 6 The flowchart for Example 3 shows the process of inputting multimodal fusion features into a gesture classification network for training.

[0176] Figure 7 This is a schematic diagram of the process of inputting training data into the gesture classification network for one round of training, as described in Example 3. Detailed Implementation

[0177] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0178] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0179] Example 1

[0180] This embodiment discloses as follows: Figure 1 The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud, as shown, includes:

[0181] S1: Acquire video data, micro-Doppler data, and point cloud sequences;

[0182] S2: Calculate the normalized values ​​of video data, micro-Doppler data, and point cloud sequence respectively to obtain the normalized values ​​of video data, micro-Doppler data, and point cloud sequence.

[0183] S3: Based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, obtain the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence.

[0184] S4: Calculate a unified effective acquisition time based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence. Extract the video data, micro-Doppler data, and point cloud sequence according to the unified effective acquisition time to obtain effective video data, micro-Doppler acquisition data, and effective point cloud sequence data.

[0185] S5: Input the effective video data, effective micro-Doppler data, and effective point cloud sequence data into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, perform feature fusion on the video modal features, micro-Doppler modal features, and point cloud sequence modal features to obtain multimodal fusion features.

[0186] S6: Construct a gesture classification network, input multimodal fusion features into the gesture classification network for training, and obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud;

[0187] S7: Obtain gesture information, and obtain the recognition result based on the gesture information through the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, thus completing gesture recognition.

[0188] This embodiment fuses the modal features of video data, micro-Doppler data, and point cloud sequences to obtain multimodal fusion features, reducing the impact of single-modal data defects on gesture recognition results and achieving complementary advantages between various modalities. This enables the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, obtained from the multimodal fusion features, to have higher accuracy in complex application scenarios.

[0189] Example 2

[0190] This embodiment further discloses information based on Embodiment 1:

[0191] In step S1, the video data, micro-Doppler data, and point cloud sequence include:

[0192] The video data is represented as follows:

[0193] VD = [VD1, VD2, ..., VD] n ,..,VD N ],

[0194]

[0195] Where VD represents video data, VD n Indicates the nth frame VD, This represents the coordinates of the m-th keypoint in the n-th frame of video data, where m = 1, 2, ..., M represents the keypoint index, and n = 1, 2, ..., N represents the frame index of the video data.

[0196] Micro-Doppler data are represented as follows:

[0197] DD = [DD1, DD2, ..., DD] s ,...,DD s ],

[0198]

[0199] Where DD represents micro-Doppler data, DD s This represents the S-th frame of micro-Doppler data, where s = 1, 2, ..., S represents the frame index of the micro-Doppler data. DD s The element value at index (rg, dp) is obtained by FFT. D (rg, dp) represents the two-dimensional frequency domain transform of the intermediate frequency signal, where rg and dp represent the two dimensions of the micro-Doppler data, and rg = 1, 2, ..., RG, dp = 1, 2, ..., DP. (FFT) R (rg, n) c ) represents the one-dimensional frequency domain transform of the intermediate frequency signal, n c =1,2,...,N c Indicates the chirp signal index;

[0200] The point cloud sequence is represented as follows:

[0201] PD = [PD1, PD2, ..., PD] l ,...,PD L ],

[0202]

[0203] Where PD is the point cloud sequence, PD l This represents the point cloud sequence of the l-th frame. Let H represent the h-th coordinate point of the l-th frame point cloud sequence, where h = 1, 2, ..., H represents the index of the coordinate point, and l = 1, 2, ..., L represents the frame index of the point cloud sequence.

[0204] like Figure 2 As shown in step S2, calculating the normalized value of the video data includes:

[0205] S2.1.1: Initialize m to 1;

[0206] S2.1.2: Calculation Distance between two adjacent frames

[0207]

[0208] in, This represents the dimension value of the m-th keypoint in the n-th frame of VD. This represents the dimension value of the m-th keypoint in the (n+1)-th frame of video data.

[0209] S2.1.3: Calculation of mean

[0210] S2.1.4: Calculate VD n superior normalized value

[0211] in, Represents all of the m-th key points The maximum value, Represents all of the m-th key points The minimum value;

[0212] S2.1.5: Calculate the normalized value of the video data:

[0213] In step S2, calculating the normalized value of the micro-Doppler data includes:

[0214] S2.2.1: rg, dp, n c All are initialized to 1;

[0215] S2.2.2: Calculation of micro-Doppler data mean

[0216]

[0217] S2.2.3: Calculation normalized value

[0218]

[0219] in, express The maximum value, express The minimum value;

[0220] S2.2.4: Calculate the normalized value of the micro-Doppler data:

[0221] In step S2, calculating the normalized value of the point cloud sequence includes:

[0222] S2.3.1: Initialize h to 1

[0223] S2.3.2: Calculation Distance between two adjacent frames

[0224]

[0225] in, This represents the dimension value of the h-th coordinate point of the PD in the l-th frame. This represents the dimension value of the h-th coordinate point in the (l+1)-th frame of the point cloud sequence;

[0226] S2.3.3: Calculation Mean distance

[0227] S2.4.4: Calculation normalized value

[0228] in, Represents all coordinates of the h-th point The maximum value, Represents all coordinates of the h-th point The minimum value;

[0229] S2.3.5: Calculate the normalized value of PD:

[0230] In step S3, based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence are obtained, including:

[0231] The effective acquisition time threshold for video data, derived from the normalized values ​​of the video data, includes:

[0232] S3.1.1: Calculate the traversal threshold TH of the video data v Threshold traversal step size for video data v :

[0233] TH v =min(VS) n )

[0234] Where, ε v Indicates the number of times the video data has been traversed;

[0235] S3.1.2: Initialize VD n Less than TH v Frame count Nv0, VD n Greater than TH v The number of frames Nv1, and all VDs in the video data less than VTh. n All VDs greater than VTh in Sv0 and VD n Sv1 and n:

[0236] Nv0=0,Sv0=0,Nv1=0,Sv1=0,n=1

[0237] S3.1.3:

[0238] S3.1.4: Determine whether n is less than or equal to N. If yes, return to step S313; otherwise, proceed to step S315.

[0239] S3.1.5: Let n = n + 1;

[0240] S3.1.6: Calculate the proportion of Nv0 (pv0), the proportion of Nv1 (pv1), and the mean of Sv0. Sv1 mean

[0241] S3.1.7: Calculation in TH v Under the condition, the inter-class variance of VD

[0242]

[0243] S3.1.8: Judgment Is it more than If yes, proceed to step S319; otherwise, proceed to step S3.1.4.

[0244] S3.1.9: Calculate the effective acquisition time threshold VTh = TH for video data. v

[0245] The effective acquisition time threshold for micro-Doppler data, derived from the normalized value of micro-Doppler data, includes:

[0246] S3.2.1: Set the effective acquisition time threshold variable TH for micro-Doppler data. d Calculate the effective acquisition time threshold DTh and the threshold traversal step size for micro-Doppler data. d :

[0247] DThh = TH d

[0248]

[0249] Where, ε d Indicates the number of times the microDoppler data has been traversed;

[0250] S3.2.2: Calculate DS s Less than TH d The proportion of frames p0(TH) d ), DS s Greater than TH d Frame count

[0251] The proportion p1(TH) d ):

[0252] Among them, DS s (ds) represents the micro-Doppler data that is numerically equal to ds, where ds represents the normalized value that varies according to the traversal step size.

[0253] S3.2.3: Calculate the values ​​of microDoppler data smaller than TH. d DS s The weighted average M g (TH d ), greater than TH in micro-Doppler data d DS s The weighted average M G (TH d ):

[0254]

[0255] S3.2.4: Calculate the values ​​of microDoppler data smaller than TH. d DS s The average value Avt0(TH) d ), greater than TH in micro-Doppler data d DS s The average value of Avr1(TH) d ):

[0256]

[0257] S3.2.5: Calculate the inter-class variance σ 2 :

[0258] S3.2.6: For σ 2 Seeking TH d The partial derivative of zp is used to calculate the local maximum and zero point zp.

[0259]

[0260] Micro-Doppler data DTh satisfies: DTh = zp;

[0261] The effective acquisition time threshold for the point cloud sequence obtained from the normalized value of the point cloud sequence includes:

[0262] S3.3.1: Calculate the traversal threshold TH of the point cloud sequence p Threshold traversal step size for point cloud sequences p :

[0263] TH p =min(PS)

[0264] Where, ε l Indicates the number of times the point cloud sequence has been traversed;

[0265] S3.3.2: Initialize PS s Less than TH p Frame rate Np0, PS s Greater than TH p The number of frames Np1 and PD is less than TH p All PS s The sum of Sp0 and PD is greater than TH p All PS s The sum of Sp1, and 1:

[0266] Np0=0,Sp0=0,Np1=0,Sp1=0,l=1

[0267] S3.3.3:

[0268]

[0269] S3.3.4: Determine if l is less than or equal to L. If yes, return to step S3.3.3; otherwise, proceed to step S3.3.5.

[0270] S3.3.5: Let l = l + 1;

[0271] S3.3.6: Calculate the proportions of Np0 (pp0), Np1 (pp1), and Sp0, and their mean values. Mean of Sp1

[0272] S.3.3.7: Calculation in TH p Under the given conditions, the inter-class variance of PD:

[0273] S3.3.8: Judgment Is it more than If yes, proceed to step S3.3.9; otherwise, proceed to step S3.3.4.

[0274] S3.3.9: Calculate the effective acquisition time threshold PTh = TH for the point cloud sequence. p

[0275] This embodiment fuses the modal features of video data, micro-Doppler data, and point cloud sequences to obtain multimodal fusion features, reducing the impact of single-modal data defects on gesture recognition results and achieving complementary advantages between various modalities. This enables the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, obtained from the multimodal fusion features, to have higher accuracy in complex application scenarios.

[0276] Example 3

[0277] This embodiment further discloses information based on Embodiment 2:

[0278] like Figure 3 In step S4, a unified effective acquisition time is calculated based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequences. The video data, micro-Doppler data, and point cloud sequences are then truncated according to this unified effective acquisition time to obtain the effective video acquisition data, effective micro-Doppler acquisition data, and effective point cloud sequence acquisition data, including:

[0279] S4.1: Based on the effective acquisition time threshold of the acquired video data, divide the effective acquisition time of the video data into segments:

[0280] Among them, VT begin Indicates the valid start time of video data acquisition, VT end ΔT represents the effective end time of video data acquisition. v This indicates the time interval for transmitting video data frames.

[0281] Based on the effective acquisition time threshold of the acquired micro-Doppler data, the effective acquisition time of micro-Doppler data is divided as follows:

[0282] Among them, DT begin DT represents the effective start time of micro-Doppler data acquisition. end ΔT represents the effective acquisition end time of micro-Doppler data. d This indicates the time interval for transmitting micro-Doppler data frames.

[0283] Based on the effective acquisition time threshold of the acquired point cloud sequence, the effective acquisition time of the point cloud sequence is divided as follows:

[0284]

[0285] Among them, PT begin PT represents the valid start time of point cloud acquisition. end ΔT represents the effective acquisition end time of the point cloud sequence. p This indicates the time interval for transmitting point cloud sequence frames.

[0286] S4.2: According to VT begin VT end DT begin DT end PT begin PT end Unified effective sampling time for the three modes:

[0287]

[0288] Among them, T begin T represents the unified and valid start time of data collection. end T represents the unified valid data collection end time. new This indicates a unified valid data collection time interval;

[0289] S4.3: According to T new The VD, DD, and PD data are extracted, and finally the valid video data V is obtained. row Effective data D of micro-Doppler data row Valid data P of point cloud sequence row :

[0290]

[0291] Where k = 1, 2, ..., K are group indices. V represents row The Kth group of data, D represents row The Kth group of data, P represents row In the Kth group of data, V rowFor valid video data, D row For valid micro-Doppler data, P row This is valid data for the point cloud sequence.

[0292] like Figure 4 As shown, in step S5, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, the video modal features, micro-Doppler modal features, and point cloud sequence modal features are fused to obtain multimodal fusion features, including:

[0293] S5.1: Divide the valid video data, valid micro-Doppler data, and valid point cloud sequence data into K groups, with each group represented as follows:

[0294]

[0295] Initialize k and the total loss value ΔW: k = 1, ΔW = 0

[0296] S5.2: For each set of data, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into the corresponding feature extraction network, and the feature matrices of the effective video data, effective micro-Doppler data, and effective point cloud sequence data are obtained respectively.

[0297] S5.3: Calculate VM i×j and DM i×j The basic product between, DM i×j and PM i×j The basic product between, DM i×j and PM i×j The fundamental product between them, where VM i×j DM is the feature matrix of the effective video data. i×j The feature matrix of effective micro-Doppler data, PM i×j The feature matrix of the valid data for the point cloud sequence:

[0298] Hvd i，j =VM i，j ×DM i，j

[0299] Hvp i,j =VM i，j ×PM i，j

[0300] Hdp i,j =DM i，j ×PM i，j

[0301] Where i = 1, 2, ..., I represents the row index of the feature matrix, and j = 1, 2, ..., J represents the column index of the feature matrix;

[0302] S5.4: Calculate the combination matrix G i，j :G i，j =Hvd i，j +Hvp i，j +Hdp i，j

[0303] S5.5: Computing VM i×j tensor product Kv p,q DM i×j tensor product Kd p,q PM i×j tensor product Kp p,q G i×j tensor product KG p,q :

[0304]

[0305] in, Let VD represent the eigentranspose matrix. This represents the eigentranspose matrix of DD. This represents the eigentranspose matrix of PD. G represents i，j The transpose of the tensor product is given by p = 1, 2, ..., where P represents the row index of the tensor product and q = 1, 2, ..., where Q represents the column index of the tensor product.

[0306] S5.6: Calculate VM i×j With and G i，j The loss values ​​between Wv and DM i×j With and G i，j The loss values ​​between Wd and PM i×j With and G i,j The loss value Wp between:

[0307]

[0308] S5.7: Calculate ΔW:

[0309]

[0310] Where δ represents the loss linearity factor, u = 1, 2, ..., U represents the number of categories, p(u) represents the predicted value of the u-th class, and y(u) represents the actual value of the u-th class;

[0311] S5.8: Determine if ΔW is less than the loss threshold W th If yes, proceed to step S5.9; otherwise, proceed to step S5.2.

[0312] S5.9: Input the combined matrix into the second feature network to extract dynamic temporal features;

[0313] S5.10: Determine if k is greater than K. If yes, proceed to step S5.12; otherwise, proceed to step S5.11.

[0314] S5.11: Let k = k + 1, and execute step S5.1;

[0315] S5.12: Input the dynamic temporal features obtained from all groups into the fully connected layer to obtain multimodal fusion features.

[0316] like Figure 5 As shown in step S6, a gesture classification network is constructed. The multimodal fusion features are input into the gesture classification network for training, resulting in a lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, including:

[0317] S6.1: Construct a gesture classification network to classify multimodal fusion features, where group C is used as training data. c Group Z is used as categorical data. z Set the initial preweight matrix WM for the gesture classification network. all Initialize the initial preweight index φ:

[0318]

[0319] WM all =[WM1,WM2,..,WM φ ,..,WM Φ-1 WM Φ ]

[0320] φ=1

[0321] in, The i×j-th data point of the c-th training data set. Let Wm represent the i×j-th data point in the z-th classification dataset, where c = 1, 2, ..., C represents the training data index, z = 1, 2, ..., Z represents the classification data index. φ This represents the φ-th initial preweight;

[0322] S6.2: Input the training data into the gesture classification network for one round of training;

[0323] S6.3: After this round of training is completed, determine whether φ is greater than Φ. If yes, execute S6.5; otherwise, execute S6.4.

[0324] S6.4: Let φ = φ + 1, then return to step S6.2;

[0325] S6.5: Obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud;

[0326] Furthermore, such as Figure 6 As shown in step S6.2, inputting the training data into the gesture classification network for one round of training includes:

[0327] S6.2.1: Initialize Train c The initial preweights Wm used in this round T Number of iterations (num):

[0328]

[0329] num=1

[0330] in, This represents the transpose of the φ-th initial preweight;

[0331] S6.2.2: Initialize c to 1, and initialize the statistics flag mark to 0;

[0332] S6.2.3: Initialize u to 1, and initialize the flag category to u:

[0333] S6.2.4: Calculate the discriminant function value r obtained in the num-th iteration for the u-th category. u (num), and the discriminant function value r of flag. flag (num):

[0334]

[0335] S6.2.5: Determine r flag Is (num) less than r? u If (num) is true, proceed to step S6.2.6; otherwise, proceed to step S6.2.7.

[0336] S6.2.6: Set mark = mark + 1, then execute S6.2.8;

[0337] S6.2.7: Calculate the weight matrix Wm for the next round of labeling categories. flag (num+1), the weight matrix Wm corresponding to the category in this round in the next round. u (num+1):

[0338] Wm flag (num+1)=Wm flag (num)+μTrain c

[0339] Wmu (num+1)=Wm u (num)-μTrain c

[0340] Among them, Wm falg (num) represents the weight matrix of the label categories in this round, Wm u (num) represents the weight matrix of class u in this round, μ represents the weight modification factor and μ>0;

[0341] S6.2.8: Determine if u is less than U. If yes, proceed to step S6.2.9; otherwise, proceed to step S6.2.10.

[0342] S6.2.9: Let u = u + 1 and flag = u, then execute step S6.2.4;

[0343] S6.2.10: Determine if c is less than C. If yes, proceed to step S6.2.11; otherwise, proceed to step S6.2.12.

[0344] S6.2.11: Let c = c + 1, num = num + 1, mark = mark + 1, and execute step S6.2.3;

[0345] S6.2.12: Determine if mark is equal to C. If yes, proceed to step S6.2.14; otherwise, proceed to step S6.2.13.

[0346] S6.2.13: Let nu, = num + 1, and execute step S6.2.2;

[0347] S6.2.14: Obtain information on using Wm T The gesture classification hyperplane obtained under the given conditions

[0348] Complete the training.

[0349] like Figure 7 As shown, in step S7, gesture information is acquired, and the recognition result is obtained through the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud based on the gesture information. The gesture recognition is completed by: S7.1: Initializing Sort z Classification result frequency statistics vector res:

[0350] Among them, res u This represents the number of times a data point is classified into class u. For example, U = 16, res = [0, 0, ..., 0], len(res) = 16; S7.2: Initialize u and φ to 1; S7.3: Calculate the discriminant function value for class u. In this embodiment, φ = 7, u = 8, z = 24. S7.4: Judgment Is it greater than 0? If yes, proceed to S7.5; otherwise, proceed to S7.6. S7.5: Calculate the statistical value res of the classification result of class u. u :res u =res u +1 Execute S7.8; S7.6: Determine if u is less than U. If yes, execute S7.7; otherwise, execute S7.8; S7.7: Set u = u + 1, execute S7.3; S7.8: Determine if φ is less than Φ. If yes, execute S7.9; otherwise, execute S7.10; S7.9: Set φ = φ + 1, execute S7.2;

[0351] S7.10: Calculate the classification index of the classification result:

[0352] result = index[max(res)]

[0353] In this embodiment, max(res) = 9 corresponds to the third type of gesture, so result = index(9) = 3.

[0354] This embodiment fuses the modal features of video data, micro-Doppler data, and point cloud sequences to obtain multimodal fusion features, reducing the impact of single-modal data defects on gesture recognition results and achieving complementary advantages between various modalities. This enables the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, obtained from the multimodal fusion features, to have higher accuracy in complex application scenarios.

[0355] In summary, this invention provides a lightweight multimodal fusion gesture recognition method based on Rayvision Cloud. It includes: acquiring video data, micro-Doppler data, and point cloud sequences; calculating the normalized values ​​and effective acquisition time thresholds for the three modalities, and dividing the effective acquisition time into uniform periods; inputting the effective acquisition data of the three modalities into corresponding feature extraction networks to obtain modal features and then performing feature fusion; and inputting the acquired multimodal fusion features into a gesture classification network for training. By using video data, micro-Doppler data, and point cloud sequences as modal data, the impact of single-modal data defects on gesture recognition results is reduced, achieving complementary advantages among various modalities and improving the accuracy of gesture recognition in complex application scenarios.

[0356] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A lightweight multimodal fusion gesture recognition method based on Rayvision Cloud, characterized in that, Includes the following steps: S1: Acquire video data, micro-Doppler data, and point cloud sequences; S2: Calculate the normalized values ​​of video data, micro-Doppler data, and point cloud sequence respectively to obtain the normalized values ​​of video data, micro-Doppler data, and point cloud sequence. S3: Based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, obtain the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence. S4: Calculate a unified effective acquisition time based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence. Extract the video data, micro-Doppler data, and point cloud sequence according to the unified effective acquisition time to obtain effective video data, micro-Doppler acquisition data, and effective point cloud sequence data. S5: Input the effective video data, effective micro-Doppler data, and effective point cloud sequence data into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, perform feature fusion on the video modal features, micro-Doppler modal features, and point cloud sequence modal features to obtain multimodal fusion features. S6: Construct a gesture classification network, input multimodal fusion features into the gesture classification network for training, and obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud; S7: Obtain gesture information, and obtain the recognition result based on the gesture information through the lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, thus completing gesture recognition.

2. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud as described in claim 1, characterized in that, In step S1, the video data, micro-Doppler data, and point cloud sequence include: The video data is represented as follows: VD=[VD1,VD2,...,VD n ,...,VD N ], Where VD represents video data, VD n Indicates the nth frame VD, This represents the coordinates of the m-th keypoint in the n-th frame of video data, where m = 1, 2, ..., M represents the keypoint index, and n = 1, 2, ..., N represents the frame index of the video data. Micro-Doppler data are represented as follows: DD=[DD1,DD2,...,DD s ,...,DD S ], Where DD represents micro-Doppler data, DD s This represents the S-th frame of micro-Doppler data, where s = 1, 2, ..., and S represents the frame index of the micro-Doppler data. DD s The element value at index (rg, dp) is obtained by FFT. D (rg, dp) represents the two-dimensional frequency domain transform of the intermediate frequency signal, where rg and dp represent the two dimensions of the micro-Doppler data, and rg = 1, 2, ..., RG, dp = 1, 2, ..., DP. (FFT) R (r g n c ) represents the one-dimensional frequency domain transform of the intermediate frequency signal, n c =1,2,...,N c Indicates the chirp signal index; The point cloud sequence is represented as follows: PD=[PD1,PD2,...,PD l ,...,PD L ], Where PD represents the point cloud sequence, PD l This represents the point cloud sequence of the j-th frame. Let H represent the h-th coordinate point in the j-th frame point cloud sequence, where h = 1, 2, ..., H represents the index of the coordinate point, and l = 1, 2, ..., L represents the frame index of the point cloud sequence.

3. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud as described in claim 1, characterized in that, In step S2, calculating the normalized value of the video data includes: S2.1.1: Initialize m to 1; S2.1.2: Calculation Distance between two adjacent frames in, This represents the dimension value of the m-th keypoint in the n-th frame of VD. This represents the dimension value of the m-th keypoint in the (n+1)-th frame of video data. S2.1.3: Calculation of mean : S2.1.4: Calculate VD n superior normalized value in, Represents all of the m-th key points The maximum value, Represents all of the m-th key points The minimum value; S2.1.5: Calculate the normalized value of the video data:

4. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S2, calculating the normalized value of the micro-Doppler data includes: S2.2.1: rg, dp, n c All are initialized to 1; S2.2.2: Calculation of micro-Doppler data mean S2.2.3: Calculation normalized value in, express The maximum value, express The minimum value; S2.2.4: Calculate the normalized value of the micro-Doppler data:

5. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S2, calculating the normalized value of the point cloud sequence includes: S2.3.1: Initialize h to 1 S2.3.2: Calculation Distance between two adjacent frames : in, This represents the dimension value of the h-th coordinate point of the PD in the l-th frame. This represents the dimension value of the h-th coordinate point in the (l+1)-th frame of the point cloud sequence; S2.3.3: Calculation Mean distance : S2.4.4: Calculation normalized value ; in, Represents all coordinates of the h-th point The maximum value, Represents all coordinates of the h-th point The minimum value; S2.3.5: Calculate the normalized value of PD:

6. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S3, based on the normalized values ​​of video data, micro-Doppler data, and point cloud sequence, the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequence are obtained, including: The effective acquisition time threshold for video data, derived from the normalized values ​​of the video data, includes: S3.1.1: Calculate the traversal threshold TH for video data. v Threshold traversal step size for video data v : TH v =min(VS n ) Where, ε v Indicates the number of times the video data has been traversed; S3.1.2: Initialize VD n Less than TH v Frame count Nv0, VD n Greater than TH v The number of frames Nv1, and all VDs in the video data less than VTh. n All VDs greater than VTh in Sv0 and VD n Sv1 and n: Nv0=0,Sv0=0,Nv1=0,Sv1=0,n=1 S3.1.3： S3.1.4: Determine if n is less than or equal to N. If yes, return to step S3.1.3; otherwise, proceed to step S3.1.

5. S3.1.5: Let n = n + 1; S3.1.6: Calculate the proportion of Nv0 (pv0), the proportion of Nv1 (pv1), and the mean of Sv0. The mean of Sv1 : S3.1.7: Calculation in TH v Under the condition, the inter-class variance of VD : S3.1.8: Judgment Is it more than If yes, proceed to step S3.1.9; otherwise, proceed to step S3.1.

4. S3.1.9: Calculate the effective acquisition time threshold VTh = TH for video data. v The effective acquisition time threshold for micro-Doppler data, derived from the normalized value of micro-Doppler data, includes: S3.2.1: Set the effective acquisition time threshold variable TH for micro-Doppler data. d Calculate the effective acquisition time threshold DTh and the threshold traversal step size for micro-Doppler data. d : DTh=TH d Where, ε d Indicates the number of times the microDoppler data has been traversed; S3.2.2: Calculate DS s Less than TH d The proportion of frames p0(TH) d ), DS s Greater than TH d The proportion of frames p1(TH) d ): Among them, DS s (ds) represents the micro-Doppler data that is numerically equal to ds, where ds represents the normalized value that varies according to the traversal step size. S3.2.3: Calculate the values ​​of microDoppler data smaller than TH. d DS s The weighted average M g (TH d ), greater than TH in micro-Doppler data d DS s The weighted average M G (TH d ): S3.2.4: Calculate the values ​​of microDoppler data smaller than TH. d DS s The average value Avr0(TH) d ), greater than TH in micro-Doppler data d DS s The average value of Avr1(TH) d ): S3.2.5: Calculate the inter-class variance σ 2 : S3.2.6: For σ 2 Seeking TH d The partial derivative of zp is used to calculate the local maximum and zero point zp. Micro-Doppler data DTh satisfies: DTh = zp; The effective acquisition time threshold for the point cloud sequence obtained from the normalized value of the point cloud sequence includes: S3.3.1: Calculate the traversal threshold TH of the point cloud sequence p Threshold traversal step size for point cloud sequences p : TH p =min(PS l ) Where, ε l Indicates the number of times the point cloud sequence has been traversed; S3.3.2: Initialize PS s Less than TH p Frame rate Np0, PS s Greater than TH p The number of frames Np1 and PD is less than TH p All PS s The sum of Sp0 and PD is greater than TH p All PS s The sum of Sp1, and 1: Np0=0,Sp0=0,Np1=0,Sp1=0,l=1 S3.3.3： S3.3.4: Determine if l is less than or equal to L. If yes, return to step S3.3.3; otherwise, proceed to step S3.3.

5. S3.3.5: Let l = l + 1; S3.3.6: Calculate the proportions of Np0 (pp0), Np1 (pp1), and Sp0, and their mean values. The mean of Sp1 : S.3.3.7: Calculation in TH p Under the given conditions, the inter-class variance of PD: S3.3.8: Judgment Is it more than If yes, proceed to step S3.3.9; otherwise, proceed to step S3.3.

4. S3.3.9: Calculate the effective acquisition time threshold PTh = TH for the point cloud sequence. p .

7. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S4, a unified effective acquisition time is calculated based on the effective acquisition time thresholds for video data, micro-Doppler data, and point cloud sequences. The video data, micro-Doppler data, and point cloud sequences are then truncated according to this unified effective acquisition time to obtain the effective acquisition data for video, micro-Doppler, and point cloud sequences, including: S4.1: Based on the effective acquisition time threshold of the acquired video data, divide the effective acquisition time of the video data into segments: Among them, VT begin Indicates the valid start time of video data acquisition, VT end ΔT represents the effective end time of video data acquisition. v Indicates the time interval for transmitting video data frames; Based on the effective acquisition time threshold of the acquired micro-Doppler data, the effective acquisition time of micro-Doppler data is divided as follows: Among them, DT begin DT represents the effective start time of micro-Doppler data acquisition. end ΔT represents the effective acquisition end time of micro-Doppler data. d Indicates the time interval for the transmission of micro-Doppler data frames; Based on the effective acquisition time threshold of the acquired point cloud sequence, the effective acquisition time of the point cloud sequence is divided as follows: Among them, PT begin PT represents the valid start time of point cloud acquisition. end ΔT represents the effective acquisition end time of the point cloud sequence. p This indicates the time interval for transmitting point cloud sequence frames; S4.2: According to VT begin VT end DT begin DT end PT begin PT end Unified effective sampling time for the three modes: Among them, T begin T represents the unified and valid start time of data collection. end T represents the unified valid data collection end time. new This indicates a unified valid data collection time interval; S4.3: According to T new The VD, DD, and PD data are extracted, and finally the valid video data V is obtained. row Effective data D of micro-Doppler data row Valid data P of point cloud sequence row : Where k = 1, 2, ..., K is the group index. V represents row The Kth group of data, D represents row The Kth group of data, P represents row In the Kth group of data, V row For valid video data, D row For valid micro-Doppler data, P row This is valid data for the point cloud sequence.

8. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S5, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into their respective feature extraction networks to obtain video modal features, micro-Doppler modal features, and point cloud sequence modal features. Then, the video modal features, micro-Doppler modal features, and point cloud sequence modal features are fused to obtain multimodal fused features, including: S5.1: Divide the valid video data, valid micro-Doppler data, and valid point cloud sequence data into K groups, with each group represented as follows: Initialize k and the total loss value ΔW: k = 1, ΔW = 0 S5.2: For each set of data, the effective video data, effective micro-Doppler data, and effective point cloud sequence data are respectively input into the corresponding feature extraction network, and the feature matrices of the effective video data, effective micro-Doppler data, and effective point cloud sequence data are obtained respectively. S5.3: Calculate VM i×j and DM i×j The basic product between, VM i×j and PM i×j The basic product between, DM i×j and PM i×j The fundamental product between them, where VM i×j DM is the feature matrix of the effective video data. i×j The feature matrix of effective micro-Doppler data, PM i×j The feature matrix of the valid data for the point cloud sequence: Hvd i，j =World Cup i，j ×DM i,j Hvp i，j =VM i,j ×PM i，j Hdp i,j =DM i,j ×PM i，j Where i = 1, 2, ..., I represents the row index of the feature matrix, and j = 1, 2, ..., J represents the column index of the feature matrix; S5.4: Calculate the combination matrix G i，j :G i，j =Hvd i，j +Hvp i，j +Hdp i，j S5.5: Computing VM i×j tensor product Kv p，q DM i×j tensor product Kd p，q PM i×j tensor product Kp p，q G i×j tensor product KG p，q : in, Let VD represent the eigentranspose matrix. This represents the eigentranspose matrix of DD. This represents the eigentranspose matrix of PD. G represents i，j The transpose of the tensor product is given by p = 1, 2, ..., where P represents the row index of the tensor product and q = 1, 2, ..., where Q represents the column index of the tensor product. S5.6: Calculate VM i×j With and G i，j The loss values ​​between Wv and DM i×j With and G i,j The loss values ​​between Wd and PM i×j With and G i，j The loss value Wp between: S5.7: Calculate ΔW: Where δ represents the loss linearity factor, u = 1, 2, ..., U represents the number of categories, p(u) represents the predicted value of the u-th class, and y(u) represents the actual value of the u-th class; S5.8: Determine if ΔW is less than the loss threshold W th If yes, proceed to step S5.9; otherwise, proceed to step S5.

2. S5.9: Input the combined matrix into the second feature network to extract dynamic temporal features; S5.10: Determine if k is greater than K. If yes, proceed to step S5.12; otherwise, proceed to step S5.

11. S5.11: Let k = k + 1, and execute step S5.1; S5.12: Input the dynamic temporal features obtained from all groups into the fully connected layer to obtain multimodal fusion features.

9. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 1, characterized in that, In step S6, a gesture classification network is constructed. The multimodal fusion features are input into the gesture classification network for training, resulting in a lightweight multimodal fusion gesture recognition model based on Rayvision Cloud, including: S6.1: Construct a gesture classification network to classify multimodal fusion features, where group C is used as training data. c Group Z is used as categorical data. z Set the initial preweight matrix WM for the gesture classification network. all Initialize the initial preweight index φ: W.M. all =[WM1,WM2,...,WM φ ,...,WM Φ-1 ,WM Φ ] φ=1 in, The i×j-th data point of the c-th training data set. Let C represent the i×j-th data point in the z-th classification dataset, where c = 1, 2, ..., C represents the training data index, z = 1, 2, ..., Z represents the classification data index, and Wm φ This represents the φ-th initial preweight; S6.2: Input the training data into the gesture classification network for one round of training; S6.3: After this round of training is completed, determine whether φ is greater than Φ. If yes, execute S6.5; otherwise, execute S6.

4. S6.4: Let φ = φ + 1, then return to step S6.2; S6.5: Obtain a lightweight multimodal fusion gesture recognition model based on Leishi Cloud.

10. The lightweight multimodal fusion gesture recognition method based on Rayvision Cloud according to claim 9, characterized in that, In step S6.2, inputting the training data into the gesture classification network for one round of training includes: S6.2.1: Initialize Train c The initial preweights Wm used in this round T Number of iterations (num): num=1 in, This represents the transpose of the φ-th initial preweight; S6.2.2: Initialize c to 1, and initialize the statistics flag mark to 0; S6.2.3: Initialize u to 1, and initialize the flag category to u: S6.2.4: Calculate the discriminant function value r obtained in the num-th iteration for the u-th category. u (num), and the discriminant function value r of flag. flag (num): S6.2.5: Determine r flag Is (num) less than r? u If (num) is true, proceed to step S6.2.6; otherwise, proceed to step S6.2.

7. S6.2.6: Set mark = mark + 1, then execute S6.2.8; S6.2.7: Calculate the weight matrix Wm for the next round of labeling categories. flag (num+1), the weight matrix Wm corresponding to the category in this round in the next round. u (num+1): Wm flag (num+1)=Wm flag (num)+μTrain c Wm u (num+1)=Wm u (num)-μTrain c Among them, Wm flag (num) represents the weight matrix of the label categories in this round, Wm u (num) represents the weight matrix of class u in this round, μ represents the weight modification factor and μ>0; S6.2.8: Determine if u is less than U. If yes, proceed to step S6.2.9; otherwise, proceed to step S6.2.

10. S6.2.9: Let u = u + 1 and flag = u, then execute step S6.2.4; S6.2.10: Determine if c is less than C. If yes, proceed to step S6.2.11; otherwise, proceed to step S6.2.

12. S6.2.11: Let c = c + 1, num = num + 1, mark = mark + 1, and execute step S6.2.3; S6.2.12: Determine if mark is equal to C. If yes, proceed to step S6.2.14; otherwise, proceed to step S6.2.

13. S6.2.13: Let num = num + 1, and execute step S6.2.2; S6.2.14: Obtain information on using Wm T The gesture classification hyperplane obtained under the given conditions Complete the training.

Images