Anti-interference gesture recognition method, system, device and medium based on memory network and generative adversarial network

Abstract

The application discloses an anti-interference gesture recognition method, system, device and medium based on a memory network and a generative adversarial network. The method comprises the following steps: first, a memory network is established, gesture features of different samples are written into different memory slots according to similarity, gesture invariants are obtained through memory reading, and the gap between different samples and the gesture invariants is reduced, so that the interference of obvious factors irrelevant to the gesture can be avoided; second, a cross-modal cycle is constructed based on the generative adversarial network, features of each mode are decomposed into mode invariants and mode specific features, the mode invariants and the mode specific features are utilized to realize cross-modal reconstruction, so as to promote information complementation between different modes; finally, the obtained gesture invariant features and mode features are jointly input into a gesture recognition network for anti-interference gesture recognition. The system, device and medium can perform anti-interference gesture recognition based on the memory network and the generative adversarial network, and can improve recognition accuracy and recognition efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of gesture recognition, and particularly relates to an anti-interference gesture recognition method, system, device and medium based on a memory network and a generative adversarial network. BACKGROUND

[0002] At present, gesture recognition based on video has been widely applied in video monitoring, intelligent transportation and the most common human-computer interaction. And with the appearance of RGB-D devices such as Kinect, multi-modal data can be captured at the same time, which is helpful for accurate recognition of gestures. Although great progress has been made in this field, the existing technology has always focused on how to distinguish different gesture categories, while ignoring the influence of gesture-independent factors such as lighting, background and appearance of people on the internal differences of the same gesture category. At the same time, for multi-modal gesture recognition, current methods generally choose to perform feature or score fusion in the last stage to combine different modal information, so that the posture-related features in different modal information may be redundant, and the complementarity of the modal is not fully utilized.

[0003] Patent application CN107025442A discloses a multi-modal fusion gesture recognition method based on color and depth information, the steps of which include: (1) using a color camera and a depth camera to simultaneously capture gestures to obtain color and depth raw data of the gestures; (2) using a Gaussian smoothing method to filter the color and depth raw data to obtain smoothed color and depth data; (3) using skin color and depth as prior knowledge to segment the smoothed color and depth data to obtain skin color segmentation regions and depth segmentation regions; (4) using dynamic weights to perform pixel layer fusion of the skin color segmentation regions and the depth segmentation regions to form a fusion segmentation map; (5) using a feature descriptor to extract features from the fusion segmentation map to form a feature map; (6) encoding the feature map to form a gesture code; and (7) recognizing the gesture code to realize recognition of the gesture.

[0004] Patent application CN108388882A discloses a gesture recognition method based on global-local RGB-D multi-modal, which mainly represents input gesture video through data modalities including bone position, RGB image, depth image and optical flow image, obtains multi-modal gesture data representation, uses convolutional neural network and recurrent neural network to express features of different modal gesture data, and uses features obtained under different modalities to classify gestures. Finally, different gesture scores obtained under different modalities are fused to obtain the final multi-modal gesture classification result. The invention can be applied to client or cloud to recognize user input gesture video, and through gesture input, computer or mobile phone software and hardware make corresponding response.

[0005] The existing technologies also have some problems that have not been solved. First, they do not take into account the interference of gesture-independent factors. In many real-world scenarios, there are two aspects of video changes: one is based on the motion of the gesture itself, which indicates the trajectory of the hand and arm; the other is the appearance change in the scene, which is related to environmental factors such as lighting, background, and appearance of the performer. The state of the performer when demonstrating the gesture, such as their speed, understanding of the gesture, and the location of their attention, can also affect the quality of the gesture demonstration. Although these appearance-based changes are not related to the gesture, they still cause internal class differences for each gesture and hinder their recognition performance.

[0006] In addition, current multi-modal gesture recognition technologies all use modal data such as RGB, depth, infrared, and optical flow to improve recognition accuracy. With respect to the interference of gesture-independent factors in different samples, modal-specific factors have a positive effect on recognition ability. For example, RGB data contains many texture details, which helps to distinguish subtle gesture differences, while depth data is only related to the distance between the camera and the object, so it can resist the interference of complex backgrounds. However, existing technologies generally choose to perform feature or score fusion in the last stage to combine different modal information, but the high-level features in the last stage have not been fully explored in terms of whether they can fully utilize the complementarity of different modalities. SUMMARY

[0007] To overcome the problems of the existing technologies described above, the purpose of the present application is to disclose an anti-interference gesture recognition method, system, device, and medium based on a memory network and a generative adversarial network. First, a memory network is established, and gesture features of different samples are written into different memory slots according to similarity. By reading the memory, gesture invariants are obtained, and the gap between different samples and gesture invariants is reduced, which can avoid obvious interference of gesture-independent factors. Second, a cross-modal cycle is constructed based on GAN (Generative Adversarial Network), and the features of each modality are decomposed into modal invariants and modal-specific features. The modal invariants and modal-specific features are used to realize cross-modal reconstruction, thereby promoting the information complementarity between different modalities. Finally, the obtained gesture invariant features and modal features are input into a gesture recognition network for anti-interference gesture recognition. This effectively solves the problem of interference of gesture-independent factors on recognition performance and fully utilizes the complementarity of different modal information to improve recognition accuracy.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] An anti-environmental interference gesture recognition method based on a memory network and a generative adversarial network, the specific operation steps are as follows:

[0010] Step 1: Extract gesture invariants through a memory network to achieve anti-environmental interference.

[0011] 1) Input different modal gesture data: Input different modal gesture data, including RGB and depth video data, into two video feature extractors at the same time to obtain the current query sample features. Where l represents the category of gesture;

[0012] 2) Establish a memory library and extract gesture invariants;

[0013] 3) Solve the weighted parameters

[0014] 4) Make q l and Align in semantic space; after multiple iterations, gradually reduce the difference between different memory slots to obtain the true gesture invariant

[0015] Step 2: Generate adversarial networks to implement cross-modal loops to fully acquire modal information.

[0016] 5) Obtaining encoded features in the attribute embedding space;

[0017] 6) Get the modal invariant g c ;

[0018] 7) Realize the loop between different modalities through cross-modal generation and back-propagation;

[0019] In step 3, the gesture invariant features obtained in step 1 and the modal features obtained in step 2 are input into the gesture recognition network for anti-interference gesture recognition.

[0020] The specific method of step 1) step 2 is:

[0021] 2.1) Design a memory bank M of size n×m×l, where m represents the number of memory slots for each type of gesture, n represents the length of the feature vector, and l represents the type of gesture.

[0022] 2.2) Memory Writing

[0023] 2.2a) First check the memory slot position indicated by the cursor variable. If the indicated memory slot is empty, you can directly transfer the current sample feature to the memory slot. Save to this memory slot;

[0024] 2.2b) If there is no free memory slot, calculate the current sample feature The similarity between the features of each memory slot and use it as the weight when writing to the memory;

[0025] 2.2c) Use the erasure matrix to ignore certain features in the memory slot. The erasure steps can be expressed as:

[0026]

[0027] where M l is the memory slot of gesture class l, t and t-1 represent the iteration number of the optimization process; e t is an erasing matrix composed of n feature vectors corresponding to n memory slots, each of which has m elements randomly generated from [0, 1] to ensure the diversity of the erasing operation;

[0028] 2.2d) After the completion of the erasing step, the current sample feature is written into the memory slot:

[0029]

[0030] where w is a weight matrix of shape n x m; i ∈ {1, 2, …, n} corresponds to the i-th vector containing m identical elements and the i-th memory slot; meanwhile, a constraint is imposed on the weight matrix w:

[0031]

[0032] 2.3) Memory reading

[0033] The memory reading is the weighted sum of the features in the memory slot:

[0034]

[0035] where, denotes the intermediate-level gesture invariant derived at iteration t

[0036] The specific method of step 1 3) is:

[0037] A hierarchical weighting mechanism is adopted, including content-based weight prediction and contextual weight prediction, to provide comprehensive weights for gesture invariant extraction;

[0038] 3.1) Measure the cosine similarity between the current sample feature and the features in each memory slot , update the memory bank using at each iteration; define the content-based weight prediction as

[0039]

[0040] where cossim(·,·) denotes the cosine similarity measure; for two matrices u and v, this measure can be expressed as:

[0041]

[0042] 3.2) According to the memory bank established in step 1, the context weight prediction is introduced:

[0043]

[0044] Where γ is a flag parameter, set to 1 when reading memory, and set to 0 when writing memory;

[0045] 3.3) When the gap between gesture samples is too large, the is regarded as a one-hot encoding, which may cause the weakening of the memory network and even cause the problem of non-differentiable, by applying adjacent time slot weights, the context weight is balanced, as follows:

[0046]

[0047] Where b Ω (i) is a bell-shaped balancing function; Similarly, a temperature parameter is also introduced to control the concentration level of the distribution, that is, a temperature factor τ for amplifying the focus degree on each memory slot, and the final weight is:

[0048]

[0049] The specific method of step 1, 4) is:

[0050] Through training network, q l is aligned in semantic space, and the loss function is as follows:

[0051] Where,

[0052] The Euclidean distance is measured, The KL divergence is measured; λ L2 and λ KLD are used to balance the weights of the two parts; At the same time, W is introduced, so as to establish the memory bank before aligning the query sample and the gesture invariant, which can be represented as: W = max(0, (1-exp(-E+ε)))

[0053] Where E represents the current training period, and ε is the preheating hyperparameter confirming

[0054] the effective moment; After multiple iterations, the difference between different memory slots is gradually reduced, and the real gesture invariant

[0055] The specific method of step 2, 5) is:​

[0056] extracting single frame from RGB video data and depth video data to fully exploit the modality information complementarity; decoupling these two modalities data and respectively: modal data decoupled into modality-invariant features based on gestures themselves modal data decoupled into modality-specific attribute features firstly compute modality-specific attribute features

[0057]

[0058] where m denotes modality, v denotes input single frame; denotes attribute feature encoder based on Resnet-18; meanwhile, an additional classifier is adopted to distinguish which modality it belongs to so as to force correct mapping to corresponding modality.

[0059] The step 2, 6) is specifically as follows:

[0060] separate modality-specific attribute features from video features and input the remaining separation result into content encoder :

[0061]

[0062] where, denotes modality-specific attribute features separated from video features, which is realized by connecting and ; in addition, only denotes intermediate-level modality-invariant features, and modality-specific attribute features of different modalities are constantly approaching through back propagation, and finally obtain modality-invariant features g c .

[0063] The step 2, 7) is specifically as follows:

[0064] after obtaining modality-invariant g c , add another modality-specific attribute feature is the inverse operation of ; then, input the combined features into decoder D m to synthesize video features ​

[0065]

[0066] By cross-modal generation, Can be fused into the current modal branch, and affect Feature distribution, ultimately realize the cycle between different modalities.

[0067] A system based on the above anti-interference gesture recognition method, comprising:

[0068] Memory write module, by similarity strategy and erase strategy to memory slot write sample features to build gesture sample memory bank constantly;

[0069] Memory reading module, by weighting sum of the same kind of gesture memory slot to derive gesture invariant;

[0070] Weight calculation module, by combining content-based weight prediction and context weight prediction to provide comprehensive weight scheme for gesture invariant extraction;

[0071] Modal attribute extraction module, by attribute encoder extracts specific attribute features of two modalities and uses additional classifier to correctly map them;

[0072] Modal invariant extraction module, by content encoder separates modal specific attribute features in video features to obtain modal invariant features;

[0073] Cross-modal cycle module, by constructing an adversarial subnetwork to realize the cycle generation between different modal data.

[0074] A device based on the above anti-interference gesture recognition method, comprising:

[0075] Memory, for storing computer programs;

[0076] Processor, for executing the computer program to realize the anti-interference gesture recognition method based on memory network and generative adversarial network of any one of steps 1 to 3.

[0077] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to be able to carry out anti-interference gesture recognition based on memory network and generative adversarial network.

[0078] The present application has the following advantages:

[0079] 1. The present application can remove the influence of gesture-independent factors on gesture recognition. Gesture-independent factors refer to the internal differences between gestures of the same type, such as lighting, background, and appearance of the performer. By memory writing to build a gesture sample memory library, and then according to memory reading to solve the weighting of features in the memory library and finally obtain gesture invariants, the interference of gesture-independent factors is effectively avoided.

[0080] 2. The present application inputs different modal images into an attribute encoder to obtain modal-specific attribute features, separates modal-specific attribute features from video features to obtain modal invariants, and finally constructs a cross-modal cycle to fully utilize the information complementarity of different modal data to improve the precision of gesture recognition.

[0081] Compared with the prior art, the present application has the following advantages:

[0082] The present application does not directly encode the modal fusion feature map to obtain gesture codes, and does not need to save the original video samples in the data set as RGB-D image sequences, but directly takes the video as the input of the gesture feature extractor to obtain the query sample, uses a hierarchical weighting mechanism including content-based weight prediction and context weight prediction to construct a memory library to extract gesture invariants, thereby reducing the influence of gesture-independent factors and effectively improving the recognition accuracy.

[0083] The present application does not need to fuse gesture scores of different categories obtained under different modalities, but extracts different modal data: takes one frame from RGB video data and depth video data as input to construct a generative adversarial network to realize cross-modal cycle to fully utilize the complementarity of different modal data, further improves the recognition accuracy, and is more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 The present application is a schematic diagram of different gesture-independent factors corresponding to the same gesture.

[0085] Figure 2 The present application is a schematic diagram of the overall structure of the network.

[0086] Figure 3 The present application is a schematic diagram of the memory network structure.

[0087] Figure 4 The present application is a schematic diagram of the cross-modal cycle generation structure.

[0088] Figure 5 The present application is a schematic diagram of the complementary relationship between different modal data information. DETAILED DESCRIPTION

[0089] The present application will be further described in detail below with reference to the accompanying drawings.

[0090] The application discloses an environmental interference-resistant gesture recognition method based on a memory network and a generative adversarial network, and the specific operation steps are as follows:

[0091] Step 1 extracts gesture invariants through a memory network to realize environmental interference resistance; see Figures 1 to 3

[0092] 1) input different modal gesture data: input different modal gesture data, including RGB and depth video data, into two video feature extractors respectively to obtain current query sample features Wherein l represents the category of the gesture;

[0093] 2) define an initial memory bank as M, and the size of M is n x m x l, wherein m represents the number of memory slots for each gesture category, n represents the length of a feature vector, and n is set to 512; the establishment of the memory bank M includes two steps: memory writing and memory reading;

[0094] 2.1) design a memory bank M, and the size of the memory bank M is n x m x l, wherein m represents the number of memory slots for each gesture category, n represents the length of a feature vector, and l represents the category of the gesture;

[0095] 2.2) memory writing

[0096] 2.2a) first, check the memory slot position indicated by the cursor variable, and when the indicated memory slot is empty, the current sample feature can be directly saved to the memory slot;

[0097] 2.2b) if there is no idle memory slot, calculate the similarity between the current sample feature and the features in each memory slot, and use the similarity as the weight during memory writing;

[0098] 2.2c) use an erasing matrix to erase certain features in the memory slot, and the erasing step can be expressed as:

[0099]

[0100] Wherein, M l represents the memory slot of the gesture category l, t and t-1 represent the iteration number of the optimization process; e t is an erasing matrix composed of n feature vectors corresponding to n memory slots, each feature vector has m elements randomly generated from [0, 1], so as to ensure the diversity of the erasing operation;

[0101] 2.2d) after the erasing step is completed, the current sample feature is written into the memory slot:

[0102]

[0103] where w is a weight matrix with shape n x m; i ∈ {1, 2, …, n} corresponds to the ith vector containing m identical elements and the ith memory slot; meanwhile, a constraint is imposed on the weight matrix w:

[0104]

[0105] 2.3) Memory reading

[0106] Memory reading is a key step to extract gesture invariants, and the memory reading is the weighted sum of the features in the memory slots:

[0107]

[0108] where, denotes the intermediate-level gesture invariants derived at iteration t

[0109] 3) Adopting a hierarchical weighting mechanism, including content-based weight prediction and context weight prediction, to provide comprehensive weights for the extraction of gesture invariants;

[0110] 3.1) Measure the cosine similarity between the current sample feature and the features in each memory slot Update the memory bank using at each iteration; define the content-based weight prediction as

[0111]

[0112] where cossim(·,·) denotes the cosine similarity measure; for two matrices u and v, this measure can be expressed as:

[0113]

[0114] 3.2) Introduce context weight prediction based on the memory bank established in step 2): both the memory writing and memory reading processes use However, the weight solving approach in these two stages is inconsistent, i.e., in memory writing, which is mentioned in step 1, 2.2b), the similarity between the sample feature and the slot feature needs to be calculated to calculate the corresponding weight feature of the memory slot, thereby maintaining the diversity of the memory slot content; while the memory reading step is to extract gesture invariants, which should be as different as possible from each gesture sample feature, so that the extracted gesture invariants can carry less gesture-independent factors, therefore, read from the memory bank should be as different as possible from q lDifferent from the previous one, the gesture-irrelevant features are highlighted; therefore, the context weight prediction is defined as:

[0115]

[0116] Among them, γ is a flag parameter, which is set to 1 when the memory is read and set to 0 when the memory is written;

[0117] 3.3) When the gap between gesture samples is too large, Treated as a one-hot encoding, this may weaken the role of the memory network and even lead to non-differentiable problems. By applying adjacent time slot weights, the context weights are balanced as follows:

[0118]

[0119] Among them, b Ω (i) is a bell-shaped equilibrium function. Similarly, a temperature parameter is introduced to control the distribution concentration level, that is, the temperature factor τ used to amplify the focusing degree on each memory slot. The final weight is:

[0120]

[0121] 4) Make q l and Aligned in semantic space, the loss function is as follows:

[0122]

[0123] in, Measuring Euclidean distance, Measures KL divergence; λ L2 and λ KLD The weight used to balance the two parts, λ L2 Set to 10, λ KLD Set to 1; at the same time, W is introduced to build a memory library before aligning the query sample and the gesture invariant, which can be expressed as:

[0124] W=max(0,(1-exp(-E+ε)))

[0125] Among them, E represents the current training cycle, ε is the confirmation Warm-up hyperparameters at the time of effectiveness; after multiple iterations, the differences between different memory slots are gradually reduced, and the true gesture invariant is obtained

[0126] Step 2 uses generative adversarial networks to implement cross-modal loops to fully obtain modal information; see Figure 4 、 Figure 5

[0127] 5) Extract single frame from RGB video data and depth video data to fully exploit the modality information complementarity; decouple these two modalities data and respectively: modal data decoupled into modality-invariant features based on gestures themselves modal data decoupled into modality-specific attribute features Firstly, compute modality-specific attribute features

[0128]

[0129] where m denotes the modality, v denotes the input single frame; denotes the attribute feature encoder based on Resnet-18; meanwhile, an additional classifier is adopted to distinguish which modality it belongs to so as to force correct mapping to the corresponding modality;

[0130] 6) Adopt separation strategy to separate modality-specific attribute features from video features and input the remaining separation results into the content encoder :

[0131]

[0132] where, denotes the modality-specific attribute features separated from the video features, realized by connecting and ; in addition, only denotes the intermediate-level modality-invariant features, and the modality-specific features of different modalities are constantly approaching through back propagation, and finally obtain the modality-invariant features g c ;

[0133] 7) After obtaining the modality-invariant g c , add another modality-specific attribute feature is the inverse operation of ; then, input the combined features into the decoder D m to synthesize video features

[0134]

[0135] through cross-modality generation,​ can be fused into the current modal branch and affect it through backpropagation feature distribution, ultimately achieving the cycle between different modalities.

[0136] Step 3 inputs the gesture invariant features obtained in step 1 and the modal features obtained in step 2 into the gesture recognition network together to perform anti-interference gesture recognition.

[0137] A system based on the above anti-interference gesture recognition method, comprising:

[0138] A memory writing module that continuously writes sample features to the memory slot to build a gesture sample memory bank through a similarity strategy and an erasing strategy;

[0139] A memory reading module that performs weighted summation on the memory slots of similar gestures:

[0140]

[0141] to derive gesture invariants;

[0142] A weight calculation module that combines content-based weight prediction

[0143]

[0144] and context weight prediction

[0145]

[0146] When the gap between gesture samples is too large, the is regarded as a one-hot encoding, which may weaken the effect of the memory network or even cause non-differentiable problems, and by applying adjacent time slot weights, the context weight is balanced

[0147]

[0148] provides a comprehensive weight scheme for gesture invariant extraction

[0149]

[0150] A modal attribute extraction module that extracts specific attribute features of two modalities of RGB video data and depth video data through an attribute encoder and uses an additional classifier to correctly map them;

[0151] A modal invariant extraction module that separates the modal-specific attribute features in the video features through a content encoder to obtain modal invariant features;

[0152] A cross-modal cycle module that realizes the cycle generation between different modal data by constructing an adversarial subnetwork.

[0153] An apparatus based on the anti-interference gesture recognition method, comprising:

[0154] a memory for storing a computer program;

[0155] a processor for implementing the anti-interference gesture recognition method based on the memory network and the generative adversarial network when executing the computer program.

[0156] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to be capable of performing anti-interference gesture recognition based on a memory network and a generative adversarial network.

Claims

1. A method for anti-environmental interference gesture recognition based on a memory network and a generative adversarial network, characterized in that, The specific operation steps are as follows: Step 1. Extract gesture invariants through a memory network to achieve anti-environmental interference; 1) Input different modal gesture data: input different modal gesture data, including RGB and depth video data, into two video feature extractors respectively to obtain current query sample features where l represents the category of the gesture; 2) Establish a memory bank and extract gesture invariants; 3) Solving for the weighting parameters 4) q by training network l With Align in semantic space; After several iterations, the differences between the different memory slots are gradually reduced, resulting in a true gesture invariant Step 2. Use a generative adversarial network to achieve cross-modal circulation to fully obtain modal information; 5) Obtain the coding features in the attribute embedding space; 6) obtaining the modal invariant g c ; Separating modal-specific attribute features from video features and inputting the remaining separation result into a content encoder : wherein, represents the modal-specific attribute feature in the separated video feature, which is achieved by connecting and ; in addition, only represents the intermediate-level modal-invariant feature, and the modal-invariant features of different modalities are obtained by continuously approaching through back propagation c ; 7) Through cross-modal generation and back propagation, the circulation between different modalities is realized; The modal invariant g is obtained c After, by Adding another modal specific attribute feature For The inverse operation; after that, the combined features are input to the decoder D m In the synthesized video features By cross-modal generation, can be fused into the current modal branch and affect feature distribution, ultimately achieving a cycle between different modalities; Step 3. The gesture invariant features obtained in step 1 and the modal features obtained in step 2 are jointly input into the gesture recognition network for anti-interference gesture recognition.

2. The method of claim 1, wherein the method is based on a memory network and a generative adversarial network. The specific method of step 1, step 2) is: 2.1) Design a memory bank M, whose size is n x m x l, where m represents the number of memory slots for each type of gesture, n represents the length of the feature vector, and l represents the category of the gesture; 2.2) Memory writing 2.2a) first look at the memory slot position indicated by the cursor variable, when the indicated memory slot is empty, the current sample feature can be directly saved to the memory slot; ​ 2.2b) if no free memory slot is present, then calculate the similarity between the current sample features and the features of each memory slot and use it as a weight for the memory writing; and the similarity between the features of each memory slot and use it as a weight for the memory writing; 2.2c) Use the erase matrix to ignore some features in the memory slot, and the erase step can be represented as: where M l is the memory slot of gesture class l, t and t-1 represent the iteration number of the optimization process; e t is an erasing matrix composed of n feature vectors corresponding to n memory slots, each of which has m elements randomly generated from [0, 1] to ensure the diversity of the erasing operation; 2.2d) After the erasing step is completed, the current sample characteristics Write memory slot: Where w is a weight matrix of shape n x m; i∈{1,2,…,n} corresponds to the i-th vector containing m identical elements and the i-th memory slot; At the same time, a constraint is imposed on the weight matrix w: 2.3) Memory reading Memory reading is the weighted sum of features in the memory slot: wherein, denotes the derived intermediate-level gesture invariant at iteration t 3.The environmental interference resistant gesture recognition method based on memory network and generative adversarial network according to claim 1, characterized in that, The specific method of step 1, step 3) is: A hierarchical weighting mechanism is adopted, including content-based weight prediction and context weight prediction, to provide comprehensive weights for gesture invariant extraction; 3.1) by the current sample features with the features in each memory slot Cosine similarity is measured between each memory slot content-based weight prediction is defined as Where cossim(·,·) represents the cosine similarity measure; For two matrices i and v, the measure can be represented as: 3.2) According to the memory bank established in step 1, step 2), introduce context weight prediction: Where γ is a flag parameter, set to 1 when reading memory, and set to 0 when writing memory; 3.3) When the gap between gesture samples is too large, consider viewed as a one-hot encoding, which can lead to a weakened memory network or even non-differentiable problems, by applying adjacent time slot weights, balancing the context weights, as follows: where b Ω (i) is a bell-shaped balancing function; also, a temperature parameter is introduced to control the level of the concentration profile, i.e. a temperature factor τ to amplify the focusing degree on each memory slot, and the final weight is: 4.The environmental interference resistant gesture recognition method based on memory network and generative adversarial network according to claim 1, characterized in that, The specific method of step 1, step 4) is: By training the network to make q l With Aligning in semantic space, the loss function is as follows: where, measuring Euclidean distance, measuring KL divergence; λ L2 and λ KLD for balancing the weights of two parts; meanwhile, introducing W to build a memory bank before aligning query samples and gesture invariants, which can be expressed as: W=max(0,(1-exp(-E+ε))) wherein E represents the current training period, and ε is a confirmation The preheating hyperparameters at the effective moment; after multiple iterations, the differences between different memory slots are gradually reduced, and the real gesture invariant is obtained 5.The environmental interference resistant gesture recognition method based on memory network and generative adversarial network according to claim 1, characterized in that, The specific method of step 2, step 5) is: extracting a single frame in the rgb data and a single frame in the depth data to take advantage of the modality information complementarity; decoupling the two modalities of data and respectively: modality data decoupled into modality invariant features based on the gesture itself modality data decoupled into modality specific attribute features first computing modality specific attribute features where m denotes the modality, v denotes the input single frame; represents the attribute feature encoder based on Resnet-18; at the same time, an additional classifier is adopted to distinguish which kind of modality so as to force to correctly map to the corresponding modality.

6. A system for gesture recognition based on the interference-free gesture recognition method according to any one of claims 1 to 5, characterized in that Including: The memory writing module continuously writes sample features to the memory slot to build a gesture sample memory bank through a similarity strategy and an erase strategy; The memory reading module derives the gesture invariants by weighting and summing the memory slots of the same type of gesture; The weight calculation module provides a comprehensive weight scheme for gesture invariant extraction by combining content-based weight prediction and context weight prediction; The modal attribute extraction module extracts specific attribute features of the two modalities through the attribute encoder and uses an additional classifier to correctly map them; The modal invariant extraction module separates the modal specific attribute features in the video features through the content encoder to obtain the modal invariant features; The cross-modal circulation module realizes the circulation generation between different modal data by constructing an adversarial subnetwork.

7. A device based on the interference-resistant gesture recognition method according to any one of claims 1 to 5, characterized in that Including: Memory for storing computer programs; Processor for executing the computer program to realize the anti-interference gesture recognition method based on memory network and generative adversarial network of any one of steps 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by a processor, can perform anti-interference gesture recognition based on the anti-interference gesture recognition method according to any one of claims 1 to 5.

Images