Efficient video gesture authentication method based on behavior feature enhancement and related device

Abstract

The application discloses a high-efficiency video gesture authentication method based on behavior feature enhancement and related equipment, relates to the technical field of biometric recognition and authentication, and comprises the following steps: acquiring a video stream containing dynamic gestures; establishing an FSTA-Net framework, wherein the FSTA-Net framework at least comprises a behavior enhancement module and a frequency domain space-time attention module with a plug-and-play feature, wherein the video stream containing dynamic gestures obtains a behavior enhancement graph after passing through the behavior enhancement module, the behavior enhancement graph obtains local identity features after being processed by a set level module of a backbone network, the local identity features realize global identity information capture and local information enhancement after being processed by the frequency domain space-time attention module, and more comprehensive identity features are obtained; and the more comprehensive identity features obtain final identity features after being processed by the set level module of the backbone network. The application can improve the authentication accuracy and operation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biometric recognition and authentication, and particularly relates to an efficient video gesture authentication method based on behavior feature enhancement and related equipment. BACKGROUND

[0002] With the advent of the Internet of Everything era, data security and information privacy issues are increasingly prominent. For ubiquitous identity authentication in life, people urgently need a more secure, friendly and efficient authentication method. Under the driving of the great development of artificial intelligence algorithms, biometric authentication technology has obvious advantages compared with traditional methods and has great development potential. However, the existing biometric authentication technology still has one or more of the problems of single feature essence, weak resistance to imitation attack, and unfriendly data collection method, thereby resulting in unsatisfactory user experience. Some biometric authentication technologies also touch on user information privacy, which makes it difficult to promote. Therefore, the present application takes dynamic gestures as a credential for identity authentication. When using gesture authentication, the user only needs to perform a gesture in the air to complete the identity authentication, and the whole process does not need to be remembered, which is easy and natural. The gesture does not involve sensitive private information, and the user friendliness is very high, and it combines physiological (such as hand shape, finger length) and behavioral (such as force, flexibility) two types of biometric modalities, which can greatly improve the attack difficulty of the system. In addition, the authentication process is non-contact, clean and sanitary, and is not disturbed by stains. It needs to be additionally explained that gesture authentication and gesture recognition are completely two different technologies. Gesture recognition focuses on the type of gesture, while gesture authentication focuses on the identity features contained in the gesture.

[0003] Dynamic gesture authentication based on video understanding involves spatio-temporal feature extraction. The current mainstream spatio-temporal feature extraction network includes three-dimensional convolutional neural network, dual-stream convolutional neural network and two-dimensional convolutional neural network. Wu et al. first introduced the dual-stream convolutional neural network into gesture authentication and used optical flow for behavior feature representation (TSCNN). Considering the low efficiency and poor effect of optical flow extraction, Song et al. replaced the optical flow with PA (ITSCNN) on this basis, which greatly reduced the authentication error rate. However, due to the dual-stream network with twice the parameter quantity and operation quantity, the running efficiency still has a large space for improvement. In addition, Liu et al. designed DHGA-Net based on three-dimensional convolutional neural network and achieved ideal authentication effect. However, three-dimensional convolution also has the problems of large parameter quantity and low running efficiency. At the same time, since three-dimensional convolution cannot display the physiological features and behavior features for targeted processing, there is still a lot of room for improvement in authentication accuracy. In order to alleviate the above problems of large operation quantity and low efficiency of dual-stream network and three-dimensional network, Song et al. customized TDS-Net and 3DTDS-Net for gesture authentication. These two methods use a light-weight symbiotic branch attached to the high-efficiency two-dimensional convolutional neural network to analyze the behavior features, and realize the complementary fusion of physiological features and behavior features through BE feature fusion and BM feature fusion modules, finally improve the problem of large operation quantity of three-dimensional network and two-dimensional network to a certain extent, and further improve the authentication accuracy. However, in TDS-Net and 3DTDS-Net, gesture behavior feature analysis is severely limited by the features provided by the backbone network and the limited nonlinear expression ability of the symbiotic branch, so the gesture behavior features are not well developed and utilized. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application provides a high-efficiency video gesture authentication method based on behavior feature enhancement and related equipment, which improves the authentication accuracy and running efficiency.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a high-efficiency video gesture authentication method based on behavior feature enhancement, which comprises:

[0007] Obtaining a video stream containing dynamic gestures;

[0008] The FSTA-Net framework at least includes a behavior enhancement module and a frequency domain space-time attention module with a plug-and-play feature, wherein the video stream containing a dynamic gesture obtains a behavior enhancement graph after passing through the behavior enhancement module, the local identity feature is obtained after the behavior enhancement graph passes through a set level module of a backbone network, and the global identity information capture and local information enhancement are realized after the local identity feature passes through the frequency domain space-time attention module, so that a more comprehensive identity feature is obtained.

[0009] The more comprehensive identity feature obtains a final identity feature after passing through a set level module of a backbone network.

[0010] In a second aspect, the present application provides an efficient video gesture authentication system based on behavior feature enhancement, which comprises:

[0011] a video acquisition unit for acquiring a video stream containing a dynamic gesture;

[0012] a processing unit for establishing an FSTA-Net framework, wherein the FSTA-Net framework at least includes a behavior enhancement module and a frequency domain space-time attention module with a plug-and-play feature, wherein the video stream containing a dynamic gesture obtains a behavior enhancement graph after passing through the behavior enhancement module, the local identity feature is obtained after the behavior enhancement graph passes through a set level module of a backbone network, and the global identity information capture and local information enhancement are realized after the local identity feature passes through the frequency domain space-time attention module, so that a more comprehensive identity feature is obtained; and

[0013] a result output unit for obtaining a final identity feature after the more comprehensive identity feature passes through a set level module of a backbone network.

[0014] In a third aspect, the present application further provides an electronic device comprising a processor and a memory.

[0015] The memory is used for storing a program.

[0016] The processor executes the program to realize the method as described above.

[0017] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to realize the method as described above.

[0018] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the foregoing method.

[0019] Compared with the prior art, the present application has the beneficial effects that:

[0020] The technical solution provided by the present application takes into account the reduction of error rate and the improvement of operation efficiency, which is embodied in that:

[0021] The present application designs two time domain analysis modules on the basis of a two-dimensional convolutional neural network to make it have a space-time information processing capability, so that the final gesture authentication model inherits the high efficiency of the two-dimensional convolutional neural network while being well applicable to the video gesture authentication task.

[0022] The present application designs a behavior enhancement (BE) module to reduce redundant information and highlight behavior details. The BE module has strong short-term space-time modeling capability, which can improve the authentication performance while greatly improving the operation efficiency.

[0023] The present application first introduces and extends the Discrete Cosine Transform (DCT) to the neural network and designs a frequency domain space-time attention (FSTA) module. Since the FSTA module has a strong theoretical basis, it has a stronger long-distance space-time modeling capability. Compared with other space-time attention models, the calculation amount and GPU memory occupation introduced by the FSTA are very appropriate. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 The FSTA-Net framework and data processing flowchart of the embodiments of the present application;

[0026] Figure 2 The FSTA module and data processing flowchart of the embodiments of the present application;

[0027] Figure 3 The flowchart of the efficient video gesture authentication method based on behavior feature enhancement of the embodiments of the present application;

[0028] Figure 4 Flowchart for obtaining more comprehensive identity features for embodiments of the present application;

[0029] Figure 5 Another flowchart for the efficient video gesture authentication method based on behavior feature enhancement for embodiments of the present application;

[0030] Figure 6 Schematic diagram of an electronic device for embodiments of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0032] Embodiments:

[0033] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] To better understand the technical solutions provided by the embodiments of the present application, some simple introductions to the inventive construction of the technical solutions provided by the embodiments of the present application are given below.

[0035] Dynamic gesture is a promising biometric feature, which combines both physiological features (e.g. palm shape and finger length) and behavioral features (e.g. force and flexibility). Therefore, dynamic gesture authentication is theoretically more accurate and secure. However, dynamic gesture authentication is a fine-grained video understanding task, which is very challenging for algorithm design. The model needs to find subtle physiological differences from a large number of similar hands and find weak motion clues from similar behavioral patterns. At the same time, the understanding of physiological features and behavioral features is somewhat conflicting. Behavioral features are reflected between video frames, while physiological features are reflected in video frames. In order to record sufficient behavioral features, high frame rate is a necessary condition for data acquisition. However, high frame rate will introduce excessive computational load and redundant information, which leads to two major problems: difficulty in learning identity features and low overall system running efficiency. Current gesture authentication methods based on video understanding mainly focus on reducing the equal error rate, while ignoring the problem of running efficiency.

[0036] The technical scheme provided by the embodiment of the application takes into account reducing the equal error rate and improving the running efficiency, which is embodied in that:

[0037] The embodiment is based on a two-dimensional convolutional neural network, and two time domain analysis modules are designed to make it have spatio-temporal information processing capability, so that the final gesture authentication model inherits the high efficiency of the two-dimensional convolutional neural network and can be well applied to the video gesture authentication task.

[0038] The embodiment designs a behavior enhancement (BE) module to reduce redundant information and highlight behavioral details. The BE module has strong short-term spatio-temporal modeling capability, which can improve authentication performance and greatly improve running efficiency.

[0039] The embodiment first introduces and extends the Discrete Cosine Transform (DCT) to the neural network and designs a frequency domain spatio-temporal attention (FSTA) module. Since the FSTA module has a strong theoretical basis, it has a stronger long-distance spatio-temporal modeling capability. Compared with other spatio-temporal attention models, the computational load and GPU memory usage introduced by FSTA are very appropriate.

[0040] Referring to Figures 1 to 5 A high-efficiency video gesture authentication method based on behavior feature enhancement, which can specifically include the following steps:

[0041] Step 100: Obtain a video stream containing dynamic gestures.

[0042] Step 200: Establishing an FSTA-Net framework, which at least includes a behavior enhancement module and a frequency domain space-time attention module with plug-and-play characteristics, wherein the video stream containing dynamic gestures obtains a behavior enhancement graph after passing through the behavior enhancement module, the local identity feature is obtained after the behavior enhancement graph is processed by a set level module of a backbone network, the global identity information capture and local information enhancement are realized after the local identity feature is processed by the frequency domain space-time attention module, and a more comprehensive identity feature is obtained.

[0043] More specifically, in certain embodiments, the video stream containing dynamic gestures obtains a behavior enhancement graph after passing through the behavior enhancement module, specifically including the steps of:

[0044] Step 201: The video stream containing dynamic gestures sequentially passes through an inter-frame time domain difference operation and a channel accumulation operation to obtain a gray difference video with behavior feature enhancement;

[0045] Step 202: The gray difference video sequentially passes through a three-dimensional convolution operation, a two-dimensional batch normalization operation, an activation function processing operation, and a two-dimensional maximum pooling operation to obtain the behavior enhancement graph.

[0046] More specifically, in certain embodiments, the local identity feature realizes global identity information capture and local information enhancement after being processed by the frequency domain space-time attention module, and a more comprehensive identity feature is obtained, specifically including the steps of:

[0047] Step 211: The local identity feature sequentially passes through different three-dimensional convolution processing to realize a base and filter generation operation, an input feature compression and adaptation operation, and an attention inquiry generation operation, thereby corresponding to obtain an important base, a compressed and adapted feature suitable for frequency domain transformation, and an attention inquiry matrix;

[0048] Step 212: Frequency domain transformation and filtering are realized by matrix multiplication of the important base and the compressed and adapted feature suitable for frequency domain transformation, and a filtered global frequency domain feature is obtained;

[0049] Step 213: Global information summary is realized by matrix multiplication of the global frequency domain feature and the attention inquiry matrix, thereby obtaining global enhancement information;

[0050] Step 214: The global enhancement information realizes self-adaptive adjustment and channel reconstruction by three-dimensional convolution, and obtains a form suitable for information fusion with local features;

[0051] Step 215: The global enhancement information and the local identity feature are added to realize information enhancement of the global enhancement information on the local identity feature, thereby obtaining a more comprehensive identity feature.

[0052] Step 300: the more comprehensive identity features are obtained after the setting level module of the backbone network is processed.

[0053] To better understand the technical solutions provided by the embodiments of the present application, the above technical solutions will be described in detail below with reference to the accompanying drawings:

[0054] The overall framework and data processing flow of the FSTA-Net are shown in Figure 1 Considering that the two-dimensional convolutional neural network is relatively efficient compared with the three-dimensional convolutional neural network and the dual-flow convolutional neural network, and there are a large number of pre-training models that can be used for weight initialization, the embodiment adopts a standard 2D ResNet18 as the backbone of the gesture authentication network, and adopts the TSN video processing paradigm for processing of the gesture video. In order to make the FSTA-Net have stronger spatiotemporal feature analysis capability, the embodiment designs a behavior enhancement (BE) module and a frequency spatiotemporal attention module (FSTA) and inserts them into the FSTA-Net.

[0055] Among them, the BE module can well promote the analysis of local identity features (especially behavior features) while removing a large amount of redundant information. As shown in Figure 2 The core operation of this module is the time domain difference and the three-dimensional convolution operation (Conv3D), wherein the weights of the Conv3D are initialized by processing the weights of the pre-training 2D ResNet18 Conv1. The Conv3D can well extract the behavior information enhanced identity features through three-dimensional convolution on the basis of the difference video. These features will contain rich and valuable local behavior features after subsequent two-dimensional convolution (mainly through the Basic Block1 of L1, L2 and L3) processing. The BE module is a very important component, and the local identity information extracted by it is also an important basis for the summary of subsequent global information.

[0056] The FSTA module can capture global identity information and be used to enhance local information, thus enabling the model to capture more comprehensive identity information (mainly through Basic Block 2 of L3 and L4). In order to enable the FSTA module to function better, embodiments need to select a suitable implantation position for it according to the abstract degree and spatial resolution of the feature. If the implantation position of the FSTA module is too shallow (for example, L2) or too deep (for example, L4), it will be difficult for the FSTA module to effectively summarize satisfactory global identity information. When the implantation position is too shallow, the local identity information cannot be fully analyzed due to fewer convolution operations. At the same time, in the shallow layer, the feature dimension is high, which will lead to difficulty in frequency domain analysis. For the implantation position that is too deep, the FSTA module will not be able to obtain sufficient position-related detailed information (which is very important for analyzing behavioral features) and the remaining convolution operations will also not be able to well process the enhanced identity features due to insufficient number. Finally, embodiments determine the position of the FSTA module between the two blocks of L3 through experiments (for the standard ResNet18, its L1, L2, L3, and L4 layers all contain two basic residual blocks (Basic Block)).

[0057] After being processed by the four layers of L1-L4, the output feature becomes (T / 2-1) x 512 x H / 32 x W / 32. Embodiments will then sequentially input the result into the global spatial domain average pooling and the fully connected layer (512 x 512) to obtain (T / 2-1) 512-dimensional vectors. Finally, embodiments perform global time domain average pooling operation to obtain the final identity feature. This feature will be used for training, registration, and verification.

[0058] To better understand the behavior enhancement module provided by the embodiments of the application, the behavior enhancement module provided by the embodiments of the application will be described in detail below in conjunction with the drawings:

[0059] There are two main problems in gesture authentication: (1) it is difficult to extract fine-grained identity features (especially behavioral features) (2) it is very time-consuming to process high-frame-rate videos and it is difficult to meet the real-time requirement. To solve the first problem, inspired by TSN, we use time domain difference operation to highlight behavioral clues, thereby guiding the model to analyze behavioral features, which can be expressed as:

[0060] ΔV(t, c, h, w) = V(t, c, h, w) - V(t+1, c, h, w) (5)

[0061] where V ∈ R T×C×H×W is a gesture video, t, c, h, and w represent the t-th frame, the c-th channel, the h-th row, and the w-th column, respectively, t ∈ {1, 2, …, N-1}, and ΔV ∈ R (T-1)×C×H×W represents the corresponding time domain difference video.

[0062] To cope with the second problem, by deeply analyzing the behavior feature and the physiological feature, embodiments find that there is a lot of redundant information in the high frame rate video. For the behavior feature, the motion information difference of the three channels of RGB is very weak. Therefore, the behavior feature is redundant in the channel dimension of the video. For the physiological feature, the gestures of the adjacent frames of the high frame rate video have little difference in angle and posture, so the physiological feature is redundant in the time domain dimension. To reduce the redundant information of the behavior feature, embodiments propose to compress the channel on the basis of the time domain difference image. Considering that the time domain difference value is usually small after the inter-frame subtraction operation, embodiments adopt a global accumulation method to compress the channel for each position, as follows:

[0063]

[0064] where ΔV c ∈R (T-1)×H×W is the compressed time domain difference video. To reduce the redundant information of the physiological feature while considering that the behavior feature is mainly contained in the continuous adjacent frames, embodiments divide j adjacent compressed time domain difference video frames into a group (so that each group contains j+1 frame gesture video frame information) to represent the minimum granularity of the identity feature. On this basis, embodiments can regard the time domain dimension of each particle as the channel dimension and perform two-dimensional convolution to extract the local identity feature.

[0065] Specifically, to capture as much behavior feature as possible, embodiments use three-dimensional convolution (Conv3D) to slide in the time domain dimension to achieve the effect of overlapping grouping. To make full use of the pre-trained two-dimensional convolutional neural network parameters, embodiments set j to 3 (actual deployment can not be restricted by this). Therefore, the time domain convolution kernel size of three-dimensional convolution is also 3. To balance efficiency and capture more behavior information, embodiments set the time domain step size of the convolution kernel to 2. Before inputting Conv3D, embodiments first need to convert ΔV c c ​shape of (1, T-1, H, W) to construct a channel dimension for 3D convolution (channel size is 1). The weights of Conv3D can be evolved from the weights of Conv1 of the pre-trained model. The weight shape of Conv1 is (64, 3, 7, 7), and the embodiment deforms it to (64, 1, 3, 7, 7) to be used as the weight of Conv3D. After the processing of Conv3D, the features will be first transposed to the feature shape for two-dimensional convolution processing, and then sequentially pass through the processing of two-dimensional batch normalization (BN2D), ReLU and two-dimensional maximum pooling (MaxPool2D), and finally the output feature shape is (T / 2-1, 64, H / 4, W / 4). The feature can be regarded as a kind of online generated behavior feature related pseudo mode, which is called behavior enhanced map (BE-Map) by the embodiment, and the specific process can be represented as:

[0066] BEM = Ψ (Transpose (Conv3D (Reshape (ΔV c )))) (7)

[0067] Where BEM represents BE-Map, and Ψ represents the combination of BN2D, ReLU and MaxPool2D. BE-Map can be directly input to the remaining part (L1-L4, FC, etc.) of the backbone network (2DResNet) for regular two-dimensional convolution processing to extract valuable identity features. When performing two-dimensional convolution processing, the T / 2-1 frames of the time domain dimension will be regarded as the size of the image batch. BE-Map is a computationally friendly pseudo mode customized for behavior feature related authentication methods. The generator BE module that generates this pseudo mode can be trained end-to-end. Compared with the original video, BE-Map not only enhances the local behavior features but also effectively reduces the computational complexity Figure 1 One example given can reduce the computational consumption by half.

[0068] In order to better understand the frequency domain space-time attention module provided by the embodiment of the application, the frequency domain space-time attention module provided by the embodiment of the application will be described in detail below in conjunction with the drawings:

[0069] Related basis about DCT and frequency domain filtering:

[0070] Video and image are a kind of complex signals, which can be considered as the superposition of multiple frequency domain signals. Frequency domain analysis is an effective method to understand this kind of complex signals. Among the numerous frequency domain transform methods, discrete cosine transform (DCT) is very suitable for image or video processing, because the result of the transform is also a real signal and the algorithm complexity is relatively low. Taking one-dimensional DCT as an example, DCT represents a finite dimensional data sequence as the weighted sum of cosine functions of different frequencies, where the weight can be represented as:

[0071]

[0072] where c(u) = 1 when u = 0, otherwise, X(i) is the input sequence, F(u) is the corresponding DCT frequency spectrum, F(u), X(i) ∈ R N For ease of calculation, formula (1) can be converted into matrix multiplication form as follows:

[0073]

[0074] where X, F ∈ R N×1 , BB T = E(B ∈ R N×N , E is the unit matrix). Correspondingly, the inverse DCT (IDCT) is:

[0075] X = B T F (3)

[0076] After obtaining the frequency spectrum F, the valuable features can be preserved and some task-irrelevant information can be removed through frequency domain filtering operation, and then converted back to the spatial domain through IDCT. The filter is a vector h ∈ R N×1 , each dimension of which represents the retention or removal of the corresponding frequency feature. The process can be represented as:

[0077]

[0078] where represents Hadamard product.

[0079] Although the BE module can improve the ability of the two-dimensional backbone network to understand local identity features, the backbone network still has deficiencies in the ability to summarize and analyze global information. Therefore, embodiments of the present disclosure design a frequency domain spatio-temporal attention based on DCT to solve this problem. The frequency domain spatio-temporal attention can well integrate global spatio-temporal information through frequency domain transformation and remove part of irrelevant information (for example, noise and unstable identity features) through frequency domain filtering. Embodiments first introduce how to generalize DCT and filtering, so that they can be implemented through a neural network and can be run online. Then embodiments introduce how to use the global frequency domain information extracted by the extended DCT to enhance local information.

[0080] Embodiments first replace the basis function of the original DCT with a more general function g(i, u), so that the generalized DCT can be represented as:

[0081]

[0082] In order to keep the value range of consistent, embodiments limit:

[0083] g(i, u) ∈ [-1, 1] (9)

[0084] Correspondingly, the basis vector of formula (2) is adjusted to b u = N(u) [ …, g(i, u), … ] T . Considering that the DCT transformation matrix B (composed of b u ) is an orthogonal matrix, i.e. BB T = E, where E is the identity matrix, embodiments first adopt a normalization function N(u) to make In other words, formula (8) should satisfy:

[0085]

[0086] Then, embodiments need to constrain However, embodiments find through experiments that this constraint is too harsh for neural networks at present and will affect the quality of normal identity feature extraction. Therefore, embodiments relax this constraint and set an upper limit for the correlation between bases, i.e. So formula (8) needs to satisfy:

[0087]

[0088] where θ ∈ [0, π / 2] is a hyper-parameter to control the tolerance of inter-basis correlation (the larger the more strict). If θ is too large, then it will interfere with the normal feature extraction of the neural network. On the contrary, it will lead to too high inter-basis correlation, eventually leading to the extracted global frequency domain features containing more redundant information. In formula (11), the reason for using the absolute value is that the embodiment considers that the redundant information brought by the included angle between the two bases is not much different. For example, a very bad case is that the two bases are collinear but in opposite directions.

[0089] Assuming that the shape of the gesture video feature map (X) input to the FSTA is C × T × H × W, the embodiment first deforms and transposes it to N × C, where N = THW, and then converts it to the frequency domain spectrum F using formula (8). In the frequency domain, the embodiment hopes to retain some globally identifiable identity features and remove some irrelevant information through filtering from a global perspective:

[0090]

[0091] where H = diag(h). Since H, B ∈ R N×N , direct processing according to formula (12) will cause a lot of calculation consumption and GPU memory occupation. Therefore, the embodiment studies the design of the filter, which can be represented as:

[0092]

[0093] where δ(u) represents the unit impulse function, and S represents the function of selecting non-zero rows. In formula (13), there are K non-zero rows, so H S ∈ R K×N (K is much smaller than N). K represents that there will be K key frequency domain components used for global information summary.

[0094] After replacing H in formula (12) with H S , we get:

[0095]

[0096] where represents the valuable global identity features summarized from K different angles. In order to obtain more careful identity features, the embodiment proposes a frequency domain space-time attention. It first generates an attention query for all space-time local features to extract their corresponding customized global enhanced features from K global features:

[0097] X global = A(S(H)BXA) (15)

[0098] Where A is the feature adaptation function, used to make the features more suitable in shape and content for fusion with local features X, thus globally enhancing feature X. global ∈R C×N Additionally, in formula (15), A∈R C×N Representing the attention query matrix:

[0099] A = [a0, ..., a i ,…,a N-1 ],a i ∈R C (16)

[0100] Where a i This represents the query for the i-th local feature. Finally, the frequency domain spatiotemporal attention performs information fusion on the local and global features through addition using an attention mechanism:

[0101] Y = X + X global

[0102] Here Y∈R C×T×H×W Since the input and output have the same shape, frequency domain spatiotemporal attention is plug-and-play.

[0103] The architecture and process of the FSTA module are as follows: Figure 2 As shown, it consists of five steps: basis and filter generation; input image compression and adaptation; frequency domain transformation and filtering; frequency domain spatiotemporal attention; global information adaptation and local information enhancement.

[0104] Given basis B∈R N×N (N = THW), generating these bases via neural networks requires significant computation and GPU memory. To address this issue, equation (15) is transformed into:

[0105]

[0106] in It is made by filter H S ∈R K×N K important bases are generated. At this point, important bases are generated. (K is much smaller than N) and attention query A∈R C×N The computational workload is quite reasonable.

[0107] In the generation of the basis and filter, the important basis is first generated by a 1×1×1 convolution β(·). (i.e., learning of the basis generation and filtering operation simultaneously). Considering that the range of each value of the basis is [-1, 1] (see equation (9)) and the bases are unit vectors (see equation (10)), embodiments use tanh activation function (Tanh) and vector normalization (Norm) for processing. In order to reduce the correlation between the bases, an additional regularization term L basis is introduced according to equation (11). Therefore, the loss function can be expressed as:

[0108]

[0109] where I is the E unit matrix, Sum represents row accumulation, and λ is a hyperparameter that controls the degree of regularization.

[0110] In input image compression and adaptation, in order to reduce FLOPs and better prepare data for frequency domain transformation, embodiments use another 1x1x1 convolution χ(·) to compress and adaptively adjust the input data to obtain the feature X ca . In frequency domain transformation and filtering, embodiments realize frequency domain transformation and filtering through matrix multiplication of X ca and , to obtain the filtered frequency domain feature (see equation (14)). Then in the frequency domain spatio-temporal attention, embodiments generate N queries customized for all local features, i.e., matrix A, through a third 1x1x1 convolution α(·) and a Softmax function. By matrix multiplication of the filtered frequency domain feature and the attention query (A), the global enhanced feature is obtained (see equation (15)). Finally, in global information adaptation and local information enhancement, embodiments use a fourth 1x1x1 convolution η(·) and 3D batch normalization (BN3D) to realize adaptive adjustment and channel reconstruction of global features (i.e., function A) and to realize enhancement processing of global information on local information through a residual connection (see equations (15) and (17)).

[0111] The technical solutions in the embodiments of the present application will be described below in conjunction with experimental data.

[0112] Embodiments compare with current related methods from multiple aspects such as authentication error rate (the lower the EER, the better), computational complexity (FLOPs), parameter quantity, GPU memory occupancy, delay and throughput. The EER involves the following two authentication modes, as follows:

[0113] MG: The system defines gesture authentication, the registered gesture is consistent with the authentication gesture, and only the system-defined gesture can be used.

[0114] UMG: User-defined gesture authentication, register gesture consistent with authentication gesture, user can design their own gesture type. The superiority of the BE module is shown in Table 1:

[0115] Table 1

[0116]

[0117] In order to prove the superiority of the BE graph, the embodiment compares it with some pseudo modalities that are also related to behavior. It is obvious that the BE graph proposed in the embodiment has significant advantages in equal error rate and computational complexity. Therefore, the BE module can greatly improve the running efficiency while reducing the equal error rate.

[0118] The superiority of the FSTA module is shown in Table 2:

[0119] Table 2

[0120]

[0121] In order to prove the superiority of the FSTA module, the embodiment successively adopts the most mainstream channel attention (SE, squeeze-and-excitation) and space-time attention (NL, non-local) methods on the basis of the BE graph. It can be seen that the method proposed in the embodiment has the lowest equal error rate. The SE channel attention method has no effect on gesture authentication. Although the NL space-time attention method can effectively further reduce the equal error rate, the computational consumption and video memory occupation required are very large, and it is difficult to meet the real-time requirements of the authentication system.

[0122] The comparison with the current leading video gesture authentication method is shown in Table 3:

[0123] Table 3

[0124]

[0125] It can be seen that the method proposed in the present application has obvious advantages in various performance evaluation indexes, which can fully prove the high efficiency and effectiveness of the method.

[0126] Based on the same inventive concept, the embodiment of the present application also provides an efficient video gesture authentication system based on behavior feature enhancement, which comprises:

[0127] A video acquisition unit for acquiring a video stream containing dynamic gestures;

[0128] a processing unit configured to establish an FSTA-Net framework, the FSTA-Net framework comprising at least a behavior enhancement module and a frequency domain space-time attention module with plug-and-play characteristics, wherein the video stream containing a dynamic gesture is processed by the behavior enhancement module to obtain a behavior enhancement graph, the behavior enhancement graph is processed by a set level module of a backbone network to obtain a local identity feature, the local identity feature is processed by the frequency domain space-time attention module to realize global identity information capture and local information enhancement, and a more comprehensive identity feature is obtained; and

[0129] a result output unit configured to process the more comprehensive identity feature by a set level module of a backbone network to obtain a final identity feature.

[0130] Since the system is a system corresponding to the behavior feature enhanced efficient video gesture authentication method of the embodiments of the present application, and the principle of solving the problem of the system is similar to that of the method, the implementation of the system can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0131] Referring to Figure 6 Based on the same inventive concept, the embodiments of the present application also provide an electronic device, which comprises a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the behavior feature enhanced efficient video gesture authentication method as described above.

[0132] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above-mentioned method embodiments, etc.; the data storage area can store data created according to the use of the server, etc.

[0133] The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be implemented by a single chip.

[0134] Since the electronic device is the electronic device corresponding to the high-efficiency video gesture authentication method based on behavior feature enhancement of the embodiment of the application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0135] Based on the same inventive concept, the embodiment of the application also provides a computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the high-efficiency video gesture authentication method based on behavior feature enhancement.

[0136] Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be instructed by programs to relevant hardware, and the programs can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be used by a computer.

[0137] Since the storage medium is the storage medium corresponding to the high-efficiency video gesture authentication method based on behavior feature enhancement, and the principle of solving the problem of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0138] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the high-efficiency video gesture authentication method based on behavior feature enhancement according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0139] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used to implement the hardware: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth, or a combination of them.

[0140] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0141] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A high-efficiency video gesture authentication method based on behavior feature enhancement, characterized in that, The method comprises the following steps: acquiring a video stream containing dynamic gestures; establishing an FSTA-Net framework, wherein the FSTA-Net framework comprises at least a behavior enhancement module and a frequency domain space-time attention module with a plug-and-play feature, and the video stream containing dynamic gestures is subjected to the behavior enhancement module to obtain a behavior enhancement map, wherein the video stream containing dynamic gestures is subjected to the behavior enhancement module to obtain the behavior enhancement map, comprising the following steps: the video stream containing dynamic gestures is subjected to an inter-frame time domain difference operation and a channel accumulation operation in sequence to obtain a behavior feature enhanced gray difference video; the gray difference video is subjected to a three-dimensional convolution operation, a two-dimensional batch normalization operation, an activation function processing operation and a two-dimensional maximum pooling operation in sequence to obtain the behavior enhancement map; the behavior enhancement map is subjected to a set level module of a backbone network to obtain a local identity feature; the local identity feature is subjected to processing of the frequency domain space-time attention module to realize global identity information capture and local information enhancement, and obtain a more comprehensive identity feature; the more comprehensive identity feature is subjected to processing of a set level module of a backbone network to obtain a final identity feature.

2. The efficient video gesture authentication method based on behavior feature enhancement according to claim 1, wherein the local identity feature is subjected to processing of the frequency domain space-time attention module to realize global identity information capture and local information enhancement, and obtain a more comprehensive identity feature, comprising the following steps: the local identity feature is subjected to different three-dimensional convolution processing in sequence to realize a base and filter generation operation, an input feature compression and adaptation operation and an attention inquiry generation operation, so as to correspondingly obtain an important base, a compressed and adapted feature suitable for frequency domain transformation and an attention inquiry matrix; frequency domain transformation and filtering are realized through matrix multiplication of the important base and the compressed and adapted feature suitable for frequency domain transformation, and a filtered global frequency domain feature is obtained; global information summary is realized through matrix multiplication of the global frequency domain feature and the attention inquiry matrix, so as to obtain global enhancement information; the global enhancement information is subjected to adaptive adjustment and channel reconstruction through three-dimensional convolution, and a form suitable for information fusion with a local feature is obtained; the global enhancement information is added to the local identity feature to realize information enhancement of the local identity feature by the global enhancement information, so as to obtain a more comprehensive identity feature.

3. The efficient video gesture authentication method based on behavior feature enhancement according to claim 2, wherein the inter-frame time domain difference operation is processed by using the following set formula: ΔV(t,c,h,w)=V(t,c,h,w)-V(t+1,c,h,w) the channel accumulation operation is processed by using the following set formula:

4. The efficient video gesture authentication method based on behavior feature enhancement according to claim 3, wherein the behavior enhancement map is processed by using the following set formula: In the formula, V ∈ R T×C×H×W is a video stream containing dynamic gestures, t, c, h, w respectively represent the t-th frame, the c-th channel, the h-th row and the w-th column, t ∈ {1, 2,..., N-1}, ΔV ∈ R (T-1)×C×H×W denotes the corresponding time domain difference video; wherein BEM represents a BE-Map behavior enhancement map, and represents a combination of a two-dimensional batch normalization operation, an activation function processing operation and a two-dimensional maximum pooling operation. where ΔV c ∈R (T-1)×H×W is the compressed time-domain difference video. The method comprises the following steps: ​ BEM = Ψ (Transpose (Conv3D (Reshape (ΔV c )))) ​ 5. A high efficient video gesture authentication system based on behavior feature enhancement, characterized in that, ​ a video acquisition unit configured to acquire a video stream containing dynamic gestures; a processing unit configured to establish an FSTA-Net framework, the FSTA-Net framework comprising at least a behavior enhancement module and a frequency domain spatio-temporal attention module with plug-and-play characteristics, and the video stream containing dynamic gestures is subjected to the behavior enhancement module to obtain a behavior enhancement graph, wherein the video stream containing dynamic gestures is subjected to the behavior enhancement module to obtain a behavior enhancement graph, comprising the steps of: the video stream containing dynamic gestures is subjected to an inter-frame time domain difference operation and a channel accumulation operation in sequence to obtain a behavior feature enhanced gray difference video; the gray difference video is subjected to a three-dimensional convolution operation, a two-dimensional batch normalization operation, an activation function processing operation and a two-dimensional maximum pooling operation in sequence to obtain the behavior enhancement graph; the behavior enhancement graph is subjected to a set level module of a backbone network to obtain a local identity feature, and the local identity feature is subjected to the frequency domain spatio-temporal attention module to realize global identity information capture and local information enhancement, and obtain a more comprehensive identity feature; and a result output unit configured to obtain a final identity feature after the more comprehensive identity feature is subjected to a set level module of a backbone network.

6. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize the efficient video gesture authentication method based on behavior feature enhancement as claimed in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize the efficient video gesture authentication method based on behavior feature enhancement as claimed in any one of claims 1 to 4.

Images