Cross-domain gesture recognition model training method, recognition method, device and storage medium

Abstract

This invention discloses a cross-domain gesture recognition model training method, recognition method, device, and storage medium, relating to the field of radio frequency signal sensing technology. The invention preprocesses multiple channel state information from different domains to obtain a training dataset. Task generation operations are performed on the training dataset to obtain single-domain and multi-domain tasks. An embedding network and a relational network are trained based on the data from the single-domain and multi-domain tasks, respectively, to improve the model's adaptability in different domains. A cross-domain gesture recognition model is obtained based on the trained embedding and relational networks. The embedding and relational networks, based on meta-learning, divide the data into support and query sets for simultaneous prediction and training of the model. This improves the model's learning ability in different domains, reduces the amount of training data required for the cross-domain gesture recognition model, and achieves high-performance gesture recognition in different target domains.

Description

Technical Field

[0001] This invention relates to the field of radio frequency signal sensing technology, and in particular to a cross-domain gesture recognition model training method, recognition method, device and storage medium. Background Technology

[0002] Currently, gesture recognition methods can be mainly divided into three types: those based on dedicated sensors, those based on video images, and those based on radio frequency (RF) signals. Among these, gesture recognition methods based on dedicated sensors require users to wear specialized sensors. The advantage of this method is that it is easy to collect gesture data and less affected by the environment, making it easy to implement. However, the need to wear sensors can be inconvenient for users. Gesture recognition methods based on video images use video image acquisition devices to capture user gestures and then process this data to achieve gesture recognition. This method does not require users to wear sensors, making it very convenient. However, the disadvantages of video image-based gesture recognition methods are that video images are sensitive to the environment, and devices such as cameras can infringe on user privacy. Gesture recognition methods based on RF signals collect and process gesture data using RF devices. Compared to gesture recognition methods based on dedicated sensors and video images, gesture recognition methods based on RF signals do not require users to wear specialized sensor devices, do not need to consider the influence of light, do not require cameras, and do not infringe on user privacy. Therefore, gesture recognition methods based on RF signals have great research potential.

[0003] However, wireless signals and their characteristics are highly specific to the environment in which the person performing the gesture is located, as well as their position and orientation (these gesture-independent factors are collectively referred to as domain factors below). A gesture classifier trained with the characteristics of wireless signals from one domain usually experiences a significant drop in recognition accuracy when it moves to another domain.

[0004] To achieve cross-domain recognition of wireless signals, related technologies require adding data from new target domains to the classification model each time, necessitating additional work in data collection and model retraining. Furthermore, given the infinite number of domains in the real world, this approach is impractical. Other related technologies extract domain-independent gesture features for gesture recognition, but these methods require deploying multiple devices and knowing the precise locations of the transceivers in advance, similarly limiting their practicality. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a cross-domain gesture recognition model training method, which can achieve gesture recognition in different target domains while reducing the workload of model training.

[0006] On one hand, embodiments of the present invention provide a cross-domain gesture recognition model training method, including the following steps:

[0007] Obtain channel state information from multiple domains;

[0008] The channel state information from multiple sources is preprocessed to obtain a training dataset;

[0009] Perform task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks;

[0010] An embedding network and a relation network are trained based on data from single-domain tasks and multi-domain tasks, respectively. The embedding network is used to divide the input data into a support set and a query set, and to extract features from the data in the support set and the query set, respectively, to obtain support set data features and query set data features. The relation network is used to determine a relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score.

[0011] A cross-domain gesture recognition model is obtained based on the trained embedding network and relational network.

[0012] According to some embodiments of the present invention, the preprocessing of the plurality of channel state information to obtain a training dataset includes the following steps:

[0013] The phase offset segments in multiple channel state information are removed by the antenna conjugate multiplication based on weights to obtain the first preprocessed dataset.

[0014] The wavelet transform algorithm is used to remove noise from the channel state information in the first preprocessed dataset to obtain the second preprocessed dataset;

[0015] Principal component analysis algorithm is used to reduce the dimensionality of the channel state information in the second preprocessed dataset to obtain the third preprocessed dataset.

[0016] The channel state information in the third preprocessed data is subjected to a short-time Fourier transform to obtain the training dataset.

[0017] According to some embodiments of the present invention, the embedded network includes a support set module and a query set module, wherein the support set module is used to extract features from data assigned to the support set, and the query set module is used to extract features from data assigned to the query set.

[0018] Both the support set module and the query set module include convolutional residual network units and convolutional pooling network units connected in sequence; the convolutional residual network unit includes an autocorrelation information extraction layer, multiple convolutional layers and a connection layer connected in sequence, and the connection layer is used to integrate the output of the last convolutional layer and the input of the autocorrelation information extraction layer to obtain the output of the convolutional residual network unit; the convolutional pooling network unit includes a convolutional layer, a normalization layer and a pooling layer connected in sequence.

[0019] According to some embodiments of the present invention, the relationship network includes a feature concatenation layer, a convolutional pooling network unit, a fully connected layer, and a classification layer connected in sequence. The feature concatenation layer is used to concatenate the support set data features and query set data features output by the embedding network to obtain a concatenated feature vector. The convolutional pooling network unit is used to perform dimensionality reduction on the concatenated feature vector. The fully connected layer is used to predict the relationship score based on the dimensionality-reduced concatenated feature vector. The classification layer is used to determine the predicted gesture classification result of the query set data based on the relationship score.

[0020] According to some embodiments of the present invention, the self-correlation information extraction layer is used to extract multiple channel features of the input matrix, and multiply one of the channel features by all the surrounding [-dU,dU]×[-dV,dV] channel features bitwise to obtain the output matrix of the self-correlation information extraction layer, wherein dU and dV are preset hyperparameters.

[0021] According to some embodiments of the present invention, the training of the embedding network and the relation network based on data from single-domain tasks and multi-domain tasks, respectively, includes the following steps:

[0022] The data of the single-domain task is sequentially input into the embedding network and the relational network for prediction and classification to obtain the first predicted gesture classification result of the single-domain task data. The first loss value of the embedding network and the relational network on the single-domain task is calculated based on the first predicted gesture classification result. The parameters of the embedding network and the relational network are updated based on the first loss value.

[0023] The data from the multi-domain task is sequentially input into the embedding network and the relational network updated based on the data from the single-domain task for prediction and classification, to obtain the second predicted gesture classification result of the multi-domain task data. The second loss value of the embedding network and the relational network on the multi-domain task is calculated based on the second predicted gesture classification result, and the parameters of the embedding network and the relational network are updated based on the second loss value.

[0024] On the other hand, embodiments of the present invention also provide a gesture recognition method, including the following steps:

[0025] Obtain the channel state information to be identified;

[0026] The channel state information is input into the cross-domain gesture recognition model to obtain the gesture classification result;

[0027] The cross-domain gesture recognition model is trained by the cross-domain gesture recognition model training method according to any one of claims 1 to 6.

[0028] On the other hand, embodiments of the present invention also provide a cross-domain gesture recognition model training device, comprising:

[0029] The first module is used to acquire multiple channel state information from different domains;

[0030] The second module is used to preprocess the multiple channel state information to obtain a training dataset;

[0031] The third module is used to perform task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks.

[0032] The fourth module is used to train an embedding network and a relation network based on data from single-domain tasks and multi-domain tasks, respectively. The embedding network is used to divide the input data into a support set and a query set, and to extract features from the data in the support set and the query set, respectively, to obtain support set data features and query set data features. The relation network is used to determine a relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score.

[0033] The fifth module is used to obtain a cross-domain gesture recognition model based on the trained embedding network and relational network.

[0034] On the other hand, embodiments of the present invention also provide an electronic device, including:

[0035] At least one processor;

[0036] At least one memory for storing at least one program;

[0037] When the at least one program is executed by the at least one processor, the at least one processor implements the cross-domain gesture recognition model training method or gesture recognition method as described above.

[0038] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the cross-domain gesture recognition model training method or gesture recognition method as described above.

[0039] The technical solution described above has at least one of the following advantages or beneficial effects: This invention acquires multiple channel state information from different domains, preprocesses the multiple channel state information to obtain a training dataset, performs task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks, trains an embedding network and a relational network based on the single-domain and multi-domain task data respectively, and obtains a cross-domain gesture recognition model based on the trained embedding network and relational network. Training the model based on single-domain and multi-domain tasks respectively can improve the model's adaptability in different domains. The embedding network divides the input data into a support set and a query set, extracts features from the data in the support set and query set respectively to obtain support set data features and query set data features, and the relational network determines the relation score based on the support set data features and query set data features. Based on the relation score, it determines the predicted gesture classification result of the query set data. The embedding network and relational network, based on the meta-learning idea, divide the data into a support set and a query set for simultaneous prediction and training of the model, which can improve the model's learning ability in different domains, reduce the requirement for training data volume for the cross-domain gesture recognition model, and achieve high-performance gesture recognition in different target domains. Attached Figure Description

[0040] Figure 1 This is a flowchart of the cross-domain gesture recognition model training method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the training process of a cross-domain gesture recognition model provided in another embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar originals or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0045] In the description of this invention, the use of terms such as "first," "second," etc., is merely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0046] The cross-domain gesture recognition model training method or gesture recognition method in this application embodiment can be applied to a terminal, a server, or software running on a terminal or server. The terminal can be a tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0047] Reference Figure 1 and Figure 2 This invention provides a cross-domain gesture recognition model training method, which includes, but is not limited to, steps S110, S120, S130, S140 and S150.

[0048] Step S110: Obtain multiple channel state information from different domains;

[0049] Step S120: Preprocess multiple channel state information to obtain a training dataset;

[0050] Step S130: Perform task generation operation on the training dataset to obtain single-domain tasks and multi-domain tasks;

[0051] Step S140: Train the embedding network and the relation network based on data from single-domain tasks and multi-domain tasks, respectively. The embedding network is used to divide the input data into a support set and a query set, and extract features from the data in the support set and the query set to obtain the corresponding support set data features and query set data features, respectively. The relation network is used to determine the relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score.

[0052] Step S150: Obtain the cross-domain gesture recognition model based on the trained embedding network and relational network.

[0053] In step S110 of some embodiments, the cross-domain gesture recognition model training method can be applied to a system architecture including a wireless signal device and a processing device. The wireless signal device includes a wireless signal transmitter and a wireless signal receiver, which can be connected to the processing device via wired or wireless means. The wireless signal transmitter transmits wireless signals to the wireless signal receiver according to the control instructions of the processing device; the wireless signal receiver generates Channel State Information (CSI) based on the received wireless signal strength, phase, and other information, and sends the CSI to the processing device; the processing device generates a training dataset based on a large amount of collected CSI, and trains the behavior detection model based on the training dataset. Specifically, the wireless network card transmitter and receiver are deployed in different environments and different subjects are positioned and oriented to collect CSI in different domains. The deployment of the wireless network card transmitter and receiver in different environments, different subjects, and different positions and orientations of the same subject correspond to different domains. Furthermore, the original CSI signal can be augmented, for example, by stretching, compressing, and adding white noise to the CSI signal.

[0054] In step S120 of some embodiments, preprocessing operations such as phase alignment, noise reduction, principal component analysis, and short-time Fourier transform are performed on the collected channel state information to obtain DFS (Doppler Frequency Shift) data, which can improve the accuracy and efficiency of the model. Further, after collecting the multiple channel state information, gesture type annotation is needed to generate a training dataset. For example, the annotation of the channel state information can be various control gesture types or no gesture. Specifically, the annotation method can be manual annotation or machine annotation.

[0055] In the manual annotation method, the processing device collects channel status information in real time at a certain sampling frequency and displays it on the display screen in real time. Technicians annotate the channel status information on the display screen according to whether there are subjects between the wireless signal transmitter and the wireless signal receiver in the actual scenario and the subjects' gestures. Then, the channel status information and the corresponding annotation information are stored in the processing device for subsequent preprocessing operations.

[0056] In the machine labeling method, the processing device is also connected to a camera, which is used to capture images between the wireless signal transmitter and receiver. The processing device acquires channel state information and images in real time at a certain sampling frequency. The processing device inputs the images into a target detection model based on computer vision technology to output visual detection information about whether there are human gestures and the type of human gestures in the images. The processing device uses the visual detection information of the images as tags for the channel state information acquired at the same time, and then saves the channel state information and the corresponding tags in the processing device for subsequent preprocessing operations.

[0057] In step S130 of some embodiments, the present invention employs the idea of ​​meta-learning to perform task generation on the training dataset, generating a large number of single-domain tasks and multi-domain tasks. Single-domain tasks represent that all gesture data comes from the same domain, while multi-domain tasks represent that the data for each gesture comes from different domains. For example, multiple channel state information collected from the same subject in the same environment at the same position and orientation is classified as a single-domain task, while multiple channel state information collected from the same subject in the same environment at different positions and orientations is classified as a multi-domain task.

[0058] In steps S140 and S150 of some embodiments, the cross-domain gesture recognition model includes an embedding network and a relational network. Data from single-domain tasks and multi-domain tasks are input into the cross-domain gesture recognition model to train it, resulting in a trained cross-domain gesture recognition model. During model training, there are two data propagation methods: one is along the path from input to output, called forward propagation, and the other is from output back to input, called back propagation. In forward propagation, channel state information is processed layer by layer through the model and transmitted to the output layer. The error between the predicted output and the true expectation is described by the loss function. Then, back propagation begins, and the partial derivatives of the loss function with respect to the weights of each neuron are calculated layer by layer, forming the weight gradient data of the loss function with respect to the weight vector. This data serves as the basis for updating the weight parameters. The model training is completed by continuously updating the weight parameters, thus obtaining a trained cross-domain gesture recognition model.

[0059] Please continue to refer to Figure 2The embedding network is used to extract features from the channel state information data after short-time Fourier transform (hereinafter referred to as DFS data or DFS matrix). The embedding network includes a support set module and a query set module. The support set module is used to extract features from the data assigned to the support set, and the query set module is used to extract features from the data assigned to the query set. Both the support set module and the query set module include convolutional residual network units and convolutional pooling network units connected in sequence. The convolutional residual network unit includes an autocorrelation information extraction layer, multiple convolutional layers, and a connection layer connected in sequence. The connection layer is used to integrate the output of the last convolutional layer and the input of the autocorrelation information extraction layer to obtain the output of the convolutional residual network unit. The convolutional pooling network unit includes a convolutional layer, a normalization layer, and a pooling layer connected in sequence. Specifically, after the task generation is completed, all tasks are divided into a support set and a query set, where both the support set and the query set contain different amounts of DFS data for various gestures. In the embedded network, the support set data and query set data will learn self-association information through the residual blocks of the self-association information extraction layer, and then be input into four convolutional layers and convolutional pooling network units for feature extraction.

[0060] Furthermore, the self-correlation information extraction layer plays a crucial role in learning key features of the DFS image, thereby improving the model's recognition accuracy. The working process of the self-correlation information extraction layer is as follows:

[0061] Extract multiple channel features from the input matrix, where the input matrix is ​​an H*W*C DFS matrix, where H is the height of the DFS matrix, W is the width of the DFS matrix, C is the number of channels in the DFS matrix, and each 1*C matrix in H*W is a channel feature.

[0062] Multiply one of the channel features bitwise with all the surrounding [-dU,dU]×[-dV,dV] channel features, and fill the matrix at the edges with zeros to obtain U*V 1*C matrices. Process the U*V 1*C matrices to obtain an output matrix with tensor size H*W*U*V*C, where dU and dV are preset hyperparameters, U=2dU+1, V=2dV+1.

[0063] Furthermore, the matrix of size H*W*U*V*C output from the self-correlation information extraction layer is input into four convolutional layers, processed into tensors of size H*W*1*1*C, then compressed back to a matrix of size H*W*C, and finally added to the DFS matrix of the convolutional layer with 1×1 convolutional kernel (residual connection) to obtain the residual block self-correlation information of the DFS matrix of the input convolutional residual network unit.

[0064] Please continue to refer to Figure 2The relation network consists of a feature concatenation layer, convolutional pooling network units, a fully connected layer, and a classification layer connected in sequence. The feature concatenation layer is used to concatenate the support set data features (i.e., the features output by the DFS matrix after passing through the support set module) and the query set data features (i.e., the features output by the DFS matrix after passing through the query set module) of the embedded network output to obtain a concatenated feature vector. The convolutional pooling network units are used to perform dimensionality reduction on the concatenated feature vector. The convolutional pooling network units consist of a convolutional layer, a normalization layer, and a pooling layer connected in sequence. The fully connected layer is used to predict the relation score based on the dimensionality-reduced concatenated feature vector. The classification layer is used to determine the predicted gesture classification result of the query set data based on the relation score.

[0065] Specifically, the role of the relation network is to enable the model to learn a transferable depth metric to establish relationships between features in the support set and the query set. If the DFS features in the support set and the query set belong to the same gesture, the output relation score is set close to 1; otherwise, the output relation score is set close to 0. Each DFS data point in the query set is concatenated with the DFS data points corresponding to all gesture categories in the support set to form a concatenated feature vector. This vector is then input into the convolutional pooling network units and fully connected layers of the relation network to obtain the relation score. For each DFS data point in the query set, it has a relation score with each category of DFS data in the support set. The gesture category of the DFS data point in the support set with the highest relation score is the classification result of the DFS data in that query set.

[0066] According to some embodiments of the present invention, in step S120, the step of preprocessing multiple channel state information to obtain a training dataset includes, but is not limited to, the following steps:

[0067] Step S210: Based on the weighted antenna conjugate multiplication, phase offset segments in multiple channel state information are removed to obtain the first preprocessed dataset;

[0068] Step S220: Use wavelet transform algorithm to remove noise from the channel state information in the first preprocessed dataset to obtain the second preprocessed dataset;

[0069] Step S230: Principal component analysis algorithm is used to reduce the dimensionality of the channel state information in the second preprocessed dataset to obtain the third preprocessed dataset;

[0070] Step S240: Perform a short-time Fourier transform on the channel state information in the third preprocessed data to obtain the training dataset.

[0071] In step S210 of some embodiments, weighted antenna conjugate multiplication can remove phase offset and some noise from the acquired multiple channel state information. The principle of weighted antenna conjugate multiplication is that two antennas on the same receiver have the same phase offset. From the antennas of the receiver, an antenna with the largest dynamic response and an antenna with the largest static response are selected. The CSI signals acquired by these two antennas are multiplied with corresponding weights and then conjugate multiplication is performed (the selection of weights is related to the CSI value of the selected antenna). Then, the CSI signals after conjugate multiplication are passed through a low-pass filter and a high-pass filter to obtain multiple channel state information with phase offset and some noise removed, forming a first preprocessed dataset.

[0072] In step S220 of some embodiments, after removing the phase offset, the CSI signal in the first preprocessed dataset still contains a significant amount of noise. This is because the effective signal has large wavelet transform coefficients at many wavelet transform scales, while the wavelet transform coefficients of noise decay rapidly with increasing wavelet transform scale. Therefore, the wavelet transform coefficients of the effective signal exhibit strong correlation at different wavelet scales, while the correlation of the wavelet transform coefficients of noise is weaker. Based on this, embodiments of the present invention decompose the CSI signal into wavelet coefficients at different scales. The wavelet coefficients at lower scales are multiplied by wavelet coefficients at a higher scale to obtain a correlation coefficient. After normalizing the correlation coefficient, its absolute value is compared with the absolute value of the wavelet coefficient at the lower scale. Specifically, for a wavelet coefficient a1 at a certain scale, if the absolute value |b1| of the normalized correlation coefficient is larger than the absolute value |a1| of the wavelet coefficient at that scale, then coefficient a1 is extracted; otherwise, it is not extracted and is set to zero. After extracting the important edge points, repeat this operation to extract the less important edge points until the energy of the unextracted points in the wavelet coefficients at that scale is approximately equal to the noise energy threshold at that scale. This operation can continue on the wavelet coefficients at the next scale until the specified wavelet scale is reached, thus obtaining the second preprocessed dataset formed after CSI signal denoising.

[0073] In step S230 of some embodiments, since the CSI signal data volume in the second preprocessing dataset is large, dimensionality reduction of the CSI signal is required. This embodiment of the invention uses Principal Component Analysis (PCA) to reduce the dimensionality of the CSI signal. The PCA algorithm first calculates the covariance matrix of the CSI signal, and then calculates the eigenvectors and corresponding eigenvalues ​​of the covariance matrix. After calculating the eigenvalues, the PCA algorithm arranges the eigenvalues ​​from largest to smallest to determine the importance level of the corresponding eigenvectors. The eigenvectors corresponding to the first two eigenvalues ​​are selected as the CSI feature matrix for subsequent processing. After denoising by the PCA algorithm, the CSI signal in the second preprocessing dataset forms the third preprocessing dataset, which can reduce the complexity of the CSI signal while discarding less important information and reducing noise.

[0074] In step S240 of some embodiments, the CSI feature matrix in the third preprocessed dataset is converted into DFS data through a short-time Fourier transform operation.

[0075] According to some embodiments of the present invention, step S140, which involves training the embedding network and the relation network based on data from single-domain tasks and multi-domain tasks respectively, includes the following steps:

[0076] Step S310: Input the data of the single-domain task into the embedding network and the relational network in sequence for prediction and classification, and obtain the first predicted gesture classification result of the single-domain task data. Calculate the first loss value of the embedding network and the relational network on the single-domain task based on the first predicted gesture classification result, and update the parameters of the embedding network and the relational network based on the first loss value.

[0077] Step S320: Input the multi-domain task data into the updated embedding network and relation network based on the single-domain task data in sequence for prediction and classification, and obtain the second predicted gesture classification result of the multi-domain task data. Calculate the second loss value of the embedding network and relation network on the multi-domain task based on the second predicted gesture classification result, and update the parameters of the embedding network and relation network based on the second loss value.

[0078] In this embodiment, the data of a single-domain task is first input into the embedding network and the relational network sequentially for training and updating the parameters of the embedding network and the relational network, which can improve the learning efficiency of the embedding network and the relational network and the accuracy of prediction for single-domain tasks. Then, the data of a multi-domain task is input into the embedding network and the relational network trained based on the single-domain task data for further training, which improves the adaptability of the embedding network and the relational network in different domains. Finally, a cross-domain gesture recognition model is formed based on the trained embedding network and the relational network.

[0079] This invention also provides a gesture recognition method, comprising the following steps:

[0080] Obtain the channel state information to be identified;

[0081] The channel state information is input into the cross-domain gesture recognition model to obtain the gesture classification result;

[0082] The cross-domain gesture recognition model is trained using the cross-domain gesture recognition model training method described in the previous embodiment. The cross-domain gesture recognition model of this invention exhibits strong adaptability to data from different domains. It can achieve accurate gesture recognition for channel state information acquired in different scenarios. Furthermore, using the gesture recognition method of this embodiment, users do not need to wear sensing devices or have their images captured to achieve gesture recognition, thus improving user privacy and security.

[0083] This invention also provides a cross-domain gesture recognition model training device, comprising:

[0084] The first module is used to acquire multiple channel state information from different domains;

[0085] The second module is used to preprocess multiple channel state information to obtain the training dataset;

[0086] The third module is used to perform task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks.

[0087] The fourth module is used to train the embedding network and the relation network based on single-domain task and multi-domain task data, respectively. The embedding network is used to divide the input data into a support set and a query set, and to extract features from the data in the support set and the query set, respectively, to obtain the support set data features and the query set data features. The relation network is used to determine the relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score.

[0088] The fifth module is used to obtain a cross-domain gesture recognition model based on the trained embedding network and relational network.

[0089] It is understood that the content of the above cross-domain gesture recognition model training method embodiments is applicable to this virtual device embodiment. The specific functions implemented by this device embodiment are the same as those of the above cross-domain gesture recognition model training method embodiments, and the beneficial effects achieved are also the same as those achieved by the above cross-domain gesture recognition model training method embodiments.

[0090] Reference Figure 3 , Figure 3This is a schematic diagram of an electronic device provided in one embodiment of the present invention. The electronic device of this embodiment includes one or more control processors and a memory. Figure 3 The example consists of a control processor and a memory.

[0091] The control processor and memory can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0092] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to a control processor, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0093] Those skilled in the art will understand that Figure 3 The device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0094] The non-transient software programs and instructions required to implement the cross-domain gesture recognition model training method or gesture recognition method applied to electronic devices in the above embodiments are stored in the memory. When executed by the controlled processor, the cross-domain gesture recognition model training method or gesture recognition method applied to electronic devices in the above embodiments is executed.

[0095] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more control processors, causing the one or more control processors to perform the cross-domain gesture recognition model training method or gesture recognition method in the above method embodiments.

[0096] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0097] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for training a cross-domain gesture recognition model, characterized in that, Includes the following steps: Obtain channel state information from multiple domains; The channel state information from multiple sources is preprocessed to obtain a training dataset; Perform task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks; An embedding network and a relation network are trained based on data from single-domain tasks and multi-domain tasks, respectively. The embedding network is used to divide the input data into a support set and a query set, and to extract features from the data in the support set and the query set, respectively, to obtain support set data features and query set data features. The relation network is used to determine a relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score. A cross-domain gesture recognition model is obtained based on the trained embedding network and relation network; The embedded network includes a support set module and a query set module. The support set module is used to extract features from the data assigned to the support set, and the query set module is used to extract features from the data assigned to the query set. Both the support set module and the query set module include convolutional residual network units and convolutional pooling network units connected in sequence; the convolutional residual network unit includes an autocorrelation information extraction layer, multiple convolutional layers and a connection layer connected in sequence, and the connection layer is used to integrate the output of the last convolutional layer and the input of the autocorrelation information extraction layer to obtain the output of the convolutional residual network unit; the convolutional pooling network unit includes a convolutional layer, a normalization layer and a pooling layer connected in sequence. The relationship network comprises a feature concatenation layer, a convolutional pooling network unit, a fully connected layer, and a classification layer connected in sequence. The feature concatenation layer is used to concatenate the support set data features and query set data features output by the embedding network to obtain a concatenated feature vector. The convolutional pooling network unit is used to perform dimensionality reduction on the concatenated feature vector. The fully connected layer is used to predict the relationship score based on the dimensionality-reduced concatenated feature vector. The classification layer is used to determine the predicted gesture classification result of the query set data based on the relationship score. The self-correlation information extraction layer is used to extract multiple channel features of the input matrix. One of the channel features is multiplied bitwise with all the surrounding channel features of [-dU,dU]×[-dV,dV] to obtain the output matrix of the self-correlation information extraction layer, where dU and dV are preset hyperparameters.

2. The cross-domain gesture recognition model training method according to claim 1, characterized in that, The preprocessing of multiple channel state information sets to obtain the training dataset includes the following steps: The phase offset segments in multiple channel state information are removed by the antenna conjugate multiplication based on weights to obtain the first preprocessed dataset. The wavelet transform algorithm is used to remove noise from the channel state information in the first preprocessed dataset to obtain the second preprocessed dataset; Principal component analysis algorithm is used to reduce the dimensionality of the channel state information in the second preprocessed dataset to obtain the third preprocessed dataset. The channel state information in the third preprocessed data is subjected to a short-time Fourier transform to obtain the training dataset.

3. The cross-domain gesture recognition model training method according to claim 1, characterized in that, The training of the embedding network and the relation network based on data from single-domain tasks and multi-domain tasks, respectively, includes the following steps: The data of the single-domain task is sequentially input into the embedding network and the relational network for prediction and classification to obtain the predicted gesture classification result of the single-domain task data. The first loss value of the embedding network and the relational network on the single-domain task is calculated based on the predicted gesture classification result of the single-domain task data. The parameters of the embedding network and the relational network are updated based on the first loss value. The data from the multi-domain task is sequentially input into the embedding network and the relational network updated based on the data from the single-domain task for prediction and classification, thereby obtaining the predicted gesture classification result of the multi-domain task data. The second loss value of the embedding network and the relational network on the multi-domain task is calculated based on the predicted gesture classification result of the multi-domain task data, and the parameters of the embedding network and the relational network are updated based on the second loss value.

4. A gesture recognition method, characterized in that, Includes the following steps: Obtain the channel state information to be identified; The channel state information is input into the cross-domain gesture recognition model to obtain the gesture classification result; The cross-domain gesture recognition model is trained by the cross-domain gesture recognition model training method according to any one of claims 1 to 3.

5. A cross-domain gesture recognition model training device, characterized in that, include: The first module is used to acquire multiple channel state information from different domains; The second module is used to preprocess the multiple channel state information to obtain a training dataset; The third module is used to perform task generation operations on the training dataset to obtain single-domain tasks and multi-domain tasks. The fourth module is used to train an embedding network and a relation network based on data from single-domain tasks and multi-domain tasks, respectively. The embedding network is used to divide the input data into a support set and a query set, and to extract features from the data in the support set and the query set, respectively, to obtain support set data features and query set data features. The relation network is used to determine a relation score based on the support set data features and the query set data features, and to determine the predicted gesture classification result of the query set data based on the relation score. The fifth module is used to obtain a cross-domain gesture recognition model based on the trained embedding network and relation network; The embedded network includes a support set module and a query set module. The support set module is used to extract features from the data assigned to the support set, and the query set module is used to extract features from the data assigned to the query set. Both the support set module and the query set module include convolutional residual network units and convolutional pooling network units connected in sequence; the convolutional residual network unit includes an autocorrelation information extraction layer, multiple convolutional layers and a connection layer connected in sequence, and the connection layer is used to integrate the output of the last convolutional layer and the input of the autocorrelation information extraction layer to obtain the output of the convolutional residual network unit; the convolutional pooling network unit includes a convolutional layer, a normalization layer and a pooling layer connected in sequence. The relationship network comprises a feature concatenation layer, a convolutional pooling network unit, a fully connected layer, and a classification layer connected in sequence. The feature concatenation layer is used to concatenate the support set data features and query set data features output by the embedding network to obtain a concatenated feature vector. The convolutional pooling network unit is used to perform dimensionality reduction on the concatenated feature vector. The fully connected layer is used to predict the relationship score based on the dimensionality-reduced concatenated feature vector. The classification layer is used to determine the predicted gesture classification result of the query set data based on the relationship score. The self-correlation information extraction layer is used to extract multiple channel features of the input matrix. One of the channel features is multiplied bitwise with all the surrounding channel features of [-dU,dU]×[-dV,dV] to obtain the output matrix of the self-correlation information extraction layer, where dU and dV are preset hyperparameters.

6. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the cross-domain gesture recognition model training method as described in any one of claims 1 to 3 or the gesture recognition method as described in claim 4.

7. A computer-readable storage medium storing a processor-executable program, characterized in that, When the processor executes the program, it is used to implement the cross-domain gesture recognition model training method as described in any one of claims 1 to 3 or the gesture recognition method as described in claim 4.

Images