Gesture recognition model establishing method and device and gesture recognition method and device

Abstract

The invention relates to the technical field of artificial intelligence, and provides a gesture recognition model establishing method and device and a gesture recognition method and device. The method comprises the following steps: acquiring inertial measurement data and gesture light current data; determining a first training sample corresponding to the inertial measurement data and a second training sample corresponding to the gesture photocurrent data; inputting the first training sample into a generator of a conditional generative adversarial network model to generate a pseudo sample; inputting the second training sample and the pseudo sample into a discriminator of a conditional generative adversarial network model to obtain a classification result; updating parameters of the conditional generative adversarial network model according to the classification result to obtain a trained conditional generative adversarial network model; and fusing the trained conditional generative adversarial network model and a pre-trained gesture classifier to obtain a gesture recognition model. Through the embodiment of the invention, the deployment cost of human body gesture recognition based on the photocurrent data can be greatly reduced on the premise of ensuring the gesture recognition precision.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to methods and apparatus for establishing gesture recognition models, and gesture recognition methods and apparatus. Background Technology

[0002] Solar cell-based gesture recognition is an emerging technology enabling contactless, battery-free human-computer interaction. It utilizes photocurrent data generated by solar cells and different photocurrent patterns associated with each gesture for gesture recognition. While this technology holds great potential in smart homes, autonomous vehicles, and interactive entertainment, its practical application faces significant challenges due to the need for pre-collecting large amounts of photocurrent data and its difficulty in recognizing unfamiliar and complex gestures. Therefore, a method for establishing a gesture recognition model is urgently needed. This model, in applications using solar cells for human gesture recognition, would not require collecting photocurrent data from solar cells. Instead, it would utilize sensor data measured by the Inertial Measurement Unit (IMU) of a wearable device to achieve cross-modal complex gesture recognition, thereby significantly reducing the deployment cost of photocurrent-based human gesture recognition while maintaining accuracy. Summary of the Invention

[0003] Given that current solar cell-based gesture recognition methods require the prior collection of large amounts of photocurrent data and struggle to recognize unfamiliar and complex gestures, this solution is proposed to overcome or at least partially solve these problems.

[0004] On the one hand, the purpose of some embodiments of this specification is to provide a method for establishing a gesture recognition model, the method comprising:

[0005] Acquire inertial measurement data and gesture photocurrent data;

[0006] Determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data;

[0007] Establish a conditional generative adversarial network model;

[0008] The first training sample is input into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise.

[0009] The second training sample and the pseudo sample are input into the discriminator of the conditional generative adversarial network model so that the discriminator can classify the second training sample and the pseudo sample to obtain the classification result.

[0010] The parameters of the conditional generative adversarial network model are updated based on the classification results;

[0011] Repeat the above process of generating pseudo-samples using the first training samples, judging the pseudo-samples and the second training samples, and updating the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained.

[0012] By fusing the trained conditional generative adversarial network model and the pre-trained gesture classifier, a gesture recognition model is obtained.

[0013] Furthermore, inertial measurement data and gesture photocurrent data are acquired, including:

[0014] Acquire inertial measurement data from smart portable devices;

[0015] The gesture photocurrent data is obtained from a smart wearable device.

[0016] Further, acquiring the gesture photocurrent data from the smart wearable device includes:

[0017] Real-time measurement of light intensity and light incident angle at the light acquisition component of the smart wearable device, as well as acceleration data of the user's target hand and the smart wearable device, during user interaction; wherein, relative motion occurs between the target hand and the smart wearable device, causing the smart wearable device to generate gesture photocurrent data;

[0018] Acquire the initial position of the target hand and the smart wearable device, the shape and size information of the target hand, and the shape and size information of the light acquisition component;

[0019] Calculate the real-time relative position between the target hand and the smart wearable device based on the initial position and acceleration data of the target hand and the smart wearable device.

[0020] Based on the real-time relative position, the shape and size information of the target hand, and the shape and size information of the light acquisition component, the angle threshold between the surface of the light acquisition component and the space that can absorb light is calculated.

[0021] The gesture photocurrent data is calculated using the light intensity, light incident angle, angle threshold, and shape and size information of the light acquisition component.

[0022] Furthermore, the generator is constructed using a convolutional neural network, and the discriminator includes at least two convolutional layers and an activation function layer.

[0023] Further, updating the parameters of the conditional generative adversarial network model based on the classification results includes:

[0024] Based on the classification results, the loss is calculated using a preset loss function;

[0025] Based on the loss, the parameters of the generator and discriminator in the conditional generative adversarial network model are updated using the backpropagation algorithm.

[0026] Furthermore, by fusing the trained conditional generative adversarial network model and the pre-trained gesture classifier, a gesture recognition model is obtained, including:

[0027] The generator in the trained conditional generative adversarial network model is cascaded with the gesture classifier to obtain a gesture recognition model.

[0028] On the other hand, some embodiments of this specification also provide a gesture recognition model building apparatus, the apparatus comprising:

[0029] The acquisition module is used to acquire inertial measurement data and gesture photocurrent data;

[0030] The determination module is used to determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data;

[0031] Establish a module for building conditional generative adversarial network models;

[0032] The generation module is used to input the first training sample into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise.

[0033] The judgment module is used to input the second training sample and the pseudo sample into the discriminator of the conditional generative adversarial network model, so that the discriminator classifies the second training sample and the pseudo sample to obtain a classification result;

[0034] An update module is used to update the parameters of the conditional generative adversarial network model based on the classification results;

[0035] The repeat module is used to repeatedly generate pseudo samples using the first training samples, judge the pseudo samples and the second training samples, and update the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained.

[0036] The fusion module is used to fuse the trained conditional generative adversarial network model and the pre-trained gesture classifier to obtain a gesture recognition model.

[0037] Based on the same inventive concept, and on the other hand, the purpose of some embodiments of this specification is to provide a method comprising:

[0038] Receive inertial measurement data to be processed;

[0039] The inertial measurement data to be processed is input into the gesture recognition model trained using the method described in any of the above embodiments to obtain the gesture classification result corresponding to the inertial measurement data to be processed.

[0040] Based on the same inventive concept, some embodiments of this specification also provide an apparatus comprising:

[0041] The receiving module is used to receive the inertial measurement data to be processed.

[0042] The recognition module is used to input the inertial measurement data to be processed into the gesture recognition model trained using the method described in any of the above embodiments, and to obtain the gesture classification result corresponding to the inertial measurement data to be processed.

[0043] On the other hand, some embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, which, when run by the processor, executes instructions for the methods described above.

[0044] On the other hand, some embodiments of this specification also provide a computer storage medium having a computer program stored thereon, which, when run by the processor of a computer device, executes instructions for the methods described above.

[0045] On the other hand, some embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the methods described above.

[0046] Some embodiments of this specification provide one or more technical solutions, which have at least the following technical effects:

[0047] The embodiments described in this specification first automatically acquire inertial measurement data and gesture photocurrent data to determine the first and second training samples. A generator then uses the first training sample as a condition and generates pseudo-samples corresponding to the condition using noise. A discriminator classifies the second training sample and pseudo-samples to obtain classification results. Based on the classification results, the parameters of the generator and discriminator are iteratively updated to obtain a trained conditional generative adversarial network (GAN) model. This quickly and accurately determines the nonlinear mapping relationship between inertial measurement data and gesture photocurrent data. The trained GAN model and a pre-trained gesture classifier are then fused to convert inertial measurement data into corresponding gesture photocurrent data. This eliminates the need to collect photocurrent data from solar cells, enabling gesture recognition using the corresponding gesture photocurrent data. While ensuring gesture recognition accuracy, this significantly reduces the deployment cost of human gesture recognition based on photocurrent data.

[0048] The above description is merely an overview of some embodiments of the technical solutions in this specification. In order to better understand the technical means of some embodiments of this specification and to implement them in accordance with the content of the specification, and to make the above and other objects, features and advantages of some embodiments of this specification more apparent and understandable, specific implementation methods of some embodiments of this specification are given below. Attached Figure Description

[0049] To more clearly illustrate some embodiments or technical solutions in the prior art of this specification, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings:

[0050] Figure 1 A schematic diagram of an implementation system for a gesture recognition model establishment method in some embodiments of this specification is shown;

[0051] Figure 2a A flowchart of a gesture recognition model establishment method is shown in some embodiments of this specification;

[0052] Figure 2b This specification shows a schematic diagram of the gesture recognition model training process in some embodiments;

[0053] Figure 3 This is a schematic diagram illustrating the steps for acquiring inertial measurement data and gesture photocurrent data in some embodiments of this specification;

[0054] Figure 4This diagram illustrates the steps for updating the parameters of a conditional generative adversarial network model in some embodiments of this specification.

[0055] Figure 5 This is a schematic diagram comparing the accuracy of gesture photocurrent data in some embodiments of this specification;

[0056] Figure 6 A flowchart of a gesture recognition method in some embodiments of this specification is shown;

[0057] Figure 7 This is a schematic diagram of the structure of a gesture recognition model establishment device in some embodiments of this specification;

[0058] Figure 8 This is a schematic diagram of the structure of a gesture recognition device in some embodiments of this specification;

[0059] Figure 9 This is a schematic diagram of the computer device structure provided in some embodiments of this specification.

[0060] [Explanation of Labels in the Attached Image]

[0061] 101. Terminal;

[0062] 102. Server;

[0063] 701. Acquisition Module;

[0064] 702. Determine the module;

[0065] 703. Create a module;

[0066] 704, Generation Module;

[0067] 705. Judgment Module;

[0068] 706. Update module;

[0069] 707. Duplicate module;

[0070] 708. Fusion Module;

[0071] 801. Receiver module;

[0072] 802. Identification module;

[0073] 902. Computer equipment;

[0074] 904, Processor;

[0075] 906. Memory;

[0076] 908. Drive mechanism;

[0077] 910. Input / output interfaces;

[0078] 912. Input devices;

[0079] 914. Output devices;

[0080] 916. Presentation equipment;

[0081] 918. Graphical User Interface;

[0082] 920. Network interface;

[0083] 922. Communication link;

[0084] 924. Communication bus. Detailed Implementation

[0085] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in some embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on some embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0086] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0087] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of relevant laws and regulations.

[0088] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0089] like Figure 1The diagram illustrates an implementation system for a gesture recognition model establishment method according to an embodiment of the present invention. The system may include a terminal 101 and a server 102. The terminal 101 and server 102 communicate via a network, which may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and is connected to a website, user equipment (e.g., a computing device), and a backend system. Staff can send a gesture recognition model establishment request to the server 102 via the terminal 101. Upon receiving the request, the server 102 retrieves data from its database for calculation and processing to obtain the modeling result.

[0090] In the embodiments of this specification, the server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0091] In an optional embodiment, terminal 101 may be an electronic device, including but not limited to smart portable devices, smart wearable devices, and other similar types. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, Windows, etc. Of course, terminal 101 is not limited to the aforementioned physical electronic devices; it may also be software running on the aforementioned electronic devices.

[0092] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, it may include multiple terminals 101, and this specification does not impose any restrictions.

[0093] Figure 2a This is a flowchart illustrating a gesture recognition model establishment method provided by an embodiment of the present invention. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel. Specifically, as shown... Figure 2a As shown, when applied to the server side described above, the method may include:

[0094] S201: Acquire inertial measurement data and gesture photocurrent data;

[0095] It can be understood that, in some embodiments, reference is made to the appendix. Figure 2b The diagram shown illustrates the training process of a gesture recognition model. Inertial measurement data, also known as IMU data, is collected from the user's device during operation. Different individuals have different operating habits, resulting in variations in the speed and acceleration of the same gesture. Therefore, IMU data can be collected from the user's device to comprehensively represent the acceleration caused by human operating habits.

[0096] In some embodiments, inertial measurement data can be represented as:

[0097]

[0098] Where H(t) represents inertial measurement data, h n Let a be the transfer function of the acceleration mapping generated by operating habits. n (t) represents the acceleration of the nth time at time t, where n is the ordinal number.

[0099] In some embodiments, the gesture photocurrent data is the photocurrent signal generated when sunlight shines on a light-collecting component such as a solar panel, and the light intensity at the light-collecting component changes due to a change in gesture (e.g., blocking sunlight at the solar panel).

[0100] See attached document Figure 3 In some embodiments, acquiring inertial measurement data and gesture photocurrent data may include:

[0101] S301: Acquire inertial measurement data from smart portable devices;

[0102] S302: Obtain the gesture photocurrent data from the smart wearable device.

[0103] It can be understood that, in some embodiments, inertial measurement data can be acquired by a smart portable device with an accelerometer, and gesture photocurrent data can be acquired by light-collecting components such as solar panels in a smart wearable device.

[0104] Furthermore, in some embodiments, acquiring the gesture photocurrent data from a smart wearable device includes:

[0105] Real-time measurement of light intensity and light incident angle at the light acquisition component in the smart wearable device, as well as acceleration data of the user's target hand and the smart wearable device, during user interaction;

[0106] In this process, relative motion occurs between the target hand and the smart wearable device, causing the smart wearable device to generate gesture photocurrent data;

[0107] Acquire the initial position of the target hand and the smart wearable device, the shape and size information of the target hand, and the shape and size information of the light acquisition component;

[0108] Calculate the real-time relative position between the target hand and the smart wearable device based on the initial position and acceleration data of the target hand and the smart wearable device.

[0109] Based on the real-time relative position, the shape and size information of the target hand, and the shape and size information of the light acquisition component, the angle threshold between the surface of the light acquisition component and the space that can absorb light is calculated.

[0110] The gesture photocurrent data is calculated using the light intensity, light incident angle, angle threshold, and shape and size information of the light acquisition component.

[0111] In some embodiments, the smart wearable device can be a smartwatch, smart glasses, or a flexible wearable haptic device. Taking a smartwatch as an example, if the user wears it on their left hand, their right hand can operate the smartwatch's touchscreen and buttons to interact with it. At this time, there will be relative movement between the user's right hand and the smartwatch. This relative movement can manifest as the right hand moving or a change in the right hand's gesture, which is not limited here. Correspondingly, the area of ​​sunlight blocked by the right hand from the smartwatch changes accordingly, and this change can be physically reflected as a change in the gesture photocurrent data.

[0112] Specifically, in some embodiments, the light-harvesting component in the smart wearable device can be a solar panel or solar cell, in which case the smart wearable device can utilize the solar cell for power supply. This document does not limit the specific shape of the light-harvesting component. Furthermore, since the photocurrent in the solar cell is affected by the intensity of the incident light and the solar cell itself, and the intensity of the incident light can be measured by photoelectric sensors and photosensitive sensors, the light-harvesting component can also include photoelectric sensors and photosensitive sensors to obtain the intensity information of the incident light, thereby calculating the gesture photocurrent data.

[0113] However, using photoelectric sensors and photosensitive sensors requires introducing additional sensor devices into smart wearable devices to obtain the intensity information of incident light across the entire area of ​​the light-collecting component in the smart wearable device. Therefore, in a preferred embodiment, the light-collecting component in the smart wearable device is a solar panel or solar cell, which can both enable the use of solar cells to power the smart wearable device, thereby improving its portability and reducing deployment costs, and also reduce the cost of introducing new sensor devices into a smart wearable device that already only has solar panels and solar cells.

[0114] Furthermore, in some embodiments, to facilitate comparison of the efficiency of different solar cells, the standard photocurrent density under standard illumination conditions known as the global standard spectrum (AM1.5g) is defined as:

[0115]

[0116] in, Let represent the standard photocurrent density, q represent the elementary charge, h is Planck's constant, c0 is the speed of light in vacuum, λ represents the wavelength of the incident light, a(λ) represents the absorption efficiency of the solar cell, and I(λ) represents the light intensity at wavelength λ.

[0117] Since there is a linear relationship between current density and light intensity, the current density K under the preset light intensity I... SC for:

[0118]

[0119] Among them, I * This represents the light radiation power under the global standard spectrum (AM1.5g).

[0120] Then, given the current density at a preset light intensity I, the standard photocurrent without obstruction can be calculated using the following formula:

[0121]

[0122] Where K represents the standard photocurrent, S represents the shape factor of the light-collecting component, and K SC θ represents the current density, and θ represents the angle between the beam and the surface normal.

[0123] Specifically, θ represents the angle of incidence. As the target hand moves, the area illuminating the light-collecting component and the angle of incidence θ change accordingly. Therefore, it is necessary to calculate the angle threshold between the surface of the light-collecting component and the space where light can be absorbed based on the movement of the target hand, thereby determining the distribution range of the angle of incidence θ. First, based on the initial position and acceleration data of the target hand and the smart wearable device, the real-time relative position between the target hand and the smart wearable device needs to be calculated. Then, using the real-time relative position, the shape and size information of the target hand, and the shape and size information of the light-collecting component, the angle threshold between the surface of the light-collecting component and the space where light can be absorbed is calculated.

[0124] Assuming the hand is positioned above the light-collecting component, with an initial distance of d between the hand and the component, and the hand moves with a constant acceleration a in a vertical direction away from the component, then at time t, the distance between the hand and the solar cell can be expressed as d + 1 / 2at. 2 At this point, the angle threshold can be calculated using the following formula:

[0125]

[0126] Where, θ th1 (t) and θ th2 (t) represents the angle threshold, R H R represents the radius of the circular surface of the hand. S Let d represent the radius of the circular surface of the solar cell, d represent the initial distance, a represent the acceleration, and t represent the time.

[0127] After obtaining the angle threshold, the gesture photocurrent data can be calculated using the following formula:

[0128]

[0129] Where K′(t) represents, θ th1 (t) and θ th2 (t) represents the angle threshold, S represents the shape factor of the light acquisition component, and K SC θ represents the current density, and θ represents the angle between the beam and the surface normal.

[0130] S202: Determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data;

[0131] It can be understood that the embodiments of this specification aim to establish a nonlinear mapping relationship model from inertial measurement data to gesture photocurrent data. However, the nonlinear mapping relationship involved is very complex. Therefore, it is necessary to use the nonlinear fitting capability of deep learning to realize the complex transformation from the first training sample corresponding to the inertial measurement data to the second training sample corresponding to the gesture photocurrent data.

[0132] Furthermore, in some embodiments, before determining the first training sample corresponding to the inertial measurement data, it is necessary to perform data preprocessing on the inertial measurement data. Preprocessing methods may include: Z-score standardization, data segmentation, and interpolation.

[0133] The Z-score standardization process is as follows:

[0134]

[0135] Where z represents standardized data, x represents raw data, μ represents the mean, and σ represents the standard deviation.

[0136] Data segmentation divides a continuous data stream into smaller, pre-defined segments or windows for further analysis; interpolation techniques are applied to fill in any missing or incomplete data points, ensuring a complete and consistent set of input signals.

[0137] S203: Establish a conditional generative adversarial network model;

[0138] In some embodiments, the generator in a conditional generative adversarial network (GAN) model is constructed using a convolutional neural network (CNN), and the discriminator includes at least two convolutional layers and an activation function layer. CNNs are commonly used for time series signal tasks. Their architecture includes multiple convolutional layers responsible for extracting features from the input data, as well as pooling or downsampling layers responsible for extracting abstract features. The activation function in the discriminator, such as the sigmoid function, maps the outputs of the two convolutional layers to probabilities between [0, 1]. These probabilities represent the likelihood that the discriminator's input is real or fake.

[0139] S204: Input the first training sample into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise;

[0140] In some embodiments, the generator receives random noise, such as Gaussian noise, in addition to the first training sample. The generator uses the first training sample as a condition, which can also be regarded as additional label feature information. The generator connects the first training sample and the noise together through a fully connected layer as the input of the hidden layer, thereby taking the real label as part of the feature and obtaining a labeled pseudo sample. That is, the pseudo sample generated by the generator is a conditional sample that can obtain the real sample corresponding to the pseudo sample.

[0141] Specifically, in some embodiments, the generator's training process takes a first training sample and a noise vector as input. The first training sample guides the generator to produce signals of a specific category (i.e., the second training sample), while the noise vector introduces randomness, prompting the generator to create diverse and realistic signals. This helps the generator create diverse and realistic pseudo-samples; therefore, it is necessary to select a noise vector suitable for the distribution and dimension of the first training sample. The generator typically consists of a multi-layer neural network, including convolutional layers, deconvolutional layers, and activation functions, to convert the first training sample and noise into a photocurrent signal that approximates reality (i.e., pseudo-samples). The generator's training objective is to generate pseudo-samples that are relevant to given conditions to fool the discriminator, and to reduce the difference between the pseudo-samples and the second training sample through a learned mapping function.

[0142] During training, a loss function is used to optimize the generator and discriminator. This typically includes generator loss and adversarial loss. The former measures the difference between the fake samples and the second training samples, while the latter helps the generator produce more realistic signals to deceive the discriminator. During training, the generator and discriminator are trained independently and alternately. The generator receives the first training samples and a noise vector to generate fake samples. These fake samples, along with the second training samples, are input into the discriminator for adversarial training. The parameters are updated by minimizing the loss function to make the fake samples more realistic. The generator's optimization objective is to minimize the loss function so that the distribution of the fake samples matches that of the second training samples. Optimization algorithms such as gradient descent are used to continuously adjust the network parameters and improve the quality of the generated fake samples.

[0143] S205: Input the second training sample and the pseudo sample into the discriminator of the conditional generative adversarial network model so that the discriminator can classify the second training sample and the pseudo sample to obtain a classification result;

[0144] In some embodiments, the discriminator learns the features of the second training sample and the pseudo sample, and determines whether the second training sample and the pseudo sample are real samples or fake samples by judging the degree of matching between the second training sample and the conditions, and the degree of matching between the pseudo sample and the conditions.

[0145] S206: Update the parameters of the conditional generative adversarial network model based on the classification results;

[0146] See attached document Figure 4 In some embodiments, updating the parameters of the conditional generative adversarial network model based on the classification result includes:

[0147] S401: Calculate the loss using a preset loss function based on the classification results;

[0148] S402: Based on the loss, update the parameters of the generator and discriminator in the conditional generative adversarial network model using the backpropagation algorithm.

[0149] In some implementations, the loss function is used to measure the prediction accuracy of the conditional generative adversarial network model. The loss is used for backpropagation updates, and the parameters of the generator and discriminator are iteratively trained multiple times to ensure the generation accuracy of the generator.

[0150] S207: Repeat the above process of generating pseudo-samples using the first training sample, judging the pseudo-samples and the second training sample, and updating the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained.

[0151] In some embodiments, the convergence condition can be that the number of convergences reaches a preset number, or the prediction accuracy reaches a preset accuracy, etc. The generator in the trained conditional generative adversarial network model can accurately convert inertial measurement data into corresponding gesture photocurrent data. (See Appendix) Figure 5 The diagram shown illustrates the accuracy comparison of gesture photocurrent data. The original signal is the actual gesture photocurrent data (second training sample), and the generated signal is the gesture photocurrent data generated by the generator (pseudo-samples output by the generator). a, b, and c represent different gestures. It can be seen that the generator's output can accurately simulate the actual gesture photocurrent data.

[0152] S208: The conditional generative adversarial network model that has been trained and the pre-trained gesture classifier are combined to obtain the gesture recognition model.

[0153] In some embodiments, a gesture recognition model can be obtained by cascading the generator in a trained conditional generative adversarial network (GAN) model with a gesture classifier. The generator's output serves as the input to the gesture classifier, which is pre-trained using real gesture photocurrent data. This classifier can be an existing model in smart wearable devices, directly accessible from solar-powered devices, and can accurately predict user gestures based on the input gesture photocurrent data.

[0154] In some embodiments, the gesture classifier may include two CNN layers, each consisting of a 64-dimensional projection and a 128-dimensional projection, respectively. During training, the gesture classifier goes through 100 epochs with a learning rate of 0.0001.

[0155] Furthermore, in some embodiments, after cascading the generator and gesture classifier in the trained conditional generative adversarial network model, the gesture classifier can be incrementally fine-tuned using the gesture photocurrent data generated by the generator. This allows the gesture classifier to be better suited for predicting user gestures using the gesture photocurrent data generated by the generator. Incremental fine-tuning involves adjusting the network parameters of the gesture classifier using a small amount of gesture photocurrent data generated by the generator, offering the advantage of low debugging costs.

[0156] The embodiments described in this specification first automatically acquire inertial measurement data and gesture photocurrent data to determine the first and second training samples. A generator then uses the first training sample as a condition and generates pseudo-samples corresponding to the condition using noise. A discriminator classifies the second training sample and pseudo-samples to obtain classification results. Based on the classification results, the parameters of the generator and discriminator are iteratively updated to obtain a trained conditional generative adversarial network (GAN) model. This quickly and accurately determines the nonlinear mapping relationship between inertial measurement data and gesture photocurrent data. The trained GAN model and a pre-trained gesture classifier are then fused to convert inertial measurement data into corresponding gesture photocurrent data. This eliminates the need to collect photocurrent data from solar cells, enabling gesture recognition using the corresponding gesture photocurrent data. While ensuring gesture recognition accuracy, this significantly reduces the deployment cost of human gesture recognition based on photocurrent data.

[0157] Based on the same inventive concept, some embodiments of this specification also provide a gesture recognition method, see reference. Figure 6 As shown, in some embodiments, the method may include:

[0158] S601: Receive inertial measurement data to be processed;

[0159] S602: Input the inertial measurement data to be processed into the gesture recognition model trained using the method described in any of the above embodiments to obtain the gesture classification result corresponding to the inertial measurement data to be processed.

[0160] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0161] Corresponding to the gesture recognition model establishment method described above, some embodiments of this specification also provide a gesture recognition model establishment device, see reference. Figure 7As shown, in some embodiments, the apparatus may include:

[0162] The acquisition module 701 is used to acquire inertial measurement data and gesture photocurrent data;

[0163] The determining module 702 is used to determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data;

[0164] Module 703 is established to build a conditional generative adversarial network model;

[0165] The generation module 704 is used to input the first training sample into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise.

[0166] The judgment module 705 is used to input the second training sample and the pseudo sample into the discriminator of the conditional generative adversarial network model, so that the discriminator classifies the second training sample and the pseudo sample to obtain a classification result;

[0167] The update module 706 is used to update the parameters of the conditional generative adversarial network model based on the classification result;

[0168] The repeat module 707 is used to repeatedly generate pseudo samples using the first training samples, judge the pseudo samples and the second training samples, and update the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained.

[0169] The fusion module 708 is used to fuse the trained conditional generative adversarial network model and the pre-trained gesture classifier to obtain a gesture recognition model.

[0170] Based on the same inventive concept, corresponding to the gesture recognition method described above, some embodiments of this specification also provide a gesture recognition device, see reference. Figure 8 As shown, in some embodiments, the apparatus may include:

[0171] The receiving module 801 is used to receive inertial measurement data to be processed;

[0172] The recognition module 802 is used to input the inertial measurement data to be processed into the gesture recognition model trained by the method described in any of the above embodiments, and to obtain the gesture classification result corresponding to the inertial measurement data to be processed.

[0173] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized and agreed upon by the user and fully authorized by all parties.

[0175] It should be noted that the computer program product described in this specification is a software product that mainly implements the methods described in this specification through a computer program.

[0176] Embodiments of this specification also provide a computer device. For example... Figure 9 As shown, in some embodiments of this specification, the computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each processing unit implementing one or more hardware threads. The computer device 902 may also include any memory 906 for storing information of any kind, such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 906 and can run on the processor 904. When the computer program is run by the processor 904, it can execute instructions described in any of the above embodiments. Non-limitingly, for example, the memory 906 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information.

[0177] Furthermore, any memory can provide volatile or non-volatile retention of information. Furthermore, any memory can represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes associated instructions stored in any memory or combination of memories, the computer device 902 can perform any operation of the associated instructions. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive, an optical disk drive, etc.

[0178] Computer device 902 may also include an input / output interface 910 (I / O) for receiving various inputs (via input device 912) and providing various outputs (via output device 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, the input / output interface 910 (I / O), input device 912, and output device 914 may be omitted, and the device may function solely as a computer device within a network. Computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0179] Communication link 922 can be implemented in any way, such as via a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0184] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0185] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0186] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0188] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0189] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0190] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0191] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for establishing a gesture recognition model, characterized in that, The method includes: Acquire inertial measurement data and gesture photocurrent data; Determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data; Establish a conditional generative adversarial network model; The first training sample is input into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise. The second training sample and the pseudo sample are input into the discriminator of the conditional generative adversarial network model so that the discriminator can classify the second training sample and the pseudo sample to obtain the classification result. The parameters of the conditional generative adversarial network model are updated based on the classification results; Repeat the above process of generating pseudo-samples using the first training samples, judging the pseudo-samples and the second training samples, and updating the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained. By fusing the trained conditional generative adversarial network model and the pre-trained gesture classifier, a gesture recognition model is obtained.

2. The method according to claim 1, characterized in that, Acquire inertial measurement data and gesture photocurrent data, including: Acquire inertial measurement data from smart portable devices; The gesture photocurrent data is obtained from a smart wearable device.

3. The method according to claim 2, characterized in that, Obtaining the gesture photocurrent data from a smart wearable device includes: Real-time measurement of light intensity and light incident angle at the light acquisition component of the smart wearable device, as well as acceleration data of the user's target hand and the smart wearable device, during user interaction; wherein, relative motion occurs between the target hand and the smart wearable device, causing the smart wearable device to generate gesture photocurrent data; Acquire the initial position of the target hand and the smart wearable device, the shape and size information of the target hand, and the shape and size information of the light acquisition component; Calculate the real-time relative position between the target hand and the smart wearable device based on the initial position and acceleration data of the target hand and the smart wearable device. Based on the real-time relative position, the shape and size information of the target hand, and the shape and size information of the light acquisition component, the angle threshold between the surface of the light acquisition component and the space that can absorb light is calculated. The gesture photocurrent data is calculated using the light intensity, light incident angle, angle threshold, and shape and size information of the light acquisition component.

4. The method according to claim 1, characterized in that, The generator is constructed using a convolutional neural network, and the discriminator includes at least two convolutional layers and an activation function layer.

5. The method according to claim 1, characterized in that, The parameters of the conditional generative adversarial network model are updated based on the classification results, including: Based on the classification results, the loss is calculated using a preset loss function; Based on the loss, the parameters of the generator and discriminator in the conditional generative adversarial network model are updated using the backpropagation algorithm.

6. The method according to claim 1, characterized in that, By fusing the trained conditional generative adversarial network model and the pre-trained gesture classifier, a gesture recognition model is obtained, including: The generator in the trained conditional generative adversarial network model is cascaded with the gesture classifier to obtain a gesture recognition model.

7. A gesture recognition method, characterized in that, The method includes: Receive inertial measurement data to be processed; The inertial measurement data to be processed is input into the gesture recognition model trained using any one of the methods in claims 1-6 to obtain the gesture classification result corresponding to the inertial measurement data to be processed.

8. A gesture recognition model building device, characterized in that, The device includes: The acquisition module is used to acquire inertial measurement data and gesture photocurrent data; The determination module is used to determine the first training sample corresponding to the inertial measurement data and the second training sample corresponding to the gesture photocurrent data; Establish a module for building conditional generative adversarial network models; The generation module is used to input the first training sample into the generator of the conditional generative adversarial network model, so that the generator uses the first training sample as a condition and generates pseudo samples corresponding to the condition using noise. The judgment module is used to input the second training sample and the pseudo sample into the discriminator of the conditional generative adversarial network model, so that the discriminator classifies the second training sample and the pseudo sample to obtain a classification result; An update module is used to update the parameters of the conditional generative adversarial network model based on the classification results; The repeat module is used to repeatedly generate pseudo samples using the first training samples, judge the pseudo samples and the second training samples, and update the parameters of the conditional generative adversarial network model according to the classification results, until the preset convergence condition is reached, and the trained conditional generative adversarial network model is obtained. The fusion module is used to fuse the trained conditional generative adversarial network model and the pre-trained gesture classifier to obtain a gesture recognition model.

9. A gesture recognition device, characterized in that, The device includes: The receiving module is used to receive the inertial measurement data to be processed. The recognition module is used to input the inertial measurement data to be processed into a gesture recognition model trained using any one of the methods of claims 1-6, and obtain a gesture classification result corresponding to the inertial measurement data to be processed.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-7.

11. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-7.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs instructions according to any one of claims 1-7.

Images