Model training method, grip posture recognition method, device and electronic equipment

Abstract

The application provides a model training method, a grip posture recognition method, a device and an electronic device, and belongs to the technical field of artificial intelligence. The model training method provided by the application comprises: acquiring sensor data and Wi-Fi CSI data corresponding to different grip postures of a first electronic device; performing feature extraction and feature fusion on the sensor data and the Wi-Fi CSI data to obtain fused features; and training a grip posture recognition model based on the fused features to obtain a trained grip posture recognition model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a model training method, a grip posture recognition method, a device, and an electronic device. Background Technology

[0002] Currently, mobile phones and other electronic devices have become indispensable in people's daily lives. Grip recognition for mobile phones helps optimize for specific grip postures, thereby improving the user experience. Related technologies primarily rely on built-in sensors in the phone for grip recognition, such as using time-series data collected by accelerometers and gyroscopes. However, sensor-based grip recognition methods can only identify grip postures based on sensor data, resulting in low accuracy. Summary of the Invention

[0003] This application provides a model training method, a grip posture recognition method, a device, and an electronic device, which can solve the problem that grip posture recognition based solely on sensor data in related technologies results in low accuracy of the recognition results.

[0004] Firstly, a model training method is provided, including:

[0005] Acquire sensor data and Wi-Fi channel status information (CSI) data of the first electronic device under different grip postures;

[0006] Feature extraction and feature fusion are performed on the sensor data and the Wi-Fi CSI data to obtain fused features;

[0007] The grip posture recognition model is trained based on the fused features to obtain the trained grip posture recognition model.

[0008] Secondly, a grip posture recognition method is provided, applied to the grip posture recognition model trained as described in the first aspect, the method comprising:

[0009] Acquire current sensor data and Wi-Fi CSI data from electronic devices;

[0010] The sensor data and the Wi-Fi CSI data are input into the trained grip posture recognition model to obtain the grip posture recognition result.

[0011] Thirdly, a model training device is provided, comprising:

[0012] The first acquisition module is used to acquire sensor data and Wi-Fi channel status information (CSI) data corresponding to different grip postures of the first electronic device.

[0013] The processing module is used to extract and fuse features from the sensor data and the Wi-Fi CSI data to obtain fused features;

[0014] The training module is used to train the grip posture recognition model based on the fused features to obtain the trained grip posture recognition model.

[0015] Fourthly, a grip posture recognition device is provided, applied to the trained grip posture recognition model described in the first aspect, the device comprising:

[0016] The second acquisition module is used to acquire the current sensor data and Wi-Fi CSI data of the electronic device;

[0017] The third acquisition module is used to input the sensor data and the Wi-Fi CSI data into the trained grip posture recognition model and obtain the grip posture recognition result.

[0018] Fifthly, an electronic device is provided, the terminal including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect, or implementing the steps of the method as described in the second aspect.

[0019] In a sixth aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect.

[0020] In a seventh aspect, a chip is provided, the chip including a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0021] Eighthly, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method as described in the first aspect, or to implement the steps of the method as described in the second aspect.

[0022] In this embodiment, sensor data and Wi-Fi CSI data under different grip postures of electronic devices can be fused to train the grip posture recognition model. The trained grip posture recognition model can then infer the grip posture of the electronic device by combining the sensor data and Wi-Fi CSI data. Understandably, sensor data accurately reflects the physical changes in the posture of the electronic device, while Wi-Fi CSI data reflects channel characteristic fluctuations caused by the hand or other objects obstructing the antenna. This effectively avoids misidentification of grip postures due to environmental factors, thereby significantly improving the accuracy of grip posture recognition. The solution provided in this application solves the problems of single data, inaccurate recognition results, and poor anti-interference performance caused by relying solely on sensor data to recognize grip postures in related technologies. Attached Figure Description

[0023] Figure 1 This is a flowchart of a model training method provided in an embodiment of this application;

[0024] Figure 2a This is a structural diagram of a grip posture recognition model applicable to embodiments of this application;

[0025] Figure 2b yes Figure 2a Structure diagram of the sensor feature extractor;

[0026] Figure 2c yes Figure 2a Structure diagram of the CSI feature extractor;

[0027] Figure 3a This is a structural diagram of the first feature fusion module in a grip posture recognition model applicable to embodiments of this application;

[0028] Figure 3b This is a structural diagram of the second feature fusion module in a grip posture recognition model applicable to embodiments of this application;

[0029] Figure 3c This is a structural diagram of the third feature fusion module in a grip posture recognition model applicable to embodiments of this application;

[0030] Figure 3d This is a structural diagram of a global feature fusion module in a grip posture recognition model applicable to embodiments of this application;

[0031] Figure 4a This is one of the schematic diagrams illustrating the wireless connection scenarios to which the model training method provided in this application is applicable;

[0032] Figure 4b This is the second schematic diagram of a wireless connection scenario to which the model training method provided in this application embodiment is applicable;

[0033] Figure 4c This is the third schematic diagram of a wireless connection scenario to which the model training method provided in this application embodiment is applicable;

[0034] Figure 4d This is the fourth schematic diagram of a wireless connection scenario to which the model training method provided in this application embodiment is applicable;

[0035] Figure 5 This is a flowchart of a grip posture recognition method provided in an embodiment of this application;

[0036] Figure 6 This is a flowchart of another grip posture recognition method provided in the embodiments of this application;

[0037] Figure 7 This is a structural diagram of a model training device provided in an embodiment of this application;

[0038] Figure 8 This is a structural diagram of a grip posture recognition device provided in an embodiment of this application;

[0039] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application;

[0040] Figure 10 This is a structural diagram of another electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0042] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0043] To better understand, the relevant concepts that may be involved in the embodiments of this application are explained below.

[0044] Received Signal Strength Indicator (RSSI): This is an indicator used to measure the total power of wireless signals received by wireless electronic devices (such as mobile phones, routers, Bluetooth modules, etc.), providing feedback on the signal strength of the current wireless link. It is widely used in wireless communication scenarios such as Wi-Fi, Bluetooth, and early cellular networks (2G / 3G) (unit: dBm).

[0045] Wi-Fi Channel State Information (Wi-Fi CSI): It is a core parameter characterizing the propagation channel characteristics of Wi-Fi signals. It contains fine-grained information such as multipath fading, path loss, phase shift, and amplitude attenuation experienced by the signal during propagation. It describes the channel state of each subcarrier in complex form (amplitude + phase). Compared with RSSI, it can more accurately reflect subtle changes in the channel environment and is the core data foundation of Wi-Fi sensing technology (unit: none, amplitude is a dimensionless relative value, phase is in rad).

[0046] Workstation / Client (STA): A terminal device in a Wi-Fi network that connects to the network (Access Point (AP) / Soft Access Point (SAP)) as a client. Examples include mobile phones (when connected to Wi-Fi), laptops, tablets, smart cameras, game consoles, and all other devices that need to connect to Wi-Fi.

[0047] AP: Access Point, the central node of a Wi-Fi network, is an infrastructure device that provides network access services. It broadcasts a Service Set Identifier (SSID) to allow devices to discover each other, enables multiple STAs to connect simultaneously, and is responsible for data forwarding, authentication management, and Internet Protocol (IP) allocation. It typically connects to a wired network to provide Internet access. Examples include home Wi-Fi routers, enterprise wireless access points, and public Wi-Fi hotspots in shopping malls / airports.

[0048] SAP: Soft Access Point, a temporary access point implemented in software, functions similarly to an access point (AP) but does not require dedicated hardware. It typically refers to a Wi-Fi hotspot created by a mobile device such as a smartphone. This type of hotspot allows for easy sharing of the mobile device's network connection with other devices. Examples include mobile hotspots (Personal Hotspot), laptop Wi-Fi sharing, and in-vehicle Wi-Fi.

[0049] Static resource files (assets): These are used to store uncompiled raw resource files required for Android applications to run. They are the standard directory for integrating static resources in Android development.

[0050] TFLite, short for TensorFlow Lite, is a lightweight machine learning inference framework developed by Google, designed specifically for mobile devices and embedded systems. It compresses and optimizes trained models and deploys them to resource-constrained devices, enabling fast local inference. It supports Android, iOS, Linux, and various microcontroller (MCU) platforms.

[0051] P2P GO + P2P GC Architecture: The core networking mode of Wi-Fi P2P technology, which allows Wi-Fi devices to establish direct peer-to-peer connections without the need for traditional wireless routers. In this architecture, devices are divided into two roles: Group Owner (P2P GO) and Group Client (P2P GC). The P2P GO acts as a wireless access point, responsible for creating and managing the Wi-Fi network, while the P2P GC connects to the network created by the GO as a client.

[0052] Sine positional encoding: A fixed numerical vector related to the position of sequence elements is generated using sine and cosine functions, and then added to the original sequence features dimension by dimension. For a sequence of length T and feature dimension d, a positional encoding vector PE(pos,) of dimension d is generated for the element at position pos (starting from 0). The encoding value of the i-th dimension is calculated alternately using sine and cosine functions, with the formulas: PE(pos,2k)=sin(pos / 10000^(2k / d)) and PE(pos,2k+1)=cos(pos / 10000^(2k / d)).

[0053] Transformer Block: The basic component that makes up the Transformer encoder / decoder. It is based on a multi-head self-attention mechanism and a feed-forward neural network (FFN), and combines layer normalization and residual connections to form a standardized feature extraction module.

[0054] Attention pooling: An adaptive feature aggregation method based on attention mechanism, replacing the fixed weight aggregation method of average / max pooling. By learning the attention weights of the feature dimensions, it performs weighted summation on the target dimension features, achieving adaptive retention of key information and suppression of redundant information.

[0055] Cross-attention weighted network: It calculates attention weights on one feature sequence (query end) and another feature sequence (key end) to establish cross-sequence / cross-modal feature associations, adaptively mines complementary information between different features, and outputs weighted features that fuse cross-domain associations. It is an important network module for multimodal fusion.

[0056] Grip posture recognition aims to determine a user's grip posture by collecting data related to the interaction between the electronic device (such as a mobile phone or tablet, which will be explained using a mobile phone as an example below). Typical grip postures include vertical grip (left hand / right hand / both hands), horizontal grip (left hand / right hand / both hands), and free space (not gripping). The recognition results can be widely applied in scenarios such as antenna power adjustment, adaptive adjustment of the user interface (UI), game control optimization, payment security verification, and intelligent power saving. The development of smartphones is evolving towards a direction that more closely resembles the natural interaction methods of the human body. Among related technologies, grip posture recognition is mainly based on the phone's built-in sensors.

[0057] Currently, extracting information from wireless signals for sensing is a major research hotspot. Both academia and industry have researched and implemented functions such as "indoor positioning," "behavior recognition and monitoring," "wireless environmental perception," and "intrusion detection" based on wireless channel information, achieving high positioning / recognition accuracy. Wireless channel information can also be introduced into the field of hand gesture recognition. By leveraging mobile phone communication modules, the influence of a user's hand on wireless signals can be precisely sensed, thereby inferring the hand gesture state. Wireless channel information offers advantages such as being non-contact, resistant to obstruction, and not infringing on user privacy. It can finely characterize the amplitude and phase changes of multiple subcarriers, resulting in higher sensing accuracy and providing a new path to overcome the limitations of existing hand gesture recognition solutions.

[0058] The model training method, grip recognition method, device, and electronic equipment provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0059] Please refer to Figure 1 , Figure 1 This is a flowchart of a model training method provided in an embodiment of this application. The method includes the following steps:

[0060] Step 101: Obtain sensor data and Wi-Fi CSI data of the first electronic device under different grip postures.

[0061] It should be noted that the grip posture of the first electronic device can be predefined according to the usage scenario of the first electronic device. For better understanding, in some subsequent embodiments, the technical solution of this application will be described using a mobile phone as an example. For example, common mobile phone usage scenarios are shown in Table 1 below.

[0062] Table 1. Common Mobile Phone Usage Scenarios

[0063]

[0064] Of course, mobile phone usage scenarios may also include other scenarios besides those shown in Table 1, and this application embodiment does not specifically limit them.

[0065] Optionally, the way you hold a mobile phone may include the 19 commonly used ways you hold it, as shown in Table 2 below.

[0066] Table 2. Common ways to hold a mobile phone

[0067]

[0068] For example, Table 2 shows a grip posture when the user holds the top of the phone vertically with their left hand, a grip posture when the user holds the top of the phone vertically with their right hand, a grip posture when the user holds the top of the phone vertically with their left and right hands together, and so on. Other grip postures are shown in Table 2 and will not be explained in detail here.

[0069] It should be noted that, in addition to the way the phone is held, the user's own state may also affect the phone's sensor data and Wi-Fi CSI data. Optionally, the user's own state includes, but is not limited to, sitting, lying flat, lying on their side, lying prone, squatting, walking, standing, going up and down stairs, riding in a vehicle, etc., and this application embodiment does not specifically limit this state.

[0070] The model training method provided in this application can be applied to electronic devices with model training capabilities, such as mobile phones and computers. For example, the electronic device acquires sensor data and Wi-Fi CSI data corresponding to at least N handhold postures of a mobile phone, which effectively enriches the model's training data and helps improve the model's inference accuracy.

[0071] It should be noted that the sensors involved in the embodiments of this application include, but are not limited to, accelerometers, gyroscopes, magnetometers, light intensity sensors, pressure sensors, touch sensors, etc.

[0072] For example, the sensors include accelerometers, gyroscopes, magnetometers, light intensity sensors, touch sensors, etc., and collect sensor data of the mobile phone in various usage environments and different grip postures (such as the 19 common grip postures mentioned in Table 2 above) at a preset frequency (e.g., 10Hz). These sensor data are shown in Table 3 below.

[0073] Table 3. Sensor Data

[0074]

[0075] Optionally, after collecting the above sensor data, the collected sensor data can be preprocessed and the preprocessed sensor data can be input into the hand gesture recognition model.

[0076] For example, it can be utilized Criteria: Calculate the mean of data for each dimension. and standard deviation It will exceed Data within a given interval is identified as outliers. Outliers are removed, and the mean of the two adjacent valid sampling points corresponding to the outlier is used as a supplement.

[0077] Optionally, to mitigate the error caused by high-frequency mechanical vibration or electromagnetic interference noise, a sliding window midpoint filter is applied to acceleration, angular velocity, and magnetic field strength (three-dimensional). For example, the window length is set to 3 sampling points (e.g., corresponding to 300ms), and the median value of the data within the window is used to replace the value of the center sampling point to eliminate interference. To mitigate the error caused by sudden environmental changes, a smoothing filter is applied to light intensity. For example, the window length is set to 5 sampling points, and the average value of the data within the window is used to replace the value of the center sampling point to avoid the impact of sudden changes in indoor lighting environment.

[0078] In addition, to eliminate anomalies in the dimensions of data from different sensors, the sensor data can be mapped to the interval [-1, 1] for normalization. The normalization formula is as follows: ,in For the normalized data, The data before normalization. and These are the maximum and minimum values ​​collected by the sensor, respectively.

[0079] Finally, the magnitudes of acceleration, angular velocity, and magnetic field strength are calculated. , , As supplementary feature values, for example .

[0080] Based on the above methods, we finally obtain ( , , , , , , , , , , A 13-dimensional feature vector .

[0081] It should be noted that the sensor data in the embodiments of this application may include the three-dimensional acceleration values ​​of the first electronic device in three-dimensional space. , , ), three-dimensional angular velocity values ​​( , , ), three-dimensional magnetic field strength value ( , , ), light intensity value 3D modulus ( , , That is, the 13-dimensional feature vector obtained above. Optionally, the sensor data in the embodiments of this application may also include other sensor values, and this application does not specifically limit them.

[0082] For Wi-Fi CSI data, taking an Android phone as an example, the phone generates CSI based on relevant protocol messages. For instance, for a 20 MHz bandwidth, the phone's Wi-Fi antenna corresponds to 64 subcarriers; for a 40 MHz bandwidth, it corresponds to 128 subcarriers; for an 80 MHz bandwidth, it corresponds to 256 subcarriers; and for a 160 MHz bandwidth, it corresponds to 512 subcarriers.

[0083] In this embodiment, Wi-Fi CSI data can be collected using the same frequency (e.g., 10Hz) as the frequency used to collect sensor data. Taking a 20 MHz bandwidth as an example, the amplitude and phase information of 64 effective subcarriers are extracted, and each frame of Wi-Fi CSI data is ( A 2D vector (number of transmit antennas × number of receive antennas × number of subcarriers × 2(amplitude + phase)). The Wi-Fi CSI data for a single subcarrier is shown below:

[0084]

[0085] in, It is a complex number that contains both amplitude and phase information.

[0086] It should be noted that the original acquired CSI phase is affected by the carrier frequency offset (CFO), resulting in phase ambiguity. Optionally, in this embodiment, a linear fitting calibration method can be used to correct the phase information in the Wi-Fi CSI data.

[0087] For example, phase correction is performed on a single antenna group, and the original phase of the 64 subcarriers is... ,in For the first The original phase of each subcarrier, unit Indexed by subcarrier (1~64) represents the x-axis and the original phase. Using the vertical axis as the ordinate, perform linear regression fitting to obtain the equation of the fitted line. , , These represent the fitting slope and fitting intercept, respectively; then the calibrated phase is calculated. The calibrated phase distribution is in [- , The interval is then processed, and finally phase normalization is performed. Map this to the interval [-1, 1]. Repeat the above steps to complete... The phase calibration of the antenna pairs outputs the calibrated phase matrix, which is the corrected phase information.

[0088] It should be noted that the amplitude data (amplitude) of some subcarriers may be affected by hand blocking the antenna, multipath fading, and signal reflection, resulting in a sudden decrease. Such abnormal data will disrupt the continuity of channel characteristics. In the embodiments of this application, the amplitude in Wi-Fi CSI data can be corrected.

[0089] For example, the average subcarrier amplitude of a single antenna pair is calculated, based on the original amplitude sequence of the 64 subcarriers of a certain antenna pair. Calculate the mean Set the threshold to Amplitudes smaller than the specified value are considered abnormal and replaced with the average amplitude of the antenna pair. The corrected amplitude sequence is then smoothed using a three-point moving average. The first and last data points are replaced by the average of themselves and the next adjacent subcarrier. Then, normalization is performed. ,in and They are respectively The maximum and minimum values ​​in the total subcarrier sequence of the antenna group are then used to obtain the corrected amplitude.

[0090] Integrating the corrected amplitude and phase information yields the corrected Wi-Fi CSI data: CSI feature vector. Expand it into a one-dimensional vector In this application embodiment, the Wi-Fi CSI data used for model training can refer to corrected Wi-Fi CSI data.

[0091] Understandably, sensor data and Wi-Fi CSI data under different grip postures are also used as training data for the grip posture recognition model.

[0092] Step 102: Perform feature extraction and feature fusion on the sensor data and the Wi-Fi CSI data to obtain fused features.

[0093] Optionally, feature extraction and feature fusion can be performed on the sensor data and the Wi-Fi CSI data based on a hand gesture recognition model to obtain fused features. Alternatively, the electronic device can automatically perform feature extraction and feature fusion on the sensor data and the Wi-Fi CSI data without using a model to obtain fused features.

[0094] For example, in some embodiments, the preprocessed sensor data (e.g., the sensor feature vectors described above) can be used. ) and corrected Wi-Fi CSI data (such as the CSI feature vectors mentioned above) Input the grip posture recognition model, which can be achieved by first extracting features from the sensor data and the Wi-Fi CSI data respectively.

[0095] As mentioned earlier, sensor data is a 13-dimensional feature vector, while Wi-Fi CSI data is a 64-dimensional feature vector. Therefore, the 13-dimensional sensor data first needs to be converted into a 64-dimensional feature vector. For example, this can be achieved by projecting the 13-dimensional sensor data into a 64-dimensional feature space through a fully connected layer in the hand gesture recognition model. This aligns the sensor data with the Wi-Fi CSI data, facilitating feature fusion between the two datasets.

[0096] Optionally, feature extraction can be performed on sensor data and Wi-Fi CSI data based on the Transformer Block in the hand gesture recognition model. For example, feature extraction can double the feature dimension of sensor data and Wi-Fi CSI data, thereby significantly increasing the feature data and obtaining more data to train the hand gesture recognition model, which helps to improve the inference accuracy of the hand gesture recognition model.

[0097] It should be noted that after feature extraction from sensor data and Wi-Fi CSI data, the sensor features obtained from feature extraction from sensor data and the Wi-Fi CSI features obtained from feature extraction from Wi-Fi CSI data are fused to obtain fused features, and the grip posture recognition model is trained based on the fused features.

[0098] Step 103: Train the grip posture recognition model based on the fused features to obtain the trained grip posture recognition model.

[0099] For example, after obtaining the fused features, the fused features can be aggregated in the time dimension using global average pooling to obtain global features of a fixed dimension. These global features are then subjected to nonlinear transformation and feature extraction using a multi-layer perceptron (MLP), thereby further enhancing the discriminative ability of the features. Finally, a classification head (composed of fully connected layers and dropout layers) can be used to map the features to the category space, outputting probability values ​​(which can also be understood as logits) for each category, thereby training the inference ability of the hand gesture recognition model.

[0100] It should be noted that the trained grip recognition model can be deployed locally on the electronic device (e.g., a mobile phone). This allows the electronic device to recognize the user's grip in real time based on the deployed, trained grip recognition model. This helps the electronic device to make targeted optimization strategies based on the recognized grip, such as adjusting the modulation and coding scheme of the electronic device's antenna array, thereby improving the transmission performance of the electronic device. Specifically, the trained grip recognition model can infer the grip of the electronic device based on the input sensor data and Wi-Fi CSI data. For example, the output of the grip recognition model is the probability of the electronic device under various grips (such as the aforementioned 19 common grips), and the grip with the highest probability is confirmed as the grip of the electronic device.

[0101] In this embodiment of the application, the electronic device used to train the grip posture recognition model may be different from the electronic device used to finally apply the trained grip posture recognition model. For example, the grip posture recognition model may be trained by a computer first, and then the trained grip posture recognition model may be deployed to various mobile phones, tablets, smartwatches and other electronic devices.

[0102] Optionally, to meet the lightweight requirements of local deployment on electronic devices (such as mobile phones), the trained hand gesture recognition model can be optimized, including but not limited to model quantization (such as INT8 quantization), redundant parameter pruning, and model format conversion (such as converting PyTorch / TensorFlow models to TFLite format), to reduce model size and inference latency and ensure compatibility with the computing power and storage resources of mobile devices. The specific implementation methods of the above optimizations can be referred to in related technologies, and will not be elaborated in this application embodiment.

[0103] It should be noted that, taking an Android phone as an example, the trained grip posture recognition model file (such as a TFLite type file) after training and format conversion can be imported into the assets directory of the Android application package (APK) for deployment. During application, the trained grip posture recognition model deployed locally on the phone is loaded through the native inference interface provided by TFLite, and grip posture recognition inference is completed by reading the phone's sensor data and Wi-Fi CSI data. Optionally, the TFLite format grip posture recognition model can also be embedded into the Android framework as a system-level application, integrating other functional modules (such as antenna and power optimization modules) for system-level functional linkage optimization and application; this will not be elaborated upon in the embodiments of this application.

[0104] In this embodiment, sensor data and Wi-Fi CSI data corresponding to different grip postures of the first electronic device are acquired. Feature extraction and feature fusion are performed on the sensor data and Wi-Fi CSI data to obtain fused features. The grip posture recognition model is then trained based on these fused features to obtain a trained grip posture recognition model. Thus, this embodiment can fuse sensor data and Wi-Fi CSI data from different grip postures of the electronic device to train the grip posture recognition model. The trained grip posture recognition model can then infer the grip posture of the electronic device by combining the sensor data and Wi-Fi CSI data. It is understood that sensor data can accurately reflect the physical changes in the posture of the electronic device, while Wi-Fi CSI data can reflect channel characteristic fluctuations caused by the hand or other objects obstructing the antenna. This effectively avoids misidentification of grip postures due to environmental factors, thereby effectively improving the recognition accuracy of the grip posture of the electronic device. The solution provided in this application solves the problems of single data, inaccurate recognition results, and poor anti-interference performance caused by relying solely on sensor data to identify grip postures in related technologies.

[0105] Optionally, the grip recognition results of electronic devices can be integrated with the application layer functions of electronic devices (such as mobile phones). For example, when the grip of an electronic device is recognized as being held with one hand, the electronic device can enlarge the edge touch area and dynamically adjust and optimize the UI, which helps to improve the intelligent interactive experience of electronic devices.

[0106] Furthermore, the hand gesture recognition method provided in this application requires no additional hardware investment, resulting in low deployment costs for the hand gesture recognition model. Relying on the built-in sensor module of the mobile phone and conventional home / commercial routers, it eliminates the need for dedicated sensing devices or customized hardware. The deployment end of the hand gesture recognition model (e.g., a mobile phone) can read Wi-Fi CSI data through the Wi-Fi driver layer interface, effectively reducing the hardware costs for technology implementation.

[0107] Optionally, the step of extracting and fusing features from the sensor data and the Wi-Fi CSI data to obtain fused features includes:

[0108] Based on the grip recognition model, feature extraction is performed on the sensor data and the Wi-Fi CSI data at least once, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features.

[0109] For example, based on the grip recognition model, feature extraction is performed twice on the sensor data and the Wi-Fi CSI data, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features.

[0110] For example, based on the grip recognition model, a first feature extraction and a second feature extraction are performed on the sensor data corresponding to different grip postures of the mobile phone to obtain first sensor features and second sensor features. Based on the grip recognition model, a first feature extraction and a second feature extraction are performed on the Wi-Fi CSI data corresponding to different grip postures of the mobile phone to obtain first Wi-Fi CSI features and second Wi-Fi CSI features. Further, the first sensor features under different grip postures are fused with the first Wi-Fi CSI features under different grip postures to obtain a first fused feature. The second sensor features under different grip postures are fused with the second Wi-Fi CSI features under different grip postures to obtain a second fused feature. Then, the first fused feature and the second fused feature are fused together to obtain the final fused feature, and the grip recognition model is trained based on this fused feature.

[0111] In this embodiment of the application, by performing feature extraction on the sensor data and the Wi-Fi CSI data at least once, the amount of data used to train the grip posture recognition model can be effectively enriched, thereby helping to improve the inference accuracy of the trained grip posture recognition model.

[0112] Optionally, the feature extraction based on the grip recognition model is performed on the sensor data and the Wi-Fi CSI data at least once, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features, including:

[0113] Based on the grip recognition model, the sensor data is subjected to three feature extractions to obtain the first sensor feature, the second sensor feature and the third sensor feature respectively.

[0114] Based on the grip recognition model, the Wi-Fi CSI data is subjected to three feature extractions to obtain the first Wi-Fi CSI feature, the second Wi-Fi CSI feature, and the third Wi-Fi CSI feature, respectively;

[0115] The first sensor feature is fused with the first Wi-Fi CSI feature, the second sensor feature is fused with the second Wi-Fi CSI feature, and the third sensor feature is fused with the third Wi-Fi CSI feature to obtain the fused feature.

[0116] For example, please refer to Figure 2a After inputting sensor data and Wi-Fi CSI data under different grip postures into the grip posture recognition model, the sensor data is extracted three times by the sensor feature extractor in the grip posture recognition model and the Wi-Fi CSI data is extracted three times by the CSI feature extractor. The sensor features and Wi-Fi CSI features obtained after each feature extraction are fused by cross-attention to obtain the final fused features.

[0117] like Figure 2a As shown, the grip posture recognition model provided in this application mainly includes five parts: a sensor feature extractor, a CSI feature extractor, a feature fusion module, a global feature aggregation module, and a classification head. These five parts will be explained in detail in conjunction with specific implementation methods.

[0118] For example, the structure of the sensor feature extractor is as follows: Figure 2bAs shown. After acquiring the sensor data (13-dimensional feature vectors), the sensor feature extractor performs dimensionality transformation on the sensor data through an embedding layer. Optionally, the embedding layer consists of a fully connected layer, capable of projecting the 13-dimensional sensor data onto a 64-dimensional feature space. It should be noted that the subsequent feature projection is also a fully connected layer, used to increase the feature dimension for feature alignment with Wi-Fi CSI features, ensuring that sensor features and Wi-Fi CSI features can be fused.

[0119] Optionally, the 64-dimensional sensor data output from the embedding layer is position-encoded based on the position encoding layer. In some embodiments, the position encoding layer uses sinusoidal position encoding.

[0120] Optionally, after location encoding, the sensor feature extractor performs the first feature extraction (i.e., shallow feature extraction) using two Transformer Blocks (directly concatenated) through a shallow feature extraction module. Here, Batch represents the number of samples in each training group, T is the length of a single sample sequence, and D is the feature dimension (here, 13). Each Block uses 4-head self-attention (H=4). Figure 2b As shown, the shallow feature extraction module (i.e., the first feature extraction) outputs shallow features (i.e., the first sensor features), and the dimension of the first sensor features is (Batch, T, 64).

[0121] After passing through the Transformer Block, average pooling with a stride of 2 is used to compress the time step from T to T / 2 during the intermediate feature extraction (i.e., the second feature extraction), and then to T / 4 during the deep feature extraction (i.e., the third feature extraction). Additionally, feature projection is performed before the first block of both the intermediate and deep feature extraction processes, doubling the number of features. Therefore, as... Figure 2bAs shown, the shallow feature extraction module (i.e., the first feature extraction) outputs shallow features (i.e., the first sensor features), with a dimension of (Batch, T, 64); the middle feature extraction module (i.e., the second feature extraction) outputs middle features (i.e., the second sensor features), with a dimension of (Batch, T / 2, 128); and the deep feature extraction module (i.e., the third feature extraction) outputs deep features (i.e., the third sensor features), with a dimension of (Batch, T / 4, 256). Thus, by performing feature extraction three times on the input sensor data using the sensor feature extractor, three sensor features of different dimensions can be obtained, effectively expanding the number of sensor features and helping to improve the training accuracy of the grip posture recognition model. Furthermore, by compressing the time step through the time pooling layer, the time length of the feature sequence can be reduced, thereby reducing the number of parameters and lowering the computational load and noise during the training process of the grip posture recognition model, which also helps to improve the training accuracy of the grip posture recognition model.

[0122] Optionally, the structure of the CSI feature extractor is as follows: Figure 2c As shown. After acquiring Wi-Fi CSI data, the CSI feature extractor uses multi-dimensional convolutional layers to perform shallow feature extraction (i.e., the first feature extraction) and mid-level feature extraction (i.e., the second feature extraction). For example, for Wi-Fi CSI data with an input dimension of (Batch, T, MN, 128), shallow feature extraction is performed on the input data through a 2D convolutional layer and a 1D temporal convolutional layer to obtain shallow features (i.e., the first Wi-Fi CSI feature) with an output dimension of (Batch, T, MN, 64). Further, for the output shallow features, the mid-level 3D convolutional layer first doubles the feature dimension to 128 through the first 3D convolutional layer, and then compresses the time step to T / 2 through the second 3D convolutional layer (with a stride of 2), outputting mid-level features (i.e., the second Wi-Fi CSI feature) with a dimension of (Batch, T / 2, MN, 128). For deep feature extraction (i.e., the third feature extraction), attention pooling compression is performed on the antenna group M*N to eliminate the antenna group dimension MN. The feature dimension is doubled to 256 through feature projection and used as the input to the Transformer Block. Then, a pooling layer with a stride of 2 is used to compress the time step to T / 4, thereby aligning it with the sensor feature dimension. It should be noted that the first and second Wi-Fi CSI features also need to be compressed to eliminate the antenna group dimension. This can still be done using attention pooling. The attention pooling compression method can be found in related technologies, and will not be described in detail in this application.

[0123] In this embodiment, the sensor data is extracted three times (shallow, medium, and deep) using the aforementioned sensor feature extractor, and the Wi-Fi CSI data is also extracted three times (shallow, medium, and deep) using the CSI feature extractor. This effectively expands the number of features and helps improve the training accuracy of the grip posture recognition model. Furthermore, the time step is compressed using a time pooling layer during the feature extraction process, reducing the time length of the feature sequences for both sensor and Wi-Fi CSI features. This reduces the number of parameters, lowers the computational load and noise during the grip posture recognition model training process, and further contributes to improving the training accuracy of the grip posture recognition model.

[0124] Optionally, the step of fusing the first sensor feature with the first Wi-Fi CSI feature, fusing the second sensor feature with the second Wi-Fi CSI feature, and fusing the third sensor feature with the third Wi-Fi CSI feature to obtain fused features includes:

[0125] The first sensor feature and the first Wi-Fi CSI feature are sequentially enhanced and fused to obtain the first fused feature;

[0126] The second sensor feature, the second Wi-Fi CSI feature, and the first fused feature are sequentially enhanced and fused to obtain the second fused feature;

[0127] The first fusion feature and the second fusion feature are fused together to obtain the third fusion feature;

[0128] The third sensor feature, the third Wi-Fi CSI feature, and the third fusion feature are sequentially enhanced and fused to obtain the fusion feature.

[0129] Optionally, the sample sequence length of the first sensor feature and the first Wi-Fi CSI feature is T, and the feature dimension is D;

[0130] The sample sequence length of the second sensor feature, the second Wi-Fi CSI feature, and the first fused feature is T / 2, and the feature dimension is 2D.

[0131] The sample sequence length of the third sensor feature, the third Wi-Fi CSI feature, and the second fusion feature is T / 4, and the feature dimension is 4D.

[0132] The sample sequence length of the fused feature is T / 4, and the feature dimension is 8D.

[0133] Optionally, the feature fusion module in the hand gesture recognition model may include a first feature fusion module (also known as a shallow feature fusion module), a second feature fusion module (also known as a mid-level feature fusion module), and a third feature fusion module (also known as a deep feature fusion module).

[0134] For example, the structure of the first feature fusion module is as follows: Figure 3a As shown, the first feature fusion module is used to sequentially enhance and fuse the first sensor feature and the first Wi-Fi CSI feature to obtain the first fused feature. Specifically, the first Wi-Fi CSI feature has a feature dimension of (B, T, MN, 64). First, the antenna group (MN) dimension is eliminated through a spatial attention pooling layer, resulting in a feature dimension aligned with the first sensor feature: (B, T, 64), which can also be understood as feature dimension D=64. Subsequently, the first sensor feature and the first Wi-Fi CSI feature are subjected to complementary information interaction through a bidirectional cross-attention module to enhance the feature, generating a first sensor enhanced feature and a first Wi-Fi CSI enhanced feature. The first sensor enhanced feature and the first Wi-Fi CSI enhanced feature are concatenated and then input into an attention weight network. The attention weight network learns the importance weights of each feature. The first sensor enhancement feature and the first Wi-Fi CSI enhancement feature are weighted and fused based on the learned weights to obtain the first fused feature. Finally, the feature is projected through a fully connected projection layer to expand the feature dimension of the first fused feature to 128 dimensions and output the first fused feature (feature dimension is (B,T,128)).

[0135] The structure of the second feature fusion module is as follows: Figure 3bAs shown, the second feature fusion module is used to sequentially enhance and fuse the second sensor feature, the second Wi-Fi CSI feature, and the first fused feature to obtain the second fused feature. Specifically, for the second Wi-Fi CSI feature, whose feature dimension is (B, T / 2, MN, 128), the antenna group (MN) dimension is first eliminated through a spatial attention pooling layer, resulting in a feature dimension aligned with the second sensor feature: (B, T / 2, 128). For the first fused feature (B, T, 128), the time step dimension is adjusted to T / 2 through time downsampling, thereby achieving dimensional alignment with the second sensor feature and the second Wi-Fi CSI feature. Subsequently, the three features (second sensor feature, second Wi-Fi CSI feature, and first fused feature) interact through a three-way cross-attention module to generate the second sensor enhanced feature, the second Wi-Fi CSI enhanced feature, and the first enhanced fused feature, respectively. After concatenating the three enhanced features, the input is fed into an attention weight network, which learns the importance weights of each enhanced feature. The second sensor enhancement feature, the second Wi-Fi CSI enhancement feature, and the first enhancement fusion feature are weighted and fused based on the learned weights to obtain the second fused feature. Figure 3b The features output by the middle layer fusion are finally projected through a fully connected projection layer to expand the feature dimension of the second fused feature to 256 dimensions, which serves as the input to the third feature fusion module.

[0136] The structure of the third feature fusion module is as follows: Figure 3c As shown, the operation is similar to that of the second feature fusion module, and the third feature fusion module adopts a similar architecture design. Specifically, the dimensions of the third sensor feature (B,T / 4,256) and the third Wi-Fi CSI feature (B,T / 4,256) are already aligned, requiring no additional processing. The second fused feature (B,T / 2,256) is adjusted to (B,T / 4,256) through temporal downsampling (stride=2). For the first fused feature (B,T,128), two downsampling operations are required, and feature dimension alignment is performed through a projection layer to adjust the dimension of the first fused feature to (B,T / 4,256). The first and second fused features are fused to obtain the third fused feature, which is then interacted with the third sensor feature and the third Wi-Fi CSI feature through three-way cross-attention to generate the third sensor enhanced feature, the third Wi-Fi CSI enhanced feature, and the third enhanced fused feature (i.e.,...). Figure 3cThe three enhancement features are then concatenated and input into an attention weight network, which learns the importance weights of each enhancement feature. Based on the learned weights, the third sensor enhancement feature, the third Wi-Fi CSI enhancement feature, and the multi-layer enhancement features are weighted and fused to obtain the final fused feature. Finally, the feature is projected through a fully connected projection layer to expand the feature dimension of the fused feature to 512 dimensions, which is then used as the final fused feature output. Figure 3c (Characteristics of mid-to-deep layer-by-layer fusion output).

[0137] Alternatively, please refer to Figure 3d The final fused feature (B,T / 4,512) output by the third feature fusion module is first aggregated in the time dimension through temporal global pooling to obtain a fixed-dimensional global feature representation (B,512). This global feature is then input into a multilayer perceptron for nonlinear transformation and feature extraction, further enhancing the discriminative ability of the features. Finally, the input features are mapped to the grip posture category space through a classification head (composed of fully connected layers and Dropout layers), outputting the logits (B,num_classes) for each grip posture category. During the inference stage, the probability of each grip posture can be obtained by applying the Softmax function to the logits, thus determining the corresponding grip posture.

[0138] In this embodiment, the first feature fusion module, the second feature fusion module and the third feature fusion module respectively enhance and fuse features of different paths. Based on the final fused features, the grip posture recognition model is trained, so that the trained grip posture recognition model can combine sensor data and Wi-Fi CSI data of electronic devices to perform grip posture recognition, which helps to improve the accuracy and anti-interference of grip posture recognition of electronic devices.

[0139] Optionally, in this embodiment of the application, obtaining sensor data and Wi-Fi CSI data corresponding to different grip postures of the first electronic device includes any one of the following:

[0140] When the first electronic device is connected to the Wi-Fi network of the access point AP, sensor data and Wi-Fi CSI data of the first electronic device under different grip postures are acquired based on the same sampling frequency;

[0141] When the first electronic device is connected to the Wi-Fi network enabled by the second electronic device as a soft access point SAP, or when the second electronic device is connected to the Wi-Fi network enabled by the first electronic device as SAP, or when the first electronic device has not established a direct connection link with the second electronic device, sensor data corresponding to the first electronic device under different grip postures are obtained based on a first sampling frequency, and Wi-Fi CSI data corresponding to the first electronic device under different grip postures are obtained based on a second sampling frequency, wherein the first sampling frequency is less than the second sampling frequency.

[0142] Understandably, there are various ways in which a first electronic device (such as a mobile phone) can access a Wi-Fi network. Depending on how the first electronic device accesses the Wi-Fi network, there can also be different ways to collect sensor data and Wi-Fi CSI data from the first electronic device.

[0143] For example, such as Figure 4a As shown, the first electronic device (mobile phone) is connected to the Wi-Fi network of the router (i.e., AP). In this case, sensor data and Wi-Fi CSI data of the first electronic device under different grip postures are collected at the same sampling frequency (e.g., 10Hz).

[0144] like Figure 4b As shown, the second electronic device 12 acts as a mobile SAP to turn on a Wi-Fi hotspot. The first electronic device 11 (one or more, acting as STAs) connects to the Wi-Fi hotspot turned on by the second electronic device 12. That is, at least one first electronic device 11 accesses the Wi-Fi network of the second electronic device 12. The second electronic device 12 can recognize the grip of all the first electronic devices 11 that access its Wi-Fi network.

[0145] In this scenario, sensor data for all first electronic devices connected to the Wi-Fi hotspot activated by the second electronic device under different hand grip postures is collected based on a first sampling frequency (e.g., 1Hz). Similarly, Wi-Fi CSI data for all first electronic devices connected to the Wi-Fi hotspot activated by the second electronic device under different hand grip postures is collected based on a second sampling frequency (e.g., 10Hz). The sample size of the sensor data is then expanded to match the sample size of the Wi-Fi CSI data. For example, if one sensor data point and ten Wi-Fi CSI data points are collected within one second, this one sensor data point can be copied ten times to obtain the same sample size as the Wi-Fi CSI data, ensuring consistency in the quantity of sensor data and Wi-Fi CSI data during model training.

[0146] Optionally, the second electronic device may train the hand gesture recognition model based on sensor data and Wi-Fi CSI data of all the first electronic devices connected to its Wi-Fi hotspot. The training method is as described above and will not be repeated here.

[0147] like Figure 4c As shown, the first electronic device 11 acts as a mobile SAP to turn on a Wi-Fi hotspot, and the second electronic device 12 connects to the Wi-Fi hotspot turned on by the first electronic device 11, that is, the second electronic device 12 accesses the Wi-Fi network of the first electronic device 11.

[0148] In this scenario, sensor data for the first electronic device under different grip postures is collected based on a first sampling frequency (e.g., 1Hz), and Wi-Fi CSI data for the first electronic device under different grip postures is collected based on a second sampling frequency (e.g., 10Hz). The sample size of the sensor data is then expanded to match the sample size of the Wi-Fi CSI data. Optionally, the second electronic device can train the grip posture recognition model based on the sensor data and Wi-Fi CSI data acquired from the first electronic device acting as a Wi-Fi hotspot. The training method is as described above and will not be repeated here.

[0149] In the above Figure 4b or Figure 4c In the scenario shown, electronic devices accessing the Wi-Fi hotspot can be distinguished by the Media Access Control (MAC) information carried in the link transmission message or the MAC information carried in the CSI. During the training process of the hand gesture recognition model, the weight of sensor data can be appropriately reduced, so that the hand gesture recognition model tends to learn CSI information and perceive wireless channel information, thereby better recognizing the hand gesture of electronic devices accessing the Wi-Fi hotspot.

[0150] It should be noted that, for example Figure 4b or Figure 4cIn the illustrated scenario, regardless of whether the first electronic device acts as an SAP-enabled Wi-Fi hotspot for other electronic devices to access, or whether the first electronic device accesses a Wi-Fi hotspot enabled by other electronic devices, if the first electronic device has a trained grip posture recognition model, it can recognize its own grip posture based on this model. Furthermore, it can recognize the grip postures of other electronic devices based on sensor data and Wi-Fi CSI data from devices with which it has a direct connection. Therefore, the grip posture recognition method provided in this application supports multi-device collaborative recognition, enabling grip posture recognition of multiple electronic devices, and is suitable for multi-device scenarios such as team collaboration and equipment monitoring. Moreover, other electronic devices with a direct connection to the first electronic device do not require additional configuration; they only need to be normally connected to the Wi-Fi hotspot for the first electronic device to recognize their grip postures.

[0151] Specifically, when the first electronic device identifies its own grip posture based on the trained grip posture recognition model, it uses the same sampling frequency to acquire sensor data and Wi-Fi CSI data corresponding to its own grip posture under different grip postures. When the first electronic device identifies the grip posture of other electronic devices with which it has a direct connection based on the trained grip posture recognition model, it acquires sensor data corresponding to the other electronic devices under different grip postures based on a first sampling frequency, and acquires Wi-Fi CSI data corresponding to the other electronic devices under different grip postures based on a second sampling frequency.

[0152] like Figure 4d As shown, the first electronic device 11 does not establish a direct link with the second electronic device 12, and the first electronic device 11 and the second electronic device 12 cannot communicate. Optionally, the first electronic device 11 can connect to the router's Wi-Fi network.

[0153] In this scenario, the second electronic device can activate Wi-Fi listening mode and capture air interface information (such as the wireless air interface message information when the first electronic device communicates via Wi-Fi) using a sniffer. It can then listen for the MAC address in the air interface message or the MAC address carried in the CSI to distinguish different electronic devices in the environment. The second electronic device can collect sensor data corresponding to different grip postures of the first electronic device based on a first sampling frequency (e.g., 1Hz), and collect Wi-Fi CSI data corresponding to different grip postures of the first electronic device based on a second sampling frequency (e.g., 10Hz). Furthermore, the second electronic device trains a grip posture recognition model based on the acquired sensor data and Wi-Fi CSI data from the first electronic device. The training method is as described above and will not be repeated here.

[0154] In this embodiment, for different ways the first electronic device accesses the Wi-Fi network, the sampling frequency for the collection of sensor data and Wi-Fi CSI data corresponding to different grip postures of the first electronic device can also be different, so that the model training method provided in this embodiment can be better applied to different wireless scenarios and also helps to expand the application scenarios of electronic device grip posture recognition.

[0155] Please refer to Figure 5 This application also provides a grip posture recognition method, applied to the trained grip posture recognition model as described above, the method comprising the following steps:

[0156] Step 201: Obtain the current sensor data and Wi-Fi CSI data of the electronic device;

[0157] Step 202: Input the sensor data and the Wi-Fi CSI data into the trained grip posture recognition model and obtain the grip posture recognition result.

[0158] The trained grip posture recognition model is based on the above. Figure 1 The model is trained using the method described above, which will not be repeated here. It should be noted that the electronic device in this embodiment is equipped with the trained hand gesture recognition model.

[0159] In this embodiment, after acquiring current sensor data and Wi-Fi CSI data, the electronic device can use the trained grip recognition model to identify its current grip based on the sensor data and Wi-Fi CSI data, thereby determining the current grip posture of the electronic device. Understandably, sensor data accurately reflects the physical changes in the electronic device's posture, while Wi-Fi CSI data reflects channel characteristic fluctuations caused by the hand or other objects obstructing the antenna. This effectively avoids misidentification of grip posture due to environmental factors, thus significantly improving the accuracy and anti-interference capability of grip posture recognition.

[0160] Please refer to Figure 6 , Figure 6 This is a flowchart of another grip posture recognition method provided in the embodiments of this application. This method can be applicable to the above-mentioned... Figures 4a-4d The communication scenario shown is illustrated, and the method is explained using a mobile phone as the first electronic device. Figure 6 As shown, the method includes the following steps:

[0161] Step 301: Obtain sensor data corresponding to the N types of hand grip postures of the mobile phone;

[0162] Step 302: Synchronously acquire Wi-Fi CSI data corresponding to the N types of hand grips of the mobile phone;

[0163] Step 303: Fuse the sensor data and the Wi-Fi CSI data and perform preprocessing;

[0164] The specific implementation of the fusion and preprocessing of the sensor data and the Wi-Fi CSI data can be referred to the foregoing. Figure 1 The descriptions in the method embodiments will not be repeated in this embodiment.

[0165] Step 304: Train the model using a machine learning model or a deep learning model to obtain the model weight file;

[0166] Step 305: Process the model weight file into a suitable model file format and deploy it on the mobile phone;

[0167] It should be noted that the machine learning / deep learning model here can be understood as the grip posture recognition model mentioned above, and the model file here can be understood as the grip posture recognition model after training.

[0168] Step 306: When the mobile phone is running, it inputs real-time sensor data and Wi-Fi CSI data into the model file, obtains the output grip recognition results, and saves them.

[0169] The grip recognition method provided in this application embodiment can be implemented in conjunction with the foregoing. Figure 1 The model training method shown and Figure 5 The specific description of the grip posture recognition method shown is provided, and it can achieve the same technical effect as the aforementioned method embodiments. To avoid repetition, this embodiment will not be described again.

[0170] The model training method provided in this application can be executed by a model training device. This application uses an example of a model training device executing the model training method to illustrate the model training device provided in this application.

[0171] Please refer to Figure 7 , Figure 7 This is a structural diagram of a model training device provided in an embodiment of this application. Figure 7 As shown, the model training device 700 includes:

[0172] The first acquisition module 701 is used to acquire sensor data and Wi-Fi channel status information (CSI) data corresponding to different grip postures of the first electronic device;

[0173] Processing module 702 is used to extract and fuse features from the sensor data and the Wi-Fi CSI data to obtain fused features;

[0174] The training module 703 is used to train the grip posture recognition model based on the fused features to obtain the trained grip posture recognition model.

[0175] Optionally, the processing module 702 is specifically used for:

[0176] Based on the grip recognition model, feature extraction is performed on the sensor data and the Wi-Fi CSI data at least once, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features.

[0177] Optionally, the processing module 702 is specifically used for:

[0178] Based on the grip recognition model, the sensor data is subjected to three feature extractions to obtain the first sensor feature, the second sensor feature and the third sensor feature respectively.

[0179] Based on the grip recognition model, the Wi-Fi CSI data is subjected to three feature extractions to obtain the first Wi-Fi CSI feature, the second Wi-Fi CSI feature, and the third Wi-Fi CSI feature, respectively;

[0180] The first sensor feature is fused with the first Wi-Fi CSI feature, the second sensor feature is fused with the second Wi-Fi CSI feature, and the third sensor feature is fused with the third Wi-Fi CSI feature to obtain the fused feature.

[0181] Optionally, the processing module 702 is specifically used for:

[0182] The first sensor feature and the first Wi-Fi CSI feature are enhanced and fused together in one step to obtain the first fused feature;

[0183] The second sensor feature, the second Wi-Fi CSI feature, and the first fused feature are enhanced and fused together to obtain the second fused feature.

[0184] The first fusion feature and the second fusion feature are fused together to obtain the third fusion feature;

[0185] The third sensor feature, the third Wi-Fi CSI feature, and the third fusion feature are enhanced and fused to obtain the fusion feature.

[0186] Optionally, the sample sequence length of the first sensor feature and the first Wi-Fi CSI feature is T, and the feature dimension is D;

[0187] The sample sequence length of the second sensor feature, the second Wi-Fi CSI feature, and the first fused feature is T / 2, and the feature dimension is 2D.

[0188] The sample sequence length of the third sensor feature, the third Wi-Fi CSI feature, and the second fusion feature is T / 4, and the feature dimension is 4D.

[0189] The sample sequence length of the fused feature is T / 4, and the feature dimension is 8D.

[0190] Optionally, the first acquisition module 701 is specifically used for any of the following:

[0191] When the first electronic device is connected to the Wi-Fi network of the access point AP, sensor data and Wi-Fi CSI data of the first electronic device under different grip postures are acquired based on the same sampling frequency;

[0192] When the first electronic device is connected to the Wi-Fi network enabled by the second electronic device as a soft access point SAP, or when the second electronic device is connected to the Wi-Fi network enabled by the first electronic device as SAP, or when the first electronic device has not established a direct connection link with the second electronic device, sensor data corresponding to the first electronic device under different grip postures are obtained based on a first sampling frequency, and Wi-Fi CSI data corresponding to the first electronic device under different grip postures are obtained based on a second sampling frequency, wherein the first sampling frequency is less than the second sampling frequency.

[0193] The model training device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0194] The grip posture recognition method provided in this application can be executed by a grip posture recognition device. This application uses a grip posture recognition device executing the grip posture recognition method as an example to illustrate the grip posture recognition device provided in this application.

[0195] Please refer to Figure 8 , Figure 8 This is a structural diagram of a grip posture recognition device provided in an embodiment of this application, as shown below. Figure 8 As shown, the grip posture recognition device 800 includes:

[0196] The second acquisition module 801 is used to acquire the current sensor data and Wi-Fi CSI data of the electronic device;

[0197] The third acquisition module 802 is used to input the sensor data and the Wi-Fi CSI data into the trained grip posture recognition model and obtain the grip posture recognition result;

[0198] The trained grip posture recognition model is based on Figure 1 The model was trained using the aforementioned training method.

[0199] The grip recognition device provided in this application embodiment can achieve... Figure 5 The various processes implemented in the method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0200] The model training device or hand gesture recognition device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0201] The model training device or grip recognition device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0202] Optionally, such as Figure 9 As shown, this application embodiment also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above-described model training method or grip recognition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0203] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0204] Figure 10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0205] The electronic device 1000 includes, but is not limited to, the following components: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0206] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 10 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0207] In one embodiment, the processor 1010 is configured to:

[0208] Acquire sensor data and Wi-Fi CSI data of the first electronic device under different grip postures;

[0209] Feature extraction and feature fusion are performed on the sensor data and the Wi-Fi CSI data to obtain fused features;

[0210] The grip posture recognition model is trained based on the fused features to obtain the trained grip posture recognition model.

[0211] Optionally, the processor 1010 is also used for:

[0212] Based on the grip recognition model, feature extraction is performed on the sensor data and the Wi-Fi CSI data at least once, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features.

[0213] Optionally, the processor 1010 is also used for:

[0214] Based on the grip recognition model, the sensor data is subjected to three feature extractions to obtain the first sensor feature, the second sensor feature and the third sensor feature respectively.

[0215] Based on the grip recognition model, the Wi-Fi CSI data is subjected to three feature extractions to obtain the first Wi-Fi CSI feature, the second Wi-Fi CSI feature, and the third Wi-Fi CSI feature, respectively;

[0216] The first sensor feature is fused with the first Wi-Fi CSI feature, the second sensor feature is fused with the second Wi-Fi CSI feature, and the third sensor feature is fused with the third Wi-Fi CSI feature to obtain the fused feature.

[0217] Optionally, the processor 1010 is also used for:

[0218] The first sensor feature and the first Wi-Fi CSI feature are sequentially enhanced and fused to obtain the first fused feature;

[0219] The second sensor feature, the second Wi-Fi CSI feature, and the first fused feature are sequentially enhanced and fused to obtain the second fused feature;

[0220] The first fusion feature and the second fusion feature are fused together to obtain the third fusion feature;

[0221] The third sensor feature, the third Wi-Fi CSI feature, and the third fusion feature are sequentially enhanced and fused to obtain the fusion feature.

[0222] Optionally, the sample sequence length of the first sensor feature and the first Wi-Fi CSI feature is T, and the feature dimension is D;

[0223] The sample sequence length of the second sensor feature, the second Wi-Fi CSI feature, and the first fused feature is T / 2, and the feature dimension is 2D.

[0224] The sample sequence length of the third sensor feature, the third Wi-Fi CSI feature, and the second fusion feature is T / 4, and the feature dimension is 4D.

[0225] The sample sequence length of the fused feature is T / 4, and the feature dimension is 8D.

[0226] Alternatively, the processor 1010 is also used for any of the following:

[0227] When the first electronic device is connected to the Wi-Fi network of the access point AP, sensor data and Wi-Fi CSI data of the first electronic device under different grip postures are acquired based on the same sampling frequency;

[0228] When the first electronic device is connected to the Wi-Fi network enabled by the second electronic device as a soft access point SAP, or when the second electronic device is connected to the Wi-Fi network enabled by the first electronic device as SAP, or when the first electronic device has not established a direct connection link with the second electronic device, sensor data corresponding to the first electronic device under different grip postures are obtained based on a first sampling frequency, and Wi-Fi CSI data corresponding to the first electronic device under different grip postures are obtained based on a second sampling frequency, wherein the first sampling frequency is less than the second sampling frequency.

[0229] In another embodiment, processor 1010 is configured to:

[0230] Acquire current sensor data and Wi-Fi CSI data from electronic devices;

[0231] The sensor data and the Wi-Fi CSI data are input into the trained grip posture recognition model to obtain the grip posture recognition result;

[0232] The trained grip posture recognition model is based on Figure 1 The model was trained using the aforementioned training method.

[0233] The electronic device 1000 provided in this application embodiment can implement each step of the above-described model training method or grip posture recognition method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0234] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1007 includes at least one of a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0235] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0236] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0237] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method or grip posture recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0238] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0239] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method or grip posture recognition method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0240] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0241] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described model training method or grip recognition method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0242] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0243] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0244] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Acquire sensor data and Wi-Fi channel status information (CSI) data of the first electronic device under different grip postures; Feature extraction and feature fusion are performed on the sensor data and the Wi-Fi CSI data to obtain fused features; The grip posture recognition model is trained based on the fused features to obtain the trained grip posture recognition model.

2. The method according to claim 1, characterized in that, The step of extracting and fusing features from the sensor data and the Wi-Fi CSI data to obtain fused features includes: Based on the grip recognition model, feature extraction is performed on the sensor data and the Wi-Fi CSI data at least once, and the features corresponding to the sensor data obtained after each feature extraction are fused with the features corresponding to the Wi-Fi CSI data to obtain fused features.

3. The method according to claim 2, characterized in that, The grip posture recognition model performs feature extraction on the sensor data and the Wi-Fi CSI data at least once each, and fuses the features corresponding to the sensor data and the features corresponding to the Wi-Fi CSI data obtained after each feature extraction to obtain fused features, including: Based on the grip recognition model, the sensor data is subjected to three feature extractions to obtain the first sensor feature, the second sensor feature and the third sensor feature respectively. Based on the grip recognition model, the Wi-Fi CSI data is subjected to three feature extractions to obtain the first Wi-Fi CSI feature, the second Wi-Fi CSI feature, and the third Wi-Fi CSI feature, respectively; The first sensor feature is fused with the first Wi-Fi CSI feature, the second sensor feature is fused with the second Wi-Fi CSI feature, and the third sensor feature is fused with the third Wi-Fi CSI feature to obtain the fused feature.

4. The method according to claim 3, characterized in that, The process of fusing the first sensor feature with the first Wi-Fi CSI feature, fusing the second sensor feature with the second Wi-Fi CSI feature, and fusing the third sensor feature with the third Wi-Fi CSI feature to obtain fused features includes: The first sensor feature and the first Wi-Fi CSI feature are sequentially enhanced and fused to obtain the first fused feature; The second sensor feature, the second Wi-Fi CSI feature, and the first fused feature are sequentially enhanced and fused to obtain the second fused feature; The first fusion feature and the second fusion feature are fused together to obtain the third fusion feature; The third sensor feature, the third Wi-Fi CSI feature, and the third fusion feature are sequentially enhanced and fused to obtain the fusion feature.

5. The method according to claim 4, characterized in that, The sample sequence length of the first sensor feature and the first Wi-Fi CSI feature is T, and the feature dimension is D; The sample sequence length of the second sensor feature, the second Wi-Fi CSI feature, and the first fused feature is T / 2, and the feature dimension is 2D. The sample sequence length of the third sensor feature, the third Wi-Fi CSI feature, and the second fusion feature is T / 4, and the feature dimension is 4D. The sample sequence length of the fused feature is T / 4, and the feature dimension is 8D.

6. The method according to any one of claims 1-5, characterized in that, The acquisition of sensor data and Wi-Fi CSI data of the first electronic device under different grip postures includes any one of the following: When the first electronic device is connected to the Wi-Fi network of the access point AP, sensor data and Wi-Fi CSI data of the first electronic device under different grip postures are acquired based on the same sampling frequency; When the first electronic device is connected to the Wi-Fi network enabled by the second electronic device as a soft access point SAP, or when the second electronic device is connected to the Wi-Fi network enabled by the first electronic device as SAP, or when the first electronic device has not established a direct connection link with the second electronic device, the corresponding sensor data of the first electronic device under different grip postures is obtained based on the first sampling frequency, and the corresponding Wi-Fi CSI data of the first electronic device under different grip postures is obtained based on the second sampling frequency, wherein the first sampling frequency is less than the second sampling frequency.

7. A grip posture recognition method, characterized in that, The method, applied to the trained grip recognition model according to any one of claims 1-6, comprises: Acquire current sensor data and Wi-Fi CSI data from electronic devices; The sensor data and the Wi-Fi CSI data are input into the trained grip posture recognition model to obtain the grip posture recognition result.

8. A model training device, characterized in that, include: The first acquisition module is used to acquire sensor data and Wi-Fi channel status information (CSI) data corresponding to different grip postures of the first electronic device. The processing module is used to extract and fuse features from the sensor data and the Wi-Fi CSI data to obtain fused features; The training module is used to train the grip posture recognition model based on the fused features to obtain the trained grip posture recognition model.

9. A grip posture recognition device, characterized in that, The device, applied to the trained grip recognition model according to any one of claims 1-6, comprises: The second acquisition module is used to acquire the current sensor data and Wi-Fi CSI data of the electronic device; The third acquisition module is used to input the sensor data and the Wi-Fi CSI data into the trained grip posture recognition model and obtain the grip posture recognition result.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the model training method as described in any one of claims 1-6, or to implement the steps of the grip recognition method as described in claim 7.

Images