Gesture control method, electronic device and system

Abstract

This application provides a gesture control method, electronic device, and system, applied to a first system. The method includes: a second device receiving a first user operation on the second device; the second device responding to the first user operation by acquiring multiple first gesture data; the first device receiving the multiple first gesture data sent by the second device; the first device training and testing a first model based on the multiple first gesture data; and the first device setting a function corresponding to a first gesture type as a first control function, with the first model used to recognize the first gesture type. This application, through the interaction between the first and second devices and the first model in the first device, achieves personalized gesture customization and gesture recognition functions, satisfying different users' different usage habits and preferences. Furthermore, the first model enables more accurate gesture recognition, improving recognition accuracy and thus enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a gesture control method, electronic device and system. Background Technology

[0002] With the widespread adoption of smart devices and the rapid development of communication technology, styluses, as optional accessories for terminals, can simulate actual handwriting, enabling functions such as instant recording and drawing through interaction between the stylus and the terminal. Currently, while stylus-terminal interaction can recognize user gestures, the number of gestures that can be recognized is limited, and the recognition accuracy is low. This fails to meet the diverse usage habits and preferences of different users, thus impacting the user experience. Summary of the Invention

[0003] This application discloses a gesture control method, electronic device, and system. Through the interaction between electronic device 100 and electronic device 200, and the first model in electronic device 100, the functions of personalized gesture customization and gesture recognition are completed to meet the different usage habits and preferences of different users. Among them, the first model can more accurately recognize gestures, improve the accuracy of recognition, and thus improve the user experience.

[0004] In a first aspect, embodiments of this application provide a gesture control method applied to a first system, the first system including a first device and a second device, the first device and the second device establishing a communication connection, the method including: the second device receiving a first user operation on the second device, the first user operation including a movement operation when the user holds the second device; the second device responding to the first user operation by acquiring multiple first gesture data; the first device receiving the multiple first gesture data sent by the second device; the first device training and testing a first model based on the multiple first gesture data, the first model being a model not trained during the development phase; the first device setting a function corresponding to a first gesture type as a first control function, wherein the first gesture type is the gesture type corresponding to the multiple first gesture data, and the first model is used to recognize the first gesture type.

[0005] In the above method, the first device and the second device do not need to touch. The user can hold the second device and operate it remotely (e.g., manually move the second device). This allows the second device to acquire multiple first gesture data input by the user based on the remote operation. After receiving the multiple first gesture data sent by the second device, the first device can train and test the first model based on the multiple first gesture data. Upon completion of the test, the function corresponding to the first gesture type can be set as the first control function according to the user's selection. Furthermore, the first model can also be used for subsequent recognition of first gesture types. Through the interaction between the first and second devices, the user can control the first device by holding the second device and operating it remotely. By collecting relatively accurate first gesture data through the second device in contact with the user, the first model built based on the more powerful first device can realize personalized gesture customization and gesture recognition functions, meeting the different usage habits and preferences of different users. Users are not limited to specific and limited gestures, giving them greater freedom in using personalized gesture functions, thereby improving the user experience. Moreover, the first model can be trained and tested based on a small sample size of user data (e.g., the multiple first gesture data mentioned above), accurately recognizing different gestures from different users, improving the accuracy of gesture recognition, and thus enhancing the user experience.

[0006] In one possible implementation, the plurality of first gesture data are data corresponding to the first gesture type that are entered multiple times, and one first gesture data is data corresponding to the first gesture type that is entered once.

[0007] In the above method, the first user operation can include multiple user operations. The user can operate the second device multiple times to enable the second device to acquire multiple first gesture data. One user operation can correspond to the acquisition of one first gesture data. Multiple entries of first gesture data corresponding to the first gesture type can be used in the gesture training process of the first model to improve the accuracy of the first model in recognizing gestures.

[0008] In one possible implementation, after the first device receives the plurality of first gesture data sent by the second device, the method further includes: when the difference between the plurality of first gesture data is less than or equal to a first threshold, the first device displays a pattern of the first gesture type corresponding to the first gesture data; and / or, when the difference between the plurality of first gesture data is greater than the first threshold, the first device displays a prompt message, the prompt message being used to prompt the user to re-enter a new gesture.

[0009] In the above method, during the gesture customization process, each time a user enters a new gesture, they need to enter data corresponding to the same type of gesture. When the difference between multiple first gesture data is less than or equal to a first threshold, it can be determined that these multiple first gesture data belong to the same gesture type (i.e., the first gesture type). At this time, the first device can display the entered first gesture type pattern. When the difference between multiple first gesture data is greater than the first threshold, it can be determined that these multiple first gesture data do not belong to the same gesture type. At this time, the first device can prompt the user to re-enter the gesture. By displaying different information through the visual interface of the first device, the user can promptly know whether multiple entered gestures are feasible, which facilitates user use and improves the user experience.

[0010] In one possible implementation, the plurality of first gesture data includes second gesture data and third gesture data, wherein the second gesture data and the third gesture data are different, and the first device trains and tests the first model based on the plurality of first gesture data, including: when the difference between the plurality of first gesture data is less than or equal to a first threshold, the first device trains the first model based on the second gesture data, and the first device tests the first model based on the third gesture data.

[0011] In the above method, during the gesture training process, the training dataset and the test dataset are different (i.e., the second gesture data and the third gesture data are different), but the second gesture data and the third gesture data correspond to the same gesture type (i.e., the first gesture type mentioned above). The first model can be trained and tested by using a small sample of user data (the multiple first gesture data mentioned above), which improves the recognition accuracy of the first model. It can accurately recognize different gestures of different users, thereby improving the user experience.

[0012] In one possible implementation, the first device sets the function corresponding to the first gesture type as the first control function, including: when the first accuracy of the first model is greater than the second threshold, the first device sets the function corresponding to the first gesture type as the first control function.

[0013] In the above method, testing the first model yields a first accuracy. When the first accuracy of the first model exceeds a second threshold, it can be determined that the trained first model is accurate and that the first gesture types corresponding to multiple first gesture data are usable. At this point, the first device can set the function corresponding to the first gesture type as the first control function according to the user's selection. In this way, it can be ensured that the first model is a well-trained model (i.e., the recognition accuracy of the first model can meet the standard for use). Using the well-trained first model can more accurately recognize the first gesture type, thereby improving the user experience.

[0014] In one possible implementation, the method further includes: when the first accuracy of the first model is less than or equal to a second threshold, the first device adjusts the first model.

[0015] In the above method, when the first accuracy of the first model is less than or equal to the second threshold, it can be determined that the trained first model is inaccurate. At this time, the first device can adjust the first model, such as adjusting the loss function, activation function, hyperparameters, etc., until the accuracy of the first model is greater than the first preset threshold. By testing the first model, it can be determined whether the current first model has been trained well (e.g., the recognition accuracy has reached the threshold, and the internal parameters of the first model have been adjusted accurately), thereby ensuring the recognition accuracy of the first model.

[0016] In one possible implementation, after the first device sets the function corresponding to the first gesture type as the first control function, the method further includes: the first device receiving fourth gesture data sent by the second device; the first device determining that the fourth gesture data matches the first gesture type based on the first model; and when the first device stores the first gesture type and the corresponding first control function, the first device executing the first control function.

[0017] In the above method, during the recognition of user gestures, the user can operate the second device (e.g., the user holds the second device and performs a gesture in the air), so that the second device can acquire the fourth gesture data input by the user. After receiving the fourth gesture data, the first device can use a first model to determine whether the input fourth gesture data matches the features of a stored gesture type. If a matching first gesture type exists, the first control function corresponding to the first gesture type is executed. Through the interaction between the first and second devices, the first device can recognize user gestures and execute corresponding operations. That is, the user can control the first device by drawing gestures while holding the second device, which also improves the user experience of using the second device.

[0018] In one possible implementation, the method further includes: when the first device stores the first gesture type and the corresponding first control function, the first device adjusts the first model according to the fourth gesture data.

[0019] In the above method, the first device can input the fourth gesture data as a training and / or test dataset into the first model to train or adjust the first model. For example, it can use the fourth gesture data as a training dataset to retrain the first model to improve its accuracy, or it can use the fourth gesture data as a test dataset to test the first model. Based on the test results, the loss function, activation function, hyperparameters, etc. of the first model can be adjusted. The adjusted first model can be better used for customizing gestures and recognizing gestures, further improving the accuracy of gesture recognition.

[0020] In one possible implementation, the method further includes: the first device displaying a prompt message, the prompt message being used to prompt the user whether to select to use the fourth gesture data to adjust the first model, the prompt message including a first control; the first device adjusting the first model according to the fourth gesture data includes: the first device adjusting the first model according to the fourth gesture data in response to a second user operation on the first control.

[0021] In the above method, the first device can display a prompt message to ask the user whether to use the fourth gesture data to adjust the first model. The first device can respond to the user's operation and adjust the first model according to the fourth gesture data. This approach gives the user the choice; the user can choose whether to adjust the first model based on their personal usage habits. For example, users with higher demands for user experience will more often choose to adjust the first model, thereby improving the user experience.

[0022] In one possible implementation, the first control function includes system functions and functions corresponding to multiple applications. The system functions include any one of the following: screenshot, go back, lock screen, unlock, mute, adjust volume, and return to the home screen. The functions corresponding to the multiple applications include general functions and specific functions corresponding to the multiple applications. The general functions corresponding to the multiple applications include any one of the following: launch application, close application, and display the main interface of the application. The specific functions corresponding to different applications are different. The specific functions include any one of the following: page turning, adding page, deleting page, and copying page for presentation applications; switching brushes, switching layers, style changing, and layer separation for drawing applications; and pausing playback, starting playback, speeding up playback, fast forwarding, and rewinding for video applications.

[0023] In the above method, the first control function can be a built-in function of the system or a function supported by multiple applications. The types of control functions are diverse and can meet the needs of users in different usage scenarios. Users can execute interactive functions on the first device by drawing different gestures with the second device. As the device version is iterated and the interactive functions are expanded and promoted, the second device can replace the user's fingers to complete all interactive functions with the first device, thereby improving the user experience.

[0024] In one possible implementation, the first device is a tablet, the second device is a stylus, the first gesture type is a circular pattern, and the first control function is to start or close the first application.

[0025] In the above method, the tablet and stylus do not need to touch. The user can hold the stylus and draw a circular pattern in the air. At this time, through the interaction between the tablet and the stylus, the tablet can respond in a timely manner and execute the operation corresponding to the user's drawn circular pattern, such as launching or closing a first application. Through the interaction between the tablet and the stylus, the tablet can recognize the user's gestures and execute corresponding operations. That is, the user can control the tablet by drawing gestures with the stylus, which also improves the user experience of using the stylus.

[0026] Secondly, embodiments of this application provide a gesture control method applied to a first device, wherein the first device and a second device establish a communication connection. The method includes: receiving a plurality of first gesture data sent by the second device, wherein the plurality of first gesture data is acquired by the second device in response to a user's first user operation on the second device, the first user operation including a movement operation performed by the user while holding the second device; training and testing a first model based on the plurality of first gesture data, wherein the first model is a model that has not been trained during the development phase; and setting a function corresponding to a first gesture type as a first control function, wherein the first gesture type is the gesture type corresponding to the plurality of first gesture data, and the first model is used to identify the first gesture type.

[0027] In one possible implementation, the plurality of first gesture data are data corresponding to the first gesture type that are entered multiple times, and one first gesture data is data corresponding to the first gesture type that is entered once.

[0028] In one possible implementation, after receiving multiple first gesture data sent by the second device, the method further includes: when the difference between the multiple first gesture data is less than or equal to a first threshold, displaying a pattern of the first gesture type corresponding to the first gesture data; and / or, when the difference between the multiple first gesture data is greater than the first threshold, displaying a prompt message, the prompt message being used to prompt the user to re-enter a new gesture.

[0029] In one possible implementation, the plurality of first gesture data includes second gesture data and third gesture data, wherein the second gesture data and the third gesture data are different, and the training and testing of the first model based on the plurality of first gesture data includes: when the difference between the plurality of first gesture data is less than or equal to a first threshold, training the first model based on the second gesture data, and testing the first model based on the third gesture data.

[0030] In one possible implementation, the first device sets the function corresponding to the first gesture type as the first control function, including: when the first accuracy of the first model is greater than the second threshold, the first device sets the function corresponding to the first gesture type as the first control function.

[0031] In one possible implementation, the method further includes: adjusting the first model when the first accuracy of the first model is less than or equal to a second threshold.

[0032] In one possible implementation, after setting the first control function corresponding to the first gesture type, the method further includes: receiving fourth gesture data sent by the second device; determining that the fourth gesture data matches the first gesture type based on the first model; and executing the first control function when the first device stores the first gesture type and the corresponding first control function.

[0033] In one possible implementation, the method further includes: when the first device stores the first gesture type and the corresponding first control function, adjusting the first model according to the fourth gesture data.

[0034] In one possible implementation, the method further includes: displaying a prompt message to prompt the user whether to use the fourth gesture data to adjust the first model, the prompt message including a first control; adjusting the first model according to the fourth gesture data includes: the first device adjusting the first model according to the fourth gesture data in response to a second user operation on the first control.

[0035] In one possible implementation, the first control function includes system functions and functions corresponding to multiple applications. The system functions include any one of the following: screenshot, go back, lock screen, unlock, mute, adjust volume, and return to the home screen. The functions corresponding to the multiple applications include general functions and specific functions corresponding to the multiple applications. The general functions corresponding to the multiple applications include any one of the following: launch application, close application, and display the main interface of the application. The specific functions corresponding to different applications are different. The specific functions include any one of the following: page turning, adding page, deleting page, and copying page for presentation applications; switching brushes, switching layers, style changing, and layer separation for drawing applications; and pausing playback, starting playback, speeding up playback, fast forwarding, and rewinding for video applications.

[0036] In one possible implementation, the first device is a tablet, the second device is a stylus, the first gesture type is a circular pattern, and the first control function is to start or close the first application.

[0037] Thirdly, embodiments of this application provide a gesture control method applied to a second device, wherein the second device and a first device establish a communication connection. The method includes: receiving a first user operation on the second device, the first user operation including a movement operation when the user holds the second device; in response to the first user operation, acquiring a plurality of first gesture data; and sending the plurality of first gesture data to the first device, wherein the plurality of first gesture data is used by the first device to train and test a first model, the first model being a model that has not been trained during the development phase, the first model being used to identify a first gesture type, the first gesture type being the gesture type corresponding to the plurality of first gesture data.

[0038] In one possible implementation, the second device includes a plurality of accelerometer-gyroscope sensors, the number of which is 2N, where N is a positive integer greater than or equal to 2. The 2N accelerometer-gyroscope sensors are arranged in a distributed layout at the 2N corners of a regular polygon, the number of which is 2N. The plurality of accelerometer-gyroscope sensors are used to acquire the plurality of first gesture data multiple times, wherein the plurality of accelerometer-gyroscope sensors are used to acquire one first gesture data at a time.

[0039] In the above method, the second device collects the user's first gesture data through multiple accelerometer-gyroscope sensors (A+GSensors). These multiple A+G sensors can be distributed to increase the direction of gesture data collection, solving the problem of inaccurate data features from a single A+G sensor, improving the accuracy of the collected gesture data, and thus improving the accuracy of gesture recognition. Furthermore, the distributed layout eliminates restrictions on the user's pen grip direction and posture during gesture input, allowing the user to be unaware of the pen grip direction during the gesture input process. It also reduces cumulative errors caused by projection calculations and installation errors, further improving the accuracy of gesture data.

[0040] In one possible implementation, the second device includes a plurality of accelerometer-gyroscope sensors, the number of which is six, and the six accelerometer-gyroscope sensors are arranged in a distributed layout at the six corners of a regular hexagon.

[0041] In one possible implementation, the second device includes a flexible circuit board and a plurality of accelerometer-gyroscope sensors. The plurality of accelerometer-gyroscope sensors are mounted on the flexible circuit board in either a first or a second placement configuration. The plurality of accelerometer-gyroscope sensors are positioned in space by a first X-axis, a first Y-axis, and a first Z-axis. The flexible circuit board is positioned in space by a second X-axis, a second Y-axis, and a second Z-axis. In the first placement configuration, a first plane defined by the first X-axis and the first Y-axis is parallel to a second plane defined by the second X-axis and the second Y-axis, and the directions of the first Z-axis and the second Z-axis are aligned. In the second placement configuration, a third plane defined by the first X-axis and the first Z-axis is parallel to the second plane, and the directions of the first Y-axis and the second Z-axis are aligned.

[0042] In the above method, when multiple A+G sensors are mounted on a flexible circuit board, different placement methods can be selected according to the different acquisition directions of the axes that need to be added. For example, when it is necessary to add acquisition directions of the X and Y axes, the first placement method can be used. Similarly, when it is necessary to add acquisition directions of the X and Z axes, the second placement method can be used. The placement of multiple A+G sensors on the flexible circuit board is flexible and can be based on the sensor performance. Furthermore, based on the number of multiple A+G sensors and the board area occupied after arrangement, priority can be given to arranging them in the available areas, and they can be arranged as compactly as possible in these areas. This ensures that multiple A+G sensors can work normally while minimizing the area of ​​the flexible circuit board and reducing costs.

[0043] In one possible implementation, the second device is any of the following: a stylus, a touchpad, a sensor, or a wearable device.

[0044] Fourthly, embodiments of this application provide a gesture control system, the system including a first device and a second device, the system being used to execute the gesture control method in any possible implementation of the first aspect, the first device being used to execute the gesture control method in any possible implementation of the second aspect, and the second device being used to execute the gesture control method in any possible implementation of the third aspect.

[0045] Fifthly, embodiments of this application provide a stylus, including a transceiver, a processor, and a memory. The memory is used to store a computer program, and the processor calls the computer program to execute the gesture control method in any possible implementation of the third aspect.

[0046] In a sixth aspect, this application provides an electronic device including a transceiver, a processor, and a memory, wherein the memory is used to store a computer program, and the processor invokes the computer program to execute the gesture control method in any possible implementation of the second or third aspect.

[0047] In a seventh aspect, this application provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the gesture control method in any possible implementation of the second or third aspect described above.

[0048] Eighthly, this application provides a computer storage medium storing a computer program that, when executed by a processor, implements the gesture control method in any of the possible implementations of any of the above aspects.

[0049] Ninthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the gesture control method in any possible implementation of any of the above aspects.

[0050] In a tenth aspect, this application provides an electronic device including a method or apparatus for performing any implementation of the second or third aspect of this application. The electronic device is, for example, a chip.

[0051] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that an embodiment can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0052] The following describes the accompanying drawings used in this application.

[0053] Figure 1 This is a schematic diagram of the architecture of a gesture control system 10 provided in this application;

[0054] Figure 2 This is an interactive schematic diagram of a gesture control system 10 provided in this application;

[0055] Figure 3 This is a schematic diagram of the hardware structure of an electronic device 100 provided in this application;

[0056] Figure 4 This is a schematic diagram of the software architecture of an electronic device 100 provided in this application;

[0057] Figure 5 This is a schematic diagram of the structure of a flexible circuit board for an electronic device 200 provided in this application;

[0058] Figure 6 (A) is a schematic diagram of an A+G sensor provided in this application;

[0059] Figure 6 (B)- Figure 6 (D) is a schematic diagram of the distributed layout of multiple A+G sensors provided in this application;

[0060] Figure 7 This is a schematic diagram of a distributed layout of multiple A+G sensors provided in this application;

[0061] Figure 8 This is a schematic diagram of another distributed layout of multiple A+G sensors provided in this application;

[0062] Figures 9-12These are schematic diagrams of some user interfaces provided in this application;

[0063] Figure 13 This is a flowchart illustrating a gesture control method provided in this application;

[0064] Figure 14 This is a flowchart illustrating another gesture control method provided in this application. Detailed Implementation

[0065] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0066] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0067] With the widespread adoption of smart devices and the rapid development of communication technology, styluses, as optional accessories for terminals, can simulate actual handwriting, enabling functions such as instant recording and drawing through interaction between the stylus and the terminal. Currently, the methods for controlling the terminal via a stylus are relatively limited, mainly including double-tap to wake and brush switching. The double-tap to wake function refers to double-tapping the terminal's screen (e.g., a capacitive touchscreen) with the stylus; when the terminal detects a change in the capacitance of the capacitive screen, it activates the display driver to wake the screen. This double-tap to wake function requires direct contact between the stylus and the terminal's screen, thus it can be called a contact-based operation, heavily reliant on contact between the stylus and the terminal's screen, resulting in a poor user experience.

[0068] The brush switching function refers to the stylus's built-in touch film sensor sensing the user's finger touch on the sensor and converting the touch operation into an electrical signal, which is then sent to the touch film chip. The touch film chip transmits the touch operation information to the microcontroller unit (MCU) within the stylus via Inter-Integrated Circuit (I2C) communication. The MCU can then send the touch operation information to the terminal via Bluetooth communication. The terminal triggers an interrupt event based on the touch operation information to execute the brush switching function corresponding to the touch operation and displays the switched brush type on the user interface. This brush switching function does not require direct contact between the stylus and the terminal's screen; the stylus can establish a communication connection only with the terminal, thus it can be called contactless operation (also known as hovering operation). Contact operation can be understood as using a stylus to replace the user's finger to operate the terminal. This application embodiment uses hovering operation of the stylus to control the terminal as an example for explanation.

[0069] Air-based operation typically relies on a touch-screen module to collect user input data. This module includes a touch sensor and a touch chip, which are mounted on a flexible printed circuit (FPC), occupying internal space in the stylus. The touch chip generates additional power consumption (e.g., a few milliamps) during operation, impacting the stylus's battery life. While interaction between the stylus and the terminal allows for gesture recognition, the number of gestures a stylus can recognize is limited, recognition accuracy is low, and personalized customization is not supported.

[0070] The user gesture data collected by the sensor in the stylus can be affected by factors such as the user's pen grip direction and posture, which can limit sensor performance and make the collected user gesture data inaccurate, thus affecting the accuracy of gesture recognition. For example, when a user makes the same gesture (such as drawing a clockwise circle) while holding the pen with both hands, the difference between the left-hand and right-hand gesture data collected by the sensor is greater than a preset threshold due to the different pen grip directions. In this case, the left-hand gesture will be recognized as inconsistent with the right-hand gesture, such as recognizing the left-hand gesture as drawing a clockwise circle and the right-hand gesture as making a checkmark, which affects the user experience.

[0071] This application proposes a gesture control method applied to a gesture control system. The gesture control system may include electronic device 100 and electronic device 200, which are connected. A user can control electronic device 100 by operating electronic device 200. The gesture control system can provide gesture customization functionality, which refers to customizing / presetting user gestures and their corresponding control functions. During gesture customization, electronic device 100 can prompt the user to repeatedly input the same type of gesture using electronic device 200, and receive multiple sets of gesture data corresponding to the same type of gesture collected by electronic device 200. Based on the multiple sets of gesture data input by the user, a first model is trained and tested. When the test passes, it is determined that the gesture type corresponding to the multiple sets of gesture data is available. Electronic device 100 prompts the user that the gesture input was successful and sets the control function corresponding to the gesture type according to the user's selection. Subsequently, during the user gesture recognition process, the electronic device 200 can acquire the gesture data input by the user and use the first model to determine whether the input gesture data matches the features of a stored gesture type. If a matching gesture type exists, the control function corresponding to that gesture type is executed, and the input gesture data is used as training / test data to continue training / adjusting the first model to improve its gesture recognition accuracy. In this way, personalized gesture customization and gesture recognition functions can be completed based on the first model, meeting the different usage habits and preferences of different users. The first model enables more accurate gesture recognition, improving accuracy and thus enhancing the user experience.

[0072] Furthermore, the electronic device 200 can collect user gesture data through multiple accelerometer-gyroscope sensors (A+G sensors). These multiple A+G sensors can be distributed to increase the direction of gesture data collection, thereby solving the problem of inaccurate gesture data caused by factors such as the user's pen grip direction and posture, improving the accuracy of the collected gesture data, and thus improving the accuracy of gesture recognition. In addition, the distributed layout can also reduce the cumulative error caused by projection calculations and installation errors, further improving the accuracy of gesture data.

[0073] In this embodiment, the control functions may include, but are not limited to, the system functions of the electronic device 100 and the functions supported by different applications. The system functions may include, but are not limited to, taking screenshots, returning to the home screen, returning to the previous screen, pulling down the notification bar, locking the screen, unlocking, increasing volume, decreasing volume, muting, and displaying the currently running application. The functions supported by different applications may include, but are not limited to, launching the application, closing the application, and displaying the application's main interface. Different applications support different functions; for example, a presentation application may support functions such as page turning, adding a page, deleting a page, and copying a page. A drawing application may support functions such as switching brushes, switching layers, separating layers, background removal, and style switching. A video application may support functions such as pausing playback, starting playback, playing at a faster speed, fast forward, and rewind.

[0074] The following describes a gesture control system 10 based on an embodiment of this application.

[0075] Figure 1 An exemplary schematic diagram of the architecture of a gesture control system 10 is shown.

[0076] like Figure 1 As shown, the gesture control system 10 may include electronic device 100 and electronic device 200. Electronic device 100 can communicate with electronic device 200 via wired (e.g., universal serial bus, twisted pair, coaxial cable, and fiber optic cable) and / or wireless (e.g., wireless local area networks, Bluetooth, and cellular communication networks) means. Users can control electronic device 100 by operating electronic device 200. Wherein:

[0077] Electronic device 100 may be a mobile phone, tablet computer, all-in-one computer, handheld computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), as well as smart home devices such as smart TVs and projectors, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), or in-vehicle devices. This application embodiment does not impose any special restrictions on the specific type of electronic device 100.

[0078] Electronic device 200 can be an external input device of electronic device 100, such as a stylus, mouse, remote control, car key, gamepad, touch mouse, touchpad, keyboard, gesture sensor, handwriting tablet, etc. Electronic device 200 can also be a wearable device such as a smart bracelet, smartwatch, or smart glasses. Electronic device 200 can also be other forms of external input devices. This application embodiment does not impose special restrictions on the specific type and form of electronic device 200.

[0079] In one implementation, electronic device 200 can acquire user gesture data and send the gesture data to electronic device 100. Electronic device 100 can perform personalized customization of the gesture data (e.g., including training, testing, and setting corresponding control functions) and recognition.

[0080] Understandably, Figure 1 The number and form of the electronic devices shown are merely examples. In a specific implementation, the number may be more or less. For example, the gesture control system 10 may not include electronic device 200, that is, electronic device 200 may be integrated on electronic device 100, such as a laptop with a touchpad.

[0081] Figure 2 An interactive schematic diagram of a gesture control system 10 is shown as an example.

[0082] like Figure 2 As shown, the electronic device 100 in the gesture control system 10 may include a communication module 101 and a system-on-chip (SOC). The SOC may include a gesture training module and a gesture recognition module. The communication module 101 may include a radio frequency antenna (ANTA). The electronic device 200 in the gesture control system 10 may include a gesture collection module (e.g., multiple accelerometer-gyroscope (A+G) sensors) and a processing module. The processing module may include an MCU and the communication module 201. The communication module 201 may include a radio frequency antenna (ANTA). In some examples, the communication module 101 and the SOC can be connected via a Universal Asynchronous Receiver / Transmitter (UART) bus or a Secure Digital Input and Output (SDIO) bus. Multiple accelerometer-gyroscope sensors and the processing module can be connected via an I2C bus.

[0083] The electronic devices 100 and 200 in the gesture control system 10 can work together to complete the gesture customization process. The gesture customization process can include the gesture training process (e.g., executed in the gesture training scenario) and the gesture recognition process (e.g., executed in the gesture recognition scenario). The specific implementation process in these two scenarios is described below.

[0084] In one implementation, during gesture training, multiple A+G sensors (e.g., a gesture collection module) in electronic device 200 collect raw gesture data and send it to the MCU. Optionally, the MCU can preprocess the raw gesture data to obtain processed gesture data. The MCU can send the processed gesture data to communication module 201, which in turn sends it to communication module 101 in electronic device 100 via an RF antenna ANTA. Communication module 101 in electronic device 100 receives the processed gesture data via the RF antenna ANTA. Communication module 101 can then send the processed gesture data to a gesture training module in the SOC. The gesture training module can train and test a first model based on the processed gesture data, save the first model, and set and store the features of the gesture and its corresponding control function. Once the gesture training process is complete, it can be used in subsequent gesture recognition scenarios.

[0085] In one implementation, in a gesture recognition scenario, electronic device 200 can collect gesture data and send the processed gesture data to communication module 101 of electronic device 100 (the implementation process is similar to the above implementation). Communication module 101 can also send the processed gesture data to the gesture recognition module in the SOC. The gesture recognition module can use a first model to determine whether the processed gesture data matches the features of a stored gesture. If a matching gesture exists, the control function corresponding to that gesture is executed. The gesture training process can run throughout the entire gesture customization process. That is, the gesture training process can also be performed during gesture recognition. In some examples, if the processed gesture data matches the features of a stored gesture, the processed gesture data can be used as a training dataset and / or a test dataset to train and test the first model. This allows for simultaneous reinforcement learning in the gesture recognition scenario, and the accuracy of gesture recognition will improve with continuous training and testing of the first model.

[0086] Understandably, Figure 2The gesture control system 10 shown may include more or fewer modules. Any module in the gesture control system 10 may be a hardware module or a software module. Any module may be a separate module, or at least one module may be integrated together. For example, the A+G sensor may be an accelerometer and a gyroscope integrated together. This application does not limit this.

[0087] The structure of the exemplary electronic device 100 provided in the embodiments of this application will be described below.

[0088] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0089] Figure 3 An exemplary schematic diagram of the hardware structure of an electronic device 100 is shown.

[0090] like Figure 3 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0091] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0092] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0093] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0094] In one implementation, the processor 110 can be used for gesture training and recognition. The processor 110 can train and test a first model, and set control functions corresponding to the gestures. Based on the first model, it can identify gestures that match the recorded gestures and execute the control functions corresponding to those gestures.

[0095] The charging management module 140 receives charging input from the charger. The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In another embodiment, the power management module 141 can also be located in the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can also be located in the same device.

[0096] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0097] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In another embodiment, the antenna can be used in conjunction with a tuning switch.

[0098] The mobile communication module 150 can provide wireless communication solutions for applications on the electronic device 100, including second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), and sixth-generation (6G) mobile communication technologies. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0099] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment, the modem processor may be a separate device. In another embodiment, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0100] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0101] In one embodiment, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS). In one embodiment, electronic device 100 can receive information sent by other electronic devices, such as gesture data sent by electronic device 200, through mobile communication module 150 and / or wireless communication module 160.

[0102] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0103] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0104] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0105] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one implementation, the ISP can be integrated into the camera 193.

[0106] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0107] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0108] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0109] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0110] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0111] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0112] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0113] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0114] The 170D headphone jack is used to connect wired headphones.

[0115] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0116] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0117] The 180C barometric pressure sensor is used to measure barometric pressure.

[0118] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover.

[0119] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0120] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In one embodiment, when shooting a scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0121] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from a nearby object. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that no object is near the electronic device 100.

[0122] The 180L ambient light sensor is used to detect ambient light intensity.

[0123] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0124] The 180J temperature sensor is used to detect temperature.

[0125] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0126] The bone conduction sensor 180M can acquire vibration signals.

[0127] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0128] Motor 191 can generate vibration alerts. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card.

[0129] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. For example, a layered architecture software system can be the Android system, the Harmony operating system (OS), or other software systems. This application embodiment uses the layered architecture Android system as an example to illustrate the software structure of electronic device 100.

[0130] Figure 4 An exemplary schematic diagram of the software architecture of an electronic device 100 is shown.

[0131] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In one implementation, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0132] The application layer can include a series of application packages.

[0133] like Figure 4 As shown, the application package can include applications such as camera, gallery, music, calendar, SMS, calling, navigation, Bluetooth, browser, and screen mirroring.

[0134] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0135] like Figure 4 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0136] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0137] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0138] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0139] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0140] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0141] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0142] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0143] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0144] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0145] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0146] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0147] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] A 2D graphics engine is a graphics engine for 2D drawing.

[0150] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0151] The following example, using a gesture training scenario, illustrates the workflow of the software and hardware of electronic device 100.

[0152] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, and the control corresponding to the click as a control of the settings application, the settings application calls the interface of the application framework layer, and then calls the kernel layer to control the display driver, displaying the main interface of the settings application on the display screen 194.

[0153] The structure of the exemplary electronic device 200 provided in the embodiments of this application will be described below.

[0154] Figure 5 An exemplary schematic diagram of the structure of a flexible circuit board of an electronic device 200 is shown.

[0155] like Figure 5 As shown, the flexible circuit board of the electronic device 200 is equipped with multiple electronic components and / or integrated circuits, such as an MTP microcontroller 501, a buck converter circuit 502, a Bluetooth Low Energy (BLE) module 503, and a grounding spring 504.

[0156] In one embodiment, the blank area 505 between the BUCK circuit 502 and the BLE module 503 can be used to install multiple A+G sensors, such as 4, 6, or 8 A+G sensors. This application does not limit the specific number of A+G sensors; for further details regarding multiple A+G sensors, please refer to [link to relevant documentation]. Figure 6 (A)- Figure 6 (D)

[0157] Not limited to Figure 5 In another implementation, as in the example, multiple A+G sensors can also be mounted in other areas of the flexible circuit board. In some examples, for instance, they can be mounted above the grounding spring 504. In other examples, some electronic components on the flexible circuit board can be removed (the removal of these electronic components does not affect the normal operation of the electronic device 200), for example, some electronic components of the BUCK circuit 502 can be removed to increase the area of ​​the blank area 505, making it easier to install multiple A+G sensors.

[0158] Understandably, the multiple electronic components and integrated circuits on the flexible circuit board are arranged in a straight line. For example, the edges of the electronic components are parallel to the X-axis and Y-axis of the flexible circuit board, respectively. This arrangement allows the components to be installed compactly and densely, thereby reducing the area of ​​the flexible circuit board and reducing costs. When multiple A+G sensors are mounted on the flexible circuit board, the available space is prioritized based on the number of sensors and the area occupied after arrangement. Within this space, they are arranged as compactly as possible to ensure the normal operation of the multiple A+G sensors while minimizing the area of ​​the flexible circuit board. For details on the placement of multiple A+G sensors on the flexible circuit board, please refer to [link to relevant documentation]. Figure 6 (A)- Figure 6 (D) will not be elaborated upon for now.

[0159] Figure 6 (A) Exemplarily illustrates a schematic diagram of an A+G sensor.

[0160] Figure 6 (B)- Figure 6 (D) illustrates a schematic diagram of a distributed layout of multiple A+G sensors.

[0161] like Figure 6 As shown in (A), an A+G sensor can be used to acquire the angular velocity and acceleration of the electronic device 200 in multiple directions. In one embodiment, the magnitude of the acceleration of the electronic device 200 in three directions (i.e., the X, Y, and Z axes) can be detected by an accelerometer. In another embodiment, the magnitude of the angular velocity of the electronic device 200 about the three directions (i.e., the X, Y, and Z axes) can be determined by a gyroscope sensor. In some examples, when the user operates the electronic device 200, the spatial position of the electronic device 200 changes, and the angular velocity and acceleration of the electronic device 200 in the three directions (i.e., the X, Y, and Z axes) can be detected by the A+G sensor.

[0162] In addition to this, the angles between the electronic device 200 and the four cardinal directions can also be measured using a magnetic sensor (which may be called a magnetometer) to measure the rotation angle and / or angular velocity of the electronic device 200.

[0163] like Figure 6As shown in (B), four A+G sensors (e.g., sensors A, B, C, and D) can be arranged in a quadrilateral topology at the four corners of the quadrilateral. Each A+G sensor is centered at a corner, with the first side of the quadrilateral as its X-axis, the direction perpendicular to the X-axis as its Y-axis, and the plane perpendicular to the X and Y axes extending outwards as its Z-axis. The placement angles between adjacent A+G sensors differ by 90 degrees (e.g., 90 degrees clockwise). For example, the placement angles between sensors A and B differ by 90 degrees, between sensors B and C differ by 90 degrees, between sensors C and D differ by 90 degrees, and between sensors D and A differ by 90 degrees. The positional relationship (including placement and angle) between two adjacent A+G sensors can be understood as follows: a sensor (e.g., sensor A) moves from one corner of the quadrilateral (the position of sensor A) along the side containing its X-axis to another adjacent corner. During this movement, the sensor rotates clockwise. When it has rotated 90 degrees clockwise, it has reached the other corner, which is the position of the adjacent sensor (e.g., sensor B). These four A+G sensors can be placed flat according to the topology of the quadrilateral (e.g., the planes containing the X and Y axes of the A+G sensors are aligned with the planes containing the X and Y axes of the flexible circuit board, and the Z-axis direction of the A+G sensors is consistent with the Z-axis direction of the flexible circuit board). Figure 5 On the blank area 505 of the flexible circuit board, the X-axis of sensor A is parallel to the X-axis of the flexible circuit board, and the Y-axis of sensor A is parallel to the Y-axis of the flexible circuit board, so that sensor A and other electronic components on the flexible circuit board are arranged in a straight line. Since sensor A and the other three sensors (sensors B, C, and D) are also arranged in a straight line, the quadrilateral topology can be arranged in a straight line on the flexible circuit board.

[0164] like Figure 6As shown in (C), six A+G sensors (e.g., sensors E, F, G, H, I, J) can be arranged in a hexagonal topology at the six corners of the hexagon. Each A+G sensor is centered at a corner, with its axis along the first side of the hexagon as the X-axis, its axis perpendicular to the X-axis as the Y-axis, and its axis extending outward from the plane containing the X and Y axes as the Z-axis. The angle between any two adjacent A+G sensors differs by 60 degrees (e.g., 60 degrees clockwise). The positional relationship between two adjacent A+G sensors can be understood as follows: a sensor (e.g., sensor E) moves from one corner of the hexagon (the position of sensor E) along the side containing its X-axis to another adjacent corner. During this movement, the sensor rotates clockwise. When it has rotated 60 degrees clockwise, it has reached the other corner, which is the position of the adjacent sensor (e.g., sensor F). The six A+G sensors can be placed flat according to their regular hexagonal topology (e.g., the planes containing the X and Y axes of the A+G sensors are placed in contact with the planes containing the X and Y axes of the flexible circuit board, and the Z-axis direction of the A+G sensors is consistent with the Z-axis direction of the flexible circuit board). Figure 5 On the blank area 505 of the flexible circuit board, the X-axis of sensor E is parallel to the X-axis of the flexible circuit board, and the Y-axis of sensor E is parallel to the Y-axis of the flexible circuit board, so that sensor E and other electronic components on the flexible circuit board are arranged in a straight line.

[0165] like Figure 6 As shown in (D), Figure 6 (D) and Figure 6 (B) and Figure 6 Similar to (C), the difference is that, Figure 6 (D) consists of eight sensors (e.g., sensors K, L, M, N, O, P, Q, R) arranged in a regular octagonal topology at the eight corners of the octagon, with the angle between any two adjacent A+G sensors differing by 45 degrees (e.g., 45 degrees clockwise). These eight A+G sensors can be laid flat according to their regular octagonal topology. Figure 5 On the blank area 505 of the flexible circuit board, the X-axis of sensor K is parallel to the X-axis of the flexible circuit board, and the Y-axis of sensor K is parallel to the Y-axis of the flexible circuit board, so that sensor K and other electronic components on the flexible circuit board are arranged in a straight line.

[0166] Understandably, the accuracy of data acquired by the A+G sensor along the Z-axis (which can be simply referred to as acquisition accuracy) is much greater than that acquired along the X-axis and Y-axis. Therefore, in this embodiment, when multiple A+G sensors are used to acquire gesture data, the acquisition directions along the X-axis and Y-axis are enriched, thereby compensating for the insufficient acquisition accuracy of the X-axis and Y-axis and improving the overall accuracy of the data acquired by the A+G sensors.

[0167] In one implementation, such as Figure 6 As shown in (B), four A+G sensors can collect gesture data in four X-axis directions and four Y-axis directions. The two centrally symmetrical A+G sensors have opposite X-axis directions (180 degrees apart) and opposite Y-axis directions (180 degrees apart). For example... Figure 6 In (B), sensors A and C are a set of centrally symmetrical sensors. The X-axis of sensor A is opposite to the X-axis of sensor C, the Y-axis of sensor A is opposite to the Y-axis of sensor C, and the Z-axis of sensor A is the same as the Z-axis of sensor C. Therefore, sensors A and C can collect data in the horizontal direction of the X-axis (including horizontal left and right), the vertical direction of the Y-axis (including vertical up and down), and the direction perpendicular to the Z-axis outwards. Similarly, sensors B and D are a set of centrally symmetrical sensors. The X-axis of sensor B is opposite to the X-axis of sensor D, the Y-axis of sensor B is opposite to the Y-axis of sensor D, and the Z-axis of sensor B is the same as the Z-axis of sensor D. Therefore, sensors B and D can collect data in the vertical direction of the X-axis (including vertical up and down), the horizontal direction of the Y-axis (including horizontal left and right), and the direction perpendicular to the Z-axis outwards. Therefore, the four A+G sensors mentioned above can collect gesture data in four X-axis directions and four Y-axis directions.

[0168] In one implementation, such as Figure 6 As shown in (C), six A+G sensors can collect gesture data in six X-axis directions and six Y-axis directions. Two A+G sensors that are centrally symmetrical have opposite X-axis directions and opposite Y-axis directions. For example... Figure 6 In (C), sensor E and sensor H form a centrally symmetrical group of sensors, sensor F and sensor I form a centrally symmetrical group of sensors, and sensor G and sensor J form a centrally symmetrical group of sensors. For detailed explanations, please refer to [link to documentation]. Figure 6 The relevant explanations for (B) will not be repeated here.

[0169] In one implementation, such as Figure 6As shown in (D), eight A+G sensors can collect gesture data in eight X-axis directions and eight Y-axis directions. Two A+G sensors that are centrally symmetrical have opposite X-axis directions and opposite Y-axis directions. For example... Figure 6 In (D), sensor K and sensor O form a centrally symmetrical sensor pair, sensor L and sensor P form a centrally symmetrical sensor pair, sensor M and sensor Q form a centrally symmetrical sensor pair, and sensor N and sensor R form a centrally symmetrical sensor pair. See the attached diagram for further details. Figure 6 The relevant explanations for (B) will not be repeated here.

[0170] Understandably, when collecting gesture data using multiple A+G sensors, each A+G sensor can collect data features in one set of directions (e.g., including one X-axis, one Y-axis, and one Z-axis). These multiple sets of directional data features collected by multiple A+G sensors (e.g., including multiple X-axis, multiple Y-axis, and one Z-axis) are used as the gesture data obtained in a single acquisition, enriching the data from multiple directions. Increasing the number of directional data features collected from multiple axes can solve the problem of inaccurate data features collected by a single A+G sensor, improving the accuracy of the data features. By distributing multiple A+G sensors, the cumulative error caused by projection calculations can be effectively reduced. Furthermore, a centrally symmetrical group of A+G sensors can offset installation errors caused by board-level stress, making multiple data features more accurate. Simultaneously, it can effectively avoid feature loss caused by preprocessing gesture data (e.g., smoothing acceleration data).

[0171] Furthermore, the distributed layout of multiple A+G sensors eliminates the restriction on the user's pen grip direction during gesture input (i.e., when the sensors collect the user's gesture data). This allows the user to be unaware of their pen grip direction during gesture input. For example, when a single A+G sensor collects the user's gesture data, it needs to restrict the user's pen grip direction, such as requiring the use of the same hand, and / or the same pen grip posture, and / or the pen grip direction to be within a preset range. The data characteristics of the same gesture collected by a single A+G sensor may differ; for example, the collected data characteristics differ when the pen grip direction is different. This means that gestures recorded using one pen grip direction can be recognized, but those using another pen grip direction cannot. However, the distributed layout of multiple A+G sensors can collect data characteristics from multiple directions, and therefore is not limited by the pen grip direction. For the same gesture, the collected data characteristics are the same. This also facilitates user operation and further enhances the user experience.

[0172] Not limited to the above Figure 6 (B)- Figure 6The number of multiple A+G sensors in example (D) is as follows. In other examples, the number of multiple A+G sensors can be more, such as 10, 12, etc. Understandably, the number of multiple A+G sensors is 2N, where N is a positive integer greater than or equal to 2.

[0173] Not limited to the above Figure 6 (B)- Figure 6 In example (D), where the X-axis of multiple A+G sensors lies on the edges of a regular polygon, in another implementation, the X-axis of the A+G sensors may not lie on the edges of the regular polygon, such as... Figure 7 As shown, Figure 7 and Figure 6 Similar to (B), the difference is that, Figure 7 The X-axis of the four sensors (e.g., sensors S, T, U, and V) is not located on the sides of the regular quadrilateral. The angle between the X-axis of each A+G sensor and the side of the regular quadrilateral is α degrees (the X-axis is obtained by rotating the quadrilateral clockwise by α degrees, starting from a side). The placement angle between any two adjacent A+G sensors differs by 90 degrees (e.g., 90 degrees clockwise). In this case, sensors S and U form a centrally symmetrical pair, and sensors T and V form a centrally symmetrical pair.

[0174] Not limited to the above Figure 6 (B)- Figure 6 Example (D) illustrates the case where multiple A+G sensors are placed on a flexible circuit board, for example... Figure 6 In (B), sensor A is arranged in a straight line with other electronic components on the flexible circuit board. Figure 6 In (C), sensor E is arranged in a straight line with other electronic components on the flexible circuit board. Figure 6 In (D), sensor K is arranged in a straight line with other electronic components on the flexible circuit board. In another embodiment, the X-axis of any one of the multiple A+G sensors may be parallel to the X-axis of the flexible circuit board, and the Y-axis of that sensor may be parallel to the Y-axis of the flexible circuit board. Not limited thereto, in another embodiment, the X-axis of any one of the multiple A+G sensors may not be parallel to the X-axis of the flexible circuit board (e.g., there is an offset angle), and the Y-axis of that sensor may not be parallel to the Y-axis of the flexible circuit board (e.g., there is an offset angle). This application does not limit the specific angle at which the multiple A+G sensors are distributed on the flexible circuit board.

[0175] Not limited to the above Figure 6 (B)- Figure 6In example (D), which adds the acquisition directions of the X-axis and Y-axis, another implementation can add acquisition directions of any two of the X, Y, and Z axes. In this case, the placement of multiple A+G sensors on the flexible circuit board will change accordingly. The following explanation uses the addition of the X and Z axis acquisition directions as an example; for a specific example, please refer to [link to example]. Figure 8 .

[0176] like Figure 8 As shown, Figure 8 The four A+G sensors (e.g., sensors A', B', C', and D') can be arranged at the four corners of a regular quadrilateral topology. The planes containing the X and Z axes of these four A+G sensors are aligned with the planes containing the X and Y axes of the flexible circuit board, and the Y-axis direction of the A+G sensors is consistent with the Z-axis direction of the flexible circuit board. This can be understood as... Figure 6 The sensor in (B) is placed flat on the flexible circuit board. Figure 8 The sensor is placed upright on a flexible circuit board. Figure 8 Each A+G sensor in the array has its X-axis centered at a corner, along the sides of a regular quadrilateral. The Z-axis, perpendicular to the X-axis and located on the opposite side of the quadrilateral, is the Z-axis. The Y-axis extends outward from the plane containing the X and Z axes. Adjacent A+G sensors are positioned at 90-degree angles (e.g., 90 degrees clockwise). Sensors A' and C' form a centrally symmetrical pair, as do Sensors B' and D'.

[0177] The following describes the application scenarios involved in the embodiments of this application and the user interface diagrams in these scenarios.

[0178] Figure 9 An example diagram of a user interface is shown.

[0179] like Figure 9As shown in (A), the electronic device 100 can display a user interface 910, which can be a settings interface for hover operation. The user interface 910 can include controls 911, text content 912, a control area 913, and controls 914. Control 911 can be used to turn the hover operation function of the electronic device 100 on or off. Control 911 in the user interface 910 can indicate that the hover operation function of the electronic device 100 is currently enabled. The hover operation function can refer to the function of controlling the electronic device 100 through non-contact operation of other devices. The text content 912 includes the characters "long press pen button" and "camera." "Long press pen button" can represent the user's operation on the stylus, and "camera" can represent the corresponding control function of the electronic device 100. In some examples, the stylus can receive the user's touch operation on the stylus (e.g., long press pen button operation) and send the touch operation information to the electronic device 100. The electronic device 100 opens the camera application based on the received touch operation information. The control area 913 may include multiple user gestures and the corresponding control functions of the electronic device 100. The multiple user gestures and their corresponding control functions in the control area 913 may be factory presets of the electronic device 100. The control 914 displays the characters "Personalized Gestures", which can indicate the gesture customization function of the electronic device 100.

[0180] In one implementation, the electronic device 100 may display a gesture-customized interface of the electronic device 100 in response to a user action (such as a touch action, such as a click action) on a control 914 in the user interface 910. Specific examples can be found in [link to specific examples]. Figure 9 The user interface 920 shown in (B) is shown.

[0181] like Figure 9As shown in (B), the user interface 920 may include recorded gestures, such as gestures 921 and 924. Gesture 921 includes the characters "Gesture 1" and "Back to Previous Level". The character "Gesture 1" indicates the name of gesture 921, and the character "Back to Previous Level" indicates that the control function of the electronic device 100 corresponding to gesture 921 is: Back to Previous Level. An icon 922 is displayed to the right of gesture 921, which indicates the shape of gesture 921. The far right of gesture 921 also includes a control 923, which displays the character "Delete" and can be used to delete gesture 921. Gesture 924 includes the characters "Gesture 2" and "Launch Video Application". The character "Gesture 2" indicates the name of gesture 924, and the character "Launch Video Application" indicates that the control function of the electronic device 100 corresponding to gesture 924 is: Launch Video Application. An icon 925 is displayed to the right of gesture 924. Icon 925 indicates the shape of gesture 924. The far right of gesture 924 also includes control 926, which displays the word "Delete" and can be used to delete gesture 924. User interface 920 also includes control 927, which displays the word "Enter Gesture" and can be used to enter new gestures.

[0182] In one implementation, the electronic device 100 may display an interface for recording new gestures in response to a user action (such as a touch action, such as a click action) on a control 927 in the user interface 920. See specific examples for further details. Figure 9 The user interface 930 shown in (C) is shown.

[0183] like Figure 9 As shown in (C), the user interface 930 includes text content 931, drawing box 932, and control 933. The text content 931 includes the characters "Please enter the gesture for the first time", which can be used to prompt the user that this is the first time to enter a new gesture. The drawing box 932 can be used to display the entered new gesture. The control 933 can be used to determine the currently entered new gesture and execute the next step of entering the new gesture.

[0184] In one embodiment, the electronic device 100 displays... Figure 9 When the user interface 930 shown in (C) is active, the user can operate the electronic device 200 to allow the electronic device 100 to record new gestures. The electronic device 200 is, for example, a... Figure 10 The stylus shown in (A) allows for user operation, for example, as follows: Figure 10The hovering gesture operation shown in (A) is, for example, a circle drawing operation (i.e., the user holds the stylus and draws a circular pattern without touching the electronic device 100). After receiving the gesture operation, the electronic device 200 can send the gesture operation information (e.g., gesture data) to the electronic device 100. The electronic device 100 can then display the information received from the gesture operation. Figure 10 User interface 1010 is shown in (B). User interface 1010 and Figure 9 The user interface 930 shown in (C) is similar, except that the drawing box 932 in the user interface 1010 displays a gesture shape 1011 (e.g., a circular shape), which is similar to... Figure 10 The gesture operation information of the electronic device 200 in (A) corresponds to this. In another embodiment, if the user does not operate the electronic device 200 to allow the electronic device 100 to record a new gesture within a preset time period (e.g., 10 seconds) after the electronic device 100 displays the user interface 930, the electronic device 100 can display a prompt message to remind the user to record the gesture in a timely manner.

[0185] In one embodiment, the electronic device 100 may, in response to a user action (e.g., a touch action, such as a click action) on a control 933 in the user interface 1010, determine the currently entered gesture shape 1011 and display it. Figure 10 User interface 1020 is shown in (C). User interface 1020 and Figure 9 Similar to the user interface 930 shown in (C), the difference is that the text content 1021 in the user interface 1020 includes the characters "Please enter the gesture for the second time", which can be used to prompt the user that this is the second time to repeat the same gesture, and the gesture entered for the second time should be the same gesture entered for the first time.

[0186] In one implementation, after the electronic device repeatedly records the same gesture more than 100 times, it can display... Figure 11 The user interface 1110 shown in (A) may include text content 1111, drawing frame 1112, and control 1114. The text content 1111 includes the characters "Please enter the gesture for the fifth time", which can be used to prompt the user that this is the fifth time to repeat the same gesture. The drawing frame 1112 displays the entered gesture shape 1113. The gesture shape 1113 may be entered and displayed by the electronic device 100 after the user operates the electronic device 200. For detailed operation instructions, please refer to Figure 10 The explanation shown in (A) will not be repeated. Control 1114 displays the characters "OK" and can be used to confirm multiple input gestures.

[0187] In one implementation, electronic device 100 may, in response to a user action (e.g., a touch action, such as a click action) on control 1114 in user interface 1110, determine multiple gestures (e.g., five gesture shapes, including...) entered multiple times (e.g., five times). Figure 10 The electronic device 100 uses gesture shapes 1011 and 1113 in user interface 1010 (as shown in (B)) to determine whether these multiple gestures are the same gesture (e.g., a circle drawing gesture). When it is determined that multiple gestures are the same gesture, the new gesture is successfully recorded. At this time, the electronic device 100 can display a prompt message on user interface 1110 to inform the user that the gesture has been successfully recorded. Then, the electronic device 100 can display... Figure 11 User interface 1120 is shown in (B). Not limited thereto, in another embodiment, when it is determined that multiple gestures are not the same gesture, the electronic device 100 can display a prompt message on the user interface 1110. This prompt message is used to remind the user that the differences between the multiple recorded gestures are too large, and to re-record the multiple gestures. Then, the electronic device 100 can display... Figure 10 The user interface 1010 shown in (B) is shown.

[0188] like Figure 11 As shown in (B), the user interface 1120 may include a window 1121 (including the title "Create Gesture"). The window 1121 is used to set the information corresponding to the recorded gesture. The window 1121 includes an icon 1122, a text box 1123 and a selection box 1124. The icon 1122 can indicate the shape of the successfully recorded gesture. The text box 1123 displays the characters "Enter Gesture Name" and can be used to receive the name of the successfully recorded gesture. The selection box 1124 can be used to select and set the control function corresponding to the successfully recorded gesture.

[0189] In one embodiment, the electronic device 100 may, in response to a user operation on a text box 1123 in the user interface 1120, display the name of the gesture entered by the user in the text box 1123. Then, the electronic device 100 may, in response to a user operation on a selection box 1124 in the user interface 1120 (the user operation being, for example, a touch operation, such as a click operation), display as shown... Figure 11 The user interface 1130 shown in (C) is shown.

[0190] like Figure 11As shown in (C), the user interface 1130 may include a text box 1131 and a selection box 1132. The text box 1131 displays the characters "Gesture 3," indicating the name of the recorded gesture. The selection box 1132 includes system functions 1133 and application functions 1134. System functions 1133 may include multiple selectable system functions of the electronic device 100, such as screenshot 11331, home screen 11332, etc. Application functions may include multiple functions supported by multiple applications, such as presentation application 1135 and camera application 1136. Each application displays multiple selectable functions supported by that application, such as launching presentation application 11351 and page turning 11352 supported by presentation application 1135. A slider 1137 is also displayed to the right of system functions 1133, which can be used to slide up and down to display more system functions. A slider 1138 is also displayed to the right of application functions 1134, which can be used to slide up and down to display more application-supported functions.

[0191] In one embodiment, the electronic device 100 may, in response to a user action (such as a touch action, such as a click action) on a screenshot 11331 in the user interface 1130, select the screenshot 11331 and display it as shown. Figure 12 User interface 1210 is shown in (A). The selection box 1211 in user interface 1210 displays the word "screenshot", which can indicate that the control function corresponding to the selected recorded gesture (i.e., gesture 3) is a screenshot. User interface 1210 also includes controls 1212 and 1213. Control 1212 displays the word "OK", which can be used to confirm the setting of information of the successfully recorded gesture (including gesture name and the control function corresponding to the gesture). Control 1213 displays the word "Cancel", which can be used to cancel the setting of information of the successfully recorded gesture.

[0192] In one implementation, the electronic device 100 may, in response to a user operation on the control 1212 (such as a touch operation, such as a click operation), determine that the name of the successfully recorded gesture is gesture 3 and the control function corresponding to the gesture is screenshot.

[0193] In one implementation, Figure 12 After (A), the electronic device 100 can display Figure 12 User interface 1220 shown in (B), user interface 1220 and Figure 9The user interface 920 shown in (B) is similar, except that the user interface 1220 also includes gestures 1221 and icons 1222. Gestures 1221 include the name of gestures 1221 and the corresponding control function, and icons 1222 can indicate the gesture shape of gestures 1221.

[0194] In one implementation, Figure 12 Following (A), when a user operates the electronic device 200, such as a hovering gesture operation, for example, drawing a circle (i.e., the user holds a stylus and draws a circular pattern without touching the electronic device 100), the electronic device 200, upon receiving the gesture operation, can send the gesture operation information (e.g., gesture data) to the electronic device 100. The electronic device 100 can then use the received gesture operation information and three stored gestures (e.g.,...) to... Figure 12 The gestures 1, 2 and 3 in the user interface 1220 shown in (B) are compared. When it is determined that the information of the received gesture operation matches a stored gesture (e.g. gesture 3), the control function corresponding to gesture 3 can be executed, that is, a screenshot of the interface currently displayed on the electronic device 100 is taken.

[0195] The aforementioned air gesture operation (also known as air gesture operation) refers to the operation of a user holding a stylus and drawing different patterns in the air without contacting the stylus and electronic device 100. When the user holds the stylus and draws a pattern, the electronic device 100 can display a movable cursor (such as a pen-shaped cursor) and drawing marks. The movable cursor can indicate the position of the stylus tip on the display screen of the electronic device 100, and the drawing marks can indicate the traces on the display screen of the electronic device 100 corresponding to the movement trajectory of the stylus when the user moves it in the air.

[0196] In some scenario examples, such as Figure 12 In the scenario shown in (C), electronic device 100 is, for example, a tablet, and electronic device 200 is, for example, a stylus. The tablet can be placed flat on a table (the tablet's display screen is parallel to the table and facing upwards). The user holds the stylus (the stylus is not in contact with the tablet), with the stylus positioned above the tablet, its tip pointing downwards and its tail upwards. The angle between the stylus and the tablet's plane is β (which can be understood as the stylus tip facing the tablet's display screen). The user can simulate writing on the tablet by using the stylus to draw patterns in the air. When the drawing is completed (for example, when the user stops moving the stylus, the stylus vibrates as feedback), the stylus can send the gesture data corresponding to the pattern to the tablet, and the tablet can display the corresponding pattern based on the gesture data. In other scenario examples, such as... Figure 12In the scenario shown in (D), the tablet can be placed flat on a table (the tablet's screen is parallel to the table and facing upwards). The user holds a stylus (the stylus is not in contact with the tablet), the stylus is not located above the tablet (for example, on the right side of the tablet), and the stylus tip is pointing upwards and the pen tip downwards (this can be understood as the pen tip not pointing towards the tablet's screen). The user can draw patterns in the air using the stylus. When the drawing is complete, the stylus can send the corresponding gesture data to the tablet, and the tablet can display the corresponding pattern based on the gesture data. This is not limited to the above. Figure 12 (C)- Figure 12 In the scenario shown in (D), in other scenario examples, the tablet's display screen can also be placed vertically on the desktop. The user can hold the stylus pen with the tip facing the tablet's display screen or not, and use the stylus pen to draw patterns in the air. This application embodiment does not limit the user's hand holding the pen (e.g., left or right hand), the angle and direction of the stylus pen in the user's hand (i.e., the user's pen-holding posture and direction), or the relative position of the stylus pen and the tablet (e.g., whether the pen tip is facing the tablet's display screen). Understandably, when using the stylus pen for gesture recording, the user is not limited by the user's pen-holding posture, pen-holding direction, or the placement of the tablet and stylus pen. The user can draw patterns in the air according to their own usage habits and preferences. Furthermore, when the drawing is completed, the stylus pen can vibrate to remind the user, allowing the user to proceed to the next step, such as confirming the drawn pattern or redrawing a new pattern. This method allows for a smooth and uninterrupted user operation (e.g., the user is unsure whether the current drawing is complete and waits for additional time), further improving the user experience.

[0197] The electronic device 100, not limited to the examples above, displays the new gesture after it is entered. Figure 9 In the case of the user interface 930 shown in (C), in another embodiment, after the electronic device 100 enables the recording of new gestures, it can first set the name of the gesture to be recorded and the corresponding control function, namely display and Figure 11 The user interface shown in (B) is similar to the user interface 1120, except that it does not include the icon 1122. After the electronic device 100 has set the name of the gesture and the corresponding control function, it can start recording new gestures. At this time, it can display as shown in the image. Figure 10 The user interface 1010 shown in (B) is shown.

[0198] Not limited to the scenario described above where multiple gestures are entered before determining if they are the same gesture, in another embodiment, the electronic device 100 can compare the currently entered gesture with previously entered gestures after each entry. If the currently entered gesture is not the same as a previously entered gesture, the electronic device 100 can display prompt message 1, indicating to the user that the difference between the currently entered gesture and the previously entered gesture is too large, and requesting that the gesture be re-entered. Then, the electronic device 100 can redisplay the user interface for the currently entered gesture. Alternatively, the electronic device 100 can also display prompt message 2, indicating to the user that the differences between multiple entered gestures are too large, and requesting that multiple gestures be re-entered. Then, the electronic device 100 can redisplay the user interface for the first entered gesture, for example, displaying... Figure 10 The user interface 1010 shown in (A) is shown.

[0199] The gesture control method provided in the embodiments of this application will be introduced next.

[0200] Please see Figure 13 , Figure 13 This is a flowchart illustrating a gesture control method provided in an embodiment of this application. This method can be applied to... Figure 1 and Figure 2 The gesture control system 10 shown. This method can be applied to... Figure 3 and Figure 4 The illustrated electronic device 100. This method can be applied to... Figure 5 The illustrated electronic device 200. This method can be applied to... Figures 6-8 The diagram shows multiple A+G sensors mounted on electronic device 200. The method may include, but is not limited to, the following steps:

[0201] S101: Electronic device 200 acquires multiple raw gesture data.

[0202] In one implementation, the original gesture data can be gesture data collected by multiple A+G sensors of electronic device 200 during gesture recording, such as, but not limited to, angular velocity and acceleration. Each A+G sensor can collect a set of data features in one direction (e.g., including an X-axis, a Y-axis, and a Z-axis). The multiple sets of data features in multiple directions (e.g., including multiple X-axis, multiple Y-axis, and a Z-axis) collected by multiple A+G sensors can be used as gesture data obtained during a single gesture recording.

[0203] In one implementation, a user can operate the electronic device 200 multiple times to allow the electronic device 200 to acquire multiple raw gesture data, wherein one operation corresponds to the acquisition of one raw gesture data. In some examples, when the user operates the electronic device 200, the spatial position of the electronic device 200 changes. At this time, the angular velocity and acceleration (i.e., raw gesture data) of the electronic device 200 in multiple directions (i.e., multiple X-axis, multiple Y-axis, and one Z-axis) can be acquired at a low frequency (e.g., 20 Hz) through multiple A+G sensors.

[0204] S102: Electronic device 200 processes multiple raw gesture data to obtain multiple first gesture data.

[0205] In one implementation, S102 is an optional step.

[0206] In one embodiment, the electronic device 200 can preprocess multiple raw gesture data, such as coordinate transformation, posture calculation, and symmetric decoupling, to obtain multiple processed first gesture data. Coordinate transformation and posture calculation can enhance the features of the gesture data, while symmetric decoupling can eliminate the influence of the grip direction on the features of the gesture data.

[0207] S103: Electronic device 200 sends multiple first gesture data to electronic device 100.

[0208] In one implementation, after receiving multiple first gesture data, the electronic device 200 can send these multiple first gesture data to the electronic device 100 at once.

[0209] Not limited to the above-described embodiments S101-S103, in another embodiment, the electronic device 200 may acquire one original gesture data at a time, process the original gesture data to obtain a first gesture data, and then send the first gesture data to the electronic device 100, and repeat the above process multiple times, so as to enable the electronic device 200 to send multiple first gesture data to the electronic device 100 multiple times.

[0210] S104: Electronic device 100 determines whether the difference between multiple first gesture data is greater than a first threshold.

[0211] In one embodiment, after receiving multiple first gesture data, the electronic device 100 can determine whether the difference between the multiple first gesture data is greater than a first threshold. For example, it can calculate the difference between any two first gesture data to obtain multiple difference values ​​(e.g., Euclidean distance). If all the multiple difference values ​​are less than or equal to the first threshold, it is determined that the difference between the multiple first gesture data is less than or equal to the first threshold. At this time, the electronic device 100 can execute S105-S107. If at least one of the multiple difference values ​​is greater than the first threshold, it is determined that the difference between the multiple first gesture data is greater than the first threshold. At this time, the electronic device 100 can display a prompt message to remind the user that the difference between the multiple recorded gestures is too large and to re-record the gestures. At this time, the electronic device 200 can re-execute S101-S103. Optionally, the electronic device 100 can send an instruction message to the electronic device 200 to make the electronic device 200 execute S101-S103.

[0212] Understandably, when the electronic device 100 determines whether the difference between multiple first gesture data is greater than a first threshold, it is actually determining whether these multiple first gesture data belong to the same type of gesture data, that is, whether the multiple recorded gestures belong to the same gesture type (e.g., the first gesture type). The same gesture type can refer to gestures of the same type, such as a circle drawing gesture, a swipe gesture from left to right, an "S" shaped gesture, a hook gesture, etc. Specifically, if the difference between multiple first gesture data is less than or equal to the first threshold, it is determined that the multiple first gesture data correspond to the same gesture type (e.g., the first gesture type); if the difference between multiple first gesture data is greater than the first threshold, it is determined that the multiple first gesture data correspond to different gesture types.

[0213] Not limited to the implementation of S104, in another implementation, the electronic device 100 can receive one first gesture data at a time and determine whether the difference between the current first gesture data and the previous first gesture data is greater than a first threshold. If the difference is greater than the first threshold, the current first gesture data is re-entered by operating the electronic device 200. If the difference is less than or equal to the first threshold, the next first gesture data is received, until multiple first gesture data are received and multiple first gesture data are judged.

[0214] S105: Electronic device 100 inputs the second gesture data into the first model for training, and obtains the trained first model.

[0215] In one implementation, the first model is, for example, a Long Short-Term Memory-Recurrent Neural Network (LSTM-RNN). This first model can be one that has not been trained by the developers (during the development phase), meaning it has not been trained or tested before its first use. This can be understood as the developers designing the first model but not training or testing it using a dataset. Understandably, when the first model is LSTM-RNN, the feature dimensions of multiple first gesture data can be different, solving the gradient vanishing problem caused by the different dimensions of the gesture data and expanding the applicable scenarios of the first model.

[0216] In one implementation, the second gesture data can be at least one first gesture selected from a plurality of first gesture data, and the at least one first gesture data included in the second gesture data can all correspond to the same gesture type (i.e., the aforementioned first gesture type). The electronic device 100 can input the second gesture data as a training dataset into a first model to train the first model and obtain a trained first model. In some examples, each first gesture data in the second gesture data is the same gesture data, for example, each first gesture data in the second gesture data corresponds to the same label, such as label "A", and the second gesture data and its corresponding label are input into the first model to enable the first model to learn / train, thereby obtaining a learned / trained first model.

[0217] Optionally, the electronic device 100 system may store preset standard gestures and their corresponding gesture data. For example, the preset standard gestures may be... Figure 9The user interface 910 shown in (A) includes multiple gestures in the control area 913. If the difference between the multiple first gesture data in S104 is less than or equal to a first threshold, the electronic device 100 can also compare the multiple first gesture data with preset standard gesture data to determine whether the difference between the multiple first gesture data and the first standard gesture data is less than or equal to the first threshold (the specific judgment process is similar to the process of judging the difference between the multiple first gesture data in S104 above). The first standard gesture data can be data corresponding to any one of the preset standard gestures. When the difference between the multiple first gesture data and the first standard gesture data is less than or equal to the first threshold, the first gesture type can be determined to be the type of the first standard gesture, that is, the gesture entered by the user is one of the standard gestures. At this time, in S105, the electronic device 100 can input both the second gesture data and the first standard gesture data as training datasets into the first model to train the first model. The training process is similar to the training process of the second gesture data above. Training the first model with preset data can increase the number of training datasets, so that the first model has enough sample size to learn, and can also make the training accuracy of the first model higher.

[0218] S106: Electronic device 100 inputs the third gesture data into the trained first model for testing to obtain the first accuracy.

[0219] In one implementation, the third gesture data can be other data among a plurality of first gesture data besides the second gesture data. Understandably, the third gesture data is different from the second gesture data.

[0220] In one implementation, the electronic device 100 can input the third gesture data as a test dataset into a trained first model, test the trained first model, and output a first accuracy. In some examples, it is assumed that multiple first gesture data include five first gesture data, such as data A, data B, data C, data D, and data E, where data A and data B are second gesture data used to train the first model, and data C, data D, and data E are third gesture data used to test the first model. The electronic device 100 can input data C, data D, and data E into the trained first model, and the first model will recognize data C, data D, and data E. For example, for data C, the label corresponding to data C can be determined based on a first algorithm. If the label corresponding to data C is determined to be "A", then data C is recognized as the same gesture data as data A and data B, and the recognition is considered accurate; if the label corresponding to data C is not determined to be "A", then data C is recognized as the same gesture data as data A and data B, and the recognition is considered inaccurate. Similarly, the above process can be performed on data D and data E to identify the label corresponding to data D and the label corresponding to data E. Finally, the electronic device 100 can determine the first accuracy based on the number of accurately identified data in the third gesture data. For example, when data C, data D, and data E in the third gesture data are all accurately identified, the first accuracy is 100%. Or, for example, when some data (e.g., data C) in the third gesture data are accurately identified, the first accuracy is 33.3%.

[0221] Optionally, in S105, if the first gesture type is the type of the first standard gesture, that is, the gesture entered by the user is one of the standard gestures, the electronic device 100 can also input the third gesture data and part of the first standard gesture data as test datasets into the trained first model to test the trained first model. In this case, the data in the first standard gesture data that is used as the training dataset cannot be used as the test dataset. That is, the first standard gesture data can be divided into training dataset and test dataset, which are used for training and testing of the first model, respectively.

[0222] S107: Electronic device 100 determines whether the first accuracy is greater than the second threshold.

[0223] In one implementation, if the first accuracy is greater than the second threshold, the trained first model is determined to be accurate and the first gesture type corresponding to the multiple first gesture data is available. At this time, the electronic device 100 can execute S109. If the first accuracy is less than or equal to the second threshold, the trained first model is determined to be inaccurate. At this time, the electronic device 100 can execute S108.

[0224] S108: The first model after training by electronic device 100.

[0225] In one implementation, in S108, if the first accuracy is less than or equal to the second threshold, the trained first model is determined to be inaccurate. The electronic device 100 can adjust the trained first model, for example, by adjusting the loss function, activation function, hyperparameters, etc. After S108, the electronic device 100 can re-execute S106-S107. If, after repeatedly executing S106-S108, the accuracy of the adjusted first model is still less than or equal to the second threshold, S101-S105 can be re-executed.

[0226] S109: Electronic device 100 sets the control function corresponding to the first gesture type.

[0227] In one implementation, if the first accuracy is greater than the second threshold, the trained first model is determined to be accurate, the electronic device 100 has successfully recorded the first gesture type, and the electronic device 100 can respond to the user operation, set the control function corresponding to the first gesture type, and store multiple first gesture data and the control function corresponding to the first gesture type.

[0228] In some examples, after the electronic device 100 records a first gesture type, it can display a user interface that sets the first gesture type, such as displaying... Figure 11 The user interface 1120 shown in (B) then allows the electronic device 100 to respond to user operations by setting the name of the first gesture type and the control function corresponding to the first gesture type, such as displaying... Figure 12 The user interface 1210 shown in (A) is named “Gesture 3” for the first gesture type, and the control function corresponding to the first gesture type is screenshot.

[0229] In addition to the above example where the electronic device 100 first acquires multiple first gesture data and then sets the control function corresponding to the first gesture type, in another embodiment, the electronic device 100 can enable the gesture customization function, first set the control function corresponding to the first gesture type (i.e., execute S109), and then acquire multiple first gesture data through the electronic device 200 (i.e., execute S101-S103).

[0230] Not limited to the case where the first model in the above example is LSTM-RNN, in other examples, the first model can also be a deep neural network (DNN) or a convolutional neural network (CNN). In this case, the feature dimensions of the multiple first gesture data input into the first model should remain the same so that the first model can function properly.

[0231] exist Figure 13 In the method shown, during gesture customization, the user can operate the electronic device 200 multiple times to obtain multiple raw gesture data. The electronic device 200 can preprocess the raw gesture data to obtain multiple processed first gesture data. Upon receiving the multiple first gesture data, the electronic device 100 can train and test the first model based on this data. If the test passes, the first gesture type corresponding to the multiple first gesture data is confirmed to be usable, and the control function corresponding to the first gesture type is set according to the user's selection. In this way, relatively accurate gesture data can be collected through the electronic device 200, which is in contact with the user. The first model, built based on the more powerful electronic device 100, can then complete the personalized gesture customization function, satisfying the different usage habits and preferences of different users. Users are not limited to specific, limited gestures, giving them greater freedom in using the customized gesture function, thereby improving the user experience. The first model can be trained and tested based on a small sample size of user data, accurately recognizing different gestures from different users, improving the accuracy of gesture recognition, and thus enhancing the user experience.

[0232] Please see Figure 14 , Figure 14 This is a flowchart illustrating another gesture control method provided in an embodiment of this application. This method can be applied to... Figure 1 and Figure 2 The gesture control system 10 shown. This method can be applied to... Figure 3 and Figure 4 The illustrated electronic device 100. This method can be applied to... Figure 5 The illustrated electronic device 200. This method can be applied to... Figures 6-8 The diagram shows multiple A+G sensors mounted on electronic device 200. The method may include, but is not limited to, the following steps:

[0233] S201: Electronic device 200 acquires fourth gesture data.

[0234] In one implementation, prior to S201, there may also be... Figure 13 S101-S109.

[0235] In one embodiment, a user can operate an electronic device 200 to acquire fourth raw gesture data. The electronic device 200 can preprocess the fourth raw gesture data to obtain processed fourth gesture data. The fourth gesture data can be gesture data corresponding to the first gesture type. For details, please refer to [link to relevant documentation]. Figure 13 The explanations of S101-S102 in the document will not be repeated here.

[0236] S202: Electronic device 200 sends fourth gesture data to electronic device 100.

[0237] S203: Electronic device 100 determines, based on the first model, whether electronic device 100 stores a first gesture type that matches the fourth gesture data.

[0238] In one implementation, after receiving the fourth gesture data, the electronic device 100 can determine the gesture type that matches the fourth gesture data based on the first model, and determine whether the electronic device 100 stores the gesture type and the corresponding control function. This can be understood as the electronic device 100 determining whether the fourth gesture data belongs to a certain stored gesture type.

[0239] In one implementation, if the electronic device 100 stores a first gesture type that matches the fourth gesture data, the electronic device 100 can execute S204-S205; if the electronic device 100 does not store a first gesture type that matches the fourth gesture data, the electronic device 200 can re-execute S201.

[0240] S204: Electronic device 100 executes the control function corresponding to the first gesture type.

[0241] In one implementation, if the electronic device 100 stores data matching a first gesture type with fourth gesture data, the electronic device 100 can perform corresponding operations based on the control function corresponding to the stored first gesture type. For example, when the control function corresponding to the first gesture type is screenshot, the electronic device 100 can take a screenshot of the currently displayed user interface.

[0242] S205: Electronic device 100 adjusts the first model based on the fourth gesture data.

[0243] In one implementation, S205 is an optional step.

[0244] In one implementation, the electronic device 100 can input the fourth gesture data as a training and / or test dataset into the first model to train or adjust the first model. For example, it can use the data as a training dataset to retrain the first model to enhance its performance, or use it as a test dataset to test the first model. Based on the test results, the loss function, activation function, hyperparameters, etc., of the first model can be adjusted, and the adjusted first model can be output. The adjusted first model can be better used for customizing gestures and recognizing gestures, thereby further improving the accuracy of gesture recognition.

[0245] In some examples, the electronic device 100 may display a prompt message that suggests whether the user may choose to use the currently entered fourth gesture data for training and / or adjustment of the first model. The electronic device 100 may adjust the first model based on the fourth gesture data in response to user operation on the first control.

[0246] In one embodiment, the electronic device 100 can store the fourth gesture data into the gesture data corresponding to the first gesture type for feature difference comparison during subsequent gesture recognition.

[0247] exist Figure 14 In the method shown, during the recognition of user gestures, the user can operate the electronic device 200 to obtain the gesture data input by the user. After receiving the gesture data, the electronic device 100 can use a first model to determine whether the input gesture data matches the features of a stored gesture type. If a matching gesture type exists, the control function corresponding to that gesture type is executed, and the input gesture data is used as training / test data to continue training / adjusting the first model to improve the accuracy of gesture recognition. Through the interaction between the electronic device 100 and the electronic device 200, the electronic device 100 can recognize user gestures and perform corresponding operations. That is, the user can control the electronic device 100 by drawing gestures while holding the electronic device 200, which also improves the user experience of using the electronic device 200. During the process of customizing and recognizing gestures, the electronic device 100 can use reinforcement learning through the first model, making the recognition accuracy of the first model increasingly higher. The accuracy of gesture recognition will also improve with the continuous training and testing of the first model, making it more convenient for users to use and further improving the user experience. In addition, users can unlock more interactive functions on electronic device 100 by drawing different gestures on the handheld electronic device 200. As the version of electronic device 200 is iterated, it can replace the user's fingers and mouse to complete all interactive functions with electronic device 100.

[0248] The methods provided in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DWD), or a semiconductor medium (e.g., solid-state drive). (disk, SSD, etc.). The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A gesture control method, characterized in that, Applied to a first system, the first system including a first device and a second device, the first device and the second device establishing a communication connection, the method includes: The second device receives a first user operation from the user, the first user operation including movement operations when the user holds the second device; The second device responds to the first user's operation by acquiring multiple first gesture data; The first device receives the plurality of first gesture data sent by the second device; The first device trains and tests the first model based on the plurality of first gesture data, and the first model is a model that has not been trained during the development phase; The first device sets the function corresponding to the first gesture type as a first control function, wherein the first gesture type is the gesture type corresponding to the plurality of first gesture data, and the first model is used to identify the first gesture type.

2. The method as described in claim 1, characterized in that, The multiple first gesture data are data corresponding to the first gesture type that are entered multiple times, and one first gesture data is data corresponding to the first gesture type that is entered once.

3. The method as described in claim 1 or 2, characterized in that, After the first device receives the plurality of first gesture data sent by the second device, the method further includes: When the difference between the plurality of first gesture data is less than or equal to a first threshold, the first device displays a pattern of the first gesture type corresponding to the first gesture data; and / or, When the difference between the multiple first gesture data is greater than the first threshold, the first device displays a prompt message, which prompts the user to re-enter a new gesture.

4. The method according to any one of claims 1-3, characterized in that, The plurality of first gesture data includes second gesture data and third gesture data, wherein the second gesture data and the third gesture data are different. The first device trains and tests the first model based on the plurality of first gesture data, including: When the difference between the plurality of first gesture data is less than or equal to a first threshold, the first device trains the first model based on the second gesture data, and the first device tests the first model based on the third gesture data.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the first accuracy of the first model is less than or equal to the second threshold, the first device adjusts the first model.

6. The method according to any one of claims 1-5, characterized in that, After the first device sets the function corresponding to the first gesture type to the first control function, the method further includes: The first device receives the fourth gesture data sent by the second device; The first device determines, based on the first model, that the fourth gesture data matches the first gesture type; When the first device stores the first gesture type and the corresponding first control function, the first device executes the first control function.

7. The method as described in claim 6, characterized in that, The method further includes: When the first device stores the first gesture type and the corresponding first control function, the first device adjusts the first model according to the fourth gesture data.

8. The method as described in claim 7, characterized in that, The method further includes: The first device displays a prompt message, which prompts the user whether to use the fourth gesture data to adjust the first model. The prompt message includes a first control. The first device adjusts the first model based on the fourth gesture data, including: The first device adjusts the first model based on the fourth gesture data in response to a second user operation on the first control.

9. The method according to any one of claims 1-8, characterized in that, The first control function includes system functions and functions corresponding to multiple applications. The system functions include any one of the following: screenshot, go back, lock screen, unlock, mute, adjust volume, and return to the home screen. The functions corresponding to multiple applications include general functions and specific functions corresponding to multiple applications. The general functions corresponding to multiple applications include any one of the following: launch application, close application, and display application's main interface. The specific functions corresponding to different applications are different. The specific functions include any one of the following: page turning, adding page, deleting page, and copying page for presentation applications; switching brushes, switching layers, style changing, and layer separation for drawing applications; and pausing playback, starting playback, speeding up playback, fast forwarding, and rewinding for video applications.

10. The method according to any one of claims 1-9, characterized in that, The first device is a tablet, the second device is a stylus, the first gesture type is a circular pattern, and the first control function is to start or close the first application.

11. A gesture control method, characterized in that, Applied to a first device, the first device and a second device establish a communication connection, the method includes: The device receives multiple first gesture data sent by the second device. The multiple first gesture data are acquired by the second device in response to a first user operation performed by the user on the second device. The first user operation includes the movement operation performed by the user when holding the second device. The first model is trained and tested based on the multiple first gesture data, and the first model is a model that has not been trained during the development phase. The function corresponding to the first gesture type is set as the first control function, wherein the first gesture type is the gesture type corresponding to the plurality of first gesture data, and the first model is used to identify the first gesture type.

12. The method as described in claim 11, characterized in that, The multiple first gesture data are data corresponding to the first gesture type that are entered multiple times, and one first gesture data is data corresponding to the first gesture type that is entered once.

13. The method as described in claim 11 or 12, characterized in that, After receiving the multiple first gesture data sent by the second device, the method further includes: When the difference between the plurality of first gesture data is less than or equal to a first threshold, a pattern corresponding to the first gesture type of the first gesture data is displayed; and / or, When the difference between the multiple first gesture data is greater than the first threshold, a prompt message is displayed, which prompts the user to re-enter a new gesture.

14. The method according to any one of claims 11-13, characterized in that, The plurality of first gesture data includes second gesture data and third gesture data, wherein the second gesture data and the third gesture data are different, and the training and testing of the first model based on the plurality of first gesture data includes: When the difference between the plurality of first gesture data is less than or equal to a first threshold, the first model is trained based on the second gesture data, and the first model is tested based on the third gesture data.

15. The method according to any one of claims 11-14, characterized in that, The method further includes: When the first accuracy of the first model is less than or equal to the second threshold, the first model is adjusted.

16. The method according to any one of claims 11-15, characterized in that, After setting the function corresponding to the first gesture type as the first control function, the method further includes: Receive the fourth gesture data sent by the second device; Based on the first model, it is determined that the fourth gesture data matches the first gesture type; When the first device stores the first gesture type and the corresponding first control function, the first control function is executed.

17. The method as described in claim 16, characterized in that, The method further includes: When the first device stores the first gesture type and the corresponding first control function, the first model is adjusted according to the fourth gesture data.

18. The method as described in claim 17, characterized in that, The method further includes: Display prompt information, the prompt information being used to prompt the user whether to select to use the fourth gesture data to adjust the first model, the prompt information including a first control; The step of adjusting the first model based on the fourth gesture data includes: The first device adjusts the first model based on the fourth gesture data in response to a second user operation on the first control.

19. The method according to any one of claims 11-18, characterized in that, The first control function includes system functions and functions corresponding to multiple applications. The system functions include any one of the following: screenshot, go back, lock screen, unlock, mute, adjust volume, and return to the home screen. The functions corresponding to multiple applications include general functions and specific functions corresponding to multiple applications. The general functions corresponding to multiple applications include any one of the following: launch application, close application, and display application's main interface. The specific functions corresponding to different applications are different. The specific functions include any one of the following: page turning, adding page, deleting page, and copying page for presentation applications; switching brushes, switching layers, style changing, and layer separation for drawing applications; and pausing playback, starting playback, speeding up playback, fast forwarding, and rewinding for video applications.

20. The method according to any one of claims 11-19, characterized in that, The first device is a tablet, the second device is a stylus, the first gesture type is a circular pattern, and the first control function is to start or close the first application.

21. A gesture control method, characterized in that, Applied to a second device, the second device and the first device establish a communication connection, the method includes: Receive a first user operation on the second device, the first user operation including a movement operation when the user holds the second device; In response to the first user's operation, acquire multiple first gesture data; The first device sends the plurality of first gesture data, which is used by the first device to train and test a first model. The first model is a model that has not been trained during the development phase. The first model is used to identify a first gesture type, which is the gesture type corresponding to the plurality of first gesture data.

22. The method as described in claim 21, characterized in that, The second device includes multiple accelerometer-gyroscope sensors, the number of which is 2N, where N is a positive integer greater than or equal to 2. The 2N accelerometer-gyroscope sensors are arranged in a distributed layout at the 2N corners of a regular polygon, the number of which is 2N. The multiple accelerometer-gyroscope sensors are used to collect multiple first gesture data multiple times, wherein the multiple accelerometer-gyroscope sensors are used to collect one first gesture data at a time.

23. The method as described in claim 21 or 22, characterized in that, The second device includes a plurality of accelerometer-gyroscope sensors, the number of which is six, and the six accelerometer-gyroscope sensors are arranged in a distributed layout at the six corners of a regular hexagon.

24. The method according to any one of claims 21-23, characterized in that, The second device includes a flexible circuit board and multiple accelerometer-gyroscope sensors. The multiple accelerometer-gyroscope sensors are mounted on the flexible circuit board in either a first or a second placement method. The multiple accelerometer-gyroscope sensors are positioned in space by a first X-axis, a first Y-axis, and a first Z-axis. The flexible circuit board is positioned in space by a second X-axis, a second Y-axis, and a second Z-axis. In the first placement method, a first plane defined by the first X-axis and the first Y-axis is parallel to a second plane defined by the second X-axis and the second Y-axis, and the directions of the first Z-axis and the second Z-axis are consistent. In the second placement method, a third plane defined by the first X-axis and the first Z-axis is parallel to the second plane, and the directions of the first Y-axis and the second Z-axis are consistent.

25. The method according to any one of claims 21-24, characterized in that, The second device is any one of the following: a stylus, a touchpad, a sensor, or a wearable device.

26. A gesture control system, characterized in that, The system includes a first device and a second device, the system being configured to perform the method as described in any one of claims 1-10, the first device being configured to perform the method as described in any one of claims 11-20, and the second device being configured to perform the method as described in any one of claims 21-25.

27. An electronic device, characterized in that, The device includes a transceiver, a processor, and a memory, the memory being used to store a computer program, and the processor calling the computer program to perform the method as described in any one of claims 11-20 or any one of claims 21-25.

28. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-25.

Images