Remote control method and remote control based on gesture recognition

Abstract

The present application provides a remote control method and remote control based on gesture recognition, which relate to the field of remote control control. In the method, when the remote control performs gesture recognition for the first time, it directly uses a standard template library to match the instructions of gesture actions, and in subsequent gesture recognition, it preferentially uses a personalized template library to match the instructions of detected gesture actions. When the corresponding gesture instructions cannot be determined based on the personalized template library, the standard template library is used to match the instructions of the gesture actions, and the personalized template is updated based on the matching results, so that the instruction matching rules of the personalized template library are continuously updated and more suitable for the user's usage habits, thereby improving the accuracy and response speed of gesture recognition without affecting the versatility, and can quickly and accurately respond to the user's gesture instructions when multiple people use the same remote control.

Description

Technical Field

[0001] The present application relates to the field of remote control, and in particular to a remote control method and a remote control based on gesture recognition. Background Art

[0002] With the development of technology, smart home appliances have become increasingly popular. Users usually use remote controls to control smart home appliances such as smart TVs. Unlike traditional remote controls, today's remote controls are becoming more and more intelligent. In recent years, remote controls that use gesture recognition technology to control smart home appliances have begun to appear. Compared with the button control of traditional remote controls, gesture control is more intuitive and natural, and does not require additional equipment, which has great advantages and application prospects.

[0003] Currently, there are two main technical routes for gesture control of smart home appliances. One is to directly integrate cameras and processing chips on smart home appliances, collect user gestures through the built-in camera, and perform gesture recognition internally to generate control signals. The other is to integrate the gesture recognition module into the remote control, collect and recognize gestures through the remote control, and then transmit the control signal to the smart home appliance.

[0004] However, compared with gesture recognition through the built-in camera on the smart home appliance, integrating the gesture recognition module into the remote control is no longer restricted by the user's location. However, the same remote control may be used by different users successively, and there may be certain differences in the gesture movements of different users when executing the same gesture instructions, resulting in low recognition accuracy and response speed, which affects the user experience. Summary of the invention

[0005] The present application provides a remote control method and remote control based on gesture recognition, which preferentially use a personalized template library to match detected gesture actions with commands. When the corresponding gesture command cannot be determined based on the personalized template library, the standard template library is used to match the gesture actions with commands, and the personalized template is updated based on the matching result, so that the command matching rules of the personalized template library are continuously updated and more suitable for the user's usage habits, thereby improving the accuracy and response speed of gesture recognition without affecting versatility, and can quickly and accurately respond to user gesture commands when multiple people use the same remote control.

[0006] In a first aspect, the present application provides a remote control method based on gesture recognition, which is applied to a remote controller, comprising:

[0007] After detecting a first gesture action, matching the first gesture action from a standard template library to obtain a first gesture instruction, wherein the standard template library stores a set of multiple standard gesture actions and corresponding gesture instructions, and the first gesture action is a gesture action detected within a preset time period of initial use;

[0008] Matching a preset number of the first gesture actions with each personalized gesture action in the personalized template library respectively, obtaining multiple gesture matching degrees corresponding to each of the first gesture actions and the proportion of the best matching degree in the maximum gesture matching degrees corresponding to all the first gesture actions, the personalized template library storing a set of multiple personalized gesture actions and corresponding gesture instructions, and the maximum gesture matching degree that is greater than a preset matching degree threshold among the multiple maximum gesture matching degrees corresponding to the multiple first gesture actions is the best matching degree;

[0009] If it is detected that the proportion of the best matching degree is greater than or equal to the preset threshold, the second gesture action detected subsequently will give priority to using the personalized template library for instruction matching to obtain a second gesture instruction, and the second gesture action is other gesture actions detected in addition to the first gesture action;

[0010] If it is detected that the proportion of the best matching degree is less than the preset threshold or when the personalized template library is used to match the second gesture action, the second gesture instruction does not exist in the personalized template library, a third gesture instruction is generated according to the matching result of the second gesture action in the standard template library;

[0011] Updating the personalized template library according to the second gesture action and the corresponding matching result;

[0012] The client is remotely controlled through the first gesture instruction, the second gesture instruction and the third gesture instruction.

[0013] Through the above embodiment, the remote controller can first use the standard template library to match the first gesture action to obtain the first gesture instruction, and then match the first gesture action with the personalized template library, and determine which template library to use for instruction matching for subsequent gesture actions according to the matching degree, so that the mapping relationship between personalized gesture actions and instructions suitable for specific users can be gradually established while ensuring universality. When the matching degree of the personalized template library is high, the personalized template library is preferentially used to match the gesture instruction, with high recognition accuracy and fast response speed. When the matching degree of the personalized template library is low or there is no corresponding instruction, the standard template library can be used for matching to avoid the situation where the gesture instruction cannot be recognized. At the same time, the matching result of the second gesture action is continuously used to update the personalized template library, so that it is continuously improved with use, and the accuracy of gesture instruction recognition will become higher and higher, which is very consistent with the actual usage habits of users. While improving the accuracy and speed of gesture recognition, the method is also very universal, so that when multiple people use the same remote controller, they can also respond to the user's gesture instructions quickly and accurately.

[0014] In some embodiments, after detecting the first gesture action, before the step of matching the first gesture action from the standard template library to obtain the first gesture instruction, the method further includes:

[0015] Collect gesture image samples corresponding to various gesture instructions for feature extraction, and determine the gesture feature templates corresponding to the various gesture instructions;

[0016] Semantic labels are matched to gesture feature templates of different categories to obtain a standard template library, where the semantic labels are gesture instructions corresponding to gesture actions.

[0017] Through the above embodiments, by collecting image samples of various gesture commands, extracting features, establishing feature templates, and then matching semantic tags, a complete standard gesture command template library is obtained. By pre-establishing a basic template library covering the main common gesture commands, it can better meet the basic needs of different users and provide users with basic gesture command recognition capabilities.

[0018] In some embodiments, after detecting the first gesture action, matching the first gesture action from a standard template library to obtain a first gesture instruction specifically includes:

[0019] After detecting the first gesture action, collecting a gesture image;

[0020] Template matching is performed on the gesture image after preprocessing and feature extraction by using a similarity measurement algorithm;

[0021] Determine the target gesture action with the greatest similarity;

[0022] If the similarity of the target gesture action is greater than a preset similarity threshold, the gesture instruction corresponding to the target gesture action is set as the first gesture instruction.

[0023] Through the above embodiment, the gesture image features can be matched and compared with the standard template features to find the gesture category with the highest similarity. At the same time, setting the similarity threshold can ensure the reliability of the output command and avoid misrecognition. This further ensures the effectiveness and reliability of using the standard template library to recognize gesture commands.

[0024] In some embodiments, after the step of determining the target gesture feature template with the greatest similarity, the method further includes:

[0025] Check whether the user has enabled the gesture display permission;

[0026] If yes, the standard gesture action and gesture instruction corresponding to the target gesture action are displayed on the display of the client;

[0027] If a user's negation instruction for the standard gesture action is detected, receiving the user's redefinition of the standard gesture action, the negation instruction includes a key negation instruction and a gesture negation instruction;

[0028] The personalized template library is updated according to the redefinition result.

[0029] Through the above embodiment, the remote control can detect whether the user has enabled the gesture display permission. If enabled, after the gesture command is recognized, the standard gesture and corresponding command are displayed through the client display. The user can judge whether the recognition is accurate based on this. If the user issues a negative command, the standard gesture can be redefined, and the remote control will update the personalized template library based on the user's redefinition. This method provides a mechanism for users to actively correct gesture recognition results, further improving the recognition accuracy of the personalized gesture template library. At the same time, through the visual display of gestures, the user's understanding of standard gestures is deepened, which helps users to better perform gesture operations.

[0030] In some embodiments, after the step of updating the personalized template library according to the second gesture action and the corresponding matching result, the method further includes:

[0031] Record the usage frequency of each personalized gesture action in the personalized template library during the preset working time, where the working time refers to the operating time from when the remote control is turned on to when it is turned off;

[0032] The personalized gesture actions and corresponding gesture instructions whose usage frequency is lower than a preset frequency threshold in the personalized template library are deleted regularly.

[0033] Through the above embodiment, the remote control will record the usage frequency of each gesture action in the personalized template library over a period of time, and regularly clean up gesture instructions with a usage frequency below the threshold, so as to automatically adapt to the long-term changes in user usage habits and eliminate infrequently used gestures, making the personalized template library more streamlined and efficient, and further improving the recognition speed. At the same time, it can also avoid the problem of reduced retrieval efficiency caused by excessive expansion of the personalized template library, and improve the timeliness of the personalized template library.

[0034] In some embodiments, after the step of updating the personalized template library according to the second gesture action and the corresponding matching result, the method further includes:

[0035] Receive new gesture actions and corresponding new gesture instructions defined by the user;

[0036] The standard template library and the personalized template library are updated according to the new gesture action and the corresponding new gesture instruction.

[0037] Through the above embodiments, new gesture actions and corresponding instructions defined by the user can be received, and used to update the standard template library and the personalized template library, thereby providing users with maximum flexibility. Based on this, users can freely define personalized gesture instructions, no longer limited to the preset instruction set, so that the remote control can adapt to a wider range of control scenarios and meet the diversified needs of users. At the same time, user-defined gesture instructions can be added to the standard template library or the personalized template library at the same time. After long-term accumulation, the standard template library or the personalized template library can be continuously enriched and improved, and the recognition accuracy and comprehensive coverage will become higher and higher.

[0038] In some embodiments, before the step of remotely controlling the client through the first gesture instruction, the second gesture instruction, and the third gesture instruction, the step further includes:

[0039] Receive user voice information;

[0040] The control instruction is voice-verified according to the voice information, and the control instruction includes the first gesture instruction, the second gesture instruction and the third gesture instruction.

[0041] Through the above-mentioned embodiment, before executing the control instruction, the user's voice information is also synchronously received for verification, which can further eliminate possible ambiguity and improve the accuracy of gesture action recognition.

[0042] In a second aspect, the present application provides a remote controller, the remote controller comprising: one or more processors and a memory;

[0043] The memory is coupled to one or more processors, and is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions so that the remote control can implement a certain method provided in the above embodiment, which will not be repeated here.

[0044] In a third aspect, the present application provides a computer-readable storage medium, including instructions. When the instructions are executed on a remote control, the remote control can implement a remote control method based on gesture recognition provided in the above embodiment, which will not be described in detail here.

[0045] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on a remote control, the remote control can implement a remote control method based on gesture recognition provided in the above embodiment, which will not be described in detail here.

[0046] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0047] 1. The remote control can first use the standard template library to match the first gesture action to obtain the first gesture instruction, and then match the first gesture action with the personalized template library, and determine which template library to use for instruction matching for subsequent gesture actions based on the matching degree. This can ensure universality while gradually establishing a mapping relationship between personalized gesture actions and instructions suitable for specific users. When the matching degree of the personalized template library is high, the personalized template library is used first to match the gesture instructions, with high recognition accuracy and fast response speed. When the matching degree of the personalized template library is low or there is no corresponding instruction, the standard template library can be used for matching to avoid the situation where the gesture instruction cannot be recognized. At the same time, the matching result of the second gesture action is continuously used to update the personalized template library, so that it is continuously improved with use, and the accuracy of gesture instruction recognition will become higher and higher, which is very consistent with the actual usage habits of users. This method improves the accuracy and speed of gesture recognition while also having very good universality.

[0048] 2. By collecting image samples of various gesture commands, extracting features, establishing feature templates, and then matching semantic tags, a complete standard gesture command template library is obtained, which can better adapt to the basic needs of different users and provide users with basic gesture command recognition capabilities. It can also receive new gesture actions and corresponding commands defined by users, and use them to update the standard template library and personalized template library, thus providing users with maximum flexibility.

[0049] 3. By matching and comparing gesture image features with standard template features, the gesture category with the highest similarity can be found. At the same time, setting a similarity threshold can ensure the reliability of output commands and avoid misrecognition, thereby further ensuring the effectiveness and reliability of using the standard template library to identify gesture commands. In addition, the remote control provides a mechanism for users to actively correct gesture recognition results, and further improves the recognition accuracy of the personalized gesture template library through a visual display of gesture actions. The remote control will also record the frequency of use of each gesture action in the personalized template library over a period of time, and regularly clean up gesture commands with a usage frequency below the threshold, so as to automatically adapt to the long-term changes in user usage habits and eliminate infrequently used gestures, making the personalized template library more streamlined and efficient, and further improving the recognition speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a remote control method based on gesture recognition in an embodiment of the present application;

[0051] Figure 2 is another flow chart of a remote control method based on gesture recognition in an embodiment of the present application;

[0052] Figure 3It is a schematic diagram of a physical device structure of a remote control in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0055] For ease of understanding, the application scenarios of embodiments of the relevant technical solutions are introduced below.

[0056] Suppose Xiao Ming is a smart home enthusiast who has just purchased a smart TV. The smart TV is equipped with the latest voice control and gesture control functions. However, when he started using gestures to control the TV, he found that the recognition accuracy was not high. For example, Xiao Ming wanted to use a waving gesture to the right to switch to the next channel, but the TV often misjudged it as waving upwards, thus turning up the volume. Xiao Ming tried several times, but the error rate was still high and the user experience was poor. Xiao Ming carefully studied the instructions and found that the gesture recognition function was realized through the built-in camera of the TV. Since everyone's gesture habits are different, the built-in gesture templates of the TV are difficult to adapt to everyone. So Xiao Ming began to look for a better solution that can not only improve the accuracy of gesture recognition, but also adapt to the personalized needs of different users.

[0057] There are currently two mainstream gesture control solutions for smart TVs: one is to collect gestures through the TV's built-in camera and process and recognize them internally; the other is to integrate the gesture recognition module into the remote control to collect and recognize gestures through the remote control. Xiao Ming bought a remote control with built-in gesture recognition function, but because the gesture template is fixed inside the remote control, it cannot be adjusted according to the habits of different users. In particular, when the same remote control is used by multiple people, since each person's gesture expression for the same action may be different, the accuracy and response speed of the remote control recognition are still not ideal.

[0058] By adopting a remote control method based on gesture recognition in the present application, it is possible to maintain two sets of gesture template libraries, one of which is a preset standard template library, which contains common gestures and command mappings and can meet the general needs of most users; the other is a personalized template library, which is initially empty and can be dynamically generated according to the user's actual gesture habits and feedback during use. Suppose Xiao Ming bought a new remote control equipped with this technical solution. When used for the first time, the remote control automatically loads the standard template library, and Xiao Ming finds that most gesture commands can be correctly recognized. However, for some of his personal gesture habits, the recognition effect is still not ideal. So he tried to customize or modify several commonly used gesture commands and add them to the personalized template library. As the use time increases, the remote control continuously optimizes the personalized template library according to Xiao Ming's actual use, so that the recognition accuracy and response speed are significantly improved. At the same time, when Xiao Ming's parents or friends visit, the remote control can automatically switch to the standard template library, still maintaining a high recognition accuracy, and basically adapting to the gesture differences between different users.

[0059] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of a remote control method based on gesture recognition in an embodiment of the present application.

[0060] S101. After detecting a first gesture action, matching the first gesture action from a standard template library to obtain a first gesture instruction.

[0061] The remote control has a pre-stored standard template library, which contains a number of common standard gestures and their corresponding control commands. These standard gestures are usually simple and intuitive, such as waving, making a fist, pointing, etc., which are easy for different users to master. And each standard gesture is mapped to a specific gesture command (control command), such as turning on / off the TV, raising / lowering the volume, changing channels, etc.

[0062] When the user uses the remote control for the first time, the remote control will match a series of gestures made by the user within a preset time period (such as 10 minutes) in the standard template library one by one. Specifically, the remote control performs feature extraction and similarity calculation on the currently collected user gesture image, finds the standard gesture that best matches the first gesture, and obtains the control instruction device corresponding to the standard gesture as the first gesture instruction. For example, suppose there is a waving right gesture in the standard template library, and the corresponding instruction is to switch to the next channel. The user made a waving right gesture within the preset time period, and the remote control set this action as the first gesture. After matching with the standard template library, it is detected that it is closest to the standard gesture of waving right, so the first gesture instruction is determined to be switching to the next channel.

[0063] It should be noted that when the user uses the remote control for the first time, each gesture action in a series of gesture actions made by the user within a preset time period is a first gesture action, and the first gesture action only exists when the user uses the remote control for the first time.

[0064] S102: Match a preset number of first gesture actions with each personalized gesture action in the personalized template library respectively to obtain multiple gesture matching degrees corresponding to each first gesture action and the proportion of the best matching degree in the maximum gesture matching degree corresponding to all first gesture actions.

[0065] For the detected user's gesture action, the remote control can use the standard template library to obtain the first gesture instruction. However, different users may have different gesture habits, and the standard template library cannot completely cover all personal preferences. In order to improve the recognition accuracy and user experience, the remote control also stores a personalized template library in addition to the standard template library.

[0066] After the remote control detects the first gesture instruction, it will execute a preset number of the first gesture instructions in the order in which the instructions are detected, and match the first gesture actions corresponding to the certain number of first gesture instructions that have been executed with the personalized template library. By calculating the similarity between each first gesture action and each personalized gesture action in the personalized template library, the gesture matching degree between each first gesture action and each personalized gesture action in the personalized template library is obtained. The maximum gesture matching degree corresponding to each first gesture action is selected and set as the best matching degree, and the proportion of the best matching degree greater than the preset matching degree threshold in all maximum gesture matching degrees is calculated.

[0067] It should be noted that the gesture matching degree can be calculated using a variety of similarity measurement algorithms, such as Euclidean distance, cosine similarity, DTW (dynamic time warping), etc. The choice of algorithm needs to be based on the specific gesture features and application scenarios, and a balance between computational efficiency and accuracy, which is not limited here.

[0068] In addition, the above-mentioned personalized template library is similar to the standard template library, and is also composed of multiple gesture actions and corresponding instructions. However, unlike the global uniformity of the standard template library, the personalized template library can vary from person to person and be dynamically adjusted to adapt to the gesture habits of specific users. Specifically, when used for the first time, the personalized template library can be empty or the same as the standard template library. As the user continues to use it, the personalized template library will gradually accumulate the user's personalized gesture data.

[0069] S103: If it is detected that the proportion of the best matching degree in the maximum gesture matching degrees corresponding to all first gesture actions is greater than or equal to a preset threshold, the second gesture action detected subsequently will preferentially use the personalized template library for instruction matching to obtain a second gesture instruction.

[0070] After the remote controller obtains the gesture matching degree between the first gesture action and each personalized gesture action in the user's personalized template library, since the gesture matching degree reflects the degree of deviation between the user's gesture habit and the personalized template. If there is a preset number of first gesture actions corresponding to the maximum gesture matching degree higher than the preset matching degree threshold, it means that the first gesture action is closer to the personalized action of the personalized template. Further, if it is detected at this time that the number of first gesture actions with the maximum gesture matching degree higher than the preset matching degree threshold accounts for a high proportion of the number of all gesture actions in the preset time period (higher than the preset proportion threshold), it means that the current user habit is closer to the personalized action in the personalized template and the degree of personalization is high. In this case, the subsequent gesture action recognition will give priority to the personalized template library. Therefore, after the remote controller detects that the proportion of the best matching degree in the maximum gesture matching degree corresponding to all first gesture actions is greater than or equal to the preset threshold, the subsequent second gesture action detected will give priority to the personalized template library for instruction matching to obtain the second gesture instruction.

[0071] It should be noted that the personalized template library stores a set of multiple personalized gesture actions and corresponding gesture instructions, and the maximum gesture matching degree corresponding to the first gesture action that is greater than a preset matching degree threshold is the best matching degree.

[0072] In addition, the second gesture action refers to other gesture actions made by the user after the first gesture action. Specifically, when the remote control detects that the user has made a new gesture action (the second gesture action), the remote control will first match it with the personalized template library. If a gesture with a high matching degree is found in the personalized template library, the corresponding control instruction is directly taken out and set as the second gesture instruction for controlling client devices such as TVs.

[0073] S104: If it is detected that the gesture matching degree is less than a preset threshold among the maximum gesture matching degree corresponding to all first gesture actions or when the personalized template library is used to match the second gesture action, and the second gesture instruction does not exist in the personalized template library, a third gesture instruction is generated based on the matching result of the second gesture action in the standard template library.

[0074] Specifically, after step S102, the remote control can obtain the gesture matching degree between the first gesture action and each personalized gesture action in the personalized template library. Because the personalized gesture actions stored in the personalized template library may be empty when used for the first time, or may not have accumulated rich personalized gesture data, and cannot cover all gesture matching situations. Therefore, after the remote control detects that the gesture matching degree is less than the preset threshold, it will adopt a fallback mechanism, perform command recognition based on the matching result of the second gesture action in the standard template library, and obtain a third gesture command for remote control of the client.

[0075] Alternatively, after step S103, the remote control detects that the gesture matching degree is greater than or equal to the preset threshold value, but when matching the subsequent user's gesture action (the second gesture action) with the personalized template, if the second gesture action does not exist in the personalized template (or the gesture matching degrees between the second gesture action and each personalized gesture action in the personalized template library are less than the preset threshold value), the remote control will execute the fallback mechanism again, perform command recognition based on the matching result of the second gesture action in the standard template library, and obtain a third gesture command for remote control of the client.

[0076] It should be noted that after each execution of the fallback mechanism, the remote controller still uses the personalized template to match the next gesture command.

[0077] S105: Update the personalized template library according to the second gesture action and the corresponding matching result.

[0078] Specifically, if the remote control detects that the personalized gesture action and gesture instruction corresponding to the second gesture action do not exist in the personalized template library, a third gesture instruction will be generated based on the matching result of the second gesture action in the standard template library. Furthermore, when the remote control finds a standard gesture action and corresponding instruction that matches the second gesture action in the standard template library, the remote control will also record the matching information in the personalized template library for subsequent use. In this way, the personalized template library can be continuously accumulated and improved according to actual conditions, and ultimately can truly achieve mobile matching of user personalized gesture habits.

[0079] For example, if it is detected that the user often uses the half-circle downward gesture to turn down the volume, the gesture action and its instructions will be frequently recorded in the personalized template library. At the same time, the weight associated with the gesture will gradually increase. Eventually, even if it cannot be matched in the standard template library, it may be directly recognized in the personalized template library.

[0080] S106: Remotely control the client through the first gesture instruction, the second gesture instruction, and the third gesture instruction.

[0081] Specifically, after the remote control detects that the user uses gesture recognition for remote control for the first time, the first gesture instruction that matches the first gesture action is sent directly to the client for remote control. For the subsequent second gesture action, if the second gesture action can be matched to the corresponding second gesture instruction in the personalized template library, the second gesture instruction is directly sent to the client for remote control, and there is no third gesture instruction at this time. If the second gesture action does not match the corresponding second gesture instruction in the personalized template library. The third gesture instruction corresponding to the second gesture action can be matched through the standard template library, or the user can customize the third gesture instruction corresponding to the second gesture action, and the client is remotely controlled according to the third gesture instruction, which is not limited here.

[0082] In the above embodiment, the remote control can first use the standard template library to match the first gesture action to obtain the first gesture instruction, and then match the first gesture action with the personalized template library, and determine which template library to use for instruction matching for subsequent gesture actions based on the matching degree, so that while ensuring universality, a mapping relationship between personalized gesture actions and instructions suitable for specific users can be gradually established. When the matching degree of the personalized template library is high, the personalized template library is used first to match the gesture instructions, with high recognition accuracy and fast response speed. When the matching degree of the personalized template library is low or there is no corresponding instruction, the standard template library can be used for matching to avoid the situation where the gesture instruction cannot be recognized. At the same time, the matching result of the second gesture action is continuously used to update the personalized template library, so that it is continuously improved with use, and the accuracy of gesture instruction recognition will become higher and higher, which is very consistent with the actual usage habits of users. This method improves the accuracy and speed of gesture recognition while also having very good universality.

[0083] After combining the above scenarios, the following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of a remote control method based on gesture recognition in an embodiment of the present application.

[0084] S201 , collecting gesture image samples corresponding to various gesture instructions to perform feature extraction, and determining gesture feature templates corresponding to various gesture instructions.

[0085] Specifically, the remote control collects image samples of various gestures, which can come from gestures performed by multiple users in multiple environments. After collection, feature extraction is performed on the image samples, and feature vectors that can represent the essence of the gesture, such as shape, contour, direction, motion pattern, etc., are extracted from the original gesture image to determine the unique features of each gesture. Classification processing is performed according to the unique features of each gesture, and one or more gestures expressing the same gesture command are classified into one category, thereby obtaining gesture feature templates corresponding to each type of gesture command.

[0086] For example, to recognize a sliding gesture command, the remote control will capture multiple image samples of users performing sliding gestures, and then use feature extraction technology to analyze the direction, speed, trajectory and other characteristics of the gesture sliding. These features will form a feature template of the sliding gesture, which uniquely represents this type of gesture in the feature space.

[0087] It should be noted that the feature extraction of image samples can adopt image processing algorithms, such as edge detection, corner detection, contour tracking and other methods, so as to convert complex gesture images into simplified feature representations.

[0088] S202: Match semantic labels to gesture feature templates of different categories to obtain a standard template library.

[0089] Specifically, the remote control associates the gesture templates obtained by feature extraction with specific semantic tags to form a standard template library. Semantic tags are descriptions and classifications of gesture actions, such as left swipe, right swipe, or click, etc., matching abstract feature information with actual operation commands so that the remote control can understand the user's gesture intentions.

[0090] For example, taking the channel switching of a smart TV as an example, the user's left and right sliding gestures correspond to the channel switching down or up, respectively. To this end, each gesture feature template can be set with a label of switching down or switching up. In actual use, when a user is detected to make a left sliding gesture, the user's intention is recognized through the matched feature template, and the corresponding channel switching operation is performed.

[0091] It should be noted that matching semantic labels to gesture feature templates of different categories to obtain a standard template library can be achieved through supervised learning methods. Specifically, a model can be built and the model can be trained using a labeled data set (including different gesture features and corresponding semantic labels), thereby associating the input gesture features with pre-defined labels to obtain a standard template library. This is not limited here.

[0092] In the above embodiment, the remote controller can obtain a complete standard gesture command template library by collecting image samples of various gesture commands, extracting features, establishing feature templates, and then matching semantic tags. By pre-establishing a basic template library covering the main common gesture commands, it can better meet the basic needs of different users and provide users with basic gesture command recognition capabilities.

[0093] S203 , performing template matching on the gesture image of the first gesture action after preprocessing and feature extraction by using a similarity measurement algorithm to determine a target gesture action with the greatest similarity.

[0094] Specifically, after detecting the first gesture action, the remote controller captures a gesture image of the first gesture action through an image capture device, such as a camera. The captured gesture image is then subjected to image denoising, scaling, rotation correction and other preprocessing processes and feature extraction. After gesture feature extraction, the similarity between the first gesture action and each standard gesture action in the standard template library is calculated based on the extracted feature information, and the standard gesture action with the greatest similarity is set as the target gesture action.

[0095] S204: If the similarity of the target gesture action is greater than a preset similarity threshold, setting the gesture instruction corresponding to the target gesture action as the first gesture instruction.

[0096] After determining the target gesture action, the remote controller compares the similarity between the target gesture action and the first gesture action with a preset similarity threshold, and if the similarity is greater than the preset similarity threshold, the gesture instruction corresponding to the target gesture action is set as the first gesture instruction. The preset similarity threshold can be determined according to actual needs and is not limited here.

[0097] In the above embodiment, the remote controller can find the gesture category with the highest similarity by matching and comparing the gesture image features with the standard template features. At the same time, setting the similarity threshold can ensure the reliability of the output command and avoid misidentification. This further ensures the effectiveness and reliability of using the standard template library to identify gesture commands.

[0098] S205: If it is detected that the user has enabled the gesture display permission, the standard gesture and gesture instructions corresponding to the target gesture are displayed on the display of the client.

[0099] Specifically, the remote control can set gesture action display permissions. When it is detected that the user has turned on the gesture action display permission, the first gesture action of the user is currently detected and the corresponding first gesture instruction is determined. The corresponding standard gesture action and the function corresponding to the gesture instruction (such as turning on music, etc.) are displayed on the client's display, so that the user can understand the gesture command information currently detected by the remote control and adjust the gesture command information.

[0100] S206: If a negative instruction of the user to the standard gesture action is detected, the user's redefinition of the standard gesture action is received, and the personalized template library is updated.

[0101] While the remote control displays the standard gesture actions and gesture instructions corresponding to the target gesture actions through the client's display, it can receive the user's negative instructions. The negative instructions can be implemented by the user clicking on the case on the remote control that can send negative instructions, or by recognizing specific gestures displayed by the user (such as making a fist with one hand, etc.), which is not limited here.

[0102] After detecting the user's negative instruction for the standard gesture action, the remote control can enter the custom mode, in which the remote control can receive the user's redefinition of the standard gesture action. Specifically, in the custom mode, the remote control can display the user's current gesture action in real time on the client's display, and receive the user's function definition of the gesture action. In response to the user sending a confirmation instruction, the remote control generates a gesture instruction based on the function definition, and adds the gesture action to the personalized template library after matching it, so as to realize the user's customized update of each personalized template library.

[0103] In the above embodiment, the remote control can detect whether the user has enabled the gesture display permission. If enabled, after the gesture command is recognized, the standard gesture and corresponding command are displayed through the client display. The user can judge whether the recognition is accurate based on this. If the user issues a negative command, the standard gesture can be redefined, and the remote control will update the personalized template library based on the user's redefinition. This method provides a mechanism for users to actively correct gesture recognition results, further improving the recognition accuracy of the personalized gesture template library. At the same time, through the visual display of gestures, the user's understanding of standard gestures is deepened, which helps users to better perform gesture operations.

[0104] S207, recording the usage frequency of each personalized gesture action in the personalized template library within the preset working hours.

[0105] Specifically, after matching a personalized gesture action, the remote control can record the usage frequency of the personalized gesture action. The usage frequency is regularly cleared based on the preset working time. For example, if the preset working time is one month, the user may use the remote control to remotely control the client multiple times during this month. Each personalized gesture action that matches the user's gesture action in the personalized template library is recorded for frequency statistics of the personalized gesture action. After the end of one month and the remote control reads the frequency statistics results, the usage frequency records of all personalized gesture actions are cleared, and the frequency statistics are restarted.

[0106] S208. Regularly delete personalized gesture actions and corresponding gesture instructions in the personalized template library whose usage frequency is lower than a preset frequency threshold.

[0107] After each preset working time period, the personalized gesture actions and corresponding gesture instructions whose usage frequency is lower than the preset frequency threshold in the personalized template library are deleted.

[0108] In the above embodiment, the remote control will record the usage frequency of each gesture action in the personalized template library over a period of time, and regularly clean up gesture instructions with a usage frequency below a threshold, thereby automatically adapting to the long-term changes in user usage habits and eliminating infrequently used gestures, making the personalized template library more streamlined and efficient, and further improving the recognition speed. At the same time, it can also avoid the problem of reduced retrieval efficiency caused by excessive expansion of the personalized template library, and improve the timeliness of the personalized template library.

[0109] S209: Receive and update the user-defined gesture action and the corresponding gesture instruction.

[0110] Specifically, the remote control can respond to the user's instruction to select a customized gesture action, receive the user's customized gesture action and the specific function corresponding to the gesture action, and generate a corresponding gesture instruction based on the specific function. For example, after the user turns on gesture customization through the button on the remote control, the remote control will detect the user's gesture actions within a preset time period, obtain gesture image samples and perform feature extraction. If it is detected that the feature information of multiple image samples obtained within the preset time period is basically the same, the current gesture action is determined to be a new gesture action, and the user is prompted through the client's display that the gesture action entry has been completed and the function definition of the new gesture action needs to be performed. Specifically, the function definition of the new gesture action can be recognized by voice recognition technology to identify the user's oral function definition, or it can be automatically generated by recording the user's actual operation. This is not limited here.

[0111] After the function definition of the new gesture action is completed, and the corresponding new gesture instruction is generated based on the content of the function definition, the new gesture action and the corresponding new gesture instruction are simultaneously recorded in the standard template library and the personalized template library.

[0112] In the above embodiment, the remote control can receive new gesture actions and corresponding instructions defined by the user, and use them to update the standard template library and the personalized template library, thereby providing users with maximum flexibility. Based on this, users can freely define personalized gesture instructions, no longer limited to the preset instruction set, so that the remote control can adapt to a wider range of control scenarios and meet the diversified needs of users. At the same time, user-defined gesture instructions can be added to the standard template library or the personalized template library at the same time. After long-term accumulation, the standard template library or the personalized template library can be continuously enriched and improved, and the recognition accuracy and comprehensive coverage will become higher and higher.

[0113] S210: Receive user voice information and perform voice verification on the control instruction.

[0114] Specifically, after the user turns on the verification function, the remote control can receive the user's voice information through voice recognition technology when executing control commands. If it is detected that the gesture command recognized from the user's voice information is different from the control command to be executed or being executed, the gesture command recognized by the voice will be selected for execution by default. Of course, the user can also adjust it to select the control command for execution by default, which is not limited here.

[0115] In the above embodiment, the remote controller can synchronously receive the user's voice information for verification before executing the control command or during the execution of the command, which can further eliminate possible ambiguity and improve the accuracy of gesture action recognition.

[0116] The remote controller of the embodiment of the present invention is an electronic device. Figure 3 A schematic diagram of the architecture of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0117] It should be noted that Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0118] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions (computer programs), or by controlling related hardware through instructions (computer programs), and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. The electronic device of this embodiment includes a storage medium and a processor, wherein a plurality of instructions are stored in the storage medium, and the instructions can be loaded by the processor to execute any step of the method provided in the embodiment of the present invention.

[0119] Specifically, the storage medium and the processor are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more signal lines. The storage medium stores computer execution instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware. The processor executes various functional applications and data processing by running the software program and module stored in the storage medium. The storage medium can be, but is not limited to, random access storage medium (Random Access Memory, referred to as: RAM), read-only storage medium (Read Only Memory, referred to as: ROM), programmable read-only storage medium (Programmable Read-Only Memory, referred to as: PROM), erasable read-only storage medium (Erasable Programmable Read-Only Memory, referred to as: EPROM), electrically erasable read-only storage medium (Electric Erasable Programmable Read-Only Memory, referred to as: EEPROM), etc. Among them, the storage medium is used to store programs, and the processor executes the program after receiving the execution instruction.

[0120] Furthermore, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip having signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., which may implement or execute the various methods, steps, and logic flow diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] Since the instructions stored in the storage medium can execute the steps in any method provided in the embodiments of the present invention, the beneficial effects of any method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0122] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A remote control method based on gesture recognition, applied to a remote controller, characterized in that: The method comprises: After detecting a first gesture action, matching the first gesture action from a standard template library to obtain a first gesture instruction, wherein the standard template library stores a set of multiple standard gesture actions and corresponding gesture instructions, and the first gesture action is a gesture action detected within a preset time period of initial use; Matching a preset number of the first gesture actions with each personalized gesture action in the personalized template library respectively, obtaining multiple gesture matching degrees corresponding to each of the first gesture actions and the proportion of the best matching degree in the maximum gesture matching degrees corresponding to all the first gesture actions, wherein the personalized template library stores a set of multiple personalized gesture actions and corresponding gesture instructions, and the maximum gesture matching degree that is greater than a preset matching degree threshold among the multiple maximum gesture matching degrees corresponding to the multiple first gesture actions is the best matching degree; If it is detected that the proportion of the best matching degree in the maximum gesture matching degrees corresponding to all the first gesture actions is greater than or equal to a preset threshold, the second gesture action detected subsequently will preferentially use the personalized template library for instruction matching to obtain a second gesture instruction, where the second gesture action is other gesture actions detected in addition to the first gesture action; If it is detected that the proportion of the best matching degree in the maximum gesture matching degrees corresponding to all the first gesture actions is less than the preset threshold or when the second gesture action is matched using the personalized template library, the second gesture instruction does not exist in the personalized template library, a third gesture instruction is generated according to the matching result of the second gesture action in the standard template library; Updating the personalized template library according to the second gesture action and the corresponding matching result; The client is remotely controlled through the first gesture instruction, the second gesture instruction and the third gesture instruction.

2. The method according to claim 1, characterized in that Before the step of matching the first gesture action from a standard template library to obtain a first gesture instruction after detecting the first gesture action, the method further includes: Collect gesture image samples corresponding to various gesture instructions to perform feature extraction, and determine gesture feature templates corresponding to the various gesture instructions; Semantic labels are matched to the gesture feature templates of different categories to obtain the standard template library, wherein the semantic labels are gesture instructions corresponding to gesture actions.

3. The method according to claim 1 or 2, characterized in that: After detecting the first gesture action, matching the first gesture action from a standard template library to obtain a first gesture instruction specifically includes: After detecting the first gesture action, collecting a gesture image; Performing template matching on the gesture image after preprocessing and feature extraction by using a similarity measurement algorithm; Determine the target gesture action with the greatest similarity; If the similarity of the target gesture mode action is greater than a preset similarity threshold, the gesture instruction corresponding to the target gesture action is set as the first gesture instruction.

4. The method according to claim 3, characterized in that After the step of determining the target gesture action with the greatest similarity, the method further includes: Check whether the user has enabled the gesture display permission; If yes, the standard gesture action and gesture instruction corresponding to the target gesture action are displayed on the display of the client; If a user's negation instruction for the standard gesture action is detected, receiving the user's redefinition of the standard gesture action, the negation instruction includes a key negation instruction and a gesture negation instruction; The personalized template library is updated according to the redefinition result.

5. The method according to claim 1, characterized in that After the step of updating the personalized template library according to the second gesture action and the corresponding matching result, the method further includes: Record the usage frequency of each personalized gesture action in the personalized template library during the preset working time, where the working time refers to the operating time from when the remote control is turned on to when it is turned off; The personalized gesture actions and corresponding gesture instructions whose usage frequency is lower than a preset frequency threshold in the personalized template library are deleted regularly.

6. The method according to claim 1, characterized in that After the step of updating the personalized template library according to the second gesture action and the corresponding matching result, the method further includes: Receive new gesture actions and corresponding new gesture instructions defined by the user; The standard template library and the personalized template library are updated according to the new gesture action and the corresponding new gesture instruction.

7. The method according to claim 1, characterized in that Before the step of remotely controlling the client through the first gesture instruction, the second gesture instruction and the third gesture instruction, the method further includes: Receive user voice information; The control instruction is voice-verified according to the voice information, and the control instruction includes the first gesture instruction, the second gesture instruction, and the third gesture instruction.

8. A remote controller, characterized in that: The remote controller includes: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the remote controller to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the remote controller, the remote controller executes the method as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a remote controller, the remote controller is enabled to execute the method according to any one of claims 1 to 7.

Images