Hand information recognition method, control method, device, electronic equipment and medium

Abstract

The present disclosure discloses a hand information recognition method, a control method, an apparatus, an electronic device and a medium. The method comprises: obtaining a current hand image of a current frame; recognizing the current hand image to obtain corresponding candidate hand information; determining target hand information according to the candidate hand information and historical hand information, the historical hand information being hand information determined based on historical hand images before the current frame, the candidate hand information comprising candidate gesture category information and candidate hand key point information, the historical hand information comprising historical gesture category information and historical hand key point information, and the target hand information comprising target gesture category information and target hand key point information. By using the method, the accuracy of determining the target hand information is improved by determining the target hand information according to the candidate hand information and the historical hand information, the success rate of human-computer interaction by using the target hand information is improved, and the experience of human-computer interaction by using the target hand information is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to hand information recognition methods, control methods, devices, electronic devices, and media. Background Technology

[0002] With the development of wireless communication technology and the mobile Internet industry, smart terminal (such as mobile phone) applications are quite common, bringing many conveniences to individuals and profoundly affecting everyone's life.

[0003] Human-computer interaction is a discipline that studies the interactive relationship between a system and a user. A system can be various types of machines, or it can be a computerized system and software, such as a smart terminal.

[0004] Currently, human-computer interaction primarily relies on touchscreens (such as mobile phones or tablets) or physical buttons (light switches or computer keyboards). However, as technology evolves and people become accustomed to the convenience of existing human-computer interaction methods, they will continue to seek more convenient ways to break free from the limitations of current methods. For example, contactless human-computer interaction through motion recognition.

[0005] However, currently, human-computer interaction through motion recognition is prone to errors, which can lead to control failures and negatively impact the user's experience. Summary of the Invention

[0006] This disclosure provides a hand information recognition method, control method, device, electronic device, and medium, which improves the accuracy of target hand information determination, increases the success rate of human-computer interaction based on target hand information, and enhances the user's experience of human-computer interaction based on target hand information.

[0007] In a first aspect, embodiments of this disclosure provide a hand information recognition method, including:

[0008] Get the current hand image in the current frame;

[0009] Identify the current hand image to obtain corresponding candidate hand information;

[0010] Based on the candidate hand information and historical hand information, the target hand information is determined, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame;

[0011] The candidate hand information includes candidate gesture category information and candidate hand key point information; the historical hand information includes historical gesture category information and historical hand key point information; and the target hand information includes target gesture category information and target hand key point information.

[0012] Secondly, embodiments of this disclosure also provide a control method, including:

[0013] Obtain hand information within a set time period;

[0014] Based on the hand information, determine the change information of the hand information within the set time period;

[0015] Based on the hand information and the transformation information, the corresponding control command is determined;

[0016] Control is performed according to the control instructions;

[0017] The hand information is determined based on the target hand information.

[0018] Thirdly, embodiments of this disclosure provide a hand information recognition device, including:

[0019] The acquisition module is used to acquire the current hand image in the current frame;

[0020] The recognition module is used to recognize the current hand image to obtain corresponding candidate hand information;

[0021] The determination module is used to determine the target hand information based on the candidate hand information and the historical hand information, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame;

[0022] The candidate hand information includes candidate gesture category information and candidate hand key point information; the historical hand information includes historical gesture category information and historical hand key point information; and the target hand information includes target gesture category information and target hand key point information.

[0023] Fourthly, embodiments of this disclosure provide a control device, including:

[0024] The acquisition module is used to acquire hand information within a set time period;

[0025] The first determining module is used to determine the change information of the hand information within the set time period based on the hand information;

[0026] The second determining module is used to determine the corresponding control command based on the hand information and the transformation information;

[0027] The control module is used to control according to the control instructions.

[0028] Fifthly, embodiments of this disclosure also provide an electronic device, including:

[0029] One or more processing devices;

[0030] Storage device for storing one or more programs;

[0031] The one or more programs are executed by the one or more processing devices, causing the one or more processing devices to implement the methods provided in the embodiments of this disclosure.

[0032] Sixthly, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the methods provided in embodiments of this disclosure.

[0033] This disclosure provides a hand information recognition method, control method, device, electronic device, and medium. First, a current hand image of the current frame is acquired; then, the current hand image is recognized to obtain corresponding candidate hand information; finally, target hand information is determined based on the candidate hand information and historical hand information, where the historical hand information is hand information determined based on historical hand images prior to the current frame. By utilizing the above technical solution, the target hand information determined through candidate hand information and historical hand information improves the accuracy of target hand information determination, increases the success rate of human-computer interaction using target hand information, and enhances the user experience of human-computer interaction using target hand information. Attached Figure Description

[0034] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0035] Figure 1 This is a flowchart illustrating a hand information recognition method provided in Embodiment 1 of this disclosure;

[0036] Figure 2 This is a flowchart illustrating a control method provided in Embodiment 2 of this disclosure;

[0037] Figure 3 This is a flowchart illustrating a control method provided in Embodiment 3 of this disclosure;

[0038] Figure 4 This is a schematic diagram of the structure of a hand information recognition device provided in Embodiment 4 of this disclosure;

[0039] Figure 5 This is a schematic diagram of the structure of a control device provided in Embodiment 5 of this disclosure;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this disclosure. Detailed Implementation

[0041] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0042] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0043] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0044] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0045] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0046] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0047] In the following embodiments, each embodiment provides optional features and examples. The various features described in the embodiments can be combined to form multiple optional solutions. Each numbered embodiment should not be regarded as only one technical solution. Furthermore, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0048] Example 1

[0049] Figure 1This is a flowchart illustrating a hand information recognition method provided in Embodiment 1 of this disclosure. This method is applicable to situations where target hand information needs to be recognized. The method can be executed by a hand information recognition device, which can be implemented by software and / or hardware and is generally integrated into an electronic device. In this embodiment, the electronic device includes, but is not limited to, devices capable of acquiring images such as mobile phones and interactive flat panels.

[0050] This disclosure enables human-computer interaction control via electronic devices. The scenarios for human-computer interaction control are not limited here. The electronic device can control the device itself or a controlled device. The controlled device can be considered as a device controlled by the electronic device.

[0051] For example, an electronic device controls the content displayed on the device based on the recognition results of hand information, such as controlling the page turning, swiping up and down of the displayed content.

[0052] For example, electronic devices control controlled devices based on the recognition results of hand information, with different controlled devices corresponding to different content. The controlled devices are not limited here. Taking a smart light as an example, different hand information can control the light to turn on and off. For instance, if the hand information indicates that the current gesture is a clenched fist and the movement trajectory of the key point of the palm is upward and then downward (similar to pulling a light cord), a light-on command can be triggered. The electronic device sends the light-on command to the controlled device, i.e., the smart light, achieving contactless light-on.

[0053] For example, this disclosure can control the content of a PowerPoint presentation displayed on the device or a controlled device (such as a touch screen all-in-one machine or a conference tablet) using hand information. For instance, the opening and closing of the palm and the position of the palm's key points can be used to simulate various functions during PowerPoint use. Turning a PowerPoint presentation to the left is controlled by hand information indicating the movement of the palm from right to left with the palm open and the palm's key points moving. Turning a PowerPoint presentation to the right is controlled by hand information indicating the movement of the palm from left to right with the palm open and the palm's key points moving. Exiting the PowerPoint presentation is controlled by hand information indicating a hand gesture from an open palm to a clenched fist.

[0054] For example, this disclosure can control a short video application viewed by a user using hand information, which can be installed on this device or a controlled device. For instance, the fingertip keypoint of the index finger can be used as a positional control signal to move a mouse on a desktop. A click operation can be triggered by hand information indicating a hand gesture from open to clenched. A scroll-up operation can be triggered by hand information indicating a hand gesture from open to closed, with the palm keypoint moving from bottom to top. A scroll-down operation can be triggered by hand information indicating a hand gesture from open to closed, with the palm keypoint moving from top to bottom. A "like" operation can use a "calling" gesture or a "like" gesture as the trigger gesture for liking a video, and the like operation is triggered by hand information indicating the trigger gesture.

[0055] like Figure 1 As shown in Embodiment 1 of this disclosure, a hand information recognition method includes the following steps:

[0056] S110. Obtain the current hand image in the current frame.

[0057] In this embodiment, the electronic device can capture the user's current hand image in real time. The current hand image can be considered as the image including the user's hand at the current moment. The specific technical means of acquiring the current hand image are not limited here. The current hand image can be the hand image corresponding to the current frame.

[0058] In one embodiment, this disclosure allows the acquisition device of an electronic device, such as a camera, to capture a user's image. The user image can be an image containing the user's hand captured by the acquisition device. Alternatively, the user image can be an image frame from a video containing the user captured during human-computer interaction. Image recognition technology can be used to identify the hand in the video, thereby enabling interactive control. This avoids physical contact with the interactive device (this device or the controlled device), making human-computer interaction more convenient.

[0059] After acquiring a user image, this disclosure allows for the detection of hands included in the user image. For example, hand position information can be detected through image recognition to extract the hand image. Specifically, the user image is scaled, and the current frame's hand position information is determined based on the detection results of the hand position information from the previous frame. The hand position information can be information representing the position of the hand in the user image.

[0060] For example, assuming no hand was detected in the previous frame of the user image, a convolutional neural network is used in this frame to detect hands and extract all hands from the user image. The hand position information is then output. If a hand was detected in the previous frame of the user image, a small convolutional neural network is used in this frame to predict a more precise position of the hand in the vicinity of the hand based on the detection results of the hand in the previous frame, and the hand position information is then output.

[0061] In this disclosure, the convolutional neural network can be a compressed and quantized network to improve its operating speed while ensuring detection performance. Furthermore, when a hand is present in the previous frame of the user image, the running convolutional neural network is smaller to improve the speed of hand information recognition. The network structure of the small convolutional neural network is not limited here, as long as it is simpler than the network structure used in the previous frame of the user image where no hand was detected.

[0062] After determining the hand position information in the user image, the hand image is extracted from the user image.

[0063] S120. Identify the current hand image to obtain corresponding candidate hand information.

[0064] Candidate hand information can be considered as hand information directly determined based on the current hand image. The content of candidate hand information is not limited here. The content of candidate hand information can be determined based on the hand information required for human-computer interaction. Candidate hand information includes candidate gesture category information and candidate hand key point information.

[0065] The candidate gesture category information can be the gesture category determined after analyzing the current hand image. Gesture category information can be considered as information representing the category to which the gesture belongs. The candidate hand keypoint information can be considered as hand keypoint information determined based on the current hand image. Hand keypoint information can be considered as information representing the location of hand keypoints.

[0066] This step can classify the hand based on the hand image and determine the key point information of the candidate hand to obtain the candidate hand information.

[0067] In one embodiment, this step may involve scaling the extracted current hand image and inputting it into a convolutional neural network for gesture classification to obtain candidate hand classification information for the current hand image; or, obtaining the original probability of the gesture corresponding to the current hand image belonging to each gesture category in the original gesture categories, and then, based on the mapping relationship between the original probabilities and the categories, determining the target probability of the gesture corresponding to the current hand image belonging to each gesture category in the target gesture categories, and finally selecting the gesture category corresponding to the target probability with the largest value as a candidate gesture category, and indicating the candidate gesture category through the hand classification information.

[0068] In one embodiment, this step inputs a hand image into a convolutional neural network for hand keypoint prediction to obtain initial keypoint location information, such as the locations of 21 finger keypoints.

[0069] It should be noted that the network structure of any convolutional neural network disclosed herein is not limited, as long as it can guarantee the implementation of the corresponding function.

[0070] S130. Determine the target hand information based on the candidate hand information and historical hand information.

[0071] Historical hand information can be considered as hand information determined based on historical hand images preceding the current frame. Historical hand images can be considered as hand images from historical moments. Historical hand images can be hand images preceding the current hand image. The number of historical hand images can be one or more, and the specific number can be determined based on the actual business scenario.

[0072] The target hand information can be considered as hand information used for human-computer interaction. This disclosure allows for the determination of control commands based on the target hand information, enabling human-computer interaction control based on these commands. The control commands can be considered as instructions for implementing human-computer interaction control.

[0073] In one embodiment, the target gesture category information and the target hand key point information can both be the gesture category information and hand key point information of the current frame.

[0074] In one embodiment, the target gesture category information can be the gesture category information of the current hand image in the current frame, and the target hand key point information can be the hand key point information of the previous frame.

[0075] In one embodiment, the target gesture category information can be the gesture category information of a set frame, and the target hand key point information can be the hand key point information of the current frame.

[0076] This embodiment can determine the target hand information based on the prior nature of gestures during human-computer interaction control (such as gesture transitions being very "smooth," meaning that only these two gestures will occur during the process of changing from one gesture to another).

[0077] In one embodiment, the candidate hand information includes candidate gesture category information and candidate hand key point information, the historical hand information includes historical gesture category information and historical hand key point information, and the target hand information includes target gesture category information and target hand key point information.

[0078] Historical gesture category information can be considered as gesture category information determined based on historical frames at a historical moment. This information can be candidate gesture category information for historical hand images determined based on historical hand images at a historical moment; or it can be target gesture category information for historical hand images determined based on historical hand images at a historical moment. When the target gesture category information is determined from a historical hand image, it can be used to determine the target gesture category information for the current frame. When the target gesture category information is not determined from a historical hand image, the candidate gesture category information can be used to determine the target gesture category information for the current frame.

[0079] The number of historical frames can be one or more. Historical frames can include the frame preceding the current frame, and can be obtained through a sliding window. When there are multiple historical frames, they are sequentially consecutive. For example, if the current frame is frame 9, then the historical frames could be frames 8, 7, and 6. The number of historical frames can be determined based on the actual scenario, such as the duration of each gesture during human-computer interaction. Historical hand keypoint information can be considered as hand keypoint information determined based on historical frames at a historical moment. Historical hand keypoint information can be candidate hand keypoint information determined based on historical hand images at a historical moment; it can also be target hand keypoint information of historical hand images determined based on historical hand images at a historical moment. When target hand keypoint information of a historical hand image exists, the historical hand keypoint information can be the target hand keypoint information of the historical hand image. When target hand keypoint information of a historical hand image is not determined, the historical hand keypoint information can be candidate hand keypoint information of the historical hand image.

[0080] When determining candidate hand information using convolutional neural networks, this information may contain noise. For example, motion blur may occur due to rapid user movement during human-computer interaction, the hand image resolution may be low, or the hand may be occluded in the image. In such cases, the predicted keypoint positions of candidate hands determined by the convolutional neural network may be inaccurate, or the gesture category indicated by the candidate gesture category information may be incorrect. To obtain relatively robust target hand information, this step can determine the target hand information based on both candidate and historical hand information; that is, by combining historical hand information from historical frames.

[0081] When the target hand information includes target gesture category information and target hand key point information, the target gesture category information can be determined based on candidate gesture category information and historical gesture category information; the target hand key point information can be determined based on candidate hand key point information and historical hand key point information.

[0082] For example, this disclosure can statistically analyze the gesture category information with the highest proportion among historical gesture category information and candidate gesture category information to obtain the target gesture category information.

[0083] For example, this disclosure can determine the motion trajectory of hand key points based on candidate hand key point information and the candidate hand key point information, so as to determine the target hand key point information based on the motion trajectory of hand key points. For example, the target hand key point information can be obtained by processing the candidate hand key point information and historical hand key point information using the least squares method.

[0084] The candidate hand keypoint locations are predicted based on a convolutional neural network. These locations will contain some noise, manifesting as jitter in the time sequence. Using jittered keypoint locations as control signals for interaction will result in a poor user experience. A common approach to remove jitter is to interpolate the current keypoint location with the keypoint location of the previous frame to limit the difference between the two frames. While this method removes jitter, it also introduces a latency issue, as the keypoint location always lags behind the actual location, resulting in a poor user experience. This disclosure introduces a "future" moment from the current time to overcome this latency. The target hand keypoint information determined in this disclosure can be the hand keypoint information of a set frame preceding the current frame, overcoming latency while making the determined target hand keypoint information smoother. Specifically, when determining the target hand keypoint information of the set frame, it is based on the hand images of each frame from the set frame to the current frame, i.e., it uses the hand images obtained at the "future" moment corresponding to the set frame to determine the target hand keypoint information of the set frame.

[0085] The number of historical hand information corresponding to different target hand information may be the same or different. For example, when the target hand information is target gesture category information, the number of historical gesture category information can be equal to the number of historical hand images collected within a first preset time period corresponding to the current frame. When the target hand information is target hand keypoint information, the number of historical hand keypoints can be the number of historical hand images collected within a second preset time period corresponding to the current frame. The relationship between the first preset time period and the second preset time period is not limited here.

[0086] The hand information recognition method provided in Embodiment 1 of this disclosure first acquires the current hand image of the current frame; then, it identifies the current hand image to obtain corresponding candidate hand information; finally, it determines the target hand information based on the candidate hand information and historical hand information. By utilizing the above technical solution, the target hand information determined through candidate hand information and historical hand information improves the accuracy of target hand information determination, increases the success rate of human-computer interaction using target hand information, and enhances the user experience of human-computer interaction using target hand information.

[0087] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0088] In one embodiment, determining the target hand information based on the candidate hand information and historical hand information includes:

[0089] Retrieve historical gesture category information;

[0090] The gesture category information with the highest proportion among the candidate gesture category information and the historical gesture category information is determined as the target gesture category information for the current frame.

[0091] The historical gesture category information can be a set number of gesture categories corresponding to the current frame up to the current time. The set number is not limited here.

[0092] This step determines the target gesture category information for the current frame by selecting the gesture category information that appears most frequently in the candidate gesture category information and the historical gesture category information, i.e., the gesture category information with the highest percentage of occurrences.

[0093] In one embodiment, obtaining historical gesture category information includes:

[0094] Use a sliding window to obtain historical gesture category information within a first preset time period.

[0095] This embodiment can obtain historical gesture category information within a first preset time period before the current moment corresponding to the current frame through a sliding window. This disclosure does not limit the step size or size of the sliding window; these can be determined based on the actual scenario. This disclosure can determine the size and step size of the sliding window based on the duration of each type of hand information during human-computer interaction.

[0096] For example, this disclosure can use interval voting to smooth the results of gesture classification, i.e., the target gesture category information. In reality, gesture transitions are very "smooth," meaning that only two gestures will occur during the process of transforming from one gesture to another. Utilizing this prior knowledge, this disclosure uses a voting algorithm, with the specific steps being: 1) Using a sliding window applied to the time series to obtain the gesture classification results (which may include candidate gesture category information and / or historical gesture category information) within a certain period (i.e., within a first preset time duration); 2) Statistically analyzing the gesture classification results within the sliding window; 3) Selecting the category with the highest statistical result as the final result. Through the voting algorithm, a gesture classification result that is naturally smooth over time can be obtained, increasing the robustness of gesture classification.

[0097] In one embodiment, identifying the current hand image to obtain corresponding candidate hand information includes:

[0098] The original probability of the gesture corresponding to the current hand image belonging to each gesture category in the original gesture category is determined by using a convolutional neural network.

[0099] Based on the original probability and category mapping relationship, determine the target probability that the gesture corresponding to the current hand image belongs to each gesture category in the target gesture category;

[0100] Determine candidate gesture category information for the current hand image, wherein the candidate gesture category information represents the gesture category corresponding to the target probability with the largest value among the target probabilities;

[0101] The target gesture category is formed by reclassifying the gesture categories included in the original gesture category based on their similarity. The number of categories included in the target gesture category is less than the number of categories included in the original gesture category. The category mapping relationship represents the mapping relationship between each gesture category in the original gesture category and each gesture category in the target gesture category.

[0102] The original gesture category can be considered a general gesture category, such as the gesture classification used in gesture control in related technologies. The target gesture classification is the gesture classification determined after reclassifying the original gesture classification. The target gesture classification includes fewer categories than the original gesture classification.

[0103] By inputting the current hand image into a convolutional neural network, this disclosure can obtain the original probability that the gesture corresponding to the hand in the current hand image belongs to each gesture category in the original gesture categories. The original probability can be considered as the probability that the gesture corresponding to the current hand image belongs to each gesture category in the original hand categories.

[0104] For example, the original hand categories include category 1, category 2, category 3, category 4, and category 5. After the current hand image is processed by the convolutional neural network, the output probability that the current hand image belongs to category 1, category 2, category 3, category 4, and category 5 is 10%, 20%, 40%, 40%, and 10%, respectively.

[0105] The category mapping relationship can be considered as the mapping relationship between each gesture category in the original gesture category and each gesture category in the target gesture category. This category mapping relationship can be considered as being determined based on the similarity between each gesture category in the original gesture category.

[0106] For example, if the similarity between category 1 and category 2 in the original hand gesture categories is greater than a threshold, the two categories can be reclassified into one category, such as category A. If the similarity between category 3 and category 4 is greater than a threshold, then category 3 and category 4 can be reclassified into one category, such as category B. The target gesture category includes categories A, B, and C. The category mapping relationship is that category 1 and category 2 correspond to category A, category 3 and category 4 correspond to category B, and category 5 corresponds to category C. The number of categories here is only an example and is not limited. For example, a convolutional neural network for gesture classification can recognize 47 common gestures. Based on the needs of practical applications, this disclosure may only require a few common gestures. Therefore, this disclosure can establish a category mapping relationship between the target gesture category and the original gesture category. The category mapping relationship can be determined based on the similarity of 47 common gestures, treating multiple categories with high similarity as one category, thereby improving the robustness of gesture classification.

[0107] When determining the target probability of each gesture category in the target gesture category, the sum of the original probabilities belonging to the same category in the original gesture classification can be used as the target probability of the corresponding category in the target gesture category.

[0108] For example, the target probability for each gesture category in the target gesture category is 30%, 80%, and 10%, respectively: Category A, Category B, and Category C. The candidate gesture category information indicates that the gesture category of the current hand image is Category B, which is the gesture category corresponding to the highest target probability among the target probabilities.

[0109] This disclosure does not specify the technical means for determining the category mapping relationship, such as directly obtaining information from user feedback in actual applications. User feedback can characterize gestures that are easily confused by the user in human-computer interaction; alternatively, it can be determined through a similarity determination model, which can be a neural network model. The sample pair input to the similarity determination model can be multiple sample hand images and the corresponding sample gesture category information for each sample hand image. After recognizing the sample gesture images, the similarity determination model obtains the identified gesture category information. The similarity determination model can compare the identified gesture category information with the sample gesture category information. If they are inconsistent, the identified gesture category information and the sample gesture category information can be grouped into one category. To further improve the accuracy of target category information classification, frequency statistics can be performed. If the number of times a sample gesture category information is recognized as another gesture category information exceeds a set threshold, the sample gesture category information and the other gesture category information are grouped into one category. The other gesture category information is gesture category information other than the sample gesture category information. The specific value of the set threshold is not limited here.

[0110] In one embodiment, determining the target hand information based on the candidate hand information and historical hand information includes:

[0111] The historical key hand information within a second preset time period is obtained through a sliding window;

[0112] The candidate hand key point information and the historical hand key point information are processed using the least squares method to obtain the target hand key point information of the set frame. The set frame is the frame collected at the time corresponding to the center of the sliding window. The target hand key point information is associated with the target gesture category information of the set frame and stored to determine the corresponding control command.

[0113] The size and step size of the sliding window for acquiring historical hand key point information can differ from those for acquiring historical hand category information. This embodiment can acquire historical hand key point information within a second preset time period preceding the current moment in the current frame. The second preset time period can be related to the size and step size of the sliding window for acquiring historical hand key point information; the specific relationship can be determined based on the actual scenario and is not limited here.

[0114] This embodiment uses a linear model to determine candidate hand keypoint information and historical hand keypoint information using the least squares method. This linear model can be considered a model of fixed order. For example, assuming the current time is t, a sliding window is used to obtain keypoint results Y over a time series, including candidate hand keypoint information and historical hand keypoint information for the current frame. The length of the sliding window is l, and its center is t.

[0115] Y = [p0, p1, ..., p] l-1 ] T

[0116] Where, p l-1 This can be considered as candidate hand keypoint information for the current frame, where the keypoint result Y, excluding p... l-1 The remaining content can be considered as key information about the historical hand.

[0117] This disclosure establishes a linear model M for the movement of key hand positions. Based on the key hand information sequence Y obtained using a sliding window, the linear model of the key points is estimated using the least squares method. Then, t is substituted into the linear model to obtain the smoothed target hand key point information.

[0118] M = (X T X) -1 X T Y

[0119] in, X is determined based on the length of the hand key point position sequence Y. The numbers in the nth column are the column numbers raised to the power of their corresponding row numbers.

[0120] If the numbers in the second column are all 0 1 =0,1 7 =1,2 1 =2……(l-1) 1 = l-1. d is the order of the linear model. If the trajectory conforms to an acceleration model, then d is 2.

[0121] After determining M, the t-column of X can be multiplied by M, with X as the independent variable of M, to obtain the smoothed value at the corresponding time, such as time t, which is the target hand key point information of the set frame corresponding to time t. By using the least squares method, the error of all target hand key point information over a period of time can be minimized. Since the hand key point information sequence Y is obtained using a sliding window, the target hand key point information determined in this disclosure solves the lag problem caused by using interpolation smoothing to determine the target hand key point information.

[0122] Example 2

[0123] Figure 2 This is a flowchart illustrating a control method provided in Embodiment 2 of this disclosure. This method is applicable to situations where control is based on hand information, such as human-computer interaction control based on hand information. The method can be executed by a control device, which can be implemented in software and / or hardware and is generally integrated into an electronic device. In this embodiment, the electronic device includes, but is not limited to, devices capable of acquiring images, such as mobile phones and interactive whiteboards.

[0124] It should be noted that the electronic device performing the hand information recognition method can be the same device as the electronic device performing the control method, or it can be a different device.

[0125] like Figure 2 As shown in the embodiments of this disclosure, the control method includes the following steps:

[0126] S210, Obtain hand information within a set time period.

[0127] The hand information includes gesture category information and key hand points information.

[0128] Hand information is determined according to the hand information recognition method described in this disclosure. It should be noted that after recognizing the current hand image of the current frame using the hand information recognition method provided in this disclosure, what is obtained is the target gesture category information of the current frame and the target hand key point information of a set frame. In this step, when obtaining hand information for a set duration, the target gesture category information and target hand key point information corresponding to the same frame can be obtained.

[0129] For example, if the current frame is frame 9, the hand information recognition method of this disclosure determines the target gesture category information of frame 9 and the target hand key point information of the preceding frame, such as frame 6. After obtaining the target gesture category information of the current frame, it can be cached until the target hand key point information of the current frame is obtained, at which point the corresponding control command is determined. After determining the target hand key point information of the preceding frame, the target gesture category information of the preceding frame can be obtained, and the corresponding control command can be determined.

[0130] The hand information disclosed herein can refer to the target gesture category information and target hand key point information in the same frame.

[0131] S220. Determine the change information of the hand information within the set time period based on the hand information.

[0132] Transformation information refers to the changes in hand information within a set time period. For example, transformation information can indicate the change of the hand from a fist to a palm within a set time period. This step can determine the transformation information of hand information in time sequence based on hand information. Transformation information can characterize the change of gesture category information in hand information, or it can characterize the change of key hand points in hand information.

[0133] If the hand information does not change within the set time period, the change information can indicate that the hand information has not changed; otherwise, the change information can indicate the change method of the hand information, such as the change information indicating that the hand information is from a fist to a palm, or the change information indicating that the hand information is moving from top to bottom.

[0134] S230. Determine the corresponding control command based on the hand information and the transformation information.

[0135] Control commands can be considered as instructions for implementing human-computer interaction control. There can be a one-to-one correspondence between hand information, change information, and control commands.

[0136] S240. Control is performed according to the control command.

[0137] After determining the control command, this embodiment can perform human-computer interaction control based on the control command. This disclosure outputs corresponding control commands through sequential hand information, which increases robustness and makes users feel more immersed when performing human-computer interaction.

[0138] It should be noted that the hand information in this embodiment can be determined using the hand information recognition method described in this disclosure, or by using related technologies.

[0139] This disclosure provides a control method in embodiment two. First, it acquires hand information within a set time period. Then, it determines the transformation information of the hand information within the set time period based on the hand information. Next, it determines a corresponding control command based on the hand information and the transformation information. Finally, it performs control according to the control command. This method effectively determines control commands for human-computer interaction based on hand information within a set time period and the transformation information indicating the transformation of hand information. This enriches the forms of human-computer interaction. Furthermore, the control commands determined by using hand information and transformation information within a set time period are more robust, improving the accuracy of human-computer interaction control.

[0140] Example 3

[0141] Figure 3 This is a flowchart illustrating a control method provided in Embodiment 3 of this disclosure. This embodiment is a specific embodiment based on the various optional solutions in the above embodiments. In this embodiment, the corresponding control command is determined based on the hand information and the transformation information, specifically including:

[0142] Based on the hand information, the transformation information, and the duration of each hand information, the corresponding control command is determined.

[0143] For details not covered in this embodiment, please refer to the embodiments described above.

[0144] like Figure 3 As shown in Embodiment 2 of this disclosure, a control method includes the following steps:

[0145] S310: Obtain hand information within a set time period.

[0146] S320. Determine the change information of the hand information within the set time period based on the hand information.

[0147] S330. Determine the corresponding control command based on the hand information, the transformation information, and the duration of each hand information.

[0148] The duration of maintenance can be considered as the duration during which each hand information remains unchanged within a set time, such as the duration during which each gesture category remains unchanged or the duration during which each key hand position remains unchanged.

[0149] The duration of the event is not limited here; it can be determined based on the number of hand information samples collected within the set duration and the frequency of user image acquisition.

[0150] In this embodiment, there is a one-to-one correspondence between hand information, change information, duration of maintenance, and control commands. Different hand information, change information, and duration of maintenance correspond to different control commands.

[0151] In one embodiment, determining the corresponding control command based on the hand information, the transformation information, and the duration of each hand information includes:

[0152] Determine the duration of each hand information before the transformation occurs;

[0153] Based on the hand information, the transformation information, and the duration of each hand information, a preset correspondence is queried to determine the corresponding control command.

[0154] In this disclosure, the duration of each hand information can be considered as the duration during which each hand information was maintained before a change. Change information can indicate the change in hand information. This disclosure does not limit the duration of each hand information; for example, the duration during which each hand information was maintained before a change can be determined by statistically analyzing the hand information within a set time period.

[0155] If the hand information is gesture category information, then each piece of hand information can be considered as gesture category information for each type. If the hand information is hand keypoint location information, then each piece of hand keypoint information can be considered as hand keypoint information at each location.

[0156] In one example, when the hand information includes at least first hand information and second hand information, the transformation information represents the transformation from the first hand information to the second hand information. The duration of each hand information includes the duration of the first hand information and the duration of the second hand information. When determining the control command, the corresponding control command can be determined by querying a preset correspondence based on the hand information, the transformation information, and the duration of each hand information. The preset correspondence represents the correspondence between different hand information, different transformation information, and different durations of maintenance with different control commands.

[0157] For example, hand information includes gesture category information (indicating the gesture as first hand information, such as a fist, and second hand information, such as a palm) and hand key point information. Transformation information represents the gesture changing from a palm to a fist, and based on the hand key point information, it determines that the hand moves from bottom to top and then down. The duration of maintenance can represent the duration of the fist and palm, or the duration of the hand key point information at each position. The corresponding control command can be a light-on command, i.e., an instruction to turn on the lights. That is, when the hand information includes a fist and a palm, the transformation information represents the gesture changing from a fist to a palm, the hand moves from bottom to top and then down, and the duration of maintenance represents the fist lasting i seconds and the palm lasting j seconds, a preset correspondence can be consulted to determine that the corresponding control command is a light-on command. i and j are positive integers.

[0158] S340. Control is performed according to the control command.

[0159] This disclosure provides a control method in embodiment three, which specifies the operation of determining control commands. Using this method, when determining control commands, the duration of each hand information segment is further considered. This enhances the robustness of determining control commands while enriching the control methods, thus improving the human-computer interaction experience.

[0160] Example 4

[0161] Figure 4 This is a schematic diagram of a hand information recognition device provided in Embodiment 4 of this disclosure. The device is applicable to situations where hand information is to be recognized. The device can be implemented by software and / or hardware and is generally integrated into an electronic device.

[0162] like Figure 4 As shown, the device includes:

[0163] The acquisition module 41 is used to acquire the current hand image in the current frame;

[0164] The recognition module 42 is used to recognize the current hand image to obtain corresponding candidate hand information;

[0165] The determining module 43 is used to determine the target hand information based on the candidate hand information and the historical hand information, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame; wherein the candidate hand information includes candidate gesture category information and candidate hand key point information, the historical hand information includes historical gesture category information and historical hand key point information, and the target hand information includes target gesture category information and target hand key point information.

[0166] In this embodiment, the device first acquires the current hand image of the current frame through the acquisition module 41; secondly, it identifies the current hand image through the recognition module 42 to obtain corresponding candidate hand information; finally, it determines the target hand information through the determination module 43 based on the candidate hand information and historical hand information, wherein the historical hand information is the hand information determined based on historical hand images before the current frame.

[0167] This embodiment provides a hand information recognition device that determines the target hand information by using candidate hand information and historical hand information, thereby improving the accuracy of target hand information determination, increasing the success rate of human-computer interaction based on target hand information, and enhancing the user's experience of human-computer interaction based on target hand information.

[0168] In one embodiment, the candidate hand information includes candidate gesture category information and candidate hand key point information, the historical hand information includes historical gesture category information and historical hand key point information, and the target hand information includes target gesture category information and target hand key point information.

[0169] In one embodiment, the determining module 43 determines the target hand information based on the candidate hand information and historical hand information, including:

[0170] Retrieve historical gesture category information;

[0171] The gesture category information with the highest proportion among the candidate gesture category information and the historical gesture category information is determined as the target gesture category information for the current frame.

[0172] In one embodiment, the determining module 43 obtains historical gesture category information, including:

[0173] Use a sliding window to obtain historical gesture category information within a first preset time period.

[0174] In one embodiment, the recognition module 42 recognizes the current hand image to obtain corresponding candidate hand information, including:

[0175] The original probability of the gesture corresponding to the current hand image belonging to each gesture category in the original gesture category is determined by using a convolutional neural network.

[0176] Based on the original probability and category mapping relationship, determine the target probability that the gesture corresponding to the current hand image belongs to each gesture category in the target gesture category;

[0177] Determine candidate gesture category information for the current hand image, wherein the candidate gesture category information represents the gesture category corresponding to the target probability with the largest value among the target probabilities;

[0178] The target gesture category is formed by reclassifying the gesture categories included in the original gesture category based on their similarity. The number of categories included in the target gesture category is less than the number of categories included in the original gesture category. The category mapping relationship represents the mapping relationship between each gesture category in the original gesture category and each gesture category in the target gesture category.

[0179] In one embodiment, the determining module 43 determines the target hand information based on the candidate hand information and historical hand information, including:

[0180] The historical key hand information within a second preset time period is obtained through a sliding window;

[0181] The candidate hand key point information and the historical hand key point information are processed using the least squares method to obtain the target hand key point information of the set frame. The set frame is the frame collected at the time corresponding to the center of the sliding window. The target hand key point information is associated with the target gesture category information of the set frame and stored to determine the corresponding control command.

[0182] The aforementioned hand information recognition device can execute the hand information recognition method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0183] Example 5

[0184] Figure 5 This is a schematic diagram of a control device provided in Embodiment 5 of this disclosure. The device is applicable to situations where control is based on hand information. The device can be implemented by software and / or hardware and is generally integrated into an electronic device.

[0185] like Figure 5 As shown, the device includes:

[0186] The acquisition module 51 is used to acquire hand information within a set time period;

[0187] The first determining module 52 is used to determine the change information of the hand information within the set time period based on the hand information;

[0188] The second determining module 53 is used to determine the corresponding control command based on the hand information and the transformation information;

[0189] The control module 54 is used to perform control according to the control instructions.

[0190] In this embodiment, the device first acquires hand information within a set time period through the acquisition module 51; then, the first determination module 52 determines the change information of the hand information within the set time period based on the hand information; next, the second determination module 53 determines the corresponding control command based on the hand information and the change information; finally, the control module 54 performs control according to the control command.

[0191] This embodiment provides a control device that effectively determines control commands for human-computer interaction based on hand information and change information indicating changes in hand information within a set time period. This enriches the forms of human-computer interaction. Furthermore, the control commands determined by using hand information and change information within a set time period are more robust, thus improving the accuracy of human-computer interaction control.

[0192] In one embodiment, the hand information includes gesture category information and hand key point information.

[0193] In one embodiment, the second determining module 53 determines the corresponding control command based on the hand information and the transformation information, including:

[0194] Based on the hand information, the transformation information, and the duration of each hand information, the corresponding control command is determined.

[0195] In one embodiment, the second determining module 53 determines a corresponding control command based on the hand information, the transformation information, and the duration of each hand information, including:

[0196] Determine the duration of each hand information before the transformation occurs;

[0197] Based on the hand information, the transformation information, and the duration of each hand information, a preset correspondence is queried to determine the corresponding control command.

[0198] The above-described control device can execute the control method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0199] Example 6

[0200] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of this disclosure. Figure 6A schematic diagram of the structure of an electronic device 400 suitable for implementing embodiments of the present disclosure is shown. The electronic device 400 in embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The illustrated electronic device 400 is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0201] like Figure 6 As shown, the electronic device 400 may include one or more processing devices (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. The one or more processing devices 401 implement the hand information recognition method or control method provided in this disclosure. Various programs and data required for the operation of the electronic device 400 are also stored in RAM 403. The processing devices 401, ROM 402, and RAM 403 are interconnected via bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0202] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc., for storing one or more programs; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 6 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0203] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by a processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0204] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0205] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as Hypertext Transfer Protocol (HTTP), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0206] The aforementioned computer-readable medium may be included in the aforementioned electronic device 400; or it may exist independently and not assembled into the electronic device 400.

[0207] The aforementioned computer-readable medium stores one or more computer programs that, when executed by a processing device, implement the following method: acquiring a current hand image of the current frame;

[0208] Identify the current hand image to obtain corresponding candidate hand information;

[0209] Based on the candidate hand information and historical hand information, target hand information is determined. The historical hand information is hand information determined based on historical hand images prior to the current frame. The candidate hand information includes candidate gesture category information and candidate hand key point information. The historical hand information includes historical gesture category information and historical hand key point information. The target hand information includes target gesture category information and target hand key point information.

[0210] The aforementioned computer-readable medium stores one or more computer programs, which, when executed by a processing device, implement the following method:

[0211] Obtain hand information within a set time period;

[0212] Based on the hand information, determine the change information of the hand information within the set time period;

[0213] Based on the hand information and the transformation information, the corresponding control command is determined;

[0214] Control is performed according to the control instructions;

[0215] The hand information is determined based on the hand information recognition method described in this disclosure.

[0216] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device 400 to: be able to write computer program code for performing the operations of this disclosure in one or more programming languages ​​or combinations thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0217] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0218] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0219] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programming Logic Device (CPLD), and so on.

[0220] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0221] According to one or more embodiments of this disclosure, Example 1 provides a hand information recognition method, including:

[0222] Get the current hand image in the current frame;

[0223] Identify the current hand image to obtain corresponding candidate hand information;

[0224] Based on the candidate hand information and historical hand information, target hand information is determined. The historical hand information is hand information determined based on historical hand images prior to the current frame. The candidate hand information includes candidate gesture category information and candidate hand key point information. The historical hand information includes historical gesture category information and historical hand key point information. The target hand information includes target gesture category information and target hand key point information.

[0225] According to one or more embodiments of this disclosure, Example 2 describes the method described in Example 1, wherein determining the target hand information based on the candidate hand information and historical hand information includes:

[0226] Retrieve historical gesture category information;

[0227] The gesture category information with the highest proportion among the candidate gesture category information and the historical gesture category information is determined as the target gesture category information for the current frame.

[0228] According to one or more embodiments of this disclosure, Example 3, based on the method described in Example 2, includes obtaining historical gesture category information, which includes:

[0229] Use a sliding window to obtain historical gesture category information within a first preset time period.

[0230] According to one or more embodiments of this disclosure, Example 4, based on the method described in Example 1, identifies the current hand image to obtain corresponding candidate hand information, including:

[0231] The original probability of the gesture corresponding to the current hand image belonging to each gesture category in the original gesture category is determined by using a convolutional neural network.

[0232] Based on the original probability and category mapping relationship, determine the target probability that the gesture corresponding to the current hand image belongs to each gesture category in the target gesture category;

[0233] Determine candidate gesture category information for the current hand image, wherein the candidate gesture category information represents the gesture category corresponding to the target probability with the largest value among the target probabilities;

[0234] The target gesture category is formed by reclassifying the gesture categories included in the original gesture category based on their similarity. The number of categories included in the target gesture category is less than the number of categories included in the original gesture category. The category mapping relationship represents the mapping relationship between each gesture category in the original gesture category and each gesture category in the target gesture category.

[0235] According to one or more embodiments of this disclosure, Example 5 describes the method described in Example 1, wherein determining the target hand information based on the candidate hand information and historical hand information includes:

[0236] The historical key hand information within a second preset time period is obtained through a sliding window;

[0237] The candidate hand key point information and the historical hand key point information are processed using the least squares method to obtain the target hand key point information of the set frame. The set frame is the frame collected at the time corresponding to the center of the sliding window. The target hand key point information is associated with the target gesture category information of the set frame and stored to determine the corresponding control command.

[0238] According to one or more embodiments of this disclosure, Example 6 provides a control method, including:

[0239] Obtain hand information within a set time period;

[0240] Based on the hand information, determine the change information of the hand information within the set time period;

[0241] Based on the hand information and the transformation information, the corresponding control command is determined;

[0242] Control is performed according to the control instructions;

[0243] The hand information is determined based on any of the methods described in Examples 1-6.

[0244] According to one or more embodiments of this disclosure, Example 7, based on the method described in Example 6, determines a corresponding control command based on the hand information and the transformation information, including:

[0245] Based on the hand information, the transformation information, and the duration of each hand information, the corresponding control command is determined.

[0246] According to one or more embodiments of this disclosure, Example 8 describes the method described in Example 7, wherein determining the corresponding control command based on the hand information, the transformation information, and the duration of each hand information includes:

[0247] Determine the duration of each hand information before the transformation occurs;

[0248] Based on the hand information, the transformation information, and the duration of each hand information, a preset correspondence is queried to determine the corresponding control command.

[0249] According to one or more embodiments of this disclosure, Example 9 provides a hand information recognition device, including:

[0250] The acquisition module is used to acquire the current hand image in the current frame;

[0251] The recognition module is used to recognize the current hand image to obtain corresponding candidate hand information;

[0252] The determination module is used to determine the target hand information based on the candidate hand information and the historical hand information, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame;

[0253] The candidate hand information includes candidate gesture category information and candidate hand key point information; the historical hand information includes historical gesture category information and historical hand key point information; and the target hand information includes target gesture category information and target hand key point information.

[0254] According to one or more embodiments of this disclosure, Example 10 provides a control device, including:

[0255] The acquisition module is used to acquire hand information within a set time period;

[0256] The first determining module is used to determine the change information of the hand information within the set time period based on the hand information;

[0257] The second determining module is used to determine the corresponding control command based on the hand information and the transformation information;

[0258] The control module is used to control according to the control instructions.

[0259] According to one or more embodiments of this disclosure, Example 11 provides an electronic device, including:

[0260] One or more processing devices;

[0261] Storage device for storing one or more programs;

[0262] When the one or more programs are executed by the one or more processing devices, the one or more processing devices perform the method as described in any of Examples 1-8.

[0263] According to one or more embodiments of the present disclosure, Example 12 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the method described in any of Examples 1-8.

[0264] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0265] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0266] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for recognizing hand information, characterized in that, include: Get the current hand image in the current frame; Identify the current hand image to obtain corresponding candidate hand information; Based on the candidate hand information and historical hand information, target hand information is determined, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame; wherein the candidate hand information includes candidate gesture category information and candidate hand key point information, the historical hand information includes historical gesture category information and historical hand key point information, and the target hand information includes target gesture category information and target hand key point information; The historical gesture category information refers to candidate gesture category information or target gesture category information of historical hand images determined based on historical hand images at historical moments.

2. The method according to claim 1, characterized in that, The step of determining the target hand information based on the candidate hand information and historical hand information includes: Retrieve historical gesture category information; The gesture category information with the highest proportion among the candidate gesture category information and the historical gesture category information is determined as the target gesture category information for the current frame.

3. The method according to claim 2, characterized in that, The acquisition of historical gesture category information includes: Use a sliding window to obtain historical gesture category information within a first preset time period.

4. The method according to claim 1, characterized in that, Identifying the current hand image to obtain corresponding candidate hand information includes: The original probability of the gesture corresponding to the current hand image belonging to each gesture category in the original gesture category is determined by using a convolutional neural network. Based on the original probability and category mapping relationship, determine the target probability that the gesture corresponding to the current hand image belongs to each gesture category in the target gesture category; Determine candidate gesture category information for the current hand image, wherein the candidate gesture category information represents the gesture category corresponding to the target probability with the largest value among the target probabilities; The target gesture category is formed by reclassifying the gesture categories included in the original gesture category based on their similarity. The number of categories included in the target gesture category is less than the number of categories included in the original gesture category. The category mapping relationship represents the mapping relationship between each gesture category in the original gesture category and each gesture category in the target gesture category.

5. The method according to claim 1, characterized in that, The step of determining the target hand information based on the candidate hand information and historical hand information includes: The historical key hand information within a second preset time period is obtained through a sliding window; The candidate hand key point information and the historical hand key point information are processed using the least squares method to obtain the target hand key point information of the set frame. The set frame is the frame collected at the time corresponding to the center of the sliding window. The target hand key point information is associated with the target gesture category information of the set frame and stored to determine the corresponding control command.

6. A control method, characterized in that, include: Obtain hand information within a set time period; Based on the hand information, determine the change information of the hand information within the set time period; Based on the hand information and the transformation information, the corresponding control command is determined; Control is performed according to the control instructions; The hand information is determined based on the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that, Based on the hand information and the transformation information, the corresponding control command is determined, including: Based on the hand information, the transformation information, and the duration of each hand information, the corresponding control command is determined.

8. The method according to claim 7, characterized in that, The step of determining the corresponding control command based on the hand information, the transformation information, and the duration of each hand information includes: Determine the duration of each hand information before the transformation occurs; Based on the hand information, the transformation information, and the duration of each hand information, a preset correspondence is queried to determine the corresponding control command.

9. A hand information recognition device, characterized in that, include: The acquisition module is used to acquire the current hand image in the current frame; The recognition module is used to recognize the current hand image to obtain corresponding candidate hand information; The determination module is used to determine the target hand information based on the candidate hand information and the historical hand information, wherein the historical hand information is hand information determined based on historical hand images prior to the current frame; The candidate hand information includes candidate gesture category information and candidate hand key point information; the historical hand information includes historical gesture category information and historical hand key point information; and the target hand information includes target gesture category information and target hand key point information. The historical gesture category information refers to candidate gesture category information or target gesture category information of historical hand images determined based on historical hand images at historical moments.

10. A control device, characterized in that, include: The acquisition module is used to acquire hand information within a set time period; The first determining module is used to determine the change information of the hand information within the set time period based on the hand information; The second determining module is used to determine the corresponding control command based on the hand information and the transformation information; The control module is used to control according to the control instructions; The hand information is determined based on the method described in any one of claims 1-5.

11. An electronic device, characterized in that, include: One or more processing devices; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processing devices, the one or more processing devices perform the method as described in any one of claims 1-8.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the method as described in any one of claims 1-8.

Images