Gesture recognition methods, electronic devices and computer-readable storage media

Abstract

This application discloses a gesture recognition method, an electronic device, and a computer-readable storage medium. The method includes: obtaining a test image corresponding to a hand to be tested; inputting the test image into a prediction model, so that the prediction model determines the gesture recognition result corresponding to the hand to be tested based on the test image and reference gesture information; wherein, the reference gesture information is determined based on the key point coordinates corresponding to a reference hand of a preset gesture, and the prediction model is obtained after pre-training based on the reference gesture information and test images of multiple training gestures corresponding to the test hand; the hand to be tested belongs to the same preset part as the reference hand in the corresponding test image, and the test hand belongs to the same preset part as the reference hand in the test image. This approach can reduce the difficulty of model training and improve the accuracy of gesture recognition.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a gesture recognition method, electronic device, and computer-readable storage medium. Background Technology

[0002] As the field of computer vision becomes increasingly specialized, gesture recognition is receiving more and more attention. Gestures can effectively and conveniently convey information, and therefore, gesture recognition has been widely used in scenarios such as those without microphones, where sound recording is difficult, or for deaf and mute individuals. Current technologies typically utilize massive amounts of training data for unconstrained training to obtain gesture recognition results. This large amount of training data makes training difficult, and the recognition performance of models obtained through unconstrained training is often poor. Therefore, how to reduce the difficulty of model training and improve the accuracy of gesture recognition has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a gesture recognition method, electronic device, and computer-readable storage medium that can reduce the difficulty of model training and improve the accuracy of gesture recognition.

[0004] To address the aforementioned technical problems, this application provides a gesture recognition method, comprising: obtaining a test image corresponding to a hand to be tested; inputting the test image into a prediction model, so that the prediction model determines the gesture recognition result corresponding to the hand to be tested based on the test image and reference gesture information; wherein, the reference gesture information is determined based on the key point coordinates corresponding to a reference hand of a preset gesture, the prediction model is obtained after pre-training based on the reference gesture information and test images of multiple training gestures corresponding to the test hand, the hand to be tested belongs to the same preset part as the reference hand in the corresponding test image, and the test hand belongs to the same preset part as the reference hand in the test image.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To solve the above-mentioned technical problems, a third aspect of this application provides a computer storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The above scheme, after obtaining the image of the hand to be tested, inputs the image into the prediction model. The prediction model then predicts the gesture corresponding to the hand based on the image and reference gesture information, obtaining the gesture recognition result. This allows for obtaining a prediction result simply by inputting the image into the model. The reference gesture information is determined based on the keypoint coordinates of a reference hand corresponding to a preset gesture. The prediction model is pre-trained using the reference gesture information and test images of various training gestures corresponding to the test hand. The reference gesture information constrains and guides the training of the prediction model. Since the test hand belongs to the same preset location as the reference hand in the test image, training only requires images corresponding to one location during the training phase, reducing the amount of training data, lowering the training difficulty, and resulting in a prediction model with better accuracy, while also improving training efficiency. In the application phase, the hand to be tested belongs to the same preset location as the reference hand in the corresponding image, ensuring that the image meets the requirements of the prediction model and improving the accuracy of gesture recognition. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0009] Figure 1 This is a flowchart illustrating one embodiment of the gesture recognition method of this application;

[0010] Figure 2 This is a flowchart illustrating another embodiment of the gesture recognition method of this application;

[0011] Figure 3 yes Figure 2 A schematic diagram of an application scenario corresponding to step S205 in the middle step;

[0012] Figure 4 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0013] Figure 5 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0015] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0016] The gesture recognition method provided in this application can be used in various application scenarios that require gesture recognition. The execution subject of the gesture recognition method in this application can be a processor or a server. When the processor or server receives the image to be tested, it can perform gesture recognition on the hand to be tested in the image to be tested.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the gesture recognition method of this application. The method includes:

[0018] S101: Obtain the image of the hand to be tested.

[0019] Specifically, the image corresponding to the hand to be tested is acquired and the part to which the hand to be tested belongs is determined, thereby determining the image to be tested corresponding to the hand to be tested.

[0020] Furthermore, the target to be tested includes a first preset part and a second preset part, and the first preset part and the second preset part are mirror images of each other. The hand to be tested belongs to either the first preset part or the second preset part. In addition, the reference hand used in subsequent steps also belongs to either the first preset part or the second preset part, as detailed in step S102. It can be understood that when the first preset part is the left hand, the second preset part is the right hand, and when the first preset part is the right hand, the second preset part is the left hand.

[0021] In one application method, an initial image corresponding to the target to be tested is obtained. A sub-image region corresponding to the hand to be tested is extracted from the initial image. The part to which the hand to be tested belongs is determined based on the sub-image region. The part to which the hand to be tested belongs is compared with the part to which a reference hand belongs, thereby determining the image to be tested corresponding to the hand to be tested, so that the hand to be tested and the reference hand displayed on the image to be tested belong to the same preset part.

[0022] In another application, the image corresponding to the hand to be tested is input into the attribute detection model to obtain the part to which the image to be tested belongs. When the part to which the image to be tested belongs is consistent with the part to which the reference hand belongs, the image to be tested is used as the image to be tested. When the part to which the image to be tested belongs is inconsistent with the part to which the reference hand belongs, the image to be tested is mirrored and used as the image to be tested.

[0023] In one application scenario, the reference hand is the left hand. After obtaining the initial image corresponding to the target, the sub-image region corresponding to the hand is extracted from the initial image. The sub-image region is then detected to determine the part of the hand to be tested displayed in the sub-image region. If the hand to be tested belongs to the left hand in the sub-image region as the reference hand, the sub-image region corresponding to the hand to be tested is used as the test image. If the hand to be tested belongs to the right hand in the sub-image region, the sub-image region corresponding to the hand to be tested is mirrored, and the mirrored sub-image region is used as the test image. This ensures that the features displayed by the hand to be tested in the test image belong to the left hand as the reference hand, so that the hand to be tested and the reference hand belong to the same preset part in the corresponding test image, thereby ensuring that the test image meets the requirements of the prediction model.

[0024] S102: Input the image to be tested into the prediction model so that the prediction model can determine the gesture recognition result corresponding to the hand to be tested based on the image to be tested and the reference gesture information. The reference gesture information is determined based on the key point coordinates corresponding to the reference hand of the preset gesture. The prediction model is obtained after pre-training based on the reference gesture information and test images of various training gestures corresponding to the test hand.

[0025] Specifically, the hand under test and the reference hand belong to the same preset location in the corresponding test image, and the test hand and the reference hand belong to the same preset location in the test image. In other words, the test hand and the hand under test both belong to the same preset location in the image displayed in the test image.

[0026] Furthermore, the reference hand belongs to the first preset part or the second preset part, and the reference hand corresponds to reference gesture information. The reference gesture information is determined based on the key point coordinates corresponding to the reference hand of the preset gesture.

[0027] In one application method, a reference hand corresponds to a preset gesture, and key points on the reference hand of the preset gesture correspond to labeled data. The coordinates of the key points obtained in three-dimensional space based on the labeled data are the reference gesture information.

[0028] Furthermore, the prediction model is pre-trained based on reference gesture information and test images of various training gestures corresponding to the test hand. During training, the reference gesture information plays a constraining and guiding role in the training of the prediction model. The prediction model can learn the offset value of the training object in the training image relative to the reference hand based on the reference gesture information, thereby reducing the difficulty of training and learning. In addition, the test hand belongs to the same preset part as the reference hand in the test image, so only the images corresponding to one part need to be trained during the training phase, reducing the amount of training data, reducing the training difficulty, obtaining a prediction model with better prediction accuracy, and improving training efficiency.

[0029] Understandably, during the application phase, the hand to be tested must belong to the same preset location as the reference hand in the corresponding image to meet the requirements of the prediction model. Therefore, when the actual location of the hand to be tested is opposite to that of the reference hand, the sub-image region corresponding to the hand to be tested needs to be mirrored to obtain the image to be tested, so that the hand to be tested belongs to the same preset location as the reference hand in the image to ensure that the image to be tested meets the requirements of the prediction model and improves the accuracy of gesture recognition.

[0030] Specifically, in the application phase, the image to be tested is input into the trained prediction model so that the prediction model can determine the gesture recognition result corresponding to the hand to be tested based on the image to be tested and the reference gesture information, thereby realizing that the gesture recognition result can be obtained simply by inputting the image to be tested into the prediction model.

[0031] In one application, the prediction model estimates the key point coordinates of the hand under test based on the offset value of the hand under test relative to the reference hand in the image under test. Then, the key point coordinates of the hand under test are analyzed to obtain the gesture recognition result of the hand under test.

[0032] In another application, the prediction model estimates the coordinates of key points of the hand based on the position of the hand in the image to be tested. The coordinates of the key points of the hand to be tested are then compared with the key point of the mouth in the reference gesture to obtain the gesture recognition result of the hand to be tested.

[0033] In one application scenario, the reference hand includes a hand with a preset gesture, which includes the state of the hand when it is open at a first preset part. The gesture recognition result includes at least one preset type of gesture. The first preset part is either the left or right hand. Gestures, as one of the most primitive forms of human body language, can convey information simply, conveniently, and effectively. Accurate and timely gesture recognition is particularly important in scenarios without microphones, where sound recording is difficult, or for deaf or mute individuals.

[0034] In a specific application scenario, the hand to be tested is the hand itself, the reference hand is the left hand, and the preset gesture is the left hand in an open state. The gesture recognition result is the gesture type determined based on hand changes. During the training phase, the prediction model is trained as a whole using test images of various training gestures and reference gesture information, enabling the trained prediction model to recognize the left hand in the test image. In the application phase, the test image corresponding to the hand to be tested is input into the prediction model, and the gesture type corresponding to the hand to be tested is directly obtained from the input to the output. It is understandable that in other application scenarios, the reference hand can also be the right hand; therefore, the hand to be tested in the test image and the test hand in the test image must match the preset part corresponding to the reference hand.

[0035] The above scheme, after obtaining the image of the hand to be tested, inputs the image into the prediction model. The prediction model then predicts the gesture corresponding to the hand based on the image and reference gesture information, obtaining the gesture recognition result. This allows for obtaining a prediction result simply by inputting the image into the model. The reference gesture information is determined based on the keypoint coordinates of a reference hand corresponding to a preset gesture. The prediction model is pre-trained using the reference gesture information and test images of various training gestures corresponding to the test hand. The reference gesture information constrains and guides the training of the prediction model. Since the test hand belongs to the same preset location as the reference hand in the test image, training only requires images corresponding to one location during the training phase, reducing the amount of training data, lowering the training difficulty, and resulting in a prediction model with better accuracy, while also improving training efficiency. In the application phase, the hand to be tested belongs to the same preset location as the reference hand in the corresponding image, ensuring that the image meets the requirements of the prediction model and improving the accuracy of gesture recognition.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the gesture recognition method of this application, the method comprising:

[0037] S201: Obtain the initial image corresponding to the target to be tested, and extract the sub-image region corresponding to the hand to be tested from the initial image.

[0038] Specifically, an initial image of the target to be tested is obtained, and the sub-image region corresponding to the hand to be tested is extracted from the initial image.

[0039] In one application, an initial image of the target to be tested is input into a sub-image extraction model, which then outputs a sub-image region corresponding to the hand to be tested. The sub-image extraction model is pre-trained based on the image of the target to be tested and is capable of identifying the hand within the target and extracting the corresponding sub-image region.

[0040] S202: Determine whether the hand to be tested in the sub-image area belongs to the same preset part as the reference hand.

[0041] Specifically, the hand to be tested belongs to either the first preset part or the second preset part, and the first preset part and the second preset part are mirror images of each other, with the reference hand belonging to the first preset part.

[0042] Furthermore, it is determined whether the hand to be tested belongs to the same preset part as the reference hand within the sub-image area, that is, whether the hand to be tested belongs to the same first preset part as the reference hand in the image displayed within the sub-image area.

[0043] In one application method, the step of determining whether the hand to be tested in a sub-image region belongs to the same preset part as the reference hand includes: inputting the sub-image region into an attribute detection model to obtain the attribute detection result corresponding to the sub-image region; the attribute detection result includes the part to which the target to be tested belongs in the sub-image region; comparing the attribute detection result with the first preset part to determine whether the hand to be tested in the sub-image region belongs to the same preset part as the reference hand; wherein, the attribute detection model is obtained after pre-training based on the test image and the mirrored test image.

[0044] Specifically, the attribute detection model is used to determine whether the hand to be tested in the sub-image region belongs to a first preset part or a second preset part. The attribute detection model is obtained by pre-training based on the test image and the mirrored test image. Therefore, the attribute detection model and the prediction model can be trained using the same training data, thereby reducing the preparation of training data. Furthermore, by comparing the training data and the mirrored training data, it is ensured that the data used to train the attribute detection model contains data showing the hand to be tested in the image belonging to both the first preset part and the second preset part, and the number of such data is the same, so as to improve the recognition accuracy of the attribute detection model.

[0045] In a specific application scenario, the attribute detection model includes a convolutional layer, a normalization layer, an activation layer, and a pooling layer, and the attribute detection model has a pyramid structure. The last layer outputs a probability. When the probability is greater than a preset threshold, it is the first preset part; otherwise, it is the second preset part. The probability corresponds to a value of 0-1, and the preset threshold can be 0.5. In other specific application scenarios, the probability and the preset threshold can also be customized to other values. This application does not impose specific restrictions on this.

[0046] S203: Use the sub-image region as the image to be tested corresponding to the hand to be tested.

[0047] Specifically, when the hand to be tested and the reference hand in the sub-image area belong to the same preset part, the sub-image area corresponding to the hand to be tested is taken as the image to be tested.

[0048] S204: Mirror the sub-image region and use the mirrored sub-image region as the image to be tested corresponding to the hand to be tested.

[0049] Specifically, when the hand to be tested and the reference hand in the sub-image area do not belong to the same preset part, the sub-image area corresponding to the hand to be tested is mirrored, and the mirrored sub-image area is used as the image to be tested corresponding to the hand to be tested, so that the image displayed by the target to be tested in the image to be tested belongs to the same preset part as the reference hand, thereby meeting the requirements of the prediction model.

[0050] S205: Input the image to be tested into the prediction model so that the prediction model can determine the gesture recognition result corresponding to the hand to be tested based on the image to be tested and the reference gesture information.

[0051] Specifically, the reference gesture information is determined based on the key point coordinates corresponding to the reference hand of the preset gesture. The prediction model is obtained after pre-training based on the reference gesture information and test images of various training gestures corresponding to the test hand. The test hand belongs to the same preset part as the reference hand in the corresponding test image, and the test hand belongs to the same preset part as the reference hand in the test image.

[0052] In one application method, the reference hand includes a hand with a preset gesture, the preset gesture includes the state of the hand when the first preset part is open, and the gesture recognition result includes at least one preset type of gesture. The first preset part is either the left or right hand.

[0053] Specifically, the hand to be tested is the hand of the target being tested, and the reference hand is the hand with a preset gesture. The preset gesture is the state of the left or right hand when it is open. When the first preset part is the left hand, the second preset part is the right hand, and vice versa. The preset gesture types include, but are not limited to, command gestures, alarm gestures, distress gestures, and confirmation gestures. Among these, gestures, as one of the most primitive forms of human body language, can convey information simply, conveniently, and effectively, making them suitable for scenarios such as traffic control, assistance for the deaf and mute, and information confirmation for the general population.

[0054] Furthermore, the reference gesture information is obtained based on the following steps: obtaining a preset number of key points on a first preset part of a plurality of reference targets when the hand is open; marking the three-dimensional coordinates of the key points and performing normalization processing to obtain normalized three-dimensional coordinates; at least some key points are related to the joints of the hand of the reference targets; using a clustering algorithm to cluster the normalized three-dimensional coordinates corresponding to each key point on the plurality of reference targets to obtain a preset number of key point coordinates as reference gesture information; wherein, the key point coordinates are normalized three-dimensional coordinates after clustering.

[0055] Specifically, the three-dimensional coordinates of key points at preset positions on multiple reference targets when the hand is open are marked in a preset number of locations in three-dimensional space. At least some of the key points are related to the joints of the hand of the reference targets. The three-dimensional coordinates of the key points corresponding to each reference target are normalized to obtain normalized three-dimensional coordinates. The K-means clustering algorithm is used to determine the coordinates of the key points corresponding to the preset number of locations. These key point coordinates are normalized three-dimensional coordinates after clustering. Each location corresponds to a normalized three-dimensional coordinate after clustering. The normalized three-dimensional coordinates after clustering are used as reference gesture information.

[0056] In a specific application scenario, the preset gesture is a naturally open left hand. One keypoint corresponds to the wrist, and one keypoint corresponds to the fingertips and each joint of the fingers excluding the thumb. Another keypoint corresponds to the fingertip and joint of the thumb, as well as the tips of the two joints. Therefore, each finger corresponds to 4 keypoints, plus one keypoint at the wrist, for a total of 21 keypoints. That is, the preset value is 21. The keypoints corresponding to the naturally open left hand of multiple reference targets are determined. The 3D coordinates of the keypoints are marked in 3D space and normalized to obtain the preset value and normalized 3D coordinates for each reference target. A k-means clustering algorithm is used to create 21 clusters. The keypoint coordinates corresponding to each of the 21 keypoints in the naturally open left hand posture are obtained as reference gesture information, which serves as a constraint and guide during the training of the prediction model. In other application scenarios, the preset gesture can also be a naturally open right hand. The process of obtaining reference gesture information is the same as in the above application scenario, and this application does not impose specific restrictions on this.

[0057] In one application scenario, please refer to Figure 3 , Figure 3 yes Figure 2 The schematic diagram of an application scenario corresponding to step S205 shows that the prediction model 30 includes a cascaded image convolutional network 300, a fusion network 301, and a graph convolutional network 302. The gesture recognition result is obtained through the following steps: The image to be tested is input into the image convolutional network 300, so that the image convolutional network 300 obtains the coordinate offset value of the hand to be tested relative to the reference gesture information based on the image to be tested and the reference gesture information; the coordinate offset value and the reference gesture information are input into the fusion network 301, so that the fusion network 301 fuses the coordinate offset value into the reference gesture information to obtain the key point prediction result corresponding to the hand to be tested; the key point prediction result is input into the graph convolutional network 302, so that the graph convolutional network 302 determines the gesture recognition result corresponding to the hand to be tested based on the key point prediction result.

[0058] Specifically, the image convolutional network 300 is used to determine the coordinate offset value of the hand to be tested relative to the reference gesture information in the test image, that is, the difference between the hand to be tested and the reference hand. The fusion network 301 fuses the coordinate offset value into the reference gesture information to obtain the key point prediction result corresponding to the hand to be tested. Then, the graph convolutional network 302 determines the probability of the key point prediction result belonging to different gesture types based on the key point prediction result, and outputs the gesture type with the highest probability as the gesture recognition result. Here, the test image corresponding to the hand is input from the input end, and the prediction model 30 can output the gesture prediction result. The image convolutional network 300, the fusion network 301 and the graph convolutional network 302 in the prediction model 30 are trained together, rather than in stages, thereby improving training efficiency and the convenience of obtaining gesture types.

[0059] In one application scenario, the reference gesture information includes normalized keypoint coordinates, and the keypoint coordinates are three-dimensional coordinates. The step of obtaining the coordinate offset values ​​of the hand to be tested relative to the reference gesture information based on the image to be tested and the reference gesture information includes: obtaining the coordinate offset values ​​between each keypoint corresponding to the hand to be tested and the matching keypoints in the reference gesture information based on the image to be tested and the reference gesture information; the coordinate offset values ​​include normalized three-dimensional coordinate offsets. The step of fusing the coordinate offset values ​​to the reference gesture information to obtain the keypoint prediction results corresponding to the hand to be tested includes: fusing the normalized three-dimensional coordinate offsets to the three-dimensional coordinates corresponding to the keypoint coordinates to obtain the keypoint prediction results corresponding to the hand to be tested; the keypoint prediction results are normalized three-dimensional coordinates.

[0060] Specifically, the normalized three-dimensional coordinates impose biological structural constraints on the prediction model 30, so that the image convolutional network 300 in the prediction model 30 only needs to learn the difference between the normalized three-dimensional coordinates corresponding to the current gesture and the reference gesture information, which reduces the learning difficulty of the network, speeds up the convergence speed, and improves the accuracy of the prediction model 30.

[0061] Furthermore, the image convolutional network 300 compares the image to be tested with the reference gesture information to determine the coordinate offset values ​​between each key point corresponding to the hand to be tested and the key points matching in the reference gesture information. The coordinate offset values ​​are the normalized three-dimensional coordinate offsets. In addition, the fusion network 301 fuses the normalized three-dimensional coordinate offsets into the three-dimensional coordinates corresponding to the key point coordinates. The obtained key point prediction results are also normalized three-dimensional coordinates. Therefore, all scales in the process of obtaining key point prediction results are uniform, thus determining that the key point prediction results corresponding to the hand to be tested are independent of camera parameters. As a result, the prediction model 30 can be applied to cameras with various parameters, improving adaptability.

[0062] In a specific application scenario, the reference gesture information is the normalized 3D coordinates of 21 key points corresponding to the left hand in a naturally open posture. The image convolutional network 300 consists of several convolutional layers, normalization layers, pooling layers, activation layers, and a fully connected layer, forming a pyramid structure. When the hand to be tested is input, the image convolutional network 300 outputs the normalized 3D coordinate offset. At this time, the normalized 3D coordinate offset is the difference between the 3D coordinates of the 21 key points corresponding to the hand to be tested and the corresponding key points in the reference gesture information, resulting in a 63-dimensional vector. The fusion network 301 fuses the normalized 3D coordinate offset into the 3D coordinates corresponding to each key point of the reference hand to obtain the key point prediction result. The key point prediction result is input into the fusion network 301. The graph convolutional network 302 consists of several graph convolutional layers and activation layers. The key point prediction result is input into the graph convolutional network 302 in the form of a joint topology graph. After passing through the activation layer, the gesture type with the highest probability is output.

[0063] Optionally, after the step of inputting the image to be tested into the prediction model 30 so that the prediction model 30 determines the gesture recognition result corresponding to the hand to be tested based on the image to be tested and the reference gesture information, the method further includes: determining the warning evaluation result corresponding to the gesture recognition result based on the gesture recognition result and the preset warning gesture; and determining whether to trigger the warning signal based on the warning evaluation result.

[0064] Specifically, the gesture recognition result is compared with a preset warning gesture. Based on the similarity between the two gestures, a warning assessment result is determined. If the similarity exceeds a similarity threshold, a warning is required; otherwise, no warning is needed. Based on the assessment result, it is then determined whether to trigger a warning signal, ensuring timely issuance of alerts for scenarios such as requests for help, commands, or alarms.

[0065] In a specific application scenario, the preset warning gestures include at least one of the following: help gesture, traffic control gesture, and prohibition of operation alarm gesture. When the similarity between the gesture recognition result and any preset warning gesture exceeds the similarity threshold, a warning assessment result is generated indicating that a warning is required so as to trigger the warning signal in a timely manner.

[0066] In this embodiment, a preset number of clustered normalized 3D coordinates are used as reference gesture information. The reference gesture information imposes biological structural constraints on the prediction model 30, so that the image convolutional network 300 in the prediction model 30 only needs to learn the difference between the normalized 3D coordinates corresponding to the current gesture and the reference gesture information. This reduces the learning difficulty of the network, speeds up the convergence, and improves the accuracy of the prediction model 30. The prediction model 30 includes an image convolutional network 300, a fusion network 301, and a graph convolutional network 302. The fusion network 301 fuses the normalized 3D coordinate offsets into the 3D coordinates corresponding to the keypoint coordinates. The obtained keypoint prediction result is also a normalized 3D coordinate. Therefore, all scales in the process of obtaining the keypoint prediction result are uniform, thereby determining that the keypoint prediction result corresponding to the hand to be tested is independent of the camera parameters. As a result, the prediction model 30 can be applied to cameras with various parameters, improving adaptability.

[0067] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 40 includes a memory 401 and a processor 402 coupled to each other. The memory 401 stores program data (not shown in the figure), and the processor 402 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0068] Please see Figure 5 , Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, it implements the method in any of the above embodiments. For related descriptions, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0069] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0071] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] It should be noted that if the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, the personal information processing rules are clearly informed through signs / information, and authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0073] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A gesture recognition method, characterized in that, The method includes: Obtain the image of the hand to be tested; The process involves inputting the image to be tested into a prediction model, so that the prediction model determines the gesture recognition result corresponding to the hand to be tested based on the image to be tested and reference gesture information. Specifically, this includes: inputting the image to be tested into an image convolutional network, so that the image convolutional network obtains the coordinate offset value of the hand to be tested relative to the reference gesture information based on the image to be tested and the reference gesture information; inputting the coordinate offset value and the reference gesture information into a fusion network, so that the fusion network fuses the coordinate offset value into the reference gesture information to obtain a keypoint prediction result corresponding to the hand to be tested; and inputting the keypoint prediction result into a graph convolutional network, so that the graph convolutional network determines the gesture recognition result corresponding to the hand to be tested based on the keypoint prediction result. The reference gesture information is determined based on the key point coordinates corresponding to the reference hand of the preset gesture. The prediction model includes a cascaded image convolutional network, a fusion network, and a graph convolutional network. The prediction model is obtained after pre-training based on the reference gesture information and test images of various training gestures corresponding to the test hand. The hand to be tested belongs to the same preset part as the reference hand in the corresponding test image, and the test hand belongs to the same preset part as the reference hand in the test image.

2. The gesture recognition method according to claim 1, characterized in that, The reference hand belongs to a first preset region, and obtaining the image of the hand to be tested includes: Obtain an initial image corresponding to the target to be tested, and extract the sub-image region corresponding to the hand to be tested from the initial image; Determine whether the hand to be tested within the sub-image area belongs to the same preset part as the reference hand; If so, the sub-image region is taken as the image to be tested corresponding to the hand to be tested; Otherwise, the sub-image region is mirrored, and the mirrored sub-image region is used as the image to be tested corresponding to the hand to be tested.

3. The gesture recognition method according to claim 2, characterized in that, The step of determining whether the hand to be tested within the sub-image area belongs to the same preset part as the reference hand includes: The sub-image region is input into the attribute detection model to obtain the attribute detection result corresponding to the sub-image region; the attribute detection result includes the part of the target to be tested within the sub-image region; By comparing the attribute detection results with the first preset part, it is determined whether the hand to be tested in the sub-image area belongs to the same preset part as the reference hand; The attribute detection model is obtained by pre-training based on the test image and the mirrored test image.

4. The gesture recognition method according to any one of claims 1-3, characterized in that, The reference hand includes the hand of the preset gesture, the preset gesture includes the state of the hand when the first preset part is open, and the gesture recognition result includes at least one preset type of gesture; the first preset part is the left hand / right hand.

5. The gesture recognition method according to claim 4, characterized in that, The reference gesture information is obtained based on the following steps: A preset number of key points are obtained on multiple reference targets when the hand of the first preset part is open. The three-dimensional coordinates of the key points are marked and normalized to obtain normalized three-dimensional coordinates. At least some of the key points are related to the joints of the hand of the reference target. Clustering algorithms are used to cluster the normalized 3D coordinates of each key point on multiple reference targets to obtain a preset number of key point coordinates, which are used as the reference gesture information; wherein, the key point coordinates are the normalized 3D coordinates after clustering.

6. The gesture recognition method according to claim 1, characterized in that, The reference gesture information includes normalized key point coordinates, and the key point coordinates are three-dimensional coordinates; The step of obtaining the coordinate offset value of the hand to be tested relative to the reference gesture information based on the image to be tested and the reference gesture information includes: Based on the image to be tested and the reference gesture information, the coordinate offset values ​​between each key point corresponding to the hand to be tested and the key points matching in the reference gesture information are obtained; the coordinate offset values ​​include normalized three-dimensional coordinate offsets. The step of fusing the coordinate offset value into the reference gesture information to obtain the key point prediction result corresponding to the hand to be tested includes: The normalized three-dimensional coordinate offset is fused into the three-dimensional coordinates corresponding to the key point coordinates to obtain the key point prediction result corresponding to the hand to be tested; the key point prediction result is the normalized three-dimensional coordinates.

7. The gesture recognition method according to claim 1, characterized in that, After inputting the image to be tested into the prediction model, so that the prediction model determines the gesture recognition result corresponding to the hand to be tested based on the image to be tested and the reference gesture information, the method further includes: Based on the gesture recognition result and the preset warning gesture, determine the warning assessment result corresponding to the gesture recognition result; Based on the warning assessment results, determine whether to trigger a warning signal.

8. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, the method as described in any one of claims 1-7 is implemented.

Images