A pen-holding posture recognition model training method, a recognition method, and related devices thereof

Abstract

This application discloses a training method, recognition method, and related apparatus for a pen grip posture recognition model, relating to the field of image recognition. It includes: acquiring an image training set; acquiring training image pairs based on the image training set; acquiring high-level semantic features of two pen grip posture images respectively; acquiring a first feature distance between the training image pairs based on the high-level semantic features of the two pen grip posture images; acquiring a first optimization loss based on the first feature distance; and updating the preset model based on the first optimization loss if the first optimization loss does not meet a first threshold. The pen grip posture recognition model according to the embodiments of this application can judge the correctness of the pen grip posture in another pen grip posture image to be recognized based on a correct pen grip posture image. Since the first convolutional neural network model and the second convolutional neural network model included in the pen grip posture recognition model share weights, it saves feature extraction and data processing time.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to a pen-holding posture recognition model training method, recognition method, and related apparatus. Background Technology

[0002] Currently, mainstream deep learning convolutional neural networks, such as VGG, ResNet, and DenseNet, are mostly used for tasks like image classification and recognition. However, they are not suitable for comparing the pen-holding posture in a given video with a standard pen-holding posture video, and for indicating whether the pen-holding posture in the given video is incorrect. There are at least three reasons for this. First, existing deep learning convolutional neural networks can only obtain the feature distance of the pen-holding posture image to be identified. Based on this feature distance, they can determine whether the pen-holding posture to be identified is incorrect. They cannot determine whether another pen-holding posture image to be identified is correct based on a correct pen-holding posture image. Second, even if existing deep learning convolutional neural networks are used to extract features from a correct pen-holding posture image and a pen-holding posture image to be identified in turn to obtain the feature distance and determine whether the pen-holding posture image to be identified is correct, the process is cumbersome and time-consuming. Third, when comparing pen-holding postures based on captured videos, a large number of pen-holding posture images need to be processed. This also means that when using existing deep learning convolutional neural networks to extract features from multiple correct pen-holding posture images and multiple pen-holding posture images to be identified in turn, a large amount of data needs to be stored and retrieved, which is time-consuming. Summary of the Invention

[0003] The purpose of this application is to provide a training method, recognition method and related device for a pen grip posture recognition model, so as to solve the technical problem that existing neural network models are not suitable for comparison based on standard pen grip posture videos under the premise of shooting videos of given pen grip postures.

[0004] To achieve the above objectives, this application provides the following technical solution:

[0005] In a first aspect, embodiments of this application provide a method for training a pen-holding posture recognition model, the method comprising:

[0006] Obtain an image training set, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures;

[0007] Based on the image training group, training image pairs are obtained, each training image pair including two pen-holding posture images, and at least one pen-holding posture image represents a correct pen-holding posture.

[0008] High-level semantic features are obtained from two pen-holding posture images. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of each key point in the hand region of the pen-holding posture image. The preset model includes at least a first convolutional neural network model and a second convolutional neural network model. The two convolutional neural network models share weight parameters. Each convolutional neural network model is used to extract the high-level semantic features of the input image.

[0009] Based on the high-level semantic features of two pen-holding posture images, the first feature distance of the training image pair is obtained;

[0010] Based on the first feature distance, obtain the first optimized loss;

[0011] If the first optimization loss does not meet the first threshold, the preset model is updated based on the first optimization loss until the first preset condition is met, and the preset model after meeting the first preset condition is used as the first target model.

[0012] In one implementation, obtaining the first feature distance of the training image pair based on the high-level semantic features of two pen-holding posture images includes:

[0013] The training image pair is preprocessed to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image.

[0014] The coordinates of each key point in the hand region of the first target hand image and the second target hand image are obtained respectively;

[0015] Based on the coordinates of each key point in the hand region of the first target hand image and the second target hand image, the first feature distance of the training image pair is obtained.

[0016] In one implementation, obtaining the first feature distance of the training image pair based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image includes:

[0017] Map the coordinates of each key point in the hand region of the first target hand image and the second target hand image to the same feature space;

[0018] Based on each key point in the feature space, a preset norm distance is obtained. The preset norm distance includes at least the distance between the corresponding key points in the first target hand image and the second target hand image in the feature space.

[0019] Based on the preset norm distance, the first feature distance of the training image pair is obtained.

[0020] In one implementation, obtaining the first optimized loss based on the first feature distance includes:

[0021]

[0022] Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance y between corresponding key points in the first target hand image and the second target hand image in the feature space. n This represents the label distance for the corresponding keypoint pair.

[0023] In one embodiment, after obtaining the image training group, the method further includes:

[0024] Based on the image training group, an error image training set is obtained; the image training set contains only images of incorrect pen grip postures, and each image of an incorrect pen grip posture has at least one error label; the error labels are pre-labeled in each image of an incorrect pen grip posture.

[0025] The third convolutional neural network model is trained based on the image training set.

[0026] The training of the third convolutional neural network model based on the image training set includes:

[0027] Based on the image training set, obtain images of pen-holding postures;

[0028] Based on the third convolutional neural network model, the second feature distance of the pen-holding posture image is obtained;

[0029] If the second feature distance does not meet the second preset condition, then the second optimized loss is obtained based on the feature distance;

[0030] Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

[0031] In one implementation, training the third convolutional neural network model based on the image training set includes:

[0032] Based on the image training set, obtain images of pen-holding postures;

[0033] Based on the third convolutional neural network model, the second feature distance of the pen-holding posture image is obtained;

[0034] If the second feature distance does not meet the second preset condition, then the second optimized loss is obtained based on the feature distance;

[0035] Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

[0036] Secondly, embodiments of this application provide a pen-holding posture recognition method, including:

[0037] Acquire the image of the gesture to be recognized;

[0038] The gesture image to be recognized is input into the first target model to obtain the gesture recognition result; the first target model is trained according to the method described in the first aspect.

[0039] After obtaining the gesture recognition result, this method further includes:

[0040] If the gesture recognition result is incorrect, the gesture image to be recognized is input into the second target model to obtain the incorrect gesture category; the second target model is trained by the method described in the first aspect.

[0041] In one embodiment, after obtaining the gesture recognition result, the method further includes:

[0042] If the gesture recognition result is incorrect, the gesture image to be recognized is input into the second target model to obtain the incorrect gesture category; the second target model is trained according to the method described in the first aspect.

[0043] Thirdly, embodiments of this application provide a pen-holding posture recognition model training device, comprising:

[0044] The first acquisition unit is used to acquire an image training set, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures; and...

[0045] Based on the image training group, training image pairs are obtained, each training image pair including two pen-holding posture images, and at least one pen-holding posture image represents a correct pen-holding posture.

[0046] The first processing unit is used to acquire high-level semantic features from two pen-holding posture images respectively. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of various key points in the hand region of the pen-holding posture image. The preset model includes at least a first convolutional neural network model and a second convolutional neural network model, with the two convolutional neural network models sharing weight parameters. Each convolutional neural network model is used to extract the high-level semantic features of the input image.

[0047] Based on the high-level semantic features of two pen-holding posture images, the first feature distance of the training image pair is obtained; and,

[0048] Based on the first feature distance, obtain the first optimized loss; and,

[0049] If the first optimization loss does not meet the first threshold, the preset model is updated based on the first optimization loss until the first preset condition is met, and the preset model after meeting the first preset condition is used as the first target model.

[0050] In one embodiment, the first processing unit is further configured to preprocess two pen-holding posture images in the training image pair to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image.

[0051] The first acquisition unit is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, respectively;

[0052] The first processing unit is further configured to obtain a first feature distance of the training image pair based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image.

[0053] In one embodiment, the first processing unit is further configured to map the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image to the same feature space; and,

[0054] Based on each key point in the feature space, a preset norm distance is obtained. This preset norm distance includes at least: the distance between corresponding key points in the first target hand image and the second target hand image within the feature space; and...

[0055] Based on the preset norm distance, a first feature distance is obtained for the training image pair. The step of obtaining a first optimization loss based on the first feature distance includes:

[0056]

[0057] Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance y between corresponding key points in the first target hand image and the second target hand image in the feature space. n This represents the label distance for the corresponding keypoint pair.

[0058] In one embodiment, the first processing unit is further configured to acquire a first target hand image based on the correct pen-holding posture image;

[0059] The first acquisition unit is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and another pen-holding posture image;

[0060] The first processing unit is further configured to adjust the size of the other pen-holding posture image based on the coordinate positions of each key point of the hand region in the first target hand image and the other pen-holding posture image, so as to reduce the area difference between the hand region in the other pen-holding posture image and the hand region in the first target hand image, so as to obtain a second hand image.

[0061] The first processing unit is further configured to acquire a second target hand image based on the second hand image, so that the first target hand image and the second target hand image are the same size.

[0062] In one embodiment, the first acquisition unit is further configured to acquire the maximum distance D1 between each key point in the hand region of the first target hand image, and the slope k1 between the two key points forming the maximum distance D1;

[0063] The first processing unit is further configured to, based on the slope k1, obtain the slope k2, where k1*k2=-1; and,

[0064] Based on the slope k2, the maximum distance D2 between each key point in the hand region of the first target hand image along the slope k2 direction is obtained;

[0065] The first acquisition unit is further configured to acquire the maximum distance D3 between each key point in the hand region of the other pen-holding posture image, and the slope k3 between the two key points forming the maximum distance D3;

[0066] The first processing unit is further configured to, based on the slope k3, obtain the slope k4, where k3*k4=-1; and,

[0067] Based on the slope k4, obtain the maximum distance D4 between each key point in the hand region of the first target hand image along the direction of the slope k4; and...

[0068] Based on the ratio of D1 and D3, the size of the other pen-holding posture image along the k3 direction is adjusted proportionally; based on the ratio of D2 and D4, the size of the other pen-holding posture image along the k4 direction is adjusted proportionally; the adjusted pen-holding posture image is obtained as the second hand image.

[0069] In one embodiment, the preset model further includes: a third convolutional neural network model, wherein the three convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract high-level semantic features of the input image;

[0070] The first acquisition unit is further configured to acquire an erroneous image training set based on the image training group; the image training set contains only images of incorrect pen-holding postures, and each image of an incorrect pen-holding posture has at least one error label; the error labels are pre-labeled in each image of an incorrect pen-holding posture.

[0071] The first processing unit is further configured to train the third convolutional neural network model based on the image training set.

[0072] In one embodiment, the first acquisition unit is further configured to acquire pen-holding posture images based on the image training set;

[0073] The first processing unit is further configured to, based on the third convolutional neural network model, obtain a second feature distance of the pen-holding posture image; and,

[0074] If the second feature distance does not meet the second preset condition, then a second optimized loss is obtained based on the feature distance; and,

[0075] Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

[0076] Fourthly, embodiments of this application provide a pen-holding posture recognition device, comprising:

[0077] The second acquisition unit is used to acquire the gesture image to be recognized;

[0078] The second processing unit is used to input the gesture image to be recognized into the first target model to obtain the gesture recognition result; the first target model is trained according to the method described in the first aspect.

[0079] In one embodiment, the second processing unit is further configured to, if the gesture recognition result is incorrect, input the gesture image to be recognized into a second target model to obtain an incorrect gesture category; the second target model is trained according to the method described in the first aspect.

[0080] Fifthly, embodiments of this application provide a pen grip posture recognition device, including: a memory and a processor; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor executes the pen grip posture recognition model training method as described in the first aspect, or the pen grip posture recognition method as described in the second aspect.

[0081] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed, can implement the pen-holding posture recognition model training method described in the first aspect, or the pen-holding posture image recognition method described in the second aspect.

[0082] Compared with the prior art, the beneficial effects of this application are:

[0083] Based on the pen-holding posture image recognition model training method, recognition method, recognition device, or equipment of this application, it is possible to determine the correctness of the pen-holding posture in another pen-holding posture image to be recognized based on a correct pen-holding posture image. Furthermore, since the first and second convolutional neural network models share weights, both models can extract features from the pen-holding posture image in the same way. That is, the first and second convolutional neural network models can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared to the sequential extraction of features from different images in existing technologies, the pen-holding posture image recognition model of this application saves time and improves efficiency. Moreover, compared to the sequential feature extraction method, parallel image feature extraction eliminates the need to store and retrieve the acquired image features, further saving time. Attached Figure Description

[0084] Figure 1 This is a schematic flowchart of a model training method according to an embodiment of this application;

[0085] Figure 2 This is a schematic diagram of key points in the hand area according to an embodiment of this application;

[0086] Figure 3 This is a schematic diagram of key points in the hand region obtained in an embodiment of this application;

[0087] Figure 4This is yet another schematic diagram of key points in the hand region obtained in an embodiment of this application;

[0088] Figure 5 This is another flowchart illustrating the model training method according to an embodiment of this application;

[0089] Figure 6 This is a schematic diagram of the structure of the pen grip posture recognition model training device according to an embodiment of this application;

[0090] Figure 7 This is a schematic diagram of the pen grip posture recognition device according to an embodiment of this application;

[0091] Figure 8 This is a schematic diagram of the pen grip posture recognition device according to an embodiment of this application. Detailed Implementation

[0092] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects (e.g., the first preset condition and the second preset condition represent different preset conditions, and so on), and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0093] It should be noted that the preset model trained in the embodiments of this application is particularly suitable for image recognition scenarios where the actual labels are the same but the image features are significantly different. That is, the model is prone to identifying two actually similar images as dissimilar. For example, in the scenario of pen-holding posture image recognition, when two input images have similar pen-holding postures, if the size of the person's hand or the shooting angle is inconsistent, it will cause the positions of various key points in the hand area of ​​the image to differ significantly, which will be judged as a large distance loss. Therefore, the two images are considered to be significantly different, and the two input images are further judged as dissimilar, thus outputting an incorrect recognition result.

[0094] Furthermore, although this application's embodiments use a pen-holding posture image recognition scenario as an example to illustrate how to train the model and extract image features, those skilled in the art can implement the model training method of this application's embodiments in other human action recognition scenarios based on the disclosure of this application's embodiments, in order to perform corresponding feature extraction. The inventive principles disclosed in this application's embodiments at least include preprocessing two input images to make the number and size of their key point regions tend to be consistent, and reducing the difference between two input images with the same label by using relative distance, thereby improving the accuracy of the preset model's recognition results.

[0095] The solutions provided in this application involve technologies such as Artificial Intelligence (AI), Natural Language Processing (NLP), and Machine Learning (ML), which are specifically illustrated through the following embodiments:

[0096] AI, or Artificial Intelligence, refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, Artificial Intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial Intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0097] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0098] NLP is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. Natural Language Processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. Natural Language Processing techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0099] Specifically, this application uses pen grip posture recognition as an application scenario to describe the model training method, recognition method and related devices in this application.

[0100] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a pen-holding posture recognition model training method provided in an embodiment of this application. Specifically, the method includes:

[0101] Step S100: Obtain an image training group, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures.

[0102] Specifically, the image training group can be obtained in various ways in this application, including from videos or from photographs taken with a camera. For example, images can be manually extracted from videos of a given pen-holding posture, or images can be extracted from videos of a given pen-holding posture using video processing software, including but not limited to CamRecorder, Animate, and kmplayer. Since this is a mature existing technology, it will not be described in detail.

[0103] Step S200: Based on the image training group, obtain training image pairs, wherein the training image pairs include two pen-holding posture images, and at least one pen-holding posture image represents the correct pen-holding posture.

[0104] Specifically, in this application, the training image pair can be two pen-holding posture images randomly obtained from the image training group, and the two pen-holding posture images form a training image pair. Of course, in some special cases, for example, in order to ensure that the obtained training image pairs have certain differences and to prevent the training model from overfitting, multiple pen-holding posture images can be pre-paired. In this case, the pen-holding posture images in the image training group form the image training group in pairs.

[0105] Step S300: Obtain high-level semantic features from the two pen-holding posture images respectively. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of each key point in the hand region of the pen-holding posture image. Figure 2 As shown in a specific embodiment of this application, the number of key points in the hand area of ​​the pen-holding posture image is 21, which are: wrist joint point 0, thumb first joint point 1, thumb second joint point 2, thumb third joint point 3, thumb tip point 4, index finger first joint point 5, index finger second joint point 6, index finger third joint point 7, index finger tip point 8, middle finger first joint point 9, middle finger second joint point 10, middle finger third joint point 11, middle finger tip point 12, ring finger first joint point 13, ring finger second joint point 14, ring finger third joint point 15, ring finger tip point 16, little finger first joint point 17, little finger second joint point 18, little finger third joint point 19, little finger tip point 20.

[0106] Of course, it should be clear that in other embodiments of this application, different key points than those described above can be selected as high-level semantic features of the pen-holding posture image. For example, some key points can be added, or some of the aforementioned key points can be removed; there are no limitations on this.

[0107] In other embodiments of this application, other features can be selected as high-level semantic features of the pen-holding posture image. For example, the distance or slope between the coordinates of various key points in the hand region of the same pen-holding posture image.

[0108] Specifically, in this application, the preset model includes at least: a first convolutional neural network model and a second convolutional neural network model, the two convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract high-level semantic features of the input image.

[0109] It's important to understand that during the training of the pre-defined model, one pen-holding posture image from the training image pair is input into the first convolutional neural network (CNN) model, and the other pen-holding posture image from the same pair is input into the second CNN model. By using two CNN models with shared weights, the high-level semantic features of the two pen-holding posture images in the training image pair can be extracted quickly, improving training speed. Simultaneously, the target model obtained based on this method can also quickly extract the high-level semantic features of the acquired training image pair during application, saving image processing time. This allows the obtained target model to be applicable to the application scenario proposed in the background technique, which involves comparing pen-holding posture videos from two captured videos, given the given pen-holding posture. Since the first and second CNN models share weight parameters, the high-level semantic features of the training image pair obtained through the first and second CNN models will not have any bias and will not affect the accuracy of the judgment.

[0110] Step S400: Based on the high-level semantic features of the two pen-holding posture images, obtain the first feature distance of the training image pair.

[0111] It is important to understand that in the embodiments of this application, the first feature distances obtained from different high-level semantic features are also different. For example, the distance between the coordinates of various key points in the hand region of an image with the same pen-holding posture is used as a high-level semantic feature, and... Figure 3 The distance between wrist joint point 0 and the second joint point 2 of the thumb is the first distance. Figure 4 The distance between wrist joint point 0 and thumb second joint point 2 is the second distance. The difference between the first distance and the second distance can be used as... Figure 3 and Figure 4 The first feature distance. Alternatively, the slope between the coordinates of various key points in the hand region of an image with the same pen-holding posture can be used as a high-level semantic feature, and Figure 3 The slope between wrist joint point 0 and the second joint point 2 of the thumb is the first slope. Figure 4 If the distance between wrist joint point 0 and thumb second joint point 2 is the second slope, then the difference between the first slope and the second slope can also be used as... Figure 3 and Figure 4 The first feature distance. Alternatively, the coordinates of each key point in the hand region of an image with the same pen-holding posture can be used as high-level semantic features, and Figure 3 The coordinates of wrist joint point 0 in the diagram are the first coordinates. Figure 4 If the coordinates of wrist joint point 0 are the second coordinates, then the difference between the first and second coordinates can also be used as... Figure 3 and Figure 4 The first characteristic distance.

[0112] Step S500: Obtain the first optimized loss based on the first feature distance.

[0113] It is important to understand that in one embodiment of this application, the similarity or dissimilarity of two pen-holding posture images is determined by whether the first feature distance is greater than or less than a certain fixed threshold. However, when different people have different hand sizes or when the shooting angle of the same person's hand is different, the feature distance of key points in the two pen-holding posture images can easily increase. Even if the two pen-holding posture images are similar, the model may judge them as having a large distance loss, thus considering them to be significantly different and classifying them as incorrect pen-holding postures. For example: Figure 3 and Figure 4 As shown, assuming Figure 3 This is the correct way to hold a pen. Figure 4 and Figure 3 The pen-holding posture is similar. However, due to Figure 3 and Figure 4The significant difference in shooting angles resulted in substantial differences in the image quality of the two images. Figure 3 and Figure 4 The first feature distance obtained is also relatively large, therefore, it is very easy for the model to mistakenly identify it. Figure 3 and Figure 4 To accommodate different pen-holding postures, Figure 4 The pen-holding posture in the text is judged as incorrect. Specifically, for example... Figure 3 and Figure 4 As shown, although the pen-holding postures in the two images are similar, the distance between the wrist joint point 0 and the second joint point 2 of the thumb, as well as the slope between them, are different, making it impossible to set a fixed threshold. In other words, even if a fixed threshold could be set, a large number of images of correct pen-holding postures would need to be collected as training samples to train the model. When recognizing the pen-holding posture image to be identified, the model needs to first find the correct pen-holding posture image with the closest shooting angle to the image to be identified, and then calculate the feature distance between the two, which is time-consuming. Therefore, in this application, a first threshold is introduced to solve this technical problem.

[0114] Step S600: If the first optimization loss does not meet the first threshold, then based on the first optimization loss, update the preset model until the first preset condition is met, and then use the preset model after the first preset condition is met as the first target model.

[0115] It is important to understand that the first threshold in this application is a specific numerical range, not a fixed value. The following section uses image recognition of pen-holding posture as an application scenario to specifically introduce the first threshold proposed in this application.

[0116] Specifically, in the embodiments of this application, the label distance of the training image pair can be directly set. For example, a fixed threshold can be used as a dividing line. When both pen-holding posture images in a training image pair are correct pen-holding posture images, the label distance of the training image pair is set to be less than the fixed threshold; when one of the two pen-holding posture images in a training image pair is incorrect, the label distance of the training image pair is set to be greater than the fixed threshold. For example, using 1 as a fixed threshold, when both pen-holding posture images in a training image pair are correct pen-holding posture images, the label distance of the training image pair is set to any value less than 1; when one of the two pen-holding posture images in a training image pair is incorrect, the label distance of the training image pair is set to any value greater than 1. Therefore, when training a preset model using this method, if the input training image pair to the preset model consists of two correct pen-holding posture images, and the first optimization loss output by the preset model is greater than 1, then the first optimization loss is considered not to meet the first threshold; if the input training image pair to the preset model contains one incorrect pen-holding posture image, and the first optimization loss output by the preset model is less than 1, then the first optimization loss is considered not to meet the first threshold.

[0117] As can be seen from the above, the first threshold in the above embodiments is a range value greater than or less than 1. Using the above model training method, after training the preset model and obtaining the first target model, if the first optimization loss between the pen-holding posture image to be identified and the correct pen-holding posture image output by the first target model is less than 1, it indicates that the pen-holding posture image to be identified is a correct pen-holding posture image; if the first optimization loss between the pen-holding posture image to be identified and the correct pen-holding posture image output by the first target model is greater than 1, it indicates that the pen-holding posture image to be identified is an incorrect pen-holding posture image.

[0118] It should be noted that in other embodiments of this application, labels can be assigned to individual pen-holding posture images. For example, a correct pen-holding posture image and a batch of pen-holding posture images to be judged can be obtained. The label of the correct pen-holding posture image is set to 0, and then the labels of other pen-holding posture images are sequentially labeled. If the pen-holding posture image to be labeled is a correct pen-holding posture, it is labeled with any label less than 1 (fixed threshold); if the pen-holding posture image to be labeled is an incorrect pen-holding posture, it is labeled with any label greater than 1 (fixed threshold). This labeling method does not need to be very precise; it only needs to ensure that the trend is correct.

[0119] It should also be clear that, in the embodiments of this application, the first preset condition can be that the preset model stops training after a certain number of iterations, for example, training the preset model 200 times and then stopping. Of course, in other embodiments of this application, the first preset condition can be that the loss value of the preset model converges and then stops.

[0120] The first target model obtained through the above training method can accurately identify the correctness of pen-holding posture images. Compared to existing technologies that require setting a series of judgment conditions, such as the thumb must be between the index and middle fingers, and the middle, ring, and little fingers must be tightly pressed together, and then judging whether the pen-holding posture is correct through manually set thresholds, its judgment conditions are relatively simple. Furthermore, when labeling pen-holding posture images or training image pairs, the value of this label does not need to be particularly accurate; only the trend needs to be correct. Compared to existing technologies that require precise setting of labels and judgment conditions, its operation is simpler.

[0121] Specifically, in one embodiment of this application, step S400, obtaining the first feature distance of the training image pair based on the high-level semantic features of the two pen-holding posture images, includes:

[0122] Step S410: Preprocess the two pen-holding posture images in the training image pair to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image.

[0123] It is important to understand that, in this embodiment, the main purpose of preprocessing the two pen-holding posture images is to reduce the differences in the hand area caused by the different sizes of the pen-holding posture images or the human hand. For example, when the two pen-holding posture images are of different sizes but the sizes of their hand areas are similar, it will result in a large difference in the coordinates of their key points. Similarly, when the two pen-holding posture images are of the same size, but one of them has been reduced in height or width, leading to a large difference in the size of the hand area, it will also result in a large difference in the coordinates of their key points. Therefore, these effects can be eliminated through preprocessing. In this application, any common existing technology can be used to preprocess the two pen-holding posture images in the training image pair.

[0124] In a specific embodiment of this application, step S410, preprocessing the two pen-holding posture images in the training image pair, includes:

[0125] Step S411: Based on the correct pen-holding posture image, obtain the first target hand image.

[0126] Specifically, the purpose of steps S411 and S414 (described below) is to ensure that the obtained first target hand image and the second target hand image are of the same size, thereby eliminating the influence of images of different sizes on the coordinates of key points in the hand region. For example, when two images with identical pen-holding postures exist, one with a width and height of 1024 pixels and the other with a width and height of 512 pixels, although the pen-holding postures are identical, the coordinates of key points in the corresponding hand regions can differ significantly. Image segmentation and cropping can then be used to make the two pen-holding posture images the same size.

[0127] Of course, in other embodiments of this application, normalization processing can be used to process the coordinates of key points in the corresponding hand areas of the two pen-holding posture images to eliminate errors caused by the image size itself. Specifically, the normalization processing is as follows: divide the x-coordinate of a key point in a pen-holding posture image by the width of the pen-holding posture image to obtain x1; divide the y-coordinate of the same key point in the same pen-holding posture image by the width of the pen-holding posture image to obtain y1, and use (x1, y1) as the new coordinates of the key point. For example, if the width and height of a pen-holding posture image in the training image pair are 512 pixels, and the coordinates of one key point in the pen-holding posture image are (256, 128), then after preprocessing, the coordinates of the key point are (0.5, 0.25), and so on for other normalized key point coordinates. The other pen-holding posture image in the training image pair has a width and height of 1024 pixels. The coordinates of the keypoint corresponding to the aforementioned keypoint in this image are (512, 256). After normalization, the coordinates of this keypoint in the same image are (0.5, 0.25). Compared to the difference between coordinates (512, 256) and (256, 128), the difference between coordinates (0.5, 0.25) and (0.5, 0.25) is significantly reduced. The first feature distance obtained based on the new coordinates is not affected by the size difference between the two pen-holding posture images themselves, and can more accurately reflect the true similarity between the two images.

[0128] As can be seen from the above, when two pen-holding posture images are the same size, but one of the pen-holding posture images has been reduced in height or width, or the difference in the subject's hand size is too large, resulting in a large difference in the size of the hand area between the two pen-holding posture images, it will also cause a large difference in the coordinates of the key points between the two pen-holding posture images. Therefore, in order to solve this technical problem, step S410 in this embodiment of the application further includes:

[0129] Step S412: Obtain the coordinates of each key point in the hand region of the first target hand image and another pen-holding posture image.

[0130] Step S413: Based on the coordinate positions of each key point in the hand region of the first target hand image and the other pen-holding posture image, adjust the size of the other pen-holding posture image to reduce the area difference between the hand region in the other pen-holding posture image and the hand region in the first target hand image, so as to obtain a second hand image.

[0131] Step S414: Segment the second hand image to obtain a second target hand image, so that the first target hand image and the second target hand image are the same size.

[0132] Specifically, in one specific embodiment of this application, step S413, adjusting the size of the other pen-holding posture image based on the coordinate positions of key points in the hand region of the first target hand image and the other pen-holding posture image to obtain the second hand image, includes:

[0133] Step S413a: Obtain the maximum distance D1 between each key point in the hand region of the first target hand image, and the slope k1 between the two key points that form the maximum distance D1;

[0134] Step S413b: Based on the slope k1, obtain the slope k2, where k1*k2=-1;

[0135] Step S413c: Based on the slope k2, obtain the maximum distance D2 between each key point in the hand region of the first target hand image along the direction of the slope k2;

[0136] Step S413d: Obtain the maximum distance D3 between each key point in the hand region of the other pen-holding posture image, and the slope k3 between the two key points that form the maximum distance D3;

[0137] Step S413e: Based on the slope k3, obtain the slope k4, where k3*k4=-1;

[0138] Step S413f: Based on the slope k4, obtain the maximum distance D4 between each key point in the hand region of the first target hand image along the direction of the slope k4;

[0139] Step S413g: Based on the ratio of D1 and D3, adjust the size of the other pen-holding posture image along the k3 direction proportionally; based on the ratio of D2 and D4, adjust the size of the other pen-holding posture image along the k4 direction proportionally.

[0140] Step S413h: Obtain the resized image of the other pen-holding posture as the second hand image.

[0141] It is important to understand that in all images of pen-holding postures, regardless of the shooting angle or the size of the subject's hand, when the pen-holding posture is correct, the maximum distance between the key points is always the distance between the tip of the thumb (4) or the tip of the index finger (8) and the wrist joint (0). In this way, by adjusting the image size in this direction and in the direction perpendicular to this direction, the difference in the area of ​​the hand region in two pen-holding posture images can be reduced.

[0142] It's important to understand that when the pen grip is correct, adjusting the hand area as described above can reduce differences caused by the size of the hand itself or by resizing the image. Conversely, when the pen grip is incorrect, for example: Figure 3 and Figure 4 As shown, the maximum distance between each key point is not the distance between the thumb tip 4 or the index finger tip 8 and the wrist joint 0. After adjustment, the above-mentioned differences can still be maintained and will not affect the recognition results.

[0143] Of course, based on the above principle, it is easy to imagine that in other embodiments of this application, one of the pen-holding posture images can be proportionally reduced or enlarged based on the area of ​​the hand region in the two pen-holding posture images, so that the area of ​​the hand region in the two pen-holding posture images is consistent.

[0144] Step S420: Obtain the coordinates of each key point in the hand region of the first target hand image and the second target hand image, respectively.

[0145] Step S430: Based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, obtain the first feature distance of the training image pair.

[0146] In a specific embodiment of this application, to facilitate the training of the preset model, step S430, based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, obtains the first feature distance of the training image pair, including:

[0147] Step S431: Map the coordinates of each key point in the hand region of the first target hand image and the second target hand image to the same feature space.

[0148] Step S432: Obtain the preset norm distance based on each key point in the feature space.

[0149] It should be clear that the aforementioned preset norm distance can be the distance formed between any two key points in the feature space when they are combined in pairs.

[0150] As a preferred embodiment of this application, the preset norm distance is the distance between corresponding key points in the first target hand image and the second target hand image in the feature space.

[0151] Step S433: Based on the preset norm distance, obtain the first feature distance of the training image pair.

[0152] Specifically, in a specific embodiment of this application, if the preset norm distance is the distance between corresponding key points in the first target hand image and the second target hand image in the feature space, then step S500, based on the first feature distance, obtains a first optimization loss, including:

[0153]

[0154] Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance y between corresponding key points in the first target hand image and the second target hand image in the feature space. n This represents the label distance for the corresponding keypoint pair.

[0155] Of course, it is easy to understand that in other embodiments of this application, the first feature distance may be obtained by any combination of the distances of each key point pair in the feature space. For example, the first feature distance may be the sum or average of the distances of any one, two, three or four key point pairs in the feature space.

[0156] It is important to understand that the above method requires labeling each image with a unique tag corresponding to a key point. In a preferred embodiment of this application, step S500, based on the first feature distance, obtains a first optimization loss, including:

[0157]

[0158] Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance between corresponding key points in the first target hand image and the second target hand image in the feature space, where D is the label distance between the two pen-holding posture images. In this way, it is only necessary to label each of the two pen-holding posture images individually, or to label the image pair composed of the two pen-holding posture images.

[0159] It is important to understand that the first target model obtained through the above model training method can only identify whether the pen grip posture in the pen grip image is correct. When the pen grip posture in the pen grip image is incorrect, it cannot identify the type of error in the incorrect pen grip posture.

[0160] In order to identify the error type of incorrect pen grip posture, in another embodiment of this application, the preset model further includes: a third convolutional neural network model. The three convolutional neural network models share weight parameters. Each convolutional neural network model is used to extract high-level semantic features of the input image. The third convolutional neural network model is mainly used to identify the error type of incorrect pen grip posture.

[0161] like Figure 5 As shown, after obtaining the image training group in step S100, this method further includes:

[0162] Step S700: Based on the image training group, obtain an incorrect image training set; the image training set contains only images of incorrect pen grip postures, and each incorrect pen grip posture image has at least one error label; the error label is pre-labeled in each incorrect pen grip posture image.

[0163] Step S800 involves training the third convolutional neural network model based on the image training set. Step S800, training the third convolutional neural network model based on the image training set, includes:

[0164] Step S810: Based on the image training set, obtain the pen-holding posture image.

[0165] Step S820: Based on the third convolutional neural network model, obtain the second feature distance of the pen-holding posture image.

[0166] Step S830: If the second feature distance does not meet the second preset condition, then obtain the second optimized loss based on the feature distance.

[0167] Step S840: Based on the second optimization loss, update the third convolutional neural network model and obtain a new pen-holding posture image until the second feature distance meets the second preset condition, and use the third convolutional neural network model when the second feature distance meets the second preset condition as the second target model.

[0168] Specifically, the second preset condition here can be any condition, such as: the third convolutional neural network model has been trained a certain number of times, or the loss value has converged.

[0169] The model obtained based on the above-described pen-holding posture recognition training method can determine the correctness of the pen-holding posture in another pen-holding posture image based on a correct pen-holding posture image. Furthermore, since the first and second convolutional neural network models share weights, they can extract features from the pen-holding posture image in the same way. That is, the first and second convolutional neural network models can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared to the sequential extraction of features from different images in existing technologies, the pen-holding posture image recognition model of this application saves time and improves efficiency. Moreover, compared to the sequential feature extraction method, parallel image feature extraction eliminates the need to store and retrieve the acquired image features, further saving time.

[0170] After introducing the pen-holding posture recognition model training method in this application, the pen-holding posture image recognition method proposed in this application will be described in detail. This method includes:

[0171] Step S910: Obtain the image of the gesture to be recognized.

[0172] Specifically, the method of acquiring the gesture image to be recognized can be arbitrary, such as: capturing an image from a video or directly taking an image using a camera, without any restrictions on the acquisition method.

[0173] Step S920: Input the gesture image to be recognized into the first target model to obtain the gesture recognition result; the first target model is trained according to any one of the above-described pen grip posture recognition model training methods.

[0174] In another embodiment of this application, after obtaining the gesture recognition result in step S920, the method further includes:

[0175] Step S930: If the gesture recognition result is incorrect, the gesture image to be recognized is input into the second target model to obtain the incorrect gesture category; the second target model is trained according to any embodiment of the above-mentioned pen grip posture recognition model training method.

[0176] Based on the aforementioned pen-holding posture image recognition method, it is possible to determine the correctness of the pen-holding posture in another pen-holding posture image based on a correct pen-holding posture image. Furthermore, since the first and second convolutional neural network models share weights, both models can extract features from the pen-holding posture image in the same way. That is, the first and second convolutional neural network models can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared to the sequential extraction of features from different images in existing technologies, the pen-holding posture image recognition model of this application saves time and improves efficiency. Moreover, compared to sequential feature extraction, parallel image feature extraction eliminates the need to store and retrieve the acquired image features, further saving time.

[0177] After introducing the method of the embodiments of this application, the following references are made. Figure 6 The pen grip posture recognition model training device 10 of this application embodiment is described below. The device includes:

[0178] The first acquisition unit 11 is used to acquire an image training group, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures; and...

[0179] Based on the image training group, training image pairs are obtained, each training image pair including two pen-holding posture images, and at least one pen-holding posture image represents a correct pen-holding posture.

[0180] The first processing unit 12 is used to acquire high-level semantic features of two pen-holding posture images respectively. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of each key point in the hand region of the pen-holding posture image. The preset model includes at least a first convolutional neural network model and a second convolutional neural network model. The two convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract the high-level semantic features of the input image.

[0181] Based on the high-level semantic features of two pen-holding posture images, the first feature distance of the training image pair is obtained; and,

[0182] Based on the first feature distance, obtain the first optimized loss; and,

[0183] If the first optimization loss does not meet the first threshold, the preset model is updated based on the first optimization loss until the first preset condition is met, and the preset model after meeting the first preset condition is used as the first target model.

[0184] In one embodiment of this application, the first processing unit 12 is further configured to preprocess two pen-holding posture images in the training image pair to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image.

[0185] The first acquisition unit 11 is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, respectively;

[0186] The first processing unit 12 is further configured to obtain a first feature distance of the training image pair based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image.

[0187] In one embodiment of this application, the first processing unit 12 is further configured to map the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image to the same feature space; and,

[0188] Based on each key point in the feature space, a preset norm distance is obtained. This preset norm distance includes at least: the distance between corresponding key points in the first target hand image and the second target hand image within the feature space; and...

[0189] Based on the preset norm distance, a first feature distance is obtained for the training image pair. The step of obtaining a first optimization loss based on the first feature distance includes:

[0190]

[0191] Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance y between corresponding key points in the first target hand image and the second target hand image in the feature space. n This represents the label distance for the corresponding keypoint pair.

[0192] In one embodiment of this application, the first processing unit 12 is further configured to obtain a first target hand image based on the correct pen-holding posture image;

[0193] The first acquisition unit 11 is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and another pen-holding posture image;

[0194] The first processing unit 12 is further configured to adjust the size of the other pen-holding posture image based on the coordinate positions of each key point of the hand region in the first target hand image and the other pen-holding posture image, so as to reduce the area difference between the hand region in the other pen-holding posture image and the hand region in the first target hand image, so as to obtain a second hand image.

[0195] The first processing unit 12 is further configured to acquire a second target hand image based on the second hand image, so that the first target hand image and the second target hand image are the same size.

[0196] In one embodiment of this application, the first acquisition unit 11 is further configured to acquire the maximum distance D1 between each key point in the hand region of the first target hand image, and the slope k1 between the two key points forming the maximum distance D1;

[0197] The first processing unit 12 is further configured to, based on the slope k1, obtain the slope k2, wherein k1*k2=-1; and,

[0198] Based on the slope k2, the maximum distance D2 between each key point in the hand region of the first target hand image along the slope k2 direction is obtained;

[0199] The first acquisition unit 11 is further configured to acquire the maximum distance D3 between each key point in the hand region of the other pen-holding posture image, and the slope k3 between the two key points forming the maximum distance D3;

[0200] The first processing unit 12 is further configured to, based on the slope k3, obtain the slope k4, wherein k3*k4=-1; and,

[0201] Based on the slope k4, obtain the maximum distance D4 between each key point in the hand region of the first target hand image along the direction of the slope k4; and...

[0202] Based on the ratio of D1 and D3, the size of the other pen-holding posture image along the k3 direction is adjusted proportionally; based on the ratio of D2 and D4, the size of the other pen-holding posture image along the k4 direction is adjusted proportionally; the adjusted pen-holding posture image is obtained as the second hand image.

[0203] In one embodiment of this application, the preset model further includes: a third convolutional neural network model, wherein the three convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract high-level semantic features of the input image;

[0204] The first acquisition unit 11 is further configured to acquire an erroneous image training set based on the image training group; the image training set contains only images of erroneous pen-holding postures, and each image of an erroneous pen-holding posture has at least one error label; the error labels are pre-labeled in each image of an erroneous pen-holding posture.

[0205] The first processing unit 12 is further configured to train the third convolutional neural network model based on the image training set.

[0206] In one embodiment of this application, the first acquisition unit 11 is further configured to acquire pen-holding posture images based on the image training set;

[0207] The first processing unit 12 is further configured to, based on the third convolutional neural network model, obtain a second feature distance of the pen-holding posture image; and,

[0208] If the second feature distance does not meet the second preset condition, then a second optimized loss is obtained based on the feature distance; and,

[0209] Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

[0210] Based on the model obtained by the pen-holding posture recognition model training device 10 described above, it is possible to judge the correctness of the pen-holding posture in another pen-holding posture image to be recognized based on a correct pen-holding posture image. Furthermore, since the first convolutional neural network model and the second convolutional neural network model share weights, the two convolutional neural network models can extract features from the pen-holding posture image in the same way. That is, the first convolutional neural network model and the second convolutional neural network model can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared with the prior art of sequentially extracting features from different images, the pen-holding posture image recognition model of this application can save time and improve efficiency. Moreover, compared with the sequential feature extraction method, the parallel extraction of image features eliminates the need to store and retrieve the acquired image features, further saving time.

[0211] After introducing the embodiment of the pen grip posture recognition model training device 10 in this application, the pen grip posture recognition device 20 in this application will be described below. Specifically, in this application, the pen grip posture recognition device 20, as... Figure 7 As shown, it includes:

[0212] The second acquisition unit 21 is used to acquire the gesture image to be recognized;

[0213] The second processing unit 22 is used to input the gesture image to be recognized into the first target model to obtain the gesture recognition result; the first target model is trained according to any one of the above-described pen-holding posture recognition model training methods.

[0214] In one embodiment of this application, the second processing unit 22 is further configured to input the gesture image to be recognized into a second target model if the gesture recognition result is incorrect, and obtain the incorrect gesture category; the second target model is trained according to any one of the above-described pen grip posture recognition model training methods.

[0215] Based on the aforementioned pen-holding posture recognition device 20, it is possible to determine the correctness of the pen-holding posture in another pen-holding posture image to be recognized based on a correct pen-holding posture image. Furthermore, since the first and second convolutional neural network models share weights, both models can extract features from the pen-holding posture image in the same way. That is, the first and second convolutional neural network models can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared to the prior art's sequential extraction of features from different images, the pen-holding posture image recognition model of this application saves time and improves efficiency. Moreover, compared to the sequential feature extraction method, parallel image feature extraction eliminates the need to store and retrieve the acquired image features, further saving time.

[0216] After introducing the device in this application, the pen-holding posture recognition device 30 of this application will now be described. Specifically, as follows... Figure 8 As shown, the device includes a memory 31 and a processor 32; wherein, the memory 31 stores executable code, and when the executable code is executed by the processor 32, the processor 32 executes the pen grip posture recognition model training method as described in any one of the above embodiments, or the pen grip posture image recognition method as described in any one of the above embodiments.

[0217] Specifically, in this application, memory 31 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0218] Based on the aforementioned pen-holding posture image recognition device 30, it is possible to determine the correctness of the pen-holding posture in another pen-holding posture image to be recognized, based on a correct pen-holding posture image. Furthermore, since the first and second convolutional neural network models share weights, both models can extract features from the pen-holding posture image in the same way. That is, the first and second convolutional neural network models can extract image features from a correct pen-holding posture image and a pen-holding posture image to be recognized in parallel. Compared to the sequential extraction of features from different images in existing technologies, the pen-holding posture image recognition model of this application saves time and improves efficiency. Moreover, compared to sequential feature extraction, parallel image feature extraction eliminates the need to store and retrieve the acquired image features, further saving time.

[0219] Having described the apparatus of this application, the computer-readable storage medium storing a computer program of this application is now described. When executed, the computer program can implement the pen-grip posture recognition model training method as described in any of the above embodiments, or the pen-grip posture image recognition method as described in any of the above embodiments.

[0220] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0221] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0222] In the embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection between devices or modules through some interfaces, and may be electrical, mechanical, or other forms.

[0223] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0225] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0226] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0227] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for training a pen grip posture recognition model, the method comprising: include: Obtain an image training set, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures; Based on the image training group, training image pairs are obtained, each training image pair including two pen-holding posture images, and at least one pen-holding posture image represents a correct pen-holding posture. High-level semantic features are obtained from two pen-holding posture images. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of each key point in the hand region of the pen-holding posture image. The preset model includes at least a first convolutional neural network model and a second convolutional neural network model. The two convolutional neural network models share weight parameters. Each convolutional neural network model is used to extract the high-level semantic features of the input image. Based on the high-level semantic features of two pen-holding posture images, the first feature distance of the training image pair is obtained; Based on the first feature distance, obtain the first optimized loss; If the first optimization loss does not meet the first threshold, then based on the first optimization loss, the preset model is updated until the first preset condition is met, and the preset model after meeting the first preset condition is used as the first target model. The method of obtaining the first feature distance of the training image pair based on the high-level semantic features of two pen-holding posture images includes: The training image pair is preprocessed to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image. The coordinates of each key point in the hand region of the first target hand image and the second target hand image are obtained respectively; Based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, the first feature distance of the training image pair is obtained; The preprocessing of the two pen-holding posture images in the training image pair includes: Based on the correct pen-holding posture image, obtain the first target hand image; Obtain the coordinates of key points in the hand region of the first target hand image and another pen-holding posture image; Based on the coordinates of each key point in the hand region of the first target hand image and the other pen-holding posture image, the size of the other pen-holding posture image is adjusted to reduce the area difference between the hand region in the other pen-holding posture image and the hand region in the first target hand image, so as to obtain a second hand image; Based on the second hand image, a second target hand image is obtained so that the first target hand image and the second target hand image are the same size; The step of adjusting the size of the other pen-holding posture image to obtain the second hand image based on the coordinate positions of key points in the hand region of the first target hand image and the other pen-holding posture image includes: Obtain the maximum distance D1 between each key point in the hand region of the first target hand image, and the slope k1 between the two key points that form the maximum distance D1; Based on the slope k1, the slope k2 is obtained, where k1 k2=-1; Based on the slope k2, the maximum distance D2 between each key point in the hand region of the first target hand image along the slope k2 direction is obtained; Obtain the maximum distance D3 between each key point in the hand region of the other pen-holding posture image, and the slope k3 between the two key points that form the maximum distance D3; Based on the slope k3, the slope k4 is obtained, wherein k3 k4 = -1; Based on the slope k4, the maximum distance D4 between each key point in the hand region of the first target hand image along the slope k4 direction is obtained; Based on the ratio of D1 and D3, the size of the other pen-holding posture image along the k3 direction is adjusted proportionally; based on the ratio of D2 and D4, the size of the other pen-holding posture image along the k4 direction is adjusted proportionally. The resized image of the other pen-holding posture is obtained as the second hand image. 2.The grip posture recognition model training method of claim 1, wherein, Based on the coordinates of various key points in the hand regions of the first target hand image and the second target hand image, the first feature distance of the training image pair is obtained, including: Map the coordinates of each key point in the hand region of the first target hand image and the second target hand image to the same feature space; Based on each key point in the feature space, a preset norm distance is obtained. The preset norm distance includes at least the distance between the corresponding key points in the first target hand image and the second target hand image in the feature space. Based on the preset norm distance, the first feature distance of the training image pair is obtained. 3.The grip posture recognition model training method of claim 2, wherein, The step of obtaining the first optimized loss based on the first feature distance includes: wherein N is the number of pairs of corresponding keypoints in the feature space of the first target hand image and the second target hand image, y pn the distance in the feature space of corresponding keypoints in the first target hand image and the second target hand image, y n is the label distance for the pair of corresponding keypoints. 4.The grip posture recognition model training method of any one of claims 1 to 3, wherein, The preset model also includes: a third convolutional neural network model. The three convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract high-level semantic features of the input image. After obtaining the image training group, this method further includes: Based on the image training group, an incorrect image training set is obtained; the image training set contains only images of incorrect pen-holding postures, and each incorrect pen-holding posture image has at least one error label; the error labels are pre-labeled in each incorrect pen-holding posture image; The third convolutional neural network model is trained based on the image training set. 5.The method of claim 4, wherein, The training of the third convolutional neural network model based on the image training set includes: Based on the image training set, obtain images of pen-holding postures; Based on the third convolutional neural network model, the second feature distance of the pen-holding posture image is obtained; If the second feature distance does not meet the second preset condition, then the second optimized loss is obtained based on the feature distance; Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

6. A pen grip posture recognition method characterized by comprising: include: Acquire the image of the gesture to be recognized; The gesture image to be recognized is input into the first target model to obtain the gesture recognition result; The first target model is trained by the method according to any one of claims 1 to 5.

7. The pen grip posture recognition method according to claim 6, characterized in that, After obtaining the gesture recognition result, this method further includes: If the gesture recognition result is incorrect, the gesture image to be recognized is input into the second target model to obtain the incorrect gesture category; the second target model is trained according to the method described in claim 4 or 5.

8. A grip posture recognition model training apparatus, characterized by comprising: include: The first acquisition unit is used to acquire an image training group, which includes multiple images of correct pen-holding postures and multiple images of incorrect pen-holding postures. as well as, Based on the image training group, training image pairs are obtained, each training image pair including two pen-holding posture images, and at least one pen-holding posture image represents a correct pen-holding posture. The first processing unit is used to acquire high-level semantic features from two pen-holding posture images respectively. The high-level semantic features are obtained by a preset model based on the pen-holding posture images. The high-level semantic features include at least the coordinate positions of various key points in the hand region of the pen-holding posture image. The preset model includes at least a first convolutional neural network model and a second convolutional neural network model, with the two convolutional neural network models sharing weight parameters. Each convolutional neural network model is used to extract the high-level semantic features of the input image. Based on the high-level semantic features of two pen-holding posture images, the first feature distance of the training image pair is obtained; and, Based on the first feature distance, obtain the first optimized loss; and, If the first optimization loss does not meet the first threshold, then based on the first optimization loss, the preset model is updated until the first preset condition is met, and the preset model after meeting the first preset condition is used as the first target model. The first processing unit is further configured to preprocess two pen-holding posture images in the training image pair to obtain a first target hand image corresponding to the correct pen-holding posture image and a second target hand image corresponding to the other pen-holding posture image. The first acquisition unit is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image, respectively; The first processing unit is further configured to obtain a first feature distance of the training image pair based on the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image; The first processing unit is further configured to acquire a first target hand image based on the correct pen-holding posture image; The first acquisition unit is further configured to acquire the coordinate positions of each key point in the hand region of the first target hand image and another pen-holding posture image; The first processing unit is further configured to adjust the size of the other pen-holding posture image based on the coordinate positions of each key point of the hand region in the first target hand image and the other pen-holding posture image, so as to reduce the area difference between the hand region in the other pen-holding posture image and the hand region in the first target hand image, so as to obtain a second hand image. The first processing unit is further configured to obtain a second target hand image based on the second hand image, so that the first target hand image and the second target hand image are the same size; The first acquisition unit is further configured to acquire the maximum distance D1 between each key point in the hand region of the first target hand image, and the slope k1 between the two key points forming the maximum distance D1; The first processing unit is further configured to obtain a slope k2 based on the slope k1, wherein k1 k2 = -1; and, Based on the slope k2, the maximum distance D2 between each key point in the hand region of the first target hand image along the slope k2 direction is obtained; The first acquisition unit is further configured to acquire the maximum distance D3 between each key point in the hand region of the other pen-holding posture image, and the slope k3 between the two key points forming the maximum distance D3; The first processing unit is further configured to obtain a slope k4 based on the slope k3, wherein k3 k4=-1; and, Based on the slope k4, obtain the maximum distance D4 between each key point in the hand region of the first target hand image along the direction of the slope k4; and... Based on the ratio of D1 and D3, the size of the other pen-holding posture image along the k3 direction is adjusted proportionally; based on the ratio of D2 and D4, the size of the other pen-holding posture image along the k4 direction is adjusted proportionally; the adjusted pen-holding posture image is obtained as the second hand image.

9. The pen grip posture recognition model training device according to claim 8, characterized in that, The first processing unit is further configured to map the coordinate positions of each key point in the hand region of the first target hand image and the second target hand image to the same feature space; as well as, Based on each key point in the feature space, a preset norm distance is obtained. This preset norm distance includes at least: the distance between corresponding key points in the first target hand image and the second target hand image within the feature space; and... Based on the preset norm distance, a first feature distance is obtained for the training image pair. The step of obtaining a first optimization loss based on the first feature distance includes: Where N is the logarithm of the corresponding key points of the first target hand image and the second target hand image in the feature space, and y pn The distance y between corresponding key points in the first target hand image and the second target hand image in the feature space. n This represents the label distance for the corresponding keypoint pair. 10.The grip posture recognition model training apparatus of claim 8 or 9, wherein The preset model also includes: a third convolutional neural network model. The three convolutional neural network models share weight parameters, and each convolutional neural network model is used to extract high-level semantic features of the input image. The first acquisition unit is further configured to acquire an erroneous image training set based on the image training group; the image training set contains only images of incorrect pen-holding postures, and each image of an incorrect pen-holding posture has at least one error label; the error labels are pre-labeled in each image of an incorrect pen-holding posture. The first processing unit is further configured to train the third convolutional neural network model based on the image training set.

11. The pen grip posture recognition model training device according to claim 10, characterized in that, The first acquisition unit is further configured to acquire pen-holding posture images based on the image training set; The first processing unit is further configured to obtain a second feature distance of the pen-holding posture image based on the third convolutional neural network model; as well as, If the second feature distance does not meet the second preset condition, then a second optimized loss is obtained based on the feature distance; and, Based on the second optimization loss, the third convolutional neural network model is updated, and a new pen-holding posture image is obtained until the second feature distance meets the second preset condition. The third convolutional neural network model when the second feature distance meets the second preset condition is used as the second target model.

12. A grip posture recognition device characterized by comprising: include: The second acquisition unit is used to acquire the gesture image to be recognized; The second processing unit is used to input the gesture image to be recognized into the first target model to obtain the gesture recognition result; The first target model is trained by the method according to any one of claims 1 to 5.

13. The pen grip posture recognition device according to claim 12, characterized in that, The second processing unit is further configured to, if the gesture recognition result is incorrect, input the gesture image to be recognized into a second target model to obtain an incorrect gesture category; the second target model is trained according to the method described in claim 4 or 5.

14. A pen grip posture recognition device, characterized by comprising: include: A memory and a processor; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the pen grip posture recognition model training method as described in any one of claims 1 to 5, or the pen grip posture recognition method as described in claim 6 or 7.

15. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed, it can implement the pen grip posture recognition model training method according to any one of claims 1 to 5, or the pen grip posture image recognition method according to claim 6 or 7.

Images