Pen grip posture correction methods, devices, storage media and electronic equipment

Abstract

This disclosure relates to a method, apparatus, storage medium, and electronic device for correcting pen grip posture. It corrects pen grip posture by recognizing gesture categories in an image and generating reminder information. The method includes: acquiring an image to be recognized; inputting the image to be recognized into a hand keypoint detection model to obtain hand keypoint images output by the hand keypoint detection model based on a convolutional structure; inputting each hand keypoint image into a gesture recognition network model to obtain the gesture category output by the gesture recognition network model based on a recognition function; and generating reminder information when the gesture category is incorrect. This disclosure uses a hand keypoint detection model to acquire hand keypoint images corresponding to each hand keypoint in the image to be recognized, uses a gesture recognition network model to identify the gesture category based on the hand keypoint images, and generates reminder information when the gesture category is incorrect to remind the user to adjust their pen grip posture, thereby correcting the pen grip posture.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and more specifically, to a pen-holding posture correction method, device, storage medium, and electronic device. Background Technology

[0002] Incorrect pen-holding posture in teenagers can lead to poor sitting posture, resulting in conditions such as hunchback and nearsightedness. Holding the pen low increases the friction between the pen tip and the paper, requiring excessive force to write, resulting in slow writing speed. Prolonged low-grip writing can also cause arm pain, affecting learning efficiency. Furthermore, writing with excessive force results in thick, stiff strokes, unsightly characters, and poor handwriting, which can negatively impact a child's self-confidence.

[0003] In existing technologies, pen grip correctors are mainly used to correct teenagers' pen grip posture. However, when there is no parental supervision, teenagers may not use pen grip correctors and write directly, which makes it impossible for pen grip correctors to correct teenagers' pen grip posture. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method, apparatus, storage medium, and device for correcting pen grip posture, which corrects pen grip posture by recognizing the type of gesture in an image and generating reminder information.

[0005] To achieve the above objectives, in a first aspect, this disclosure provides a method for correcting pen-holding posture, the method comprising:

[0006] Acquire the image to be recognized;

[0007] The image to be identified is input into the hand key point detection model to obtain the hand key point image output by the hand key point detection model based on the convolution structure;

[0008] Each of the aforementioned key hand point images is input into the gesture recognition network model to obtain the gesture category output by the gesture recognition network model based on the recognition function;

[0009] If the gesture category is "gesture error", a reminder message will be generated.

[0010] Optionally, the hand key point detection model includes a first sampling network, a scaling module, and a second sampling network;

[0011] The first sampling network is used to obtain a first number of candidate images of hand key points in the image to be identified based on the first convolutional structure;

[0012] The scaling module is used to scale the image to be recognized to obtain a second number of scaled images;

[0013] The second sampling network is used to obtain a final hand key point image based on the first number of hand key point candidate images and the second number of scaled images using the second convolutional structure. The hand key point image includes hand key point images other than the first number of hand key point candidate images identified from the second number of scaled images based on the second number of scaled images and the first number of hand key point candidate images.

[0014] Optionally, the first sampling network is used to acquire multiple initial images of hand key points based on the first convolutional structure;

[0015] Based on the confidence level of each of the initial images of the hand key points, a first number of candidate images of hand key points are determined from the plurality of initial images of hand key points in descending order of confidence level.

[0016] Optionally, the first convolutional structure includes three convolutional layers with a stride of 2, wherein each convolutional layer is 3*3*6, 3*3*12, and 3*3*21 respectively.

[0017] Optionally, the second convolutional structure includes three convolutional layers with a stride of 1, wherein each convolutional layer is 3*3*14, 3*3*18, and 3*3*21 respectively.

[0018] Optionally, the gesture recognition network model includes a connection matrix and a recognition function;

[0019] The connection matrix is ​​used to generate a gesture image based on each of the hand key point images;

[0020] The recognition function is used to output a gesture category based on the gesture image, and the gesture category includes gesture error and gesture correctness.

[0021] Optionally, the connection matrix is ​​used to generate a gesture image based on each of the hand keypoint images, including:

[0022] The connection matrix obtains the connection relationship of the hand key points in each hand key point image based on the positional relationship between the target hand key point and the hand key points in each hand key point image, and generates a gesture image. The target hand key point includes the hand key point of any hand key point image in each hand key point image.

[0023] Optionally, the hand key points in the connection matrix whose connection value with the target hand key point is a first preset value are the adjacent points of the target hand key point;

[0024] The hand key points whose connection value to the target hand key points is a second preset value are the interval points of the target hand key points;

[0025] The first preset value is greater than the second preset value.

[0026] Optionally, the recognition function is obtained in the following manner:

[0027] Summing the values ​​of each row of the connection matrix yields the sum of the values ​​corresponding to each row of the connection matrix.

[0028] Construct a diagonal matrix using the sums of the given values ​​as diagonals;

[0029] The difference between the diagonal matrix and the connection matrix is ​​taken as the Laplace matrix;

[0030] The recognition function is obtained by diagonally expanding the Laplacian matrix and introducing preset parameters.

[0031] Secondly, this disclosure provides a pen-holding posture correction device, the device comprising:

[0032] The acquisition module is configured to acquire the image to be recognized;

[0033] The extraction module is configured to input the image to be identified into the hand key point detection model to obtain the hand key point image output by the hand key point detection model based on the convolution structure;

[0034] The recognition module is configured to input each of the hand key point images into the gesture recognition network model to obtain the gesture category output by the gesture recognition network model based on the recognition function;

[0035] The reminder module is configured to generate a reminder message when the gesture category is a gesture error.

[0036] Thirdly, this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.

[0037] Fourthly, this disclosure provides an electronic device, comprising:

[0038] A memory on which computer programs are stored;

[0039] A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect.

[0040] Through the above technical solution, the hand key point detection model in this disclosure obtains multiple hand key point images in the image to be recognized based on the convolution structure. The gesture recognition network obtains the gesture category based on the recognition function according to each hand key point image. When the gesture category is a gesture error, a reminder message is generated only from the image of the teenager in the learning state, so as to correct the teenager's pen-holding posture.

[0041] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a flowchart illustrating a pen-holding posture correction method according to exemplary embodiments of the present disclosure;

[0044] Figure 2 This is a schematic diagram illustrating an existing pen grip posture recognition according to exemplary embodiments of the present disclosure;

[0045] Figure 3 This is a schematic diagram illustrating a pen grip posture recognition according to an exemplary embodiment of the present disclosure;

[0046] Figure 4 This is a schematic diagram of key points of a hand according to exemplary embodiments of the present disclosure;

[0047] Figure 5 This is a schematic diagram of connecting key points of a hand according to an exemplary embodiment of the present disclosure;

[0048] Figure 6 This is a hand gesture image illustrating a pen-holding posture according to exemplary embodiments of the present disclosure;

[0049] Figure 7 This is a block diagram illustrating a pen grip posture correction device according to exemplary embodiments of the present disclosure;

[0050] Figure 8 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0051] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0052] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0053] As mentioned in the background section, teenagers mainly use pen grip correctors to correct their pen grip posture. However, when parents are not supervising, teenagers may write directly without using the pen grip corrector, which would prevent the pen grip corrector from correcting the teenager's pen grip posture.

[0054] In view of this, the present disclosure provides a pen grip posture correction method, device, storage medium and electronic device, which generates reminder information by recognizing the gesture category in the image to remind the user to correct the pen grip posture according to the reminder information.

[0055] Figure 1 This is a flowchart illustrating a pen-holding posture correction method according to an exemplary embodiment of the present disclosure. Taking the application of this method to devices with cameras, such as learning machines, laptops, and tablets, as an example, refer to... Figure 1 The method includes the following steps:

[0056] In step S101, the image to be identified is acquired.

[0057] Specifically, the images to be identified include images of the user's hands captured by the learning machine's camera, or half-body images of the user showing the state of their hands.

[0058] In step S102, the image to be identified is input into the hand key point detection model to obtain the hand key point image output by the hand key point detection model based on the convolution structure.

[0059] Specifically, the hand key point detection model identifies hand key points in the image to be identified based on a convolutional structure and outputs a heatmap of each hand key point, i.e., a hand key point image.

[0060] In step S103, the images of each hand key point are input into the gesture recognition network model to obtain the gesture category output by the gesture recognition network model based on the recognition function.

[0061] Specifically, the gesture recognition network model identifies the gesture category corresponding to each key hand point image based on the recognition function and outputs the result.

[0062] In step S104, if the gesture category is gesture error, a reminder message is generated.

[0063] Specifically, the reminder information generated in the event of incorrect gestures includes information on how to correct the user's pen grip posture. For example, if the current pen grip position is too close to the writing surface, the reminder information includes prompting the user to adjust the pen grip position; if the current pen grip posture, including the thumb, index finger, and middle finger, does not conform to the standard pen grip posture, the reminder information includes prompting the user to adjust the posture of the thumb, index finger, and middle finger, and so on.

[0064] The pen-holding posture correction method disclosed herein acquires an image containing the user's hand state as the image to be recognized. A hand key point detection model identifies the images corresponding to each key point of the hand in the image to be recognized and uses them as hand key point images. A gesture recognition network model outputs a gesture category based on each hand key point image. If the gesture category is incorrect, a reminder message is generated to remind the user to correct the pen-holding posture of the teenager. This disclosure does not require a separate pen-holding correction device, reducing detection costs. Pen-holding posture correction can be achieved through an electronic device with a camera function, without the need for constant parental supervision.

[0065] To enable those skilled in the art to better understand the pen grip posture correction method provided in this disclosure, the steps of the above method are illustrated in detail below.

[0066] Reference Figure 2 Traditional hand keypoint detection models directly identify the image to be recognized to obtain heatmaps of each hand keypoint. However, in real-world scenarios, many hand keypoints in the image to be recognized are occluded. Traditional hand keypoint detection models cannot identify occluded hand keypoints, resulting in unreliable output results.

[0067] In one feasible embodiment, refer to Figure 3 The hand key point detection model includes a first sampling model, a scaling module, and a second sampling network;

[0068] The first sampling model is used to obtain a first number of candidate images of hand key points in the image to be identified based on the first convolutional structure;

[0069] The scaling module is used to scale the image to be recognized to obtain a second number of scaled images;

[0070] The second sampling network is used to obtain the final hand keypoint image based on the second convolutional structure for a first number of hand keypoint candidate images and a second number of scaled images. The hand keypoint image includes hand keypoint images other than the first number of keypoint candidate images identified from the second number of scaled images based on the second number of scaled images and the first number of hand keypoint candidate images.

[0071] Specifically, both the first sampling model and the second sampling model are neural network models. The first sampling model obtains a first number of candidate hand keypoint images, including the hand keypoint images corresponding to the unoccluded hand keypoints in the image to be identified. The second sampling model obtains the final hand keypoint images, including the hand keypoint images corresponding to the occluded hand keypoints in the image to be identified.

[0072] Specifically, the training process of the first sampling model includes acquiring a first training sample, which includes hand keypoint images labeled with each keypoint at an unoccluded location, and the labeling method is as follows: Figure 4 As shown, the first training sample is input into the first initial sampling model until the first initial sampling model can output the hand key point image corresponding to the unoccluded hand key point, and the first initial sampling model is used as the first sampling model.

[0073] Among them, reference Figure 4 The annotation method involves setting 21 key hand points, including the interphalangeal joints and fingertips of the thumb, index finger, middle finger, ring finger, and little finger, as well as the wrist. The wrist is designated as the first key hand point, numbered 0. The fingertips of the thumb are then numbered sequentially from bottom to top as 1, 2, 3, and 4. Similarly, the interphalangeal joints and fingertips of the index, middle, ring, and little fingers are numbered sequentially from bottom to top, from 5 to 20. The result is as follows: Figure 4 The annotation results shown do not distinguish between left and right hands.

[0074] Specifically, the training process of the second sampling model includes: acquiring second training samples, which include thumbnails of the target image after resizing. The target image includes, for example,... Figure 5 The image shown includes the line connections between key points, and candidate hand key points corresponding to key points whose locations are occluded. The annotation method is as follows: Figure 4 As shown, each thumbnail is associated with each candidate image of hand key points; the second training sample is input into the second initial sampling model until the second initial sampling model can output the hand key point image corresponding to the occluded hand key points, and the second initial sampling model is used as the second sampling model.

[0075] Specifically, by training the first initial sampling model and the second initial sampling model respectively, multiple first sampling models and multiple second sampling models are obtained. Any group of the first sampling models and the second sampling models is combined to obtain multiple initial hand key point detection models. The overall loss function of each initial hand key point detection model is calculated. The initial hand key point detection model corresponding to the minimum value of the loss function is taken as the target hand key point detection model.

[0076] The calculation of the overall loss function of the initial hand keypoint detection model includes:

[0077] L = L1 + L2;

[0078] Where L1 represents the least squares loss between the output of the first sampling network model and the output of the initial hand keypoint detection model; L2 represents the least squares loss between the output of the initial hand keypoint detection model and the true label value.

[0079] For example, refer to Figure 3 The first sampling model is Network A, and the second sampling model is Network B. The image to be recognized is input into the hand keypoint detection model. Network A obtains candidate images of 10 unoccluded hand keypoints from the image to be recognized. The scaling module resizes the image to be recognized to obtain three scaled images. Figure 3 As shown in the diagram, a 32*32*3 feature map is obtained by concatenating 10 candidate images of hand key points and 3 scaled images together in the third dimension. The second sampling network then obtains the final hand key point image based on the 32*32*13 feature map.

[0080] This disclosure uses neural network A to detect candidate images of unoccluded hand key points in the image to be identified. The candidate images of hand key points are then concatenated with the thumbnail of the resized image to be identified in the third dimension to obtain a 32*32*13 feature map. Neural network B is then used to identify the 32*32*13 feature map to obtain the hand key point images corresponding to occluded hand key points in the image to be identified, thereby improving the recognition accuracy of occluded hand key points in the image to be identified.

[0081] In one feasible embodiment, the first sampling network is used to acquire initial images of multiple hand key points based on a first convolutional structure;

[0082] Based on the confidence level of each initial image of the hand keypoint, candidate images of the hand keypoints are determined from multiple initial images of the hand keypoints in descending order of confidence level, excluding a first number of candidate images of the hand keypoints.

[0083] For example, refer to Figure 3 Network A uses the stride of the first convolutional structure to sample from the image to be identified, obtaining 21 initial candidate images of hand key points. Based on the ranking of the confidence of each candidate image of hand key points from high to low, the top 10 initial candidate images of hand key points with the highest confidence are determined as candidate images of hand key points.

[0084] This disclosure uses the top 10 initial hand keypoint candidate images with the highest confidence output of neural network A as input to neural network B, thereby improving the accuracy of the recognition results of neural network B.

[0085] In one feasible embodiment, the first convolutional structure includes three convolutional layers with a stride of 2, wherein each convolutional layer is 3*3*6, 3*3*12, and 3*3*21 respectively.

[0086] Specifically, neural network A uses a first convolutional structure with a large stride to quickly identify unoccluded key points of the hand from the image to be recognized.

[0087] In one feasible embodiment, the second convolutional structure includes three convolutional layers with a stride of 1, wherein the convolutional layers are 3*3*14, 3*3*18, and 3*3*21 respectively.

[0088] Specifically, neural network B uses a second convolutional structure with a smaller stride to improve the accuracy of identifying occluded hand key points from the image to be recognized.

[0089] In one feasible embodiment, the gesture recognition network model includes a connection matrix and a recognition function;

[0090] The connection matrix is ​​used to generate a gesture image based on each hand key point image;

[0091] The recognition function is used to output the gesture category based on the gesture image. The gesture category includes incorrect gesture and correct gesture.

[0092] Specifically, since the output of the hand keypoint detection model is a hand keypoint image corresponding to 21 keypoints, the connection matrix is ​​a 21*21 matrix.

[0093] This disclosure generates a gesture image by connecting the images of key hand points using a connection matrix, and then outputs the gesture category based on the gesture image using a recognition function, thereby realizing gesture recognition in a convenient and quick manner.

[0094] In one feasible embodiment, the connection matrix is ​​used to generate a gesture image based on each hand keypoint image, including:

[0095] The connection matrix obtains the connection relationship between the hand key points in each hand key point image based on the positional relationship between the target hand key points and the hand key points in each hand key point image, and generates a gesture image. The target hand key points include the hand key points in any hand key point image.

[0096] For example, refer to Figure 6A planar coordinate system is constructed with the hand keypoint numbered 0 as the origin. The coordinate information of each of the remaining hand keypoints in this planar coordinate system is determined. A 21*2 matrix is ​​constructed according to the numbering order.

[0097]

[0098] One image to be recognized corresponds to 21 hand key points. Therefore, the connection matrix A is a 21*21 matrix with an initial parameter of 0. The coordinate information corresponding to the 21 hand key points is filled into the connection matrix and the connection values ​​between each hand key point are set to generate a gesture image.

[0099] In one feasible embodiment, the hand key points in the connection matrix whose connection value with the target hand key point is a first preset value are the adjacent points of the target hand key point;

[0100] The hand key points whose connection value to the target hand key points is the second preset value are the interval points of the target hand key points;

[0101] The first preset value is greater than the second preset value.

[0102] For example, taking the hand keypoint numbered 8 as an example, the connection value between the hand keypoint numbered 7, which is adjacent to the hand keypoint numbered 8, is set to 2, and the connection value between the hand keypoint numbered 6, which is separated from the hand keypoint numbered 8, is set to 1, resulting in the following... Figure 6 The gesture image shown.

[0103] In one feasible embodiment, the identification function is obtained as follows:

[0104] Summing the values ​​of each row of the connection matrix yields the sum of the values ​​for each row.

[0105] Construct a diagonal matrix using the sums as diagonals;

[0106] The difference between the diagonal matrix and the link matrix is ​​taken as the Laplace matrix;

[0107] The recognition function is obtained by diagonally expanding the Laplacian matrix and introducing preset parameters.

[0108] For example, consider a 3x3 connection matrix A:

[0109]

[0110] Summing the values ​​of each row of the connection matrix A yields a sum of 1, 2, and 1. Using 1, 2, and 1 as the diagonal, construct a diagonal matrix D:

[0111]

[0112] Then the Laplace matrix L = DA:

[0113]

[0114] Diagonally expanding the Laplace matrix L yields L = UΛU T , where Λ represents the eigenvector of L.

[0115] Introducing a parameter θ such that if the input is x, the output is y, we get y = θ (L(*,where θ only depends on Λ, for y= θ (L)* is transformed to obtain y=Uf θ (Λ)U T *, thus obtaining the recognition function f θ (L)=∑0.9 k θ k L k Where, the higher the power of L, the greater the f θ (L) The smaller the weight, the more accurately the gesture recognition network model can extract local features, thus improving the accuracy of its recognition results.

[0116] This disclosure obtains hand key point images corresponding to each hand key point in the image to be identified through a hand key point detection model, identifies the gesture category based on the hand key point images through a gesture recognition network model, and generates a reminder message when the gesture category is incorrect, so as to remind the user to adjust the pen-holding posture and achieve pen-holding posture correction.

[0117] Based on the same inventive concept, this disclosure also provides a pen-holding posture correction device, such as... Figure 7 As shown, the device 700 includes an acquisition module 701, an extraction module 702, an identification module 703, and an alert module 704.

[0118] The acquisition module 701 is configured to acquire the image to be recognized.

[0119] The extraction module 702 is configured to input the image to be recognized into the hand keypoint detection model to obtain the hand keypoint image output by the hand keypoint detection model based on the convolution structure.

[0120] The recognition module 703 is configured to input images of each hand key point into the gesture recognition network model to obtain the gesture category output by the gesture recognition network model based on the recognition function.

[0121] The reminder module 704 is configured to generate a reminder message when the gesture category is gesture error.

[0122] This disclosure obtains hand key point images corresponding to each hand key point in the image to be identified through a hand key point detection model, identifies the gesture category based on the hand key point images through a gesture recognition network model, and generates a reminder message when the gesture category is incorrect, so as to remind the user to adjust the pen-holding posture and achieve pen-holding posture correction.

[0123] Furthermore, the device 700 includes a detection module, which is equipped with a hand key point detection model. The hand key point detection model includes a first sampling network, a scaling module, and a second sampling network.

[0124] The first sampling network includes a first convolutional structure, which is configured to acquire a first number of candidate images of hand key points in the image to be recognized.

[0125] The scaling module is configured to scale the image to be recognized to obtain a second number of scaled images;

[0126] The second sampling network includes a second convolutional structure configured to obtain a final hand keypoint image from a first number of hand keypoint candidate images and a second number of scaled images. The hand keypoint image includes hand keypoint images other than the first number of hand keypoint candidate images identified from the second number of scaled images based on the second number of scaled images and the first number of hand keypoint candidate images.

[0127] Furthermore, the first convolutional structure is used to obtain initial images of multiple hand key points;

[0128] Based on the confidence level of each of the initial images of the hand key points, a first number of candidate images of hand key points are determined from the plurality of initial images of hand key points in descending order of confidence level.

[0129] Furthermore, the first convolutional structure includes three convolutional layers with a stride of 2, namely 3*3*6, 3*3*12, and 3*3*21 respectively.

[0130] Furthermore, the second convolutional structure includes three convolutional layers with a stride of 1, namely 3*3*14, 3*3*18, and 3*3*21 respectively.

[0131] Furthermore, the gesture recognition network model includes a connection matrix and a recognition function module;

[0132] The connection matrix is ​​configured to generate a gesture image based on each of the hand keypoint images;

[0133] The recognition function module is configured to output a gesture category based on the gesture image, which includes gesture error and gesture correctness.

[0134] Furthermore, the connection matrix is ​​configured to generate a gesture image based on each of the said hand keypoint images, including:

[0135] The connection matrix obtains the connection relationship between the hand key points in each hand key point image based on the positional relationship between the target hand key points and the hand key points in each hand key point image, and generates a gesture image. The target hand key points include the hand key points in any hand key point image.

[0136] Furthermore, in the connection matrix, the hand key points with a connection value of the first preset value to the target hand key points are the adjacent points of the target hand key points;

[0137] The hand key points whose connection value to the target hand key points is the second preset value are the interval points of the target hand key points;

[0138] The first preset value is greater than the second preset value.

[0139] Furthermore, the recognition function module is constructed in the following way:

[0140] Summing the values ​​of each row of the connection matrix yields the sum of the values ​​corresponding to each row of the connection matrix.

[0141] Construct a diagonal matrix using the sums of the given values ​​as diagonals;

[0142] The difference between the diagonal matrix and the connection matrix is ​​taken as the Laplace matrix;

[0143] The recognition function is obtained by diagonally expanding the Laplacian matrix and introducing preset parameters.

[0144] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0145] Based on the same inventive concept, this disclosure also provides an electronic device, comprising:

[0146] A processor, on which computer programs are stored;

[0147] A processor is used to execute computer programs stored in memory to implement the steps of a pen grip posture correction method.

[0148] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example... Figure 8As shown, the electronic device 800 may include a processor 801 and a memory 802. The electronic device 800 may also include one or more of a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805.

[0149] The processor 801 controls the overall operation of the electronic device 800 to complete all or part of the steps in the pen-holding posture correction method described above. The memory 802 stores various types of data to support the operation of the electronic device 800. This data may include, for example, instructions for any application or method operating on the electronic device 800, and application-related data, such as images to be recognized, gesture images, gesture categories, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 802 or transmitted via communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 805 is used for wired or wireless communication between the electronic device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 805 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0150] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the pen-holding posture correction method described above.

[0151] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the pen grip posture correction method described above. For example, the computer-readable storage medium may be the memory 802 including the program instructions described above, which may be executed by the processor 801 of the electronic device 800 to complete the pen grip posture correction method described above.

[0152] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0153] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0154] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method of correcting a pen-holding posture, characterized by, The method comprises: acquiring an image to be recognized; inputting the image to be recognized into a hand key point detection model to obtain a hand key point image output by the hand key point detection model based on a convolution structure; inputting each hand key point image into a gesture recognition network model to obtain a gesture category output by the gesture recognition network model based on a recognition function; generating a prompt information in a case that the gesture category is a gesture error; the hand key point detection model comprises a first sampling network, a scaling module and a second sampling network; the first sampling network is configured to acquire a first number of hand key point candidate images in the image to be recognized based on a first convolution structure; the scaling module is configured to perform scaling processing on the image to be recognized to obtain a second number of scaled images; the second sampling network is configured to acquire a final hand key point image based on a second convolution structure for the first number of hand key point candidate images and the second number of scaled images, wherein the hand key point image comprises a hand key point image other than the first number of hand key point candidate images identified from the second number of scaled images based on the second number of scaled images and the first number of hand key point candidate images.

2. The method of claim 1, wherein, the first sampling network is configured to acquire a plurality of hand key point initial images based on the first convolution structure; the first number of hand key point candidate images are determined from the plurality of hand key point initial images from high to low according to the confidence of each hand key point initial image.

3. The method of claim 1, wherein, the first convolution structure comprises three convolution layers with a step of 2, and each convolution layer is 3×3×6, 3×3×12 and 3×3×21 in turn.

4. The method of claim 1, wherein, the second convolution structure comprises three convolution layers with a step of 1, and each convolution layer is 3×3×14, 3×3×18 and 3×3×21 in turn.

5. The method of claim 1, wherein, the gesture recognition network model comprises a connection matrix and a recognition function; the connection matrix is configured to generate a gesture image according to each hand key point image; the recognition function is configured to output a gesture category according to a gesture image, wherein the gesture category comprises a gesture error and a gesture correct.

6. The method of claim 5, wherein, the connection matrix is configured to generate a gesture image according to each hand key point image, comprising: the connection matrix obtains a connection relationship of hand key points in each hand key point image according to a position relationship of a target hand key point and the hand key points in each hand key point image to generate a gesture image, wherein the target hand key point comprises a hand key point in any hand key point image.

7. The method of claim 6, wherein, a hand key point in the connection matrix with a connection value of the target hand key point as a first preset value is a neighboring point of the target hand key point; a hand key point in the connection matrix with a connection value of the target hand key point as a second preset value is a spaced point of the target hand key point; the first preset value is greater than the second preset value.

8. The method of claim 7, wherein, the recognition function is obtained by: summing each row value of the connection matrix to obtain a sum value corresponding to each row of the connection matrix; constructing a diagonal matrix by taking each sum value as a diagonal line; obtaining a difference between a diagonal matrix and the connection matrix as a Laplacian matrix; diagonally unfolding the Laplacian matrix and introducing a preset parameter to obtain the identification function.

9. A pen-holding posture correcting device characterized by comprising: The device comprises: an acquisition module configured to acquire an image to be identified; an extraction module configured to input the image to be identified into a hand key point detection model to obtain a hand key point image output by the hand key point detection model based on a convolution structure; an identification module configured to input each hand key point image into a gesture identification network model to obtain a gesture category output by the gesture identification network model based on an identification function; a reminding module configured to generate reminding information in a case where the gesture category is a gesture error; The device further comprises a detection module, and the detection module is provided with a hand key point detection model. The hand key point detection model comprises a first sampling network, a scaling module and a second sampling network. The first sampling network comprises a first convolution structure configured to acquire a first number of hand key point candidate images in an image to be identified. The scaling module is configured to perform scaling processing on the image to be identified to obtain a second number of scaled images. The second sampling network comprises a second convolution structure configured to acquire a final hand key point image for the first number of hand key point candidate images and the second number of scaled images. The hand key point image comprises a hand key point image other than the first number of hand key point candidate images, which is identified from the second number of scaled images based on the second number of scaled images and the first number of hand key point candidate images.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-8.

11. An electronic device, comprising: comprise: a memory having a computer program stored thereon; a processor configured to execute the computer program in the memory to implement the steps of the method of any one of claims 1-8.

Images