Pen holding posture recognition method and device, storage medium and equipment
By using the position and posture category information of the target video frame in pen-holding posture recognition, combining the tracker and classification model, and processing only the key frames, the problem of high computing resource consumption in the existing technology is solved, and efficient and accurate pen-holding posture recognition is achieved.
Patent Information
- Application Number
- CN202410329596.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-21
- Publication Date
- 2025-09-23
AI Technical Summary
Existing pen-holding posture recognition methods require recognition of each frame of video, resulting in excessive consumption of computing resources and low recognition efficiency.
By obtaining the position information and posture category information of the pen-holding hand in the target video frame, the tracker is used to track the subsequent N video frames. Combined with the pen-holding posture classification model, only key frames are identified to reduce computing resource consumption.
The efficiency and accuracy of pen-holding posture recognition are improved, and the computing resource consumption of the device is reduced.
Smart Images
Figure CN120689922A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and equipment for recognizing pen-holding posture. Background Art
[0002] Bad pen-holding posture can cause people to have many problems such as hunchback, myopia, hand muscle pain, etc. Long-term use of incorrect pen-holding posture can easily have adverse effects on the growth and development of the user's hands, and pen-holding posture also greatly affects the beauty of a person's handwriting. Therefore, timely identification of the user's pen-holding posture can help the user quickly correct the pen-holding posture and improve the user's writing ability.
[0003] Existing pen-holding posture recognition methods usually use a pen-holding posture recognition model to identify each frame of video, and determine the user's pen-holding posture based on the pen-holding posture recognition results of all video frames. The above recognition method requires a large amount of computing resources and has low recognition efficiency. Summary of the Invention
[0004] The embodiments of the present application provide a method, apparatus, storage medium, and device for pen-holding posture recognition, which can improve the efficiency and accuracy of pen-holding posture recognition.
[0005] In a first aspect, an embodiment of the present application provides a method for identifying a pen-holding posture, comprising the following steps:
[0006] In response to a pen-holding gesture recognition instruction, acquiring a target video;
[0007] identifying a target video frame in which the pen-holding behavior exists in the target video based on the pen-holding behavior recognition model, and acquiring first position information and first pen-holding posture category information of the target video frame;
[0008] Initializing a tracker based on the first position information and the first pen-holding posture category information, and using the tracker to track second position information of the pen-holding hand in N video frames following the target video frame; wherein N>1;
[0009] acquiring second pen-holding posture category information of the N video frames based on the pen-holding posture classification model and the second position information;
[0010] The pen-holding posture is determined according to the first pen-holding posture category information and the second pen-holding posture category information.
[0011] In a second aspect, an embodiment of the present application provides a device for recognizing a pen-holding posture, the device comprising:
[0012] A target video acquisition module, configured to acquire a target video in response to a pen-holding posture recognition instruction;
[0013] a pen-holding behavior recognition module, configured to identify a target video frame in which a pen-holding behavior is present in the target video based on a pen-holding behavior recognition model, and obtain first position information and first pen-holding posture category information of the target video frame;
[0014] a tracking module, configured to initialize a tracker based on the first position information and the first pen-holding posture category information, and use the tracker to track second position information of the pen-holding hand in N video frames following the target video frame; wherein N>1;
[0015] a classification module, configured to obtain second pen-holding posture category information of the N video frames based on a pen-holding posture classification model and the second position information;
[0016] The pen-holding posture determining module is configured to determine the pen-holding posture according to the first pen-holding posture category information and the second pen-holding posture category information.
[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the pen-holding posture recognition method as described in any one of the above are implemented.
[0018] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor;
[0019] When the processor executes the computer program, the steps of the pen-holding posture recognition method as described in any one of the above are implemented.
[0020] In an embodiment of the present application, a target video is obtained by responding to a pen-holding posture recognition instruction; a target video frame in which a pen-holding behavior exists in the target video is identified based on a pen-holding behavior recognition model, and first position information of the pen-holding hand and first pen-holding posture category information of the target video frame are obtained; a tracker is initialized based on the first position information and the first pen-holding posture category information, and the tracker is used to track the second position information of the pen-holding hand of N video frames after the target video frame; and second pen-holding posture category information of the N video frames is obtained based on a pen-holding posture classification model. Compared with the prior art, the present application does not need to perform pen-holding posture recognition on all video frames of the target video, and does not need to use a pen-holding posture recognition model that consumes a large amount of computing resources for each video frame, which can effectively reduce the consumption of computing resources of the device and improve the efficiency of pen-holding posture recognition. At the same time, the present application determines the pen-holding posture based on the pen-holding posture category information of the target video frame and the subsequent N video frames, thereby improving the accuracy of pen-holding posture recognition.
[0021] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a flow chart of a method for recognizing a pen-holding posture according to an embodiment of the present invention;
[0023] Figure 2 This is a flow chart of a method for recognizing a pen-holding posture according to another embodiment of the present invention;
[0024] Figure 3 is a schematic diagram of a non-standard writing posture in one embodiment of the present invention;
[0025] Figure 4 is a schematic diagram of a non-standard writing posture in another embodiment of the present invention;
[0026] Figure 5 This is a flow chart of a method for recognizing a pen-holding posture according to another embodiment of the present invention;
[0027] Figure 6 is a schematic diagram of an abnormal pen-holding posture according to an embodiment of the present invention;
[0028] Figure 7 This is a flow chart of a method for recognizing a pen-holding posture according to another embodiment of the present invention;
[0029] Figure 8 This is a schematic structural diagram of a device for recognizing pen-holding posture according to an embodiment of the present invention;
[0030] Figure 9 The figure is a schematic structural diagram of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
[0032] It should be clear that the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.
[0033] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0034] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.
[0035] In addition, in this application, unless otherwise specified, "several" refers to two or more. "And / or" describes a corresponding relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0036] In related technologies, when identifying the user's pen-holding posture in a video, it is usually necessary to combine hardware devices and software to realize the recognition of the user's pen-holding posture. The hardware devices usually include a stylus and a wearable device. The hardware devices collect hand signals through sensors to identify the pen-holding posture. The software analyzes the key points of the fingers based on vision and models the hand posture to identify the pen-holding posture.
[0037] The hardware equipment needs to be equipped with special sensors and gyroscopes, which are relatively expensive; the software is easily blocked by fingers or other external factors during visual recognition, which can easily lead to misrecognition.
[0038] Moreover, in related computer technologies, when identifying the pen-holding posture of a user in a video, a pen-holding posture recognition model is usually used to identify each video frame in the video. However, using the pen-holding posture recognition model for each video frame consumes a lot of computing resources. Therefore, in response to the above problem, the present application proposes a pen-holding posture recognition method, which obtains a target video by responding to a pen-holding posture recognition instruction; identifies a target video frame with a pen-holding behavior in the target video based on a pen-holding behavior recognition model, obtains the first position information and the first pen-holding posture category information of the pen-holding hand in the target video frame, and obtains the first position information and the first pen-holding posture category information of the pen-holding hand based on the first position information and the first pen-holding posture category information. The tracker is initialized with identification information, and the tracker is used to track the second position information of the pen-holding hand in N video frames after the target video frame. The second pen-holding posture category information of the N video frames is obtained based on the pen-holding posture classification model. Compared with the existing technology, the present application does not need to perform pen-holding posture recognition on all video frames of the target video, and does not need to use a pen-holding posture recognition model that consumes a lot of computing resources for each video frame. It can effectively reduce the consumption of computing resources of the device and improve the efficiency of pen-holding posture recognition. At the same time, the present application determines the pen-holding posture according to the pen-holding posture category information of the target video frame and the subsequent N video frames, thereby improving the accuracy of pen-holding posture recognition.
[0039] See also Figure 1 The present application provides a pen-holding posture recognition method. The pen-holding posture recognition method can be applied to a pen-holding posture recognition device or an electronic device equipped with a pen-holding posture recognition device. The method does not require the purchase of specific hardware, thereby reducing the hardware cost of pen-holding posture recognition. The method includes the following steps:
[0040] S110: Responding to the pen-holding posture recognition instruction, acquiring a target video;
[0041] The pen-holding posture recognition instruction is used to control the pen-holding posture recognition device to start the pen-holding posture recognition workflow.
[0042] The target video may be a video captured by a video acquisition device. In the present application, the target video may be a real-time video captured by the video acquisition device in real time.
[0043] S120: Identify a target video frame in which the pen-holding behavior occurs in the target video based on the pen-holding behavior recognition model, and obtain first position information and first pen-holding posture category information of the target video frame;
[0044] There may or may not be a pen-holding behavior in the target video. The pen-holding behavior refers to the behavior of the user holding a pen to write. In this application, the pen-holding behavior can be determined based on whether there is a user or a pen-holding hand in the target video frame, or the pen-holding hand posture of the user in the target video frame. For example, when it is recognized that there is no user in the target video frame, it is determined that there is no pen-holding behavior; when it is recognized that there is a pen-holding hand in the target video frame, it is determined that there is a pen-holding behavior, or when it is recognized that there is a pen-holding hand in the target video frame and the pen-holding hand is in a writing state, it is determined that there is a pen-holding behavior. Based on the pen-holding behavior recognition model, it is determined whether there is a pen-holding behavior in the target video, and the video frame with the pen-holding behavior is used as the target video frame.
[0045] Preferably, the video frames of the target video can be acquired at intervals of a set number of frames to identify whether there is a pen-holding behavior in the video frame, thereby reducing the amount of data for pen-holding behavior recognition and improving the efficiency of pen-holding behavior recognition. The set number of frames can be determined according to the number of frames of the target video. In this application, the set number of frames can be 10 frames.
[0046] The pen-holding behavior recognition model can be a model obtained by pre-training a deep learning model using a first sample training set. The first sample training set may include various sample training images and corresponding pen-holding posture category labels and pen-holding hand position labels in the sample training images. The sample training images are input into the pen-holding behavior recognition model to obtain the pen-holding posture category and pen-holding hand position of the sample training images. The category loss and position loss are further calculated based on the pen-holding posture category label and the pen-holding hand position label. The model parameters of the pen-holding behavior recognition model are updated based on the category loss and position loss. The pen-holding behavior recognition model is iteratively trained in sequence using each sample training image in the first sample training set until the pen-holding behavior recognition model meets the preset convergence conditions to obtain a trained pen-holding behavior recognition model.
[0047] The pen-holding behavior recognition model can identify the pen-holding behavior in the video frame and obtain the position information of the pen-holding hand in the target video frame and the corresponding pen-holding posture category.
[0048] The pen-holding posture category may include a standard pen-holding posture and a non-standard pen-holding posture. The standard pen-holding posture is a pen-holding hand posture under standard circumstances, and the non-standard pen-holding posture is a pen-holding posture under non-standard circumstances.
[0049] Alternatively, in another embodiment, the pen-holding posture category may further include multiple types of pen-holding postures, such as a standard pen-holding posture, a "horizontal type" pen-holding posture, a "head-down type" pen-holding posture, and an "incorrect type" pen-holding posture. For multiple types of pen-holding postures, a first sample training set having sample training images of the multiple types of pen-holding postures may be used to train a pen-holding behavior recognition model, and a second sample training set having sample training images of the multiple types of pen-holding postures may be used to train a pen-holding posture classification model, so that the pen-holding behavior recognition model and the pen-holding posture classification model can recognize multiple types of pen-holding postures.
[0050] S130: Initializing a tracker based on the first position information and the first pen-holding posture category information, and using the tracker to track second position information of the pen-holding hand in N video frames after the target video frame; wherein N>1;
[0051] N is used to determine the number of video frames that the tracker needs to track. It can be set according to user needs. For example, in this application, N can be 5. By tracking the pen-holding posture category information of the 5 video frames after the target video frame, the final pen-holding posture is determined based on the pen-holding posture category information of all video frames.
[0052] The tracker is used to track the position of the pen-holding hand. The tracker can determine the position information of the pen-holding hand in the current video frame by feature comparison based on the pen-holding hand features and the position information of the pen-holding hand in the previous video frame.
[0053] S140: Acquire second pen-holding posture category information of the N video frames based on the pen-holding posture classification model and the second position information;
[0054] The pen-holding posture classification model is used to obtain pen-holding posture category information based on the second position information. The pen-holding posture classification model can be a model obtained by pre-training the classification model using the second sample training set. The second sample training set can include various sample training images and corresponding pen-holding posture category labels in the sample training images. The sample training images are input into the pen-holding posture classification model to obtain the pen-holding posture category of the sample training images. The category loss is further calculated based on the pen-holding posture category label. The model parameters of the pen-holding posture classification model are updated based on the category loss. The pen-holding posture classification model is iteratively trained in sequence using each sample training image in the second sample training set until the pen-holding posture classification model meets the preset convergence condition, thereby obtaining a trained pen-holding posture classification model.
[0055] Specifically, in this application, after obtaining the first position information and the first pen-holding posture category information, a tracker is initialized based on the first position information and the first pen-holding posture category information, and the tracker is used to track the second position information of the pen-holding hand in the next video frame after the target video frame. Then, based on the second position information of the pen-holding hand in the next video frame, the second position information of the pen-holding hand in the next video frame is tracked. The above process is repeated to obtain the second position information of the pen-holding hand in N video frames after the target video frame.
[0056] S150: Determine a pen-holding posture according to the first pen-holding posture category information and the second pen-holding posture category information.
[0057] Specifically, the pen-holding posture can be determined based on the pen-holding posture category information of the target video frame and the N subsequent video frames. Specifically, if the pen-holding posture category information of all video frames is standard, the pen-holding posture is determined to be standard; if the pen-holding posture category information of at least M video frames is non-standard, the pen-holding posture is determined to be non-standard. Where 1≤M≤N, and M is a natural number.
[0058] In an embodiment of the present application, a target video is obtained by responding to a pen-holding posture recognition instruction; a target video frame in which a pen-holding behavior exists in the target video is identified based on a pen-holding behavior recognition model, and the first position information of the pen-holding hand and the first pen-holding posture category information of the target video frame are obtained; a tracker is initialized based on the first position information and the first pen-holding posture category information, and the tracker is used to track the second position information of the pen-holding hand of N video frames after the target video frame; and the second pen-holding posture category information of the N video frames is obtained based on a pen-holding posture classification model. Compared with the prior art, the present application does not need to perform pen-holding posture recognition on all video frames of the target video, and does not need to use a pen-holding posture recognition model that consumes a large amount of computing resources for each video frame. This can effectively reduce the consumption of computing resources of the device and improve the efficiency of pen-holding posture recognition. At the same time, the present application determines the pen-holding posture based on the pen-holding posture category information of the target video frame and the subsequent N video frames, thereby improving the accuracy of pen-holding posture recognition.
[0059] In step S110, the target video is a video continuously captured by the video capture device. The target video may or may not contain the pen-holding behavior. If the target video does not contain the pen-holding behavior, subsequent target videos may be continuously identified based on the pen-holding behavior recognition model to obtain target video frames containing the pen-holding behavior. Alternatively, in one embodiment, after obtaining the target video, the following steps are further included:
[0060] If the target video frame containing the pen-holding behavior in the target video is not identified based on the pen-holding behavior recognition model, the target video is reacquired after waiting for the first target time, and the target video frame containing the pen-holding behavior in the reacquired target video is identified based on the pen-holding behavior recognition model.
[0061] The first target time may be a waiting time preset by the user. In this application, the first target time may be set to 30 seconds.
[0062] During the first target time, pen-holding behavior recognition is not performed on the target video of the target time to avoid repeated processing of invalid data and save computing resources. After the first target time, the target video after the first target time is acquired again, and the pen-holding behavior recognition model is used to identify whether there is pen-holding behavior in the target video, thereby reducing the number of pen-holding behavior recognitions and computing resource consumption, and improving the efficiency of pen-holding behavior recognition.
[0063] In step S120 , the pen-holding posture category may include a standard pen-holding posture and a non-standard pen-holding posture. The non-standard pen-holding posture may be caused by a user's incorrect writing posture or an incorrect pen-holding gesture.
[0064] Therefore, if Figure 2 As shown, in one embodiment, after identifying the target video frame in which the pen holding behavior exists in the target video based on the pen holding behavior recognition model, the method further includes:
[0065] S210: Identifying position information of at least one writing posture key point in the target video frame based on a first key point detection model;
[0066] S220: Determining, based on the position information of the at least one writing posture key point, whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture;
[0067] S230: If the writing posture of the target video frame is a standard writing posture, obtain first position information and first pen-holding posture category information of the target video frame.
[0068] In this application, an incorrect writing posture may be caused by an incorrect posture of the user's arm and wrist, or it may be caused by an incorrect posture of the user's fingers or pen holding. An incorrect writing posture usually only presents the characteristics of the back of the hand, obscures the finger characteristics, and affects the judgment of the pen holding posture.
[0069] Therefore, in this application, the incorrect writing posture may refer to the incorrect posture of the user's arm, wrist, thumb, index finger, middle finger and other writing parts, and the key points of the writing posture may be the key points of the elbow, wrist, hand, thumb, index finger, top of the pen, pen tip and other positions of the arm.
[0070] The first key point detection model is used to detect at least one writing posture key point and its position information in the target video frame. The first key point detection model can be a model obtained by pre-training the first key point detection model using a third sample training set. The third sample training set may include sample training images of various writing posture key points and position labels of the writing posture key points. The sample training images are input into the first key point detection model to obtain the positions of the writing posture key points in the sample training images. The position loss is further calculated based on the position labels of the writing posture key points. The model parameters of the first key point detection model are updated based on the position loss. The first key point detection model is iteratively trained in sequence using each sample training image in the third sample training set until the first key point detection model meets the preset convergence conditions, thereby obtaining a trained first key point detection model.
[0071] When determining whether the writing posture is a standard writing posture or a non-standard writing posture, the position of the key point of the current writing posture can be compared with the position of the key point of the standard writing posture to determine. Specifically, when the position of the key point of the current writing posture is consistent with the position of the key point of the standard writing posture, the writing posture is determined to be a standard writing posture. If the position of the key point of the current writing posture is inconsistent with the position of the key point of the standard writing posture, the writing posture is determined to be a non-standard writing posture.
[0072] Alternatively, in another embodiment, the user's writing posture can be determined based on the position information of each key point of the writing posture, thereby determining whether the user's writing posture is incorrect. The error judgment of the writing posture can be specifically set based on common incorrect writing postures.
[0073] like Figure 3 As shown in FIG, it is a schematic diagram of a non-standard writing posture in one embodiment. Figure 3 As shown, the writing posture key points include the elbow key points and the wrist key points, and the step of determining whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture specifically includes:
[0074] A first angle between a line connecting the elbow key point and the wrist key point and a horizontal line is obtained. If the first angle is less than a first angle threshold, it is determined that the writing posture of the target video frame is a non-standard writing posture.
[0075] In this application, when the first angle between the line connecting the elbow key point and the wrist key point and the horizontal line is less than the first angle threshold, it means that the user's arm is placed horizontally, which is an incorrect writing posture.
[0076] Or, as Figure 4 As shown in FIG, it is a schematic diagram of a non-standard writing posture in another embodiment. Figure 4 As shown, the writing posture key points include elbow key points, wrist key points and hand key points; the step of determining whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture specifically includes:
[0077] Obtain a second angle of a line connecting the wrist key point and the hand key point and a second angle of a line connecting the elbow key point and the wrist key point. If the second angle is less than a second angle threshold, determine that the writing posture of the target video frame is a non-standard writing posture.
[0078] In this application, when the second angle of the line connecting the wrist key point and the hand key point and the line connecting the elbow key point and the wrist key point is less than the second angle threshold, it indicates a "twisting" pen-holding posture, which is an incorrect writing posture.
[0079] In an embodiment of the present application, after identifying the target video frame in which the pen-holding behavior exists in the target video based on the pen-holding behavior recognition model, the position information of at least one writing posture key point in the target video frame is identified through the first key point detection model, and based on the position information of at least one writing posture key point, it is determined whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture. When the writing posture of the target video frame is a standard writing posture, the pen-holding posture of the target video frame is identified again, thereby avoiding repeated processing of invalid data and saving computing power resources.
[0080] like Figure 5 As shown, in a preferred embodiment, after determining that the writing posture of the target video frame is a standard writing posture or a non-standard writing posture, the following steps are further included:
[0081] S240: If the writing posture of the target video frame is a non-standard writing posture, display a writing posture error prompt message;
[0082] S250: reacquire the target video and target video frame after waiting for a second target time, and re-determine whether the writing posture of the reacquired target video frame is a standard writing posture or a non-standard writing posture.
[0083] The writing posture error prompt information is used to inform the user that the current writing posture is incorrect, thereby reminding the user to correct the writing posture in a timely manner. Among them, the display content and display method of the writing posture error prompt information can be set according to user needs. For example, the writing posture error prompt information in this application may include error information and correction information. The error information can be used to inform the user where the writing posture is incorrect, such as an angle that is too large, and the correction information is used to prompt the user to correct the writing posture, such as "Please straighten your arms." The writing posture error prompt information can be set to be displayed in the form of a pop-up window on the user terminal.
[0084] The second target time may be a waiting time pre-set by the user. In this application, the second target time may be set to 30 seconds.
[0085] No operation is performed within the second target time to avoid repeated processing of invalid data and save computing resources; after the second target time, the target video and target video frame after the second target time are re-acquired, and the position information of at least one writing posture key point in the re-acquired target video frame is identified based on the first key point detection model; based on the position information of the at least one writing posture key point, the writing posture of the target video frame is re-determined as a standard writing posture or a non-standard writing posture, thereby reducing the number of writing posture recognition times and computing resource consumption, and improving the writing posture recognition efficiency.
[0086] The pen holding posture of the present application may include a standard pen holding posture and a non-standard pen holding posture. The existing standard pen holding posture usually only involves the posture of the hand when holding the pen, and lacks the recognition of the posture of the fingers when holding the pen, such as Figure 6 As shown, when the Figure 6 When the pen holding posture is shown, the existing pen holding posture detection model is likely to judge the current pen holding posture as the standard pen holding posture, which will affect the user's writing habits in the long run.
[0087] Therefore, if Figure 7 As shown, in one embodiment, after determining that the pen-holding posture of the target video frame is a standard pen-holding posture, the following steps are further included:
[0088] S310: Identifying position information of at least one key point of the pen-holding gesture of the target video frame based on a second key point detection model;
[0089] The key points of the pen-holding gesture are used to determine whether the user's pen-holding gesture is correct. In this application, the key points of the pen-holding gesture can be used to determine the posture of the user's fingers holding the pen. The key points of the pen-holding gesture can include the thumb, index finger, top of the pen shaft, and pen tip, four key points related to the user's finger holding pen posture.
[0090] The second key point detection model is used to detect at least one pen-holding gesture key point and its position information in the target video frame. The second key point detection model can adopt a model obtained by pre-training the second key point detection model using the fourth sample training set. The fourth sample training set may include sample training images of various pen-holding gesture key points and position labels of the pen-holding gesture key points. The sample training images are input into the second key point detection model to obtain the positions of the pen-holding gesture key points in the sample training images. The position loss is further calculated based on the position labels of the pen-holding gesture key points. The model parameters of the second key point detection model are updated based on the position loss. The first key point detection model is iteratively trained in sequence using each sample training image in the fourth sample training set until the second key point detection model meets the preset convergence conditions, thereby obtaining a trained second key point detection model.
[0091] S320: Score the pen-holding posture of the target video frame based on the position information of the at least one pen-holding gesture key point.
[0092] When scoring the pen-holding posture of the target video frame, the pen-holding posture of the target video frame may be scored according to scoring conditions preset by the user.
[0093] In the embodiment of the present application, the key points of the pen holding gesture may include the thumb key point, the index finger key point, the pen shaft top key point, and the pen tip key point. The scoring conditions may be:
[0094] a. The thumb and index finger key points are located on both sides of the pen top key point and the pen tip key point.
[0095] b. The angle between the line connecting the thumb key point and the index finger key point and the horizontal line is less than the threshold
[0096] c. The distance between the center point of the line connecting the thumb key point and the index finger key point and the pen tip key point is within the distance threshold;
[0097] It should be noted that, in actual application, users can add or reduce scoring conditions according to their scoring requirements for pen-holding posture, and this application does not specifically limit the scoring conditions.
[0098] When the key points of the pen-holding gesture meet the above-mentioned scoring conditions at the same time, the pen-holding posture is determined to be a standard pen-holding posture, and the score of the pen-holding posture can be full marks. When the key points of the pen-holding gesture do not meet the above-mentioned scoring conditions, the pen-holding posture is determined to be a non-standard pen-holding posture, and the score of the pen-holding posture can be zero. When the key points of the pen-holding gesture meet any of the above-mentioned scoring conditions, the pen-holding posture can be scored according to the number of scoring conditions that the key points of the pen-holding gesture meet.
[0099] Preferably, after scoring the pen-holding posture, scoring information can be output to the user terminal, so that the user can know the scoring status of the pen-holding posture and correct the pen-holding posture in time.
[0100] In this application, the user's finger pen-holding posture is scored by combining the scoring conditions and the position information of the thumb key point, index finger key point, pen shaft top key point and pen tip key point, thereby ensuring that the correct pen-holding posture is a posture with standard hand posture and standard finger pen-holding posture, reducing the misjudgment of the standard pen-holding posture, and thus improving the accuracy of the user's pen-holding posture detection.
[0101] like Figure 8 As shown, the embodiment of the present application further provides a device for recognizing a pen-holding posture, the device comprising:
[0102] A target video acquisition module 110 is configured to acquire a target video in response to a pen-holding gesture recognition instruction;
[0103] a pen-holding behavior recognition module 120 for identifying a target video frame in which a pen-holding behavior is present in the target video based on a pen-holding behavior recognition model, and obtaining first position information and first pen-holding posture category information of the target video frame;
[0104] a tracking module 130 configured to initialize a tracker based on the first position information and the first pen-holding posture category information, and use the tracker to track second position information of the pen-holding hand in N video frames following the target video frame, where N>1;
[0105] a classification module 140, configured to obtain second pen-holding posture category information of the N video frames based on a pen-holding posture classification model and the second position information;
[0106] The pen-holding posture determining module 150 is configured to determine the pen-holding posture according to the first pen-holding posture category information and the second pen-holding posture category information.
[0107] In one embodiment, the apparatus comprises:
[0108] The pen-holding behavior re-identification module is used to re-acquire the target video after waiting for a first target time if the target video frame containing the pen-holding behavior in the target video is not identified based on the pen-holding behavior recognition model, and to identify the target video frame containing the pen-holding behavior in the re-acquired target video based on the pen-holding behavior recognition model.
[0109] In one embodiment, the apparatus comprises:
[0110] A key point detection module, configured to identify position information of at least one writing posture key point in the target video frame based on a first key point detection model;
[0111] a writing posture determining module, configured to determine whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture based on the position information of the at least one writing posture key point;
[0112] The pen-holding behavior recognition module is used to obtain first position information and first pen-holding posture category information of the pen-holding hand in the target video frame if the writing posture of the target video frame is a standard writing posture.
[0113] In one embodiment, the apparatus comprises:
[0114] a prompt information display module, configured to display a prompt information indicating a writing posture error if the writing posture of the target video frame is a non-standard writing posture;
[0115] The writing posture re-identification module is used to re-acquire the target video and target video frame after waiting for a second target time, and re-determine whether the writing posture of the re-acquired target video frame is a standard writing posture or a non-standard writing posture.
[0116] In one embodiment, the writing posture key points include elbow key points and wrist key points, and the writing posture determination module includes:
[0117] The first recognition unit is used to obtain a first angle between a line connecting the elbow key point and the wrist key point and a horizontal line. If the first angle is less than a first angle threshold, it is determined that the writing posture of the target video frame is a non-standard writing posture.
[0118] In one embodiment, the writing posture key points include elbow key points, wrist key points, and hand key points; and the writing posture determination module includes:
[0119] The second recognition unit is used to obtain a second angle of a line connecting the wrist key point and the hand key point and a second angle of a line connecting the elbow key point and the wrist key point. If the second angle is less than a second angle threshold, it is determined that the writing posture of the target video frame is a non-standard writing posture.
[0120] In one embodiment, the pen-holding posture includes a standard pen-holding posture and a non-standard pen-holding posture. After the pen-holding posture determination module determines that the pen-holding posture of the target video frame is a standard pen-holding posture, the apparatus includes:
[0121] a pen-holding gesture key point detection module, configured to identify position information of at least one pen-holding gesture key point of the target video frame based on a second key point detection model;
[0122] A scoring module is used to score the pen-holding posture of the target video frame based on the position information of the at least one pen-holding gesture key point.
[0123] It should be noted that the pen-holding posture recognition device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the pen-holding posture recognition method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the pen-holding posture recognition device provided in the above embodiment and the pen-holding posture recognition method in the above embodiment are of the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.
[0124] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the pen-holding posture recognition method as described in any one of the above.
[0125] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0126] like Figure 9 As shown, the present application further provides an electronic device 200, comprising a memory 210, a processor 220, and a computer program stored in the memory 210 and executable by the processor 220;
[0127] When the processor 220 executes the computer program, the steps of the pen-holding posture recognition method as described in any one of the above are implemented.
[0128] The memory 210 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0129] The processor 220 is the control core (Control Unit) of the electronic device 200. It uses various interfaces and lines to connect the various components of the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing the programs or modules stored in the memory, and calling the data stored in the memory 210. For example, when the processor 220 executes the computer program stored in the memory 210, it implements all or part of the steps of the pen-holding posture recognition method described in the embodiment of the present application; or implements all or part of the functions of the pen-holding posture recognition device. The processor 220 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors and a combination of various control chips.
[0130] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications of the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims of the present invention and equivalent technologies, the present invention is also intended to include these changes and modifications.
Claims
1. A method for recognizing a pen-holding posture, characterized in that: The following steps are involved: In response to a pen-holding gesture recognition instruction, acquiring a target video; identifying a target video frame in which the pen-holding behavior exists in the target video based on the pen-holding behavior recognition model, and acquiring first position information and first pen-holding posture category information of the target video frame; Initializing a tracker based on the first position information and the first pen-holding posture category information, and using the tracker to track second position information of the pen-holding hand in N video frames following the target video frame; wherein N>1; acquiring second pen-holding posture category information of the N video frames based on the pen-holding posture classification model and the second position information; The pen-holding posture is determined according to the first pen-holding posture category information and the second pen-holding posture category information.
2. The pen-holding posture recognition method according to claim 1, characterized in that: After obtaining the target video, the following steps are also included: If the target video frame containing the pen-holding behavior in the target video is not identified based on the pen-holding behavior recognition model, the target video is reacquired after waiting for the first target time, and the target video frame containing the pen-holding behavior in the reacquired target video is identified based on the pen-holding behavior recognition model.
3. The pen-holding posture recognition method according to claim 1, characterized in that: After identifying the target video frame containing the pen-holding behavior in the target video based on the pen-holding behavior recognition model, the method further includes: Identifying position information of at least one writing posture key point in the target video frame based on a first key point detection model; determining, based on position information of the at least one writing posture key point, whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture; If the writing posture of the target video frame is a standard writing posture, first position information of the pen-holding hand and first pen-holding posture category information of the target video frame are obtained.
4. The pen-holding posture recognition method according to claim 3, characterized in that: After determining whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture, the following steps are further included: If the writing posture of the target video frame is a non-standard writing posture, displaying a writing posture error prompt message; After waiting for a second target time, the target video and the target video frame are reacquired, and the writing posture of the reacquired target video frame is re-determined as a standard writing posture or a non-standard writing posture.
5. The pen-holding posture recognition method according to claim 3, characterized in that: The writing posture key points include elbow key points and wrist key points. The step of determining whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture specifically includes: A first angle between a line connecting the elbow key point and the wrist key point and a horizontal line is obtained. If the first angle is less than a first angle threshold, it is determined that the writing posture of the target video frame is a non-standard writing posture.
6. The pen-holding posture recognition method according to claim 3, characterized in that: The writing posture key points include elbow key points, wrist key points and hand key points; and the step of determining whether the writing posture of the target video frame is a standard writing posture or a non-standard writing posture specifically includes: Obtain a second angle of a line connecting the wrist key point and the hand key point and a second angle of a line connecting the elbow key point and the wrist key point. If the second angle is less than a second angle threshold, determine that the writing posture of the target video frame is a non-standard writing posture.
7. The pen-holding posture recognition method according to claim 1, characterized in that: The pen-holding posture includes a standard pen-holding posture and a non-standard pen-holding posture. After determining that the pen-holding posture of the target video frame is the standard pen-holding posture, the following steps are further included: Identifying position information of at least one key point of the pen-holding gesture in the target video frame based on a second key point detection model; The pen-holding posture of the target video frame is scored based on the position information of the at least one pen-holding gesture key point.
8. A pen-holding posture recognition device, characterized in that: The device comprises: A target video acquisition module, configured to acquire a target video in response to a pen-holding posture recognition instruction; a pen-holding behavior recognition module, configured to identify a target video frame in which a pen-holding behavior is present in the target video based on a pen-holding behavior recognition model, and obtain first position information and first pen-holding posture category information of the target video frame; a tracking module, configured to initialize a tracker based on the first position information and the first pen-holding posture category information, and use the tracker to track second position information of the pen-holding hand in N video frames following the target video frame; wherein N>1; a classification module, configured to obtain second pen-holding posture category information of the N video frames based on a pen-holding posture classification model and the second position information; The pen-holding posture determining module is configured to determine the pen-holding posture according to the first pen-holding posture category information and the second pen-holding posture category information.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the pen-holding posture recognition method according to any one of claims 1 to 7 are implemented.
10. An electronic device, characterized in that: comprising a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, the steps of the pen-holding posture recognition method according to any one of claims 1 to 7 are implemented.