MediaPipe Handles-based video frame image gesture training action abnormity processing method and system for elderly patients

Abstract

The invention discloses a MediaPipe Handds-based video frame image gesture training motion abnormity processing method and system for elderly patients, and the method comprises the following operation steps: collecting a gesture video record, obtaining each frame of video image from the gesture video record, and obtaining a to-be-detected frame image; extracting a posture behavior area from each frame of to-be-detected frame image; carrying out attitude feature extraction on the attitude behavior region to obtain an attitude feature image; and matching the gesture feature image with a historical database, and identifying the gesture behavior of the gesture.

Description

Technical Field

[0001] The present invention relates to the field of human hand motion posture recognition, and in particular to a method and system for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipeHands. Background Art

[0002] With the rapid development of computer vision and artificial intelligence, human hand gesture recognition technology in video frame images has gradually become a research hotspot.

[0003] This technology aims to accurately recognize various gestures by analyzing and understanding the changes in hand images in video frames, thereby promoting progress in multiple fields such as human-computer interaction, virtual reality, intelligent monitoring, and medical rehabilitation.

[0004] As a natural, intuitive and expressive way of communication, gestures play an important role in human daily communication, work and entertainment.

[0005] However, since human hands have a high degree of freedom, can make complex and diverse movements, and present different shapes under different lighting, angles, occlusion and other conditions, gesture recognition technology is extremely challenging.

[0006] In the above-mentioned medical rehabilitation field, continuous training gestures can be implemented by relying on a gesture recognition software platform system to detect whether elderly patients have abnormal dominant gesture movements, and then use this to determine whether the symptoms of Alzheimer's disease have been alleviated and recovered; further researchers have found that in vision-based gesture recognition technology, depending on the processing object, it can be divided into static gesture recognition and dynamic gesture recognition.

[0007] Static gesture recognition mainly focuses on the hand area detected in a certain frame extracted from the image sequence, analyzing and identifying the posture information it contains; while dynamic gesture recognition needs to track the moving hands in the image sequence, extract the movement trajectory of the hands, and then recognize different dynamic gestures. In particular, the recognition of fine training action gestures requires very high recognition accuracy.

[0008] In order to improve the accuracy of gesture recognition, more complex algorithms and models are often needed. However, these algorithms and models usually have high computational complexity and are difficult to meet real-time requirements.

[0009] Especially when processing high-resolution videos and complex gestures, the computational burden will be further increased, resulting in a decrease in the response speed of the gesture recognition software platform system. Summary of the invention

[0010] The purpose of the present invention is to provide a method and system for handling abnormal movements of elderly patients in video frame image gesture training based on MediaPipe Hands, which solves the above-mentioned technical problems pointed out in the prior art.

[0011] The present invention provides a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, comprising the following steps: Collecting gesture video recordings, obtaining each frame of the gesture video recordings, and obtaining the frame image to be detected; Extracting a posture behavior region from each frame of the to-be-detected frame image; extracting posture features from the posture behavior region to obtain a posture feature image; The gesture feature image is matched with a historical database to identify the gesture behavior.

[0012] Preferably, the gesture feature image is matched with a historical database to identify the gesture behavior of the hand gesture, and the specific operation steps are as follows: Use the YOLO algorithm to select the candidate frame of each frame to be detected, and obtain multiple region candidate frames; Calculate the confidence score for each region candidate box; Sort the confidence scores of the candidate boxes of each segmented image, and select the candidate box with the highest confidence score as the posture behavior area; Extracting hand key points from the posture behavior area; connecting the hand key points to obtain a hand skeleton structure, and using the hand skeleton structure as a posture feature image.

[0013] Preferably, the hand key points are extracted from the posture behavior area; the hand key points are connected to obtain a hand skeleton structure, and the hand skeleton structure is used as a posture feature image. The specific operation steps are as follows: The Harris corner algorithm is used to calculate the sum of the squares of the gradients and the sum of the products of the two in the x-direction and the y-direction of each pixel point in the posture behavior area; a sliding window is performed on the posture behavior area to construct a second-order Harris matrix; ; In the formula, It is expressed as calculating the sum of the squared gradients in the x direction of each pixel; It is expressed as calculating the sum of the products of the x and y gradients of each pixel; It is expressed as calculating the sum of the squared gradients in the y direction of each pixel; And calculate the Harris response value R, preset the response value threshold, and determine whether the Harris response value R is greater than the response value threshold; Among them, Harris response value ;in is the determinant of the matrix, is the trace of the matrix, is a constant.

[0014] If yes, the pixel point corresponding to the Harris response value R is taken as the pixel point of the local maximum value, and the pixel point is taken as the corner point of the window, and the coordinates are extracted as the corner point coordinates, thus ending the corner point confirmation processing operation; Pre-build a hand model based on the MediaPipe Hands algorithm; Using a pre-built hand model, extracting key points of the hand from the gesture behavior area; The corner points and hand key points detected by each algorithm are mutually verified to obtain the hand key points, and the hand key points are connected to obtain the hand bone structure as a posture feature image.

[0015] Preferably, a hand model is pre-built according to the MediaPipe Hands algorithm, and the specific operation steps are as follows: Using the MediaPipe Hands algorithm, normalize and resize the frame image to be detected of each frame of the video recording, and convert it into an RGB image to obtain an RPG image to be detected; Predicting key points of the RPG image to be detected to obtain a preliminary hand skeleton model; Performing confidence scoring on each predicted key point in the preliminary hand skeleton model; predicting a confidence score threshold; and determining whether the confidence score of each predicted key point is lower than the confidence score threshold; If not, the predicted key points are retained as hand key points; If so, the predicted key point is the ground confidence score, the predicted key point with the low confidence score is determined with adjacent predicted key points with high confidence scores, and the interpolated position of the predicted key point with the low confidence score is calculated by weighted average of the adjacent predicted key points with high confidence scores; the position of the predicted key point with the low confidence score is updated with the interpolated position to obtain the hand key point; The hand model is constructed by constructing all the obtained hand key points.

[0016] Preferably, the corner points and hand key points detected by each algorithm are mutually verified to obtain the hand key points, and the hand key points are connected to obtain the hand bone structure. The specific operation steps are as follows: Construct the correspondence matrix M between the key points of the hand and the corner points; use the KNN algorithm to find the closest corner point for each key point of the hand; The matching degree between the corner point coordinates and the hand key point coordinates is calculated based on the distance between the hand key point and each closest corner point and the grayscale of the corner point, which is used as the matching degree index xc; Preset a matching degree threshold o; determine whether the matching degree between the corner point coordinates and the hand key point coordinates is greater than the matching degree threshold o; If yes, the matching point pair of the corner point coordinates and the hand key point coordinates is successfully matched; For the matching point pairs of successfully matched corner points and hand key points, the average distance between the hand key points and the corner points is calculated as the position deviation index J; Calculate the edge gradient of the successfully matched corner points and the matching points of the hand key points using the Sobel operator, and calculate the difference between the edge gradients as the edge change feature p; The matching index xc, the position deviation index J and the edge change feature p are combined into a fused feature vector F, wherein the fused feature vector F=[xc, J, p]; A decision tree model is constructed using random forest. A classifier is constructed using the decision tree model, and the classifier is input with the fusion feature vector F to obtain the identified hand key points, thereby connecting the hand bone structure.

[0017] Preferably, a decision tree model is constructed using random forest, and the specific steps are as follows: Collecting multiple sample gesture images, and labeling each collected sample gesture image with a gesture category; extracting features from the multiple sample gesture images, wherein the extracted features include key point positions, corner point positions, and edge features; The extracted features are used as a data set, divided into a training set and a test set, the training set is trained, and a decision tree is constructed; Selecting a feature and a threshold as splitting data for the node of the decision tree; recursively splitting data for each node from the root node of the decision tree, and stopping node splitting when the sample gesture images of the nodes in the decision tree all belong to the same gesture category; The decision tree is evaluated using the test set to obtain a decision tree model.

[0018] Preferably, the gesture feature image is matched with a historical database to identify the gesture behavior of the hand gesture, and the specific operation steps are as follows: Acquire a historical database, and collect sample posture images from the historical database; match the posture feature image with the sample posture images in the historical database, and calculate the matching degree; Preset a matching degree threshold; determine whether the matching degree is greater than the matching degree threshold; If yes, then the historical database contains a known gesture, and a sample gesture image with the highest matching degree is obtained as the gesture behavior of the gesture; If not, the historical database does not contain the posture feature image, and the historical database is manually updated.

[0019] Accordingly, the present invention also proposes a system for handling abnormal motions of elderly patients based on MediaPipe Hands for implementing video frame image gesture training, including: an acquisition module; an acquisition module; a matching module; The acquisition module is used to collect gesture video recordings, obtain each frame of the gesture video recordings, and obtain the frame image to be detected; The acquisition module is used to extract the posture behavior area of ​​each frame image to be detected; extract the posture feature of the posture behavior area to obtain a posture feature image; The matching module is used to match the gesture feature image with the historical database to identify the gesture behavior.

[0020] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages: It can be seen from the analysis of the above-mentioned method for handling abnormal movements of gesture training for elderly patients based on MediaPipe Hands provided by the present invention that, in specific applications, the gesture video recording is collected, and the image of each frame is collected as the frame image to be detected, so that the gesture posture can be recognized more quickly; Furthermore, the candidate frames of the image to be detected are selected, and the candidate frames with the highest confidence scores are selected by using the confidence scores as the posture behavior areas; this can quickly find the human hand, save time for subsequent recognition, and improve the speed; Furthermore, the Harris corner algorithm is first used to calculate the square sum product of the gradient in the x-direction and y-direction of each pixel in the posture behavior area, so that the gradient of the posture behavior area can be calculated, and then a second-order Harris matrix is ​​constructed by the calculated gradient and the sliding window; the Harris response value is calculated by the second-order Harris matrix, so that the corner points in each sliding window in the posture behavior area can be found; Furthermore, the MediaPipe Hands algorithm is used to pre-build a hand model, which can obtain the key points of the complete hand and the hand posture formed by each key point of the hand. The corner points and the key points of the hand are then verified against each other to form the joint points of the hand. By connecting the joint points, the hand posture feature map can be obtained, and the hand posture can be known. The posture feature image is matched with the sample posture images in the historical database, and finally the posture of the gesture is identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is an overall flow chart of a method for handling abnormal motions in video frame image gesture training for elderly patients based on MediaPipe Hands in Example 1; Figure 2 It is a flowchart of obtaining posture feature images of an elderly patient in a method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands in Example 1; Figure 3 A flowchart of an embodiment of the present invention for an elderly patient to obtain a hand skeleton structure by mutual verification of corner points and hand key points in a method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands; Figure 4 A flowchart of obtaining hand key points of an elderly patient in a method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands in Example 1; Figure 5 A flowchart of a decision tree model for constructing a method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands for elderly patients in Example 1; Figure 6 A schematic diagram of obtaining hand key points of an elderly patient in a method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands in Example 1; Figure 7 A list of training gesture actions in a method for handling abnormal video frame image gesture training actions based on MediaPipe Hands for an elderly patient in Embodiment 1; Figure 8 This is a schematic diagram of a system for handling abnormal motions in video frame image gesture training for elderly patients based on MediaPipe Hands in Example 2; Fig. 9 A structural schematic diagram of a storage medium for applying the above-mentioned method for handling abnormal motions in video frame image gesture training based on MediaPipe Hands for elderly patients.

[0022] Reference numerals: acquisition module 10 ; acquisition module 20 ; matching module 30 ; processor 1110 ; communication interface 1120 ; memory 1130 ; computer storage medium 1140 . DETAILED DESCRIPTION

[0023] The following will verify the technical solution of the present invention clearly and completely with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The present invention is further described in detail below through specific embodiments and accompanying drawings for mutual verification.

[0025] Embodiment 1

[0026] like Figure 1 As shown, this embodiment provides a method for handling abnormal motions of elderly patients in implementing video frame image gesture training based on MediaPipe Hands, including the following steps: S10: Collecting gesture video recordings, obtaining each frame of video images from the gesture video recordings, and obtaining a frame image to be detected; It should be noted that when collecting gesture video recordings, the frame image to be detected is obtained for each frame of the video recording. With the dynamic behavior of the hand, when making gestures, the video recording frame may have unclear residual images (that is, when making standard training gestures, there may be unclear image residual images), so it is necessary to collect and obtain each frame of the frame image to be detected to extract the gesture behavior area (that is, the position of the hand); S20: extracting a posture behavior region from each frame of the to-be-detected frame image; extracting posture features from the posture behavior region to obtain a posture feature image; It should be noted that for each frame of the image to be detected, the approximate posture behavior area of ​​the human body (i.e., the position of the hand. When making a gesture, the position of the hand may move, so it is necessary to first determine the position of the hand, and then determine the posture characteristics of the hand (i.e., the gesture)); by first identifying the posture behavior area to determine the position to be captured, and then extracting specific features of the posture through the captured position, thereby reducing the calculation time and reducing the speed of directly identifying the posture; S30: Matching the gesture feature image with a history database to identify gesture behavior.

[0027] It should be noted that a historical database is obtained, and sample posture images are collected from the historical database; the posture feature image is matched with the sample posture images in the historical database, and the matching degree is calculated; a matching degree threshold is preset; and it is determined whether the matching degree is greater than the matching degree threshold; if so, there is a known gesture in the historical database, and the sample posture image with the highest matching degree is obtained as the posture behavior of the gesture; if not, the posture feature image does not exist in the historical database, and the historical database is manually updated.

[0028] The extracted posture feature image is matched with the sample posture image in the historical database, and a matching threshold is set. If the matching degree is higher than the matching threshold, there is a known gesture in the historical database (that is, the gesture exists in the historical database, and you only need to select the sample posture image with the highest matching degree to know what the gesture is). The sample posture image with the highest matching degree with the posture feature image is screened out as the posture behavior of the gesture; if the matching degree is lower than the matching threshold, it means that there is no posture behavior that matches the sample in the historical database, so the historical database is updated.

[0029] Specifically, Figure 2 As shown, in step S20, the posture behavior region is extracted from each frame of the frame image to be detected; the posture feature is extracted from the posture behavior region to obtain a posture feature image. The specific operation steps are as follows: S21: Using the YOLO algorithm to select a candidate frame of the image to be detected in each frame, and obtaining multiple region candidate frames; Calculate the confidence score for each region candidate box; It should be noted that many region candidate frames can be obtained by selecting the candidate frames of the frame image to be detected. Each region candidate frame contains a human body part, such as the head, arm, neck, etc. The confidence score of the region candidate frame is calculated, and the region candidate frame of the hand is selected as the gesture behavior region by the confidence score (that is, the position of the hand is selected, and the gesture posture can be quickly recognized in the subsequent process to improve the detection speed). The YOLO algorithm is used to select the candidate frame of each frame image to be detected, and the entire video recording can be carefully screened, and the gesture behavior position of the hand can also be quickly identified by the candidate frame of the hand. It should be noted that by segmenting each frame of the image to be detected through the YOLO algorithm, the posture behaviors of each part can be distinguished, the detection and recognition can be faster, and preparation is made for the subsequent candidate box selection to extract the posture behavior area; S22: sorting the confidence scores of the candidate frames of each segmented image, and selecting the candidate frame with the highest confidence score as the posture behavior region (i.e., the body posture behavior region, the hand posture behavior region); It should be noted that the segmented part of the image is framed by the YOLO algorithm candidate frame, so that a large number of candidate frames will appear in each segmented image, and the confidence score of each candidate frame is used for screening, so as to find the candidate frame with the highest score in the body image and the hand image as the final posture behavior area determined by the image (i.e., the body image and the hand image) (i.e., for an arm image, the arm image itself includes not only the upper arm and the lower arm, but also the hand, which may be selected by the candidate frame, so it is necessary to calculate the confidence score of each candidate frame and select the one with the highest confidence score to determine the hand position); S23: extracting hand key points from the posture behavior area (i.e., the hand key points are the joints between all the phalanges of the hand and the joints at the wrist); connecting the hand key points to obtain a hand skeleton structure, and using the hand skeleton structure as a posture feature image; See Figure 7 This embodiment specifically designs gesture training actions for elderly patients; for example, it includes the OK gesture, the fist gesture, the goodbye gesture, the gesture of the fingertips of one hand facing each other, the gesture of one hand making a fist and the other hand spreading five fingers, and even includes the gesture of one hand palm on the chest and the other hand making a fist and stretching forward, the gesture of one hand palm pressing the fingertips of one hand, the gesture of the thumbs and index fingers of both hands climbing the ladder, and the gesture of one gun shooting four birds; The gesture of the fingertips of one hand facing each other: This gesture requires the patient to align the fingertips of one hand with the fingertips of the other hand, which helps to exercise the fine motor skills of the hand. The gesture of making a fist with one hand and spreading the five fingers of the other hand: This gesture helps to improve the coordination and control of the hand. The gesture of placing the palm of one hand on the chest and stretching the other hand forward with a fist: This action can enhance the strength and stability of the upper limbs. The gesture of pressing the fingertips of one hand with the palm of one hand: By pressing the fingertips, the tactile sensitivity and fine operation ability of the hand can be improved. The gesture of making ladder climbing movements with the thumbs and index fingers of both hands: This gesture helps to improve the flexibility and coordination of the hand. The gesture of shooting four birds with one gun: This is a finger game. By simulating the shooting action, it can test the finger flexibility of elderly patients, and even his hand-eye coordination and quick reaction ability.

[0030] Specifically, Figure 3 As shown, in step S23, the hand key points are extracted from the posture behavior area; the hand key points are connected to obtain the hand skeleton structure, and the hand skeleton structure is used as the posture feature image; the specific operation steps are as follows: S231: using the Harris corner algorithm to calculate the sum of the squares of the gradients and the sum of the products of the two in the x-direction and the y-direction of each pixel point in the gesture behavior area; performing a sliding window on the gesture behavior area to construct a second-order Harris matrix; ; In the formula, It is expressed as calculating the sum of the squared gradients in the x direction of each pixel; It is expressed as calculating the sum of the products of the x and y gradients of each pixel; It is expressed as calculating the sum of the squared gradients in the y direction of each pixel; And calculate the Harris response value R, preset the response value threshold, and determine whether the Harris response value R is greater than the response value threshold; Among them, Harris response value ;in is the determinant of the matrix, is the trace of the matrix, is a constant.

[0031] If yes, the pixel point corresponding to the Harris response value R is taken as the pixel point of the local maximum value, and the pixel point is taken as the corner point of the window, and the coordinates are extracted as the corner point coordinates, thus ending the corner point confirmation processing operation; Below are the steps of the algorithm and a detailed description of each step: It should be noted that the Harris corner detection algorithm is particularly suitable for detecting corner points in images, which can be used in a variety of applications such as image registration, target tracking, and feature matching.

[0032] Calculate the gradient: For each pixel in the image, first calculate its gradient in the x-direction and y-direction. This is usually done by using the Sobel operator or other gradient operators.

[0033] and Represent the gradient images in the x direction and y direction respectively.

[0034] Then calculate the square and product of the gradients: For each pixel, calculate the square of the gradient and the product of the gradients .

[0035] These values ​​will be used to construct the Harris matrix.

[0036] The square of the x-direction gradient of each pixel; The square of the y-direction gradient of each pixel; The product of the x and y gradients of each pixel; within a sliding window, the above three quantities are summed (that is, at each pixel position in the gesture behavior area, the gradient information of these areas is calculated for the pixels in the local area through a sliding window, so that the Harris matrix can be constructed); Then implement a sliding window and sum: use a sliding window to move across the image, and for each image point within the window, calculate the sum of the squares and products of the above gradients. This can be done by and This is achieved by integrating.

[0037] Construct the Harris matrix: For each window, construct a The Harris matrix M, whose elements are defined as: The summation here is performed within a sliding window.

[0038] Calculation of Harris response value: Harris response value R can be calculated by the following formula: ;in is the determinant of the matrix, is the trace of the matrix, is an empirical constant, usually ranging from 0.04 to 0.06>.

[0039] Threshold judgment and corner point extraction: Set a threshold. If the Harris response value R of a window is greater than this threshold, it is considered that there is a corner point in the window.

[0040] Then, the coordinates of the corner points can be extracted from the window whose response value is greater than the threshold, that is, the pixel point corresponding to the Harris response value R is taken as the pixel point of the local maximum value, and the pixel point is taken as the corner point of the window, and the coordinates are extracted as the corner point coordinates.

[0041] The corner point response function is filtered according to the preset response value threshold, and only the pixels whose response function is greater than the threshold are retained. These pixels are the detected corner points.

[0042] For example: suppose a simple image contains obvious corners.

[0043] We first calculate the gradient of the image and then slide a window.

[0044] In the standard implementation of Harris corner detection, a Harris response value is calculated for each pixel.

[0045] For each pixel in the image, we consider a window (usually a square region) centered at that pixel.

[0046] Within this window, the statistics of the gradients are calculated ( and and).

[0047] Construct a Harris matrix for the center pixel using the above statistics.

[0048] This matrix is ​​then used to calculate the Harris response value for the center pixel.

[0049] The sliding window plays an important role in this process. The window is used to calculate the gradient statistics of the local area around each pixel, which is used to construct the Harris matrix.

[0050] For each window, the embodiment of the present application calculates and The sum.

[0051] Next, construct the Harris matrix and calculate the response value R.

[0052] If R is greater than the preset response value threshold, the center point of the window is marked as a corner point.

[0053] The above process can be implemented programmatically, for example using Python and OpenCV library.

[0054] Here is a simplified code example: Python import cv2 import numpy as np # Read the image image = cv2.imread('path_to_image.jpg', 0) # 0 means reading in grayscale mode # Calculate gradients Ix = cv2.Sobel(image, cv2.CV_64F, 1, 0, ksize=3) Iy = cv2.Sobel(image, cv2.CV_64F, 0, 1, ksize=3) # Calculate the square and product of the gradient Ix2 = Ix**2; Iy2 = Iy**2; Ixy = Ix*Iy; It should be noted that Ix**2: The power operator ** is used here, which means Ix squared (that is, Ix multiplied by itself). Iy**2: Similarly, Iy**2 is also Iy squared. Ixy = Ix*Iy: The multiplication operator * is used here, which means the product of Ix and Iy.

[0055] # Harris corner detection dst = cv2.cornerHarris(image, blockSize=2, ksize=3, k=0.04) #Thresholding and marking corners dst = cv2.dilate(dst, None) image[dst>0.01*dst.max()] = [0, 0, 255] # Display results cv2.imshow('Harris Corners', image) cv2.waitKey(0) cv2.destroyAllWindows() This code will read an image, calculate its Harris corner points, and mark those corner points on the image.

[0056] By adjusting the threshold, you can control the sensitivity of corner detection and ultimately determine which are the final corner points.

[0057] S232: Pre-build hand model based on MediaPipe Hands algorithm: Using the MediaPipe Hands algorithm, normalize and resize the frame image to be detected of each frame of the video recording, and convert it into an RGB image to obtain an RPG image to be detected; Predicting key points of the RPG image to be detected to obtain a preliminary hand skeleton model; Performing confidence scoring on each predicted key point in the preliminary hand skeleton model; predicting a confidence score threshold; and determining whether the confidence score of each predicted key point is lower than the confidence score threshold; If not, the predicted key points are retained as hand key points; If so, the predicted key point is the ground confidence score, the predicted key point with the low confidence score is determined with adjacent predicted key points with high confidence scores, and the interpolated position of the predicted key point with the low confidence score is calculated by weighted average of the adjacent predicted key points with high confidence scores; the position of the predicted key point with the low confidence score is updated with the interpolated position to obtain the hand key point; The hand model is constructed by all the obtained hand key points; Extracting hand key points from the gesture behavior area using a pre-built hand model (i.e., the pre-built hand model may include the shape, posture, and motion pattern of the hand; thus, the hand can be recognized and tracked in complex or changing scenarios); It should be noted that the MediaPipe Hands algorithm can be used to normalize the RPG image to be detected and adjust the image size (that is, the gesture behavior area can be adjusted to ), which helps to standardize the input data and ensure the consistency of the network input; because the MediaPipe algorithm usually processes RGB images, it is necessary to convert the posture behavior area into RGB images; predict the key points of the hand for the RPG image to be detected. There are 21 key points in the hand, each key point represents a different position of the hand, and these predicted key points are combined into a preliminary hand model; At the same time, a confidence score is given to each predicted key point. The predicted key points with high confidence scores indicate that the key points in the hand are most accurately located and do not need to be updated. However, there may be positional deviations for the predicted key points with low confidence scores, because when the hand is moving, there may be some unclear areas in the video recording, such as bright light, occlusion, hand movements, etc. Therefore, it is necessary to find the adjacent predicted key points with high confidence scores for the predicted key points with low confidence scores and calculate the weighted average interpolation to update the predicted key points with low confidence scores to the correct position. For all the hand key points with confirmed positions and the updated hand key points, an accurate hand model is constructed (i.e., for each detected hand area, use MediaPipe The hand key point prediction model of Hands predicts 21 3D key points to build an initial hand model; refines the initial hand model: calculates the confidence scores of the 21 key points; for key points with confidence lower than 0.5, uses the position information of adjacent high-confidence key points for interpolation adjustment; based on the adjusted key point positions, updates the hand skeleton structure; repeats the above steps up to 3 times until all key point confidences are higher than 0.5 or the maximum number of iterations is reached), which can improve the accuracy of the hand model and ensure the precise position of the key points; It should be noted that the MediaPipe Hands algorithm is used to pre-build a hand model. The pre-built hand model can quickly capture the hand in the gesture behavior area of ​​the moving video recording, and recognize the key point coordinates of the captured hand according to the pre-built hand model (i.e., the hand frame); S233: verifying the corner points and hand key points detected by each algorithm to obtain the hand key points, connecting the hand key points to obtain the hand bone structure, and using it as a posture feature image; It should be noted that the Harris corner algorithm can detect the corner points of the hand in the gesture behavior area, and further operations can be performed to confirm the coordinate positions of the corner points; the MediaPipe Hands algorithm can provide the key points of the human hand; and the joint points of the hand and the hand bone structure can be obtained by continuing to identify through their respective algorithms. However, the Harris corner algorithm is mainly used to detect corner points in images, and can detect hand corner points in the gesture behavior area, but its limitation is that it is not specifically for the hand structure, so it may not be able to accurately capture the complex details of the hand; while the MediaPipe Hands algorithm has a good detection effect on hand key point detection and can provide detailed key points of the hand, but it is highly dependent on the quality and diversity of the training data, and may not be accurate enough for hand gestures that are not in the training data; therefore, it is necessary to double-identify and merge the corner points and key points to obtain the hand key points, and identify the hand bone structure with a three-dimensional and clear structure through the hand key points; Mutual verification of corner points and hand key points can improve the accuracy of gesture recognition; corner points usually represent local features, while hand key points provide the overall structural information of the hand. Mutual verification of the two can better capture the dynamic changes of the hand; for example, in gesture recognition, corner points can help identify specific movements of fingers, while key points provide the overall framework of the hand (i.e., for complex gestures, hand key points alone may not be able to accurately capture the details. After mutual verification of corner points, the relative positions and movements between fingers can be better described, thereby realizing the interpretation of complex gestures); specifically, Figure 4 As shown, in step S233, the corner points and hand key points detected by each algorithm are mutually verified to obtain the hand key points, and the hand key points are connected to obtain the hand bone structure. The specific operation steps are as follows: S2331: constructing a correspondence matrix M between the key points of the hand and the corner points; using the KNN algorithm to find the closest corner point for each key point of the hand; It should be noted that, assuming that the MediaPipe Hands algorithm obtains 21 hand key points and the Harris corner point detection algorithm obtains 50 corner points, the correspondence matrix M between the hand key points and the corner points is constructed. It means that the corresponding matrix M has 21 rows and 50 columns; the rows of the corresponding matrix M represent the 21 key points of the hand, and each row corresponds to a key point; the columns of the corresponding matrix M represent each of the 50 corner points; each element of the corresponding matrix M represents the relationship between a key point and a corner point (that is, each element in the matrix usually stores distance and similarity, which is used to represent the relationship between a key point and a corner point); Through 21 hand key points (K1, K2...K21) and 50 corner points (A1, A2...A50); for each hand key point (such as K1), find the 5 nearest corner points of each hand key point. If the relationship between a hand key point and a corner point is determined by the nearest neighbor algorithm, then for each hand key point, the corresponding matrix M will store its relationship information with the 5 nearest corner points; for example, if K1 is closest to A5, A12, A19, A23, A35, then in the row where K1 is located, the values ​​of these 5 columns may be marked as 1 or store the distances of these corner points; by constructing a corresponding matrix M, it is used to represent each hand key point and The relationship between all corner points, the corresponding matrix M can analyze the matching relationship between the hand key points and the corner points (that is, the hand key points are connected in the subsequent processing to obtain the hand bone structure (that is, gesture recognition, motion tracking, etc. of the gesture feature image)); S2332: Calculate the matching degree of the corner point coordinates and the hand key point coordinates according to the distance between the hand key point and each of the closest corner points and the grayscale of the corner point, as the matching degree index xc (that is, for the same gesture recognition, the corner points and hand key points are obtained by different algorithms, and the corner point coordinates and the hand key point coordinates may overlap or be very close, so the matching degree is calculated to determine whether the corner point coordinates and the hand key point coordinates overlap or are very close); Preset a matching degree threshold o; determine whether the matching degree between the corner point coordinates and the hand key point coordinates is greater than the matching degree threshold o; If yes, the matching point pair of the corner point coordinates and the hand key point coordinates is successfully matched; It should be noted that for each corner point coordinate, the distance to the nearest hand key point coordinate can be calculated using the Euclidean distance. The distance between the two reflects the proximity between the corner point and the hand key point. A smaller distance means that the corner point is closer to the key point, making the match more accurate. The gray value of the corner point reflects the brightness or contrast of the corner point in the image. Corner points are usually where the grayscale changes greatly in the image. Therefore, the gray value can be used to evaluate the significance of the corner point. Mutual verification of distance and grayscale value can more comprehensively evaluate the matching of corner points and key points. The distance reflects the geometric proximity, while the grayscale value reflects the saliency and contrast of the corner points, which can further evaluate the matching quality of corner points and key points. S2333: Calculate the average distance between the hand key point and the corner point for the matching point pairs of the successfully matched corner points and the hand key points, as the position deviation index J; It should be noted that the average distance can measure the overall deviation of all matching point pairs; if the average distance is small, it means that the matching is more accurate and the position information of key points and corner points is more consistent; conversely, if the average distance is large, it means that there may be large errors in the matching.

[0058] S2334: Calculate edge gradients for the successfully matched corner points and the matching points of the hand key points using the Sobel operator, and calculate the difference between the edge gradients as the edge change feature p; It should be noted that by comparing the edge gradient differences between the hand key points and the corner points, it is possible to capture subtle changes in position and more accurately determine the matching correspondence between the hand key points and the corner points (i.e., using the Sobel operator to calculate the image gradient, and then calculating the absolute average of the gradient differences between the hand key points and the corresponding corner points); S2335: Combining the matching index xc, the position deviation index J and the edge change feature p into a fused feature vector F, wherein the fused feature vector F=[xc, J, p]; It should be noted that all calculated eigenvalues ​​[xc, J, p] are combined into a vector F; this eigenvector is a comprehensive description of the key points and corner points in the image; S2336: constructing a decision tree model using random forest; constructing a classifier using the decision tree model, inputting the fusion feature vector F into the classifier, obtaining the identified hand key points, and then connecting to obtain the hand bone structure; It should be noted that the [xc, J, p] values ​​are extracted from the key nodes and corner points of the hand as features describing the gesture behavior area; these feature values ​​are combined into a fused feature vector F = [xc, J, p] to comprehensively describe the content of the gesture behavior area (i.e., xc, J and p are combined into a 3D feature vector; the feature vector is input into a pre-trained random forest classifier (containing 50 decision trees), which can recognize different common sign language gestures); The classifier uses the decision rules learned during its training process (for example, in a random forest, the voting results of multiple decision trees are input to the fused feature vector F, and can identify the hand posture (i.e., the hand bone structure) composed of the key points of the hand in the posture behavior area. After the hand bone structure is identified, the hand posture can be directly identified, which is the training gesture (i.e., OK gesture, fist gesture, goodbye gesture)); specifically, Figure 5 As shown, Figure 6 As shown, in step S2336, a decision tree model is constructed using random forest, and the specific operation steps are as follows: S23361: Collect multiple sample gesture images, and label each collected sample gesture image with a gesture category; extract features from the multiple sample gesture images, where the extracted features include key point positions, corner point positions, and edge features; It should be noted that, in order to pre-build a decision tree, a large number of sample gesture images must first be collected, each of which contains different gestures, such as OK gesture, goodbye, etc.; these sample gesture images are respectively labeled with the categories of gestures such as "OK gesture, goodbye, etc."; Before training the decision tree, it is necessary to extract features from the sample gesture image and use the extracted features as input data for model training. The extracted key point positions, corner point positions and edge features are used as the basis for building the decision tree model. The extracted features are the information that best represents the content of the sample gesture image. S23362: taking the extracted features as a data set, dividing it into a training set and a test set, training the training set, and constructing a decision tree; S23363: selecting a feature and a threshold as splitting data for the node of the decision tree; recursively splitting data for each node from the root node of the decision tree, and stopping node splitting when the sample gesture images of the node in the decision tree all belong to the same gesture category (i.e., if the samples in the node already completely belong to the same category, no further splitting is required because the node has reached the maximum purity; for example, if all the sample gesture images in a node are "handshake" gestures, the decision tree will mark this node as the "handshake" category and stop further splitting); It should be noted that the decision tree contains many nodes, and a feature and a threshold are selected for each node (that is, the threshold is a specific value used to divide the data into different subsets, and the value set for a specific feature is used to divide the data set into two parts; by selecting a suitable threshold, the decision tree can divide the data more accurately at each node, so that the samples in each subset are concentrated in the same category as much as possible, thereby improving the prediction accuracy and efficiency of the model; for example: when making a gesture, the angle of the hand is 45°. If the angle of a gesture is less than or equal to 45 degrees, then the gesture will be divided into one subset; and the gesture with an angle greater than 45 degrees will be divided into another subset; in this way, by setting the threshold, the gesture data can be divided into two groups) as split data, for example: "hands above the shoulders" may be more inclined to the "waving" gesture, while "hands not above the shoulders" may be more inclined to the "handshake" gesture; by splitting the data, each subset is more clearly split in the gesture category, thereby improving the classification accuracy of the decision tree for gestures, and ensuring that the samples in the child nodes after the split are concentrated in the same category as much as possible; S23364: Evaluate the decision tree using the test set to obtain a decision tree model; It should be noted that the obtained decision tree is verified through the test set, and whether the decision tree model is trained and can be used is determined based on the accuracy and precision of gesture recognition.

[0059] Embodiment 2 like Figure 8 As shown, this embodiment 2 also provides a system for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, including: a collection module 10; an acquisition module 20; a matching module 30; The acquisition module 10 is used to acquire the gesture video recording, obtain each frame of the gesture video recording, and obtain the frame image to be detected; The acquisition module 20 is used to extract the posture behavior area of ​​each frame image to be detected; extract the posture feature of the posture behavior area to obtain a posture feature image; The matching module 30 is used to match the gesture feature image with the history database to identify the gesture behavior.

[0060] Embodiment 3 On the other hand, the third embodiment is based on the method for handling abnormal motions of elderly patients implementing video frame image gesture training based on MediaPipeHands provided in the first embodiment of the invention, and also provides a computer storage medium 1140 (referred to as storage medium).

[0061] like Fig. 9FIG. 1 is a schematic diagram of a computer storage medium structure framework provided in Embodiment 3 of the present invention, which includes: A memory 1130, used for storing computer programs; The communication interface 1120 is used to connect the memory 1130 to the processor 1110; The processor 1110 is used to execute a computer program to implement an embodiment disclosed in combination with any of the above-mentioned embodiments, involving a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands.

[0062] In summary, the example of the present invention proposes a method and system for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands. By collecting gesture video recordings and collecting the image of each frame to be detected, the gesture posture can be recognized more quickly. Furthermore, the candidate frames of the image to be detected are selected, and the candidate frames with the highest confidence scores are selected by using the confidence scores as the posture behavior areas; this can quickly find the human hand, save time for subsequent recognition, and improve the speed; Furthermore, the Harris corner algorithm is first used to calculate the square sum product of the gradient in the x-direction and y-direction of each pixel in the posture behavior area, so that the gradient of the posture behavior area can be calculated, and then a second-order Harris matrix is ​​constructed by the calculated gradient and the sliding window; the Harris response value is calculated by the second-order Harris matrix, so that the corner points in each sliding window in the posture behavior area can be found; Furthermore, the MediaPipe Hands algorithm is used to pre-build a hand model, which can obtain the key points of the complete hand and the hand posture formed by each key point of the hand. The corner points and the key points of the hand are then verified against each other to form the joint points of the hand. By connecting the joint points, the hand posture feature map can be obtained, and the hand posture can be known. The posture feature image is matched with the sample posture images in the historical database, and finally the posture of the gesture is identified.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, characterized in that: The steps are as follows: Collecting gesture video recordings, obtaining each frame of the gesture video recordings, and obtaining the frame image to be detected; Extracting a posture behavior region from each frame of the to-be-detected frame image; extracting posture features from the posture behavior region to obtain a posture feature image; The gesture feature image is matched with a historical database to identify the gesture behavior.

2. According to claim 1, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, characterized in that: The gesture feature image is matched with the historical database to identify the gesture behavior. The specific operation steps are as follows: Use the YOLO algorithm to select the candidate frame of each frame to be detected, and obtain multiple region candidate frames; Calculate the confidence score for each region candidate box; Sort the confidence scores of the candidate boxes of each segmented image, and select the candidate box with the highest confidence score as the posture behavior area; Extracting hand key points from the posture behavior area; The key points of the hand are connected to obtain a hand skeleton structure, and the hand skeleton structure is used as a posture feature image.

3. According to claim 2, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, characterized in that: Extracting hand key points from the posture behavior area; connecting the hand key points to obtain a hand skeleton structure, and using the hand skeleton structure as a posture feature image. The specific operation steps are as follows: The Harris corner algorithm is used to calculate the sum of the squares of the gradients and the sum of the products of the two in the x-direction and the y-direction of each pixel point in the posture behavior area; a sliding window is performed on the posture behavior area to construct a second-order Harris matrix; ； In the formula, It is expressed as calculating the sum of the squared gradients in the x direction of each pixel; It is expressed as calculating the sum of the products of the x and y gradients of each pixel; It is expressed as calculating the sum of the squared gradients in the y direction of each pixel; And calculate the Harris response value R, preset the response value threshold, and determine whether the Harris response value R is greater than the response value threshold; Among them, Harris response value ;in is the determinant of the matrix, is the trace of the matrix, is a constant; If yes, the pixel point corresponding to the Harris response value R is taken as the pixel point of the local maximum value, and the pixel point is taken as the corner point of the window, and the coordinates are extracted as the corner point coordinates, thus ending the corner point confirmation processing operation; Pre-build a hand model based on the MediaPipe Hands algorithm; Using a pre-built hand model, extracting key points of the hand from the gesture behavior area; The corner points and hand key points detected by each algorithm are mutually verified to obtain the hand key points, and the hand key points are connected to obtain the hand bone structure as a posture feature image.

4. According to claim 3, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, characterized in that: Pre-build the hand model according to the MediaPipe Hands algorithm. The specific steps are as follows: Using the MediaPipe Hands algorithm, normalize and resize the frame image to be detected of each frame of the video recording, and convert it into an RGB image to obtain an RPG image to be detected; Predicting key points of the RPG image to be detected to obtain a preliminary hand skeleton model; Performing confidence scoring on each predicted key point in the preliminary hand skeleton model; predicting a confidence score threshold; and determining whether the confidence score of each predicted key point is lower than the confidence score threshold; If not, the predicted key points are retained as hand key points; If so, the predicted key point is the ground confidence score, the predicted key point with the low confidence score is determined with adjacent predicted key points with high confidence scores, and the interpolated position of the predicted key point with the low confidence score is calculated by weighted average of the adjacent predicted key points with high confidence scores; the position of the predicted key point with the low confidence score is updated with the interpolated position to obtain the hand key point; The hand model is constructed by constructing all the obtained hand key points.

5. According to claim 4, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands is characterized in that: The corner points detected by each algorithm are mutually verified with the hand key points to obtain the hand key points, and the hand key points are connected to obtain the hand bone structure. The specific operation steps are as follows: Construct the correspondence matrix M between the key points of the hand and the corner points; use the KNN algorithm to find the closest corner point for each key point of the hand; The matching degree between the corner point coordinates and the hand key point coordinates is calculated based on the distance between the hand key point and each closest corner point and the grayscale of the corner point, which is used as the matching degree index xc; Preset a matching degree threshold o; determine whether the matching degree between the corner point coordinates and the hand key point coordinates is greater than the matching degree threshold o; If yes, the matching point pair of the corner point coordinates and the hand key point coordinates is successfully matched; For the matching point pairs of successfully matched corner points and hand key points, the average distance between the hand key points and the corner points is calculated as the position deviation index J; Calculate the edge gradient of the successfully matched corner points and the matching points of the hand key points using the Sobel operator, and calculate the difference between the edge gradients as the edge change feature p; The matching index xc, the position deviation index J and the edge change feature p are combined into a fused feature vector F, wherein the fused feature vector F=[xc, J, p]; A decision tree model is constructed using random forest; a classifier is constructed using the decision tree model, and the classifier is input with the fusion feature vector F to obtain the identified hand key points, thereby connecting the hand bone structure.

6. According to claim 5, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands is characterized in that: Use random forest to build a decision tree model. The specific steps are as follows: Collecting multiple sample gesture images, and labeling each collected sample gesture image with a gesture category; extracting features from the multiple sample gesture images, wherein the extracted features include key point positions, corner point positions, and edge features; The extracted features are used as a data set, divided into a training set and a test set, the training set is trained, and a decision tree is constructed; Selecting a feature and a threshold as splitting data for the nodes of the decision tree; Recursively splitting data for each node from the root node of the decision tree, and stopping node splitting when the sample gesture images of the nodes in the decision tree all belong to the same gesture category; The decision tree is evaluated using the test set to obtain a decision tree model.

7. According to claim 6, a method for handling abnormal motions of elderly patients in video frame image gesture training based on MediaPipe Hands, characterized in that: The gesture feature image is matched with the historical database to identify the gesture behavior. The specific operation steps are as follows: Acquire a historical database, and collect sample posture images from the historical database; match the posture feature image with the sample posture images in the historical database, and calculate the matching degree; Preset a matching degree threshold; determine whether the matching degree is greater than the matching degree threshold; If yes, then the historical database contains a known gesture, and a sample gesture image with the highest matching degree is obtained as the gesture behavior; If not, the historical database does not contain the posture feature image, and the historical database is manually updated.

8. A system for abnormal motion processing of video frame image gesture training for elderly patients based on MediaPipe Hands, comprising: Acquisition module; Get module; Matching module; The acquisition module is used to collect gesture video recordings, obtain each frame of the gesture video recordings, and obtain the frame image to be detected; The acquisition module is used to extract the posture behavior area of ​​each frame image to be detected; extract the posture feature of the posture behavior area to obtain a posture feature image; The matching module is used to match the gesture feature image with the historical database to identify the gesture behavior.

Images