Method, system, medium and electronic device for enhancing stability of hand key points

Abstract

This disclosure provides a method, system, medium, and electronic device for enhancing the stability of hand keypoints. The method includes: inputting a video stream containing a target hand into a preset hand target detection model to determine an image of the target hand; the preset hand target detection model includes a temporal feature fusion module; inputting the image of the target hand into a keypoint detection model to determine a preset number of 2D keypoint coordinates and a preset number of 3D keypoint coordinates of the target hand; and performing adaptive filtering on the preset number of 2D keypoint coordinates and the preset number of 3D keypoint coordinates of the target hand at each time step in the temporal domain to determine the target keypoint coordinates of the target hand at each time step. This disclosure improves the input and output through temporal feature fusion and adaptive filtering, thereby enhancing the stability of hand keypoint detection.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a method, system, medium, and electronic device for enhancing the stability of key points on a hand. Background Technology

[0002] Hand keypoint recognition technology is an important branch of computer vision, possessing high academic research value and broad market application prospects. Hand keypoint detection is a task that extracts the coordinates of hand skeletal points from images or videos. This task models the human hand as a pose composed of a set of important hand keypoints and detects the position coordinates of these keypoints in images or videos containing the hand. Based on the dimension of the output keypoint positions, it is divided into 2D keypoint detection and 3D keypoint detection. 2D hand keypoint detection outputs the 2D coordinates of the keypoints, while 3D hand keypoint detection outputs the 3D coordinates of the keypoints.

[0003] Commonly used hand keypoint detection techniques include traditional methods and deep learning methods. Deep learning methods, in particular, obtain high-level features that can express high-level semantic information about the hand, possessing stronger descriptive capabilities. Using deep learning for hand keypoint detection can significantly improve detection accuracy. Therefore, deep learning has become the most important method for hand keypoint detection.

[0004] While deep learning-based hand keypoint detection has broad application prospects, the stability of the detected hand keypoints suffers from significant issues. In practical applications, the positions of hand keypoints detected by deep learning models exhibit considerable jitter, resulting in poor stability of the keypoints output by the model. Since the internal structure of deep learning models cannot be manually controlled, improving the stability of the model itself is extremely difficult. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a method, system, medium and electronic device for enhancing the stability of key points of the hand.

[0006] To achieve the above objectives, according to a first aspect of this disclosure, a method for enhancing the stability of key points of the hand is provided, comprising:

[0007] A video stream containing a target hand is input into a preset hand target detection model to determine the image of the target hand. The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame and the current frame of the video stream.

[0008] The image of the target hand is input into the key point detection model to determine a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand.

[0009] The coordinates of a preset number of 2D key points and a preset number of 3D key points of the target hand at each moment are adaptively filtered in the time domain to determine the coordinates of the target key points of the target hand at each moment.

[0010] Optionally, the step of inputting a video stream containing the target hand into a preset hand target detection model to determine the image of the target hand includes:

[0011] A hand detection algorithm is used to extract feature images from each frame of the video stream;

[0012] The temporal feature fusion module is used to fuse the feature images of the previous frame and the current frame of the video stream to determine the fused feature image.

[0013] The fused feature image is input into the target detection head network to determine the bounding box of the hand region and the confidence level corresponding to the bounding box of the hand region;

[0014] The target hand region bounding box is determined based on the confidence level corresponding to the bounding box of the hand region and the preset confidence level threshold.

[0015] The bounding box of the target hand region is cropped to determine the image of the target hand.

[0016] Optionally, the step of fusing the feature images of the previous frame and the current frame of the video stream using the temporal feature fusion module to determine the fused feature image includes:

[0017]

[0018] in, w represents the fused feature image at time i. i-1→i F represents the weighted average of the feature images of the image corresponding to the (i-1)th time frame. i-1 w represents the feature image of the corresponding frame at time i-1. i→i F represents the weighted average of the feature images of the image corresponding to the i-th frame. i This represents the feature image of the frame corresponding to time i.

[0019] Optionally, determining the target hand region bounding box based on the confidence level corresponding to the hand region bounding box and a preset confidence threshold includes:

[0020] The bounding box of the hand region corresponding to the confidence level being greater than the preset confidence level threshold is determined as the bounding box of the target hand region.

[0021] The bounding box of the hand region corresponding to the confidence level not exceeding the preset confidence level threshold is determined as the non-target hand region bounding box.

[0022] Optionally, the key point detection model includes a 2D key point detection model and a 3D key point detection model;

[0023] The step of inputting the image of the target hand into the key point detection model to determine the position coordinates of a preset number of 2D key points and a preset number of 3D key points of the target hand includes:

[0024] The image of the target hand is input into the 2D key point detection model to determine the position coordinates of a preset number of 2D key points of the target hand.

[0025] The image of the target hand is input into the 3D key point detection model to determine the position coordinates of a preset number of 3D key points of the target hand.

[0026] Optionally, the step of performing adaptive filtering on the preset number of 2D keypoint coordinates of the target hand at each moment and the preset number of 3D keypoint coordinates of the target hand at each moment in the time domain to determine the target keypoint coordinates of the target hand at each moment includes:

[0027] Based on the preset number of 2D key point position coordinates and the preset number of 3D key point position coordinates of the target hand at each moment, the original coordinate sequence of the key points of the target hand is obtained, and the original coordinate sequence includes the 2D key point position coordinates and the 3D key point position coordinates.

[0028] Based on the original coordinate sequence of the key points of the target hand and the position change time of the key points of the target hand, determine the change rate of the key points of the target hand at each moment;

[0029] Based on a preset constant cutoff frequency, the rate of change of the key points of the target hand at each moment is smoothed to determine the smoothing rate of the key points of the target hand at each moment.

[0030] The adaptive smoothing factor for each moment is determined based on the smoothing speed of the key points of the target hand at each moment.

[0031] Based on the adaptive smoothing factor at each time step, the key point coordinates of the target hand at each time step are adaptively filtered to determine the target key point coordinates of the target hand at each time step.

[0032] Optionally, the step of adaptively filtering the key point coordinates of the target hand at each time step based on the adaptive smoothing factor at each time step to determine the target key point coordinates of the target hand at each time step includes:

[0033]

[0034] in, X represents the coordinates of the target key points of the target hand at time i after smoothing with an adaptive smoothing factor, α represents the adaptive smoothing factor at time i, and X represents the target key points of the target hand at time i. i The coordinates of key points of the target hand at the i-th time before smoothing are given. The coordinates of the key points of the target hand at time i-1 after smoothing are represented.

[0035] According to a second aspect of this disclosure, a stability enhancement system for key hand points is provided, comprising:

[0036] The target detection module is used to input a video stream containing a target hand into a preset hand target detection model to determine the image of the target hand. The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame image and the feature images of the current frame image of the video stream.

[0037] The key point detection module is used to input the image of the target hand into the key point detection model to determine a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand.

[0038] An adaptive filtering module performs adaptive filtering on a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand at each time step in the time domain to determine the target key point coordinates of the target hand.

[0039] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method for enhancing the stability of key hand points provided in the first aspect of this disclosure.

[0040] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0041] A memory on which computer programs are stored;

[0042] A processor is configured to execute the computer program in the memory to implement the steps of the method for enhancing the stability of key hand points provided in the first aspect of this disclosure.

[0043] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:

[0044] The above technical solution employs a hand target detection model that includes a temporal feature fusion module to process video streams containing target hands. By utilizing the correlation information of feature images from adjacent frames of the video stream, it determines temporally stable images of target hands, preventing instability caused by hand movements and filtering out background interference from the input level, thus improving the input stability of the keypoint detection model. An adaptive filtering method is used to process the keypoint coordinates of the target hand, improving temporal stability from the output level and determining smooth and stable target keypoint outputs, effectively enhancing the stability of hand keypoints. Attached Figure Description

[0045] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0046] Figure 1 This is a flowchart illustrating a method for enhancing the stability of key points of a hand according to an exemplary embodiment.

[0047] Figure 2 This is a flowchart illustrating a method for determining an image of a target hand according to an exemplary embodiment.

[0048] Figure 3 This is a flowchart illustrating a method for determining the coordinates of key points of a target hand at each moment, according to an exemplary embodiment.

[0049] Figure 4 This is a comparison chart of the stability of data curves for key points of the hand, according to an exemplary embodiment.

[0050] Figure 5 This is a block diagram illustrating a stability enhancement system for key hand points according to an exemplary embodiment.

[0051] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0052] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.

[0053] Figure 1This is a flowchart illustrating a method for enhancing the stability of key points in a hand, according to an exemplary embodiment. Figure 1 As shown, a method for enhancing the stability of key points of the hand includes steps S11 to S13.

[0054] S11, input the video stream containing the target hand into the preset hand target detection model to determine the image of the target hand.

[0055] The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame and the current frame of the video stream.

[0056] As an example, a pre-defined hand target detection model is used to detect each frame I(x,y) of a video stream containing a target hand, resulting in the bounding box (x,y) of the target hand region. min ,y min ,x max ,y max The image I of the target hand is obtained by cropping along the bounding box of the target hand region. h (x,y).

[0057] S12, input the image of the target hand into the key point detection model, and determine the position coordinates of a preset number of 2D key points and a preset number of 3D key points of the target hand.

[0058] The keypoint detection model includes a 2D keypoint detection model and a 3D keypoint detection model.

[0059] The image of the target hand is input into both the 2D keypoint detection model and the 3D keypoint detection model:

[0060] As an example, an image of the target hand is input into a 2D keypoint detection model to determine the coordinates of a preset number of 2D keypoints on the target hand.

[0061] In this disclosure, a deep learning-based 2D hand keypoint detection model is used, such as the mmpose 2D hand keypoint detection algorithm.

[0062] The preset quantity k can be 21, determining the image I of the target hand. h The coordinates of the k 2D keypoints (x, y) are {P1(x1, y1),...P k (x k ,y k )}.

[0063] As another example, an image of the target hand is input into a 3D keypoint detection model to determine the coordinates of a preset number of 3D keypoints on the target hand.

[0064] In this disclosure, a deep learning-based 3D hand keypoint detection model is used, such as the FrankMoCap 3D hand keypoint detection algorithm.

[0065] The preset quantity k can be 21, determining the image I of the target hand. h The coordinates of the k 3D keypoints in (x,y) are {P1(x1,y1,z1),...P k (x k ,y k ,z k )}.

[0066] S13, perform adaptive filtering on the preset number of 2D key point coordinates of the target hand at each moment and the preset number of 3D key point coordinates of the target hand at each moment in the time domain to determine the target key point coordinates of the target hand at each moment.

[0067] Specifically, a preset number of 2D keypoint coordinates and a preset number of 3D keypoint coordinates of the target hand in each frame of the video stream are stored and processed. The preset number of 2D keypoint coordinates of the target hand in each frame represents the preset number of 2D keypoint coordinates of the target hand at each moment, and the preset number of 3D keypoint coordinates of the target hand in each frame represents the preset number of 3D keypoint coordinates of the target hand at each moment.

[0068] Adaptive filtering is applied to the coordinates of each key point, and the coordinates of the key points after adaptive filtering are used as the stable target key point coordinates of the hand.

[0069] The above technical solution employs a hand target detection model that includes a temporal feature fusion module to process video streams containing target hands. By utilizing the correlation information of feature images from adjacent frames of the video stream, it determines temporally stable images of target hands, preventing instability caused by hand movements and filtering out background interference from the input level, thus improving the input stability of the keypoint detection model. An adaptive filtering method is used to process the keypoint coordinates of the target hand, improving temporal stability from the output level and determining smooth and stable target keypoint outputs, effectively enhancing the stability of hand keypoints.

[0070] Figure 2 This is a flowchart illustrating a method for determining an image of a target hand according to an exemplary embodiment.

[0071] like Figure 2As shown, in some possible embodiments, a video stream containing the target hand is input into a preset hand target detection model to determine the image of the target hand, including S21 to S25.

[0072] S21, a hand detection algorithm is used to extract feature images from each frame of the video stream.

[0073] Among them, the hand detection algorithm can adopt a hand detection algorithm based on the Yolov3 network to extract the feature image F of each frame of the video stream. i .

[0074] S22, the temporal feature fusion module is used to fuse the feature images of the previous frame and the current frame of the video stream to determine the fused feature image.

[0075] In one possible embodiment, the first

[0076]

[0077] in, w represents the fused feature image at time i. i- 1 →i F represents the weighted average of the feature images of the image corresponding to the (i-1)th time frame. i-1 w represents the feature image of the corresponding frame at time i-1. i→i F represents the weighted average of the feature images of the image corresponding to the i-th frame. i This represents the feature image of the frame corresponding to time i.

[0078] In one possible embodiment, the weight w is used to control the weighting of adjacent frames. The weighting of each frame is determined by normalizing the similarity between the feature image of each frame in the video stream and the feature image of the current frame.

[0079] Step 1: Use a convolutional layer to map the feature image of each frame in the video stream to a similarity metric space:

[0080] F e =conv(F)

[0081] Among them, F e denoted as the feature image mapped to the similarity metric space, F represents the feature image of the image in the video stream, and conv represents the convolutional layer.

[0082] Step 2: For a spatial position p adjacent to the feature image of the j-th frame, the weighted weights of the feature images of the j-th frame are:

[0083]

[0084] Among them, w j→i The weights represent the weighted sum of the feature images of the j-th frame at spatial location p. Let represent the feature image of the j-th frame in the similarity metric space. Let i represent the feature image of the i-th frame in the similarity metric space. Let represent the feature image of the u-th frame in the similarity metric space.

[0085] S23, input the fused feature image into the target detection head network to determine the bounding box of the hand region and the confidence level corresponding to the bounding box of the hand region.

[0086] The target detection head network is a detection head based on the Yolov3 network.

[0087] fuse feature images Input a Yolov3-based detection head and output the bounding box (x) of the hand region. min ,y min ,x max ,y max ) and the confidence level s corresponding to the bounding box of the hand region.

[0088] S24. Determine the target hand region bounding box based on the confidence level corresponding to the hand region bounding box and the preset confidence level threshold.

[0089] Among them, the preset confidence threshold s min It can be s min =0.65.

[0090] As an example, the bounding box of the hand region corresponding to the confidence level greater than the preset confidence level threshold is determined as the bounding box of the target hand region.

[0091] As another example, the bounding boxes of the hand regions corresponding to confidence levels not greater than a preset confidence threshold are identified as non-target hand region bounding boxes.

[0092] The bounding boxes of the hand region corresponding to confidence scores lower than the preset confidence threshold are filtered out as invalid bounding boxes, thereby reducing the amount of data and improving the accuracy of the target key point coordinates of the target hand.

[0093] S25, crop the bounding box of the target hand region to determine the image of the target hand.

[0094] Among them, the bounding box of the target hand region (x min ,y min ,x max ,y maxThe image of the target hand is obtained by cropping. h (x,y).

[0095] The above technical solution employs a hand detection algorithm based on temporal feature fusion. By utilizing the correlation information of the target hand in adjacent frames of the video stream, a temporally stable bounding box of the target hand region is obtained, effectively improving the stability of the hand region bounding box and thus obtaining a stable image of the target hand. This improves the stability of hand keypoint detection compared to previous processing methods and filters out interference from backgrounds unrelated to the target hand.

[0096] Figure 3 This is a flowchart illustrating a method for determining the coordinates of key points of a target hand at each moment, according to an exemplary embodiment.

[0097] like Figure 3 As shown, in some possible embodiments, the coordinates of a preset number of 2D key points of the target hand at each moment and the coordinates of a preset number of 3D key points of the target hand at each moment are adaptively filtered in the time domain to determine the coordinates of the target key points of the target hand at each moment, including S31 to S35.

[0098] S31. Based on the preset number of 2D key point coordinates of the target hand at each moment and the preset number of 3D key point coordinates of the target hand at each moment, obtain the original coordinate sequence of the key points of the target hand.

[0099] The original coordinate sequence includes 2D keypoint position coordinates and 3D keypoint position coordinates.

[0100] Based on the coordinate values ​​of each key point, obtain the original coordinate sequence {X0,...X... t}, where X represents the 2D keypoint coordinates (x,y) or the 3D keypoint coordinates (x,y,z).

[0101] S32, based on the original coordinate sequence of the key points of the target hand and the position change time of the key points of the target hand, determine the change rate of the key points of the target hand at each moment.

[0102] In one possible embodiment, the rate of change of key points on the target hand at time i is:

[0103]

[0104] in, X represents the rate of change of key points on the target hand at time i. i This represents the coordinates of key points on the target hand at time i. T represents the coordinates of the key points of the target hand at time i-1 after smoothing. e This represents the time interval from time i-1 to time i when the position of the key points of the target hand changes.

[0105] S33, based on a preset constant cutoff frequency, performs speed smoothing on the change rate of key points of the target hand at each moment, and determines the smoothing speed of key points of the target hand at each moment.

[0106] The preset constant cutoff frequency is: The preset constant cutoff frequency corresponding to the 2D keypoint is The preset constant cutoff frequency corresponding to the 3D key points is

[0107] In one possible implementation, the rate of change of key points on the target hand at time i is smoothed:

[0108]

[0109]

[0110]

[0111] in, The smoothing velocity of key points on the target hand at time i is represented, where the dot at the top of the symbol indicates velocity and ^ indicates smoothing. α represents the rate of change of key points on the target hand at time i-1. d τ represents the velocity smoothing factor. d This represents the time constant calculated using a constant cutoff frequency.

[0112] S34, determine the adaptive smoothing factor at each moment based on the smoothing speed of the key points of the target hand at each moment.

[0113] In one possible embodiment, the adaptive smoothing factor at time i is determined based on the smoothing speed of the key points of the target hand at time i.

[0114]

[0115] T e =T i -T i-1

[0116]

[0117]

[0118] Where α represents the adaptive smoothing factor at time i, τ represents the time constant calculated using the adaptive cutoff frequency, and T i Let T represent the time at time i. i-1 T represents the time at time i-1. e f represents the time interval from time i-1 to time i, where f is the position change of key points on the target hand. c This indicates the adaptive cutoff frequency calculated based on the velocity. This represents the minimum adaptive cutoff frequency, and β represents the velocity coefficient. The modulus of the smoothed velocity of the key points of the target hand at time i is represented.

[0119] In this disclosure, the minimum cutoff frequencies of the 2D keypoints and 3D keypoints of the target hand are both The velocity coefficient of the 2D keypoints of the target hand is β = 0.007, and the velocity coefficient of the 3D keypoints of the target hand is β = 7.

[0120] The greater the smoothing speed of the key points of the target hand, the greater its corresponding adaptive smoothing factor; the smaller the smoothing speed of the key points of the target hand, the smaller its corresponding adaptive smoothing factor.

[0121] S35, based on the adaptive smoothing factor at each moment, performs adaptive filtering on the key point coordinates of the target hand at each moment to determine the target key point coordinates of the target hand at each moment.

[0122] In one possible embodiment, the key point coordinates of the target hand at time i are adaptively filtered using an adaptive smoothing factor at time i.

[0123]

[0124] in, X represents the coordinates of the target key points of the target hand at time i after smoothing with an adaptive smoothing factor, α represents the adaptive smoothing factor at time i, and X represents the target key points of the target hand at time i. i Represents the coordinates of key points on the unsmoothed target hand at time i. This represents the coordinates of the key points of the target hand after smoothing at time i-1.

[0125] When the smoothing speed of the key points of the target hand is large, it indicates that the target hand is in a state of high-speed motion. To prevent lag caused by strong filtering, the adaptive smoothing factor should be larger and the smoothing force should be smaller. When the smoothing speed of the key points of the target hand is small, it indicates that the target hand is in a state of low-speed motion. To ensure the stability of the position coordinates of the key points of the target hand, the smoothing force should be increased and the adaptive smoothing factor should be smaller.

[0126] The above technical solution determines the corresponding adaptive smoothing factor based on the smoothing speed of the key points of the target hand at each moment, thereby achieving adaptive filtering processing with different smoothing strengths. The subsequent processing improves the temporal stability from the output level, resulting in a smooth and stable output of the target key points of the target hand, thus improving the stability of the key points of the hand.

[0127] Figure 4 This is a comparison chart of the stability of data curves for key points of the hand, according to an exemplary embodiment.

[0128] like Figure 4 As shown, a stability test for keypoint detection was performed on a video of hand movements. Figure 4 The left image shows the keypoint data curves obtained by the original hand keypoint detection method, i.e., the data curves without stability enhancement. Figure 4 The right figure in the figure shows the key point data curve obtained by the stability enhancement method of the hand key points of this disclosure, that is, the data curve using stability enhancement.

[0129] Experiments show that after using the hand keypoint stability enhancement method proposed in this disclosure to improve stability, the stability of the keypoint data curve is significantly improved. Compared with directly using the hand keypoint detection algorithm, the hand keypoint stability enhancement method proposed in this disclosure can obtain a smoother and more stable keypoint data curve, effectively improving the stability of hand keypoint detection.

[0130] The solution disclosed herein can improve input and output through pre-processing and post-processing, while enhancing the stability of hand keypoint detection.

[0131] Figure 5 This is a block diagram illustrating a stability enhancement system for key hand points according to an exemplary embodiment.

[0132] Based on the same concept, this disclosure also provides a stability enhancement system for key hand points, referring to... Figure 5 The hand key point stability enhancement system 100 includes a target detection module 110, a key point detection module 120, and an adaptive filtering module 130.

[0133] The target detection module 110 is used to input a video stream containing a target hand into a preset hand target detection model to determine the image of the target hand. The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame image and the feature images of the current frame image of the video stream.

[0134] The key point detection module 120 is used to input the image of the target hand into the key point detection model to determine a preset number of 2D key point position coordinates and a preset number of 3D key point position coordinates of the target hand.

[0135] The adaptive filtering module 130 performs adaptive filtering on the preset number of 2D key point coordinates of the target hand at each time moment and the preset number of 3D key point coordinates of the target hand at each time moment in the time domain to determine the target key point coordinates of the target hand.

[0136] Optionally, the key point detection module 120 includes:

[0137] The feature extraction submodule is used to extract feature images of each frame of the video stream using a hand detection algorithm;

[0138] The fusion submodule is used to fuse the feature images of the previous frame and the current frame of the video stream using the temporal feature fusion module to determine the fused feature image.

[0139] The target detection submodule is used to input the fused feature image into the target detection head network to determine the bounding box of the hand region and the confidence level corresponding to the bounding box of the hand region;

[0140] The filtering submodule is used to determine the target hand region bounding box based on the confidence level corresponding to the hand region bounding box and a preset confidence level threshold.

[0141] The cropping submodule is used to crop the bounding box of the target hand region to determine the image of the target hand.

[0142] Optionally, the fusion submodule includes:

[0143]

[0144] in, w represents the fused feature image at time i. i- 1 →i F represents the weighted average of the feature images of the image corresponding to the (i-1)th time frame. i-1 w represents the feature image of the corresponding frame at time i-1. i→i F represents the weighted average of the feature images of the image corresponding to the i-th frame. i This represents the feature image of the frame corresponding to time i.

[0145] Optionally, the filtering submodule includes:

[0146] The first filtering submodule is used to determine the bounding boxes of the hand regions corresponding to those with a confidence level greater than a preset confidence level threshold as the target hand region bounding boxes.

[0147] The second filtering submodule is used to determine the bounding boxes of the hand regions corresponding to the confidence level not greater than the preset confidence level threshold as non-target hand region bounding boxes.

[0148] Optionally, the keypoint detection model includes a 2D keypoint detection model and a 3D keypoint detection model.

[0149] Optionally, the key point detection module 120 includes:

[0150] The first key point detection submodule is used to input the image of the target hand into the 2D key point detection model to determine the position coordinates of a preset number of 2D key points of the target hand.

[0151] The second key point detection submodule is used to input the image of the target hand into the 3D key point detection model to determine the position coordinates of a preset number of 3D key points of the target hand.

[0152] Optionally, the adaptive filtering module 130 includes:

[0153] The acquisition submodule is used to acquire the original coordinate sequence of key points of the target hand based on a preset number of 2D key point position coordinates and a preset number of 3D key point position coordinates of the target hand at each time moment. The original coordinate sequence includes the 2D key point position coordinates and the 3D key point position coordinates.

[0154] The rate of change determination submodule is used to determine the rate of change of the key points of the target hand at each moment based on the original coordinate sequence of the key points of the target hand and the position change time of the key points of the target hand;

[0155] The smoothing submodule is used to perform speed smoothing processing on the change rate of the key points of the target hand at each moment according to a preset constant cutoff frequency, and to determine the smoothing speed of the key points of the target hand at each moment.

[0156] The first determining submodule is used to determine the adaptive smoothing factor at each moment based on the smoothing speed of the key points of the target hand at each moment.

[0157] The second determining submodule is used to perform adaptive filtering on the key point coordinates of the target hand at each time step based on the adaptive smoothing factor at each time step, and to determine the target key point coordinates of the target hand at each time step.

[0158] Optionally, the second determining submodule includes:

[0159]

[0160] in, X represents the coordinates of the target key points of the target hand at time i after smoothing with an adaptive smoothing factor, α represents the adaptive smoothing factor at time i, and X represents the target key points of the target hand at time i. i The coordinates of key points of the target hand at the i-th time before smoothing are given. The coordinates of the key points of the target hand at time i-1 after smoothing are represented.

[0161] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0162] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0163] like Figure 6 As shown, in some possible embodiments, this disclosure may also provide an electronic device 600, which may include: a processor 601, a memory 602. The electronic device 600 may also include one or more of a multimedia component 603, an input / output interface 604, and a communication component 605.

[0164] The processor 601 controls the overall operation of the electronic device 600 to complete all or part of the steps in the hand key point stability enhancement method described in the first aspect. The memory 602 stores various types of data to support the operation of the electronic device 600. This data may include, for example, instructions for any application or method operating on the electronic device 600, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 603 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 602 or transmitted via communication component 605. The audio component also includes at least one speaker for outputting audio signals. Input / output interface 604 provides an interface between processor 601 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0165] In another exemplary embodiment, a non-transitory computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the hand keypoint stability enhancement method described in the first aspect. For example, the computer-readable storage medium may be the aforementioned memory including program instructions, which may be executed by a processor of an electronic device to perform the hand keypoint stability enhancement method.

[0166] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described method for enhancing the stability of hand key points when executed by the programmable device.

[0167] The specific embodiments of this disclosure have been described above. It should be understood that this disclosure is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this disclosure. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for enhancing the stability of key points in the hand, characterized in that, include: A video stream containing a target hand is input into a preset hand target detection model to determine the image of the target hand. The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame and the current frame of the video stream. The image of the target hand is input into the key point detection model to determine a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand. The coordinates of a preset number of 2D key points of the target hand at each moment and the coordinates of a preset number of 3D key points of the target hand at each moment are adaptively filtered in the time domain to determine the coordinates of the target key points of the target hand at each moment. The step of performing adaptive filtering on the coordinates of a preset number of 2D key points of the target hand at each moment and the coordinates of a preset number of 3D key points of the target hand at each moment in the time domain to determine the target key point coordinates of the target hand at each moment includes: Based on the preset number of 2D key point position coordinates and the preset number of 3D key point position coordinates of the target hand at each moment, the original coordinate sequence of the key points of the target hand is obtained, and the original coordinate sequence includes the 2D key point position coordinates and the 3D key point position coordinates. Based on the original coordinate sequence of the key points of the target hand and the position change time of the key points of the target hand, determine the change rate of the key points of the target hand at each moment; Based on a preset constant cutoff frequency, the rate of change of the key points of the target hand at each moment is smoothed to determine the smoothing rate of the key points of the target hand at each moment. The adaptive smoothing factor for each moment is determined based on the smoothing speed of the key points of the target hand at each moment. Based on the adaptive smoothing factor at each time step, the key point coordinates of the target hand at each time step are adaptively filtered to determine the target key point coordinates of the target hand at each time step.

2. The method according to claim 1, characterized in that, The step of inputting a video stream containing the target hand into a preset hand target detection model to determine the image of the target hand includes: A hand detection algorithm is used to extract feature images from each frame of the video stream; The temporal feature fusion module is used to fuse the feature images of the previous frame and the current frame of the video stream to determine the fused feature image. The fused feature image is input into the target detection head network to determine the bounding box of the hand region and the confidence level corresponding to the bounding box of the hand region; The target hand region bounding box is determined based on the confidence level corresponding to the bounding box of the hand region and the preset confidence level threshold. The bounding box of the target hand region is cropped to determine the image of the target hand.

3. The method according to claim 2, characterized in that, The step of fusing the feature images of the previous frame and the current frame of the video stream using the temporal feature fusion module to determine the fused feature image includes: in, w represents the fused feature image at time i. i-1→i F represents the weighted average of the feature images of the image corresponding to the (i-1)th time frame. i-1 w represents the feature image of the corresponding frame at time i-1. i→i F represents the weighted average of the feature images of the image corresponding to the i-th frame. i This represents the feature image of the frame corresponding to time i.

4. The method according to claim 2, characterized in that, The step of determining the target hand region bounding box based on the confidence level corresponding to the hand region bounding box and a preset confidence threshold includes: The bounding box of the hand region corresponding to the confidence level being greater than the preset confidence level threshold is determined as the bounding box of the target hand region. The bounding box of the hand region corresponding to the confidence level not exceeding the preset confidence level threshold is determined as the non-target hand region bounding box.

5. The method according to claim 1, characterized in that, The key point detection model includes a 2D key point detection model and a 3D key point detection model; The step of inputting the image of the target hand into the key point detection model to determine the position coordinates of a preset number of 2D key points and a preset number of 3D key points of the target hand includes: The image of the target hand is input into the 2D key point detection model to determine the position coordinates of a preset number of 2D key points of the target hand. The image of the target hand is input into the 3D key point detection model to determine the position coordinates of a preset number of 3D key points of the target hand.

6. The method according to claim 1, characterized in that, The step of adaptively filtering the key point coordinates of the target hand at each time step based on the adaptive smoothing factor at each time step to determine the target key point coordinates at each time step includes: in, X represents the coordinates of the target key points of the target hand at time i after smoothing with an adaptive smoothing factor, α represents the adaptive smoothing factor at time i, and X represents the target key points of the target hand at time i. i The coordinates of key points of the target hand at the i-th time before smoothing are given. The coordinates of the key points of the target hand at time i-1 after smoothing are represented.

7. A stability enhancement system for key hand points, characterized in that, include: The target detection module is used to input a video stream containing a target hand into a preset hand target detection model to determine the image of the target hand. The preset hand target detection model includes a temporal feature fusion module, which is used to fuse the feature images of the previous frame image and the feature images of the current frame image of the video stream. The key point detection module is used to input the image of the target hand into the key point detection model to determine a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand. An adaptive filtering module is used to perform adaptive filtering on a preset number of 2D key point coordinates and a preset number of 3D key point coordinates of the target hand at each time moment in the time domain to determine the target key point coordinates of the target hand. The adaptive filtering module is used for: Based on the preset number of 2D key point position coordinates and the preset number of 3D key point position coordinates of the target hand at each moment, the original coordinate sequence of the key points of the target hand is obtained, and the original coordinate sequence includes the 2D key point position coordinates and the 3D key point position coordinates. Based on the original coordinate sequence of the key points of the target hand and the position change time of the key points of the target hand, determine the change rate of the key points of the target hand at each moment; Based on a preset constant cutoff frequency, the rate of change of the key points of the target hand at each moment is smoothed to determine the smoothing rate of the key points of the target hand at each moment. The adaptive smoothing factor for each moment is determined based on the smoothing speed of the key points of the target hand at each moment. Based on the adaptive smoothing factor at each time step, the key point coordinates of the target hand at each time step are adaptively filtered to determine the target key point coordinates of the target hand at each time step.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-6.

Images