Gesture tracking method, apparatus, terminal device, and computer-readable storage medium

Abstract

The application is suitable for the technical field of image processing, and provides a gesture tracking method and device, terminal equipment and computer readable storage medium, the method comprises the following steps: inputting a target image into a gesture tracking network model for processing, and outputting a gesture tracking result; if the gesture tracking result is that the target image contains a gesture, determining a first gesture box corresponding to the target image, and the inside of the first gesture box contains target gesture content. The above method can improve the accuracy of the terminal equipment in tracking the hand. And by performing frame skipping processing on the video image, the processing amount of the terminal equipment in processing the video image can be reduced, and the processing efficiency of the terminal equipment can be improved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a gesture tracking method, apparatus, terminal device and computer-readable storage medium. Background Technology

[0002] Currently, gesture recognition is applied in various human-computer interaction fields. For example, the smart screen recognition function on mobile phones captures images of the user's hands and extracts the user's gestures. Then, it responds accordingly to the gestures, such as swiping or locking the screen.

[0003] However, when capturing images of a user's hand, hand movements are generally not static, thus involving gesture tracking issues. For example, when a user moves their hand quickly, hand images captured at different times will leave afterimages in the image, causing distortion of the hand. This distortion will cause drastic changes in hand tracking coordinates, thereby reducing the accuracy of the phone's hand tracking. Summary of the Invention

[0004] This application provides a gesture tracking method, apparatus, terminal device, and computer-readable storage medium, which can solve the problem of low hand tracking accuracy when performing hand tracking on acquired hand images in the prior art.

[0005] In a first aspect, embodiments of this application provide a gesture tracking method, including:

[0006] The target image is input into the gesture tracking network model for processing, and the gesture tracking result is output.

[0007] If the gesture tracking result indicates that the target image contains a gesture, then the first gesture box corresponding to the target image is determined, and the interior of the first gesture box contains the target gesture content.

[0008] Secondly, embodiments of this application provide a gesture tracking device, including:

[0009] The first processing module is used to input the target image into the gesture tracking network model for processing and output the gesture tracking result.

[0010] The first determining module is used to determine the first gesture box corresponding to the target image if the gesture tracking result is that the target image contains a gesture, and the interior of the first gesture box contains the target gesture content.

[0011] Thirdly, embodiments of this application provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0013] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in the first aspect.

[0014] The beneficial effects of this application's embodiments compared to existing technologies are as follows: The target image is input into a gesture tracking network model for processing, and the gesture tracking result is output. If the gesture tracking result indicates that the target image contains a gesture, a first gesture bounding box corresponding to the target image is determined. The first gesture bounding box contains the gesture, thus achieving a more precise determination of the first gesture bounding box from the target image, improving the accuracy of gesture tracking in the current frame. Furthermore, the terminal device can reduce the amount of video image processing required by performing frame skipping on the video image, thereby improving the processing efficiency of the terminal device. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the implementation of a gesture tracking method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart illustrating the implementation of another gesture tracking method provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram illustrating an implementation of S101B of a gesture tracking method provided in an embodiment of this application;

[0019] Figure 4 This is a schematic diagram illustrating an implementation of S102 of a gesture tracking method provided in an embodiment of this application;

[0020] Figure 5 This is a flowchart illustrating the implementation of another gesture tracking method provided in one embodiment of this application;

[0021] Figure 6 This is a structural block diagram of a gesture tracking device provided in one embodiment of this application;

[0022] Figure 7This is a structural block diagram of a terminal device provided in one embodiment of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] The gesture tracking method provided in this application can be applied to terminal devices such as mobile phones, tablets, laptops, and netbooks. This application does not impose any restrictions on the specific type of terminal device.

[0028] Figure 1 A flowchart illustrating the implementation of the gesture tracking method provided in an embodiment of the present invention is shown below, and is described in detail below:

[0029] S101. The terminal device inputs the target image into the gesture tracking network model for processing and outputs the gesture tracking result.

[0030] In application, the target image mentioned above can be an image acquired by the terminal device at the current moment. This target image can be a video image pre-stored within the terminal device. That is, the terminal device pre-stores video, and this video contains multiple frames of video images. The video image acquired by the terminal device from the video at this moment is the target image. Alternatively, the target image can be considered as an image that may contain gestures, output by the terminal device after processing the current video image. It is understood that a single current video image may contain multiple target images.

[0031] In application, the aforementioned gesture tracking network model is also a pre-set model within the terminal device, which can be used to accurately output gesture tracking results after feature processing of the target image. The gesture tracking results include both cases where the target image contains a gesture and cases where the target image does not contain a gesture. Furthermore, when the gesture tracking result indicates that the target image contains a gesture, the gesture tracking network model returns a more precise first gesture bounding box determined from the target image after processing.

[0032] S102. If the gesture tracking result is that the target image contains a gesture, the terminal device determines the first gesture box corresponding to the target image, and the inside of the first gesture box contains the target gesture content.

[0033] In the application, the first gesture box mentioned above has already been described in S101, and will not be explained again.

[0034] It is understandable that if the target image is an image obtained after the terminal device processes the current video image, then the current video image may contain multiple target images containing gestures. It should be noted that the aforementioned target gesture content can be content specifically containing gesture information.

[0035] Specifically, a target image may contain multiple bounding boxes, meaning there are multiple bounding boxes containing gestures. The gesture tracking network model can output the probability that each target image contains a gesture. Then, the terminal device determines a more specific first bounding box from target images with probabilities greater than a preset probability value. This improves the accuracy of the terminal device in determining the first bounding box from multiple target images.

[0036] Please refer to Figure 2 In one specific embodiment, before inputting the target image into the gesture tracking network model for processing in S101 and outputting the gesture tracking result, the following steps S101A-S101C are further included, as detailed below:

[0037] S101A. If the current frame image meets the gesture detection conditions, the terminal device inputs the current frame image into the gesture detection network for processing and outputs the first predicted gesture box.

[0038] In application, the aforementioned current frame image can be considered as the video image currently acquired by the terminal device from the video. The aforementioned gesture detection conditions can be detection conditions pre-set by the user within the terminal device. For example, for multiple consecutive video frames contained in a video, the aforementioned gesture detection conditions can be that the terminal device determines the current video image (current frame image) meets the gesture detection conditions every preset frame number. The preset frame number can be set by the user according to actual conditions. For example, the preset frame number can be set to 3. Alternatively, the terminal device can detect whether the current frame image carries a specific identifier; if the current frame image carries a specific identifier, it determines that the target image meets the gesture detection conditions, without limitation. The specific identifier can also be pre-added by the user to the selected video image.

[0039] Understandably, for a video, the terminal device can determine whether to perform the aforementioned steps S101A to S101C on each frame of the video by detecting whether the current frame meets the gesture detection conditions. This reduces the amount of video image processing required by the terminal device and improves its processing efficiency. Furthermore, since the gesture detection conditions are only performed every preset number of frames, the detected gestures can be considered continuous. That is, while improving the processing efficiency of the terminal device in processing video images, it can also accurately obtain the first gesture frame containing continuous gestures in the video.

[0040] In applications, the aforementioned gesture detection network can be a pre-trained network model for the user. The gesture detection network can use either MobileNet V1 or MobileNet-v2 as the feature extraction network, without limitation. In this embodiment, MobileNet-v2 is used as the feature extraction network. Compared to using MobileNet-v1, MobileNet-v2 requires fewer network parameters to process the current frame image, resulting in faster feature extraction. Furthermore, the user can pre-set the image size for each current frame image and change the number of channels in each convolutional layer of the feature extraction network. This allows the terminal device to achieve a reasonable balance between speed and accuracy in obtaining the first predicted gesture box from the current frame image.

[0041] S101B, The terminal device determines a second predicted gesture frame based on the first predicted gesture frame, and the second predicted gesture frame surrounds the first predicted gesture frame.

[0042] In applications, the first predicted gesture bounding box may not contain all the information of the gesture. If the terminal device subsequently performs S101-S102 processing based on the current first predicted gesture bounding box and determines that it includes the first gesture bounding box, the accuracy of the obtained first gesture bounding box may be inaccurate, preventing the terminal device from accurately tracking the gesture. Therefore, the terminal device determines a second predicted gesture bounding box based on the first predicted gesture bounding box by expanding the first predicted gesture bounding box to obtain the expanded second predicted gesture bounding box. Then, by inputting the target image determined from the second predicted gesture bounding box into the gesture tracking network model, the first gesture bounding box containing the gesture can be further accurately obtained from the second predicted gesture bounding box. That is, compared to the coverage area of ​​the expanded first predicted gesture bounding box, the coverage area of ​​the first gesture bounding box is relatively smaller, and the accuracy of containing the gesture is relatively higher.

[0043] It should be added that the gesture detection network can output a first predicted gesture bounding box that may contain a gesture in the current frame image. At this time, it can also output the coordinate information of the first predicted gesture bounding box in the current frame image. Based on this, the specific process of expanding the bounding box of the first predicted gesture bounding box can be described as follows:

[0044] Specifically, any given frame image consists of multiple pixels. Based on this, the terminal device can use any point in the current frame image as the origin and establish X-axis and Y-axis parallel to both sides of the current frame image to generate an image coordinate system. Then, the terminal device can generate the coordinates of each pixel based on the image coordinate system. The origin can be the center point of the input image or the pixel at the top left corner of the input image; there are no restrictions on this. It is understood that the first predicted gesture bounding box can be a portion of the current frame image, and the shape of the first predicted gesture bounding box can be a square or a rectangle.

[0045] For example, the terminal device can use the coordinates of the pixel at the top left corner of the first predicted gesture box as the coordinates of the first predicted gesture box in the current frame image. Then, the terminal device can determine the length and width of the first predicted gesture box based on the area it covers in the current frame image. Thus, the terminal device can use the coordinates, length, and width as the coordinate information of the first predicted gesture box in the current frame image. For example, the coordinate information can be (x, y, w, l), where x and y represent the horizontal and vertical coordinates of the pixel at the top left corner of the first predicted gesture box, respectively; w represents the width of the first predicted gesture box (extending w distance to the right from the top left corner pixel); and l represents the length of the first predicted gesture box (extending l distance downwards from the top left corner pixel).

[0046] Then, the terminal device can process each value (horizontal coordinate, vertical coordinate, width, and length) in the coordinate information according to the preset coefficients. Specifically, if the preset coefficient is 'a' and the coordinate information is (x, y, w, l), the preset coefficient 'a' can be multiplied by the width and length respectively to obtain the values ​​'aw' and 'al'. Then, the terminal device can use the difference between the horizontal coordinate 'x' and 'aw' in the coordinate information as the horizontal coordinate of the expanded second predicted gesture box; the difference between the vertical coordinate 'y' and 'al' in the coordinate information as the vertical coordinate of the expanded second predicted gesture box; the sum of the width in the coordinate information and twice the value of 'aw' as the width of the expanded second predicted gesture box; and the sum of the length in the coordinate information and twice the value of 'al' as the length of the expanded second predicted gesture box. For example, the coordinate information of the expanded second predicted gesture box is (xa*w, ya*h, (1+2*a)*w, (1+2*a)*l). It is understandable that, compared to the initial coordinate information (x, y, w, l) of the first predicted gesture box, the expanded second predicted gesture box in the current frame image has already surrounded the original first predicted gesture box.

[0047] S101C, the terminal device determines the target image based on the second predicted gesture frame, and the size of the target image is the preset image size.

[0048] In application, the second predicted gesture box is obtained after expansion processing. Specifically, the length and width of the expanded second predicted gesture box are increased by 2*a*l or 2*a*w compared to the length and width of the first predicted gesture box, and the coordinates of the top-left pixel are shifted upwards by a*h and to the left by a*w compared to the top-left pixel of the first predicted gesture box. Therefore, the coordinates of the top-left pixel of the expanded second predicted gesture box may be outside the current frame image, and / or the coverage area of ​​the expanded second predicted gesture box may exceed the current frame image. Specifically, the terminal device can make a judgment based on the coordinate information of the expanded first predicted gesture box and the coordinate information in the current frame image. After determining that the second predicted gesture box exceeds the current frame image, the terminal device can truncate the second predicted gesture box to obtain the target image. The size of the target image is then the image size of the current frame image. Therefore, the terminal device can consider the preset image size as the image size of the current frame image.

[0049] In application, when it is determined that the expanded second predicted gesture box exceeds the current frame image, the terminal device can only obtain the intersection box between the expanded second predicted gesture box and the current frame image and perform truncation processing. That is, the part of the expanded second predicted gesture box that exceeds the current frame image is considered to have no gesture information, and the terminal device only takes the part of the second predicted gesture box that overlaps with the current frame image as the final second predicted gesture box.

[0050] It should be added that if the expanded second predicted gesture box does not exceed the current frame image, it means that the expanded second gesture box is completely within the current frame image. In this case, the terminal device can directly process the expanded second predicted gesture box as the target image. Therefore, the terminal device can consider the image size of the target image to be the image size of the second predicted gesture box, that is, the preset image size is the image size of the second predicted gesture box.

[0051] In this embodiment, a gesture detection network is used to initially determine a first predicted gesture bounding box containing the gesture from the current frame image. Then, the first predicted gesture bounding box is expanded so that the resulting second predicted gesture bounding box more completely contains all the information of the gesture. Next, the overlapping portion of the current frame image and the second predicted gesture bounding box is used as the target image and input into the gesture tracking network model for processing. This allows for the determination of a more accurate first gesture bounding box from the current frame image, improving the accuracy of gesture tracking in the current frame image.

[0052] Please refer to Figure 3 In one specific embodiment, S101B determines the second predicted gesture box based on the first predicted gesture box, and further includes the following sub-steps S101B1-S101B3, which are detailed below:

[0053] S101B1 If the first predicted gesture frame is square, the terminal device obtains the coordinates of the calibration point, length and width of the first predicted gesture frame.

[0054] In application, S101B above already explains that the terminal device can obtain the coordinate information (including the coordinates of the top-left corner, length, and width) of the first predicted gesture frame based on the gesture detection network, and the top-left corner coordinates, length, and width of the first predicted gesture frame are explained in the description, so they will not be explained again. It is understood that the aforementioned square includes, but is not limited to, shapes such as squares and rectangles. In this case, the calibration point coordinates are the aforementioned top-left corner coordinates. However, in practical situations, when the first predicted gesture frame is square, the calibration point coordinates include, but are not limited to, the coordinates of each corner of the square and the coordinates of the center point of the square.

[0055] It should be added that the first predicted gesture frame can also be a circle, a rhombus, or other shapes, and there is no limitation on this. In this case, when the shape of the first predicted gesture frame is as described above, the center point of the first predicted gesture frame can be used as the calibration point.

[0056] S101B2. The terminal device determines the coordinates, length, and width of the expanded frame based on the coordinates, length, width, and expansion coefficient of the first predicted gesture frame.

[0057] S101B3. The terminal device determines the second predicted gesture frame based on the coordinates, length, and width of the expanded calibration point.

[0058] In application, the determination of the coordinates, length, and width of the calibration point after the frame expansion is specifically referred to in the example above S101B, which calculates the coordinates (calibration point coordinates), length, and width of the upper left corner based on the preset coefficient a (frame expansion coefficient) to obtain the coordinates (calibration point coordinates), length, and width of the upper left corner of the second predicted gesture frame after the frame expansion. This will not be explained further.

[0059] Please refer to Figure 4 In one specific embodiment, S102, which determines the first gesture box corresponding to the target image, further includes the following sub-steps S1021-S1024, detailed below:

[0060] S1021. If the current frame image is not the first frame image, the terminal device obtains the gesture box in the target image.

[0061] In applications, videos typically contain multiple frames. If the current frame is not the first frame, after processing the current frame through steps S101 (including S101A-S101B), the terminal device can initially obtain at least one first predicted gesture bounding box from the gesture detection network outputting the current frame, thus obtaining a target image. Subsequently, when the gesture tracking network model determines that the gesture tracking result of the target image contains a gesture, the terminal device can obtain the gesture bounding box in the target image for further processing.

[0062] It is understandable that when the current frame image is the first frame image, it can directly perform the processing of the current frame image in S101 (including S101A-S101C)-S102.

[0063] S1022, The terminal device obtains the second gesture box corresponding to the previous frame image of the current frame image.

[0064] In application, when the current frame image is not the first frame image, it can be assumed that the terminal device has already processed the previous frame image preceding the current frame image and determined the second gesture box containing the gesture from the previous frame image. It is understood that the previous frame image and the current frame image are relative concepts. That is, when the current frame image changes, the previous frame image also changes relatively. It is understood that the second gesture box corresponding to the previous frame image is equivalent to the first gesture box determined by the current frame image. It should be added that the previous frame image of the current frame image is preferably the previous frame image adjacent to the current frame image.

[0065] In this application, the previous frame image is an image that has already been processed by the terminal device, which can store the processed result (the second gesture box of the previous frame image). Therefore, the terminal device can directly obtain the second gesture box of each frame image preceding the current frame image, as well as the coordinate information of the second gesture box of each frame image in the corresponding image. At this time, the terminal device only needs to determine the second gesture box corresponding to the previous frame image adjacent to the current frame image from the second gesture box in each frame image.

[0066] S1023. The terminal device calculates the intersection area and the union area of ​​the gesture frame and the second gesture frame in the target image, respectively.

[0067] In the application, as described in S101B2 above, the coordinate information of the second gesture box in the corresponding previous frame image (including the coordinates of the upper left corner, length, and width of the second gesture box) can be obtained simultaneously. Based on this, the terminal device can calculate the area of ​​the second gesture box. Meanwhile, the target image in S101C is the image obtained after truncating the second predicted gesture box; in this case, the target image can be considered the gesture box. Based on this, the terminal device can calculate the area of ​​the target image as the area of ​​the gesture box based on the coordinate information of the target image (including the coordinates of the upper left corner, length, and width of the target image).

[0068] In application, after determining the coordinate information of the two frames, the area of ​​the intersection region of the two frames can also be calculated simultaneously. At this time, the intersection area is the area of ​​the intersection region between the two frames, and the union area is the sum of the area of ​​the second gesture frame and the area of ​​the gesture frame of the target image, minus the intersection area.

[0069] It is understandable that the target image is obtained by processing the current frame image based on the gesture detection network. Therefore, there may be cases where the gesture detection network identifies multiple target images from the current frame image, and the previous frame image may also have multiple second gesture bounding boxes. Based on this, it is necessary to calculate the intersection area and union area between each target image and each second gesture bounding box separately.

[0070] S1024. If the ratio of the intersection area to the union area is less than the first preset ratio, the terminal device determines the gesture box in the target image as the first gesture box corresponding to the target image.

[0071] In application, the ratio of the intersection area to the union area can be calculated using the following formula: IOU = (A∩B) / (A∪B), where A is the area of ​​the second gesture box, B is the area of ​​the gesture box of the target image, A∩B is the intersection area of ​​the second gesture box and the gesture box of the target image, and A∪B is the union area of ​​the second gesture box and the gesture box of the target image.

[0072] In application, the aforementioned first preset ratio can be set by the user according to the actual situation, and there are no limitations on it. It is understood that when the calculated ratio is greater than the first preset ratio, it indicates that there is overlap between the second gesture frame and the gesture frame of the target image, and the degree of overlap is high. In this case, the terminal device can directly delete the gesture frame of the target image. If the ratio is less than the first preset ratio, it indicates that the overlap between the second gesture frame and the gesture frame of the target image is low. In this case, the terminal device can directly determine the first gesture frame from the target image.

[0073] It should be noted that the terminal device can use the gesture frame of the target image at this time as the first gesture frame, or it can further determine a more precise first gesture frame from the target image.

[0074] It should be added that when the ratio of the intersection area to the union area is greater than or equal to a first preset ratio, the terminal device may abandon the determination of the first gesture box from the target image. Since the ratio of the intersection area to the union area is greater than or equal to the first preset ratio, it indicates that the gesture box in the target image highly overlaps with or is similar to the second gesture box in the previous frame image. In this case, the terminal device has already determined the second gesture box when processing the previous frame image. Based on this, the terminal device can abandon the operation of determining the first gesture box from the target image, reducing the storage requirements of the terminal device.

[0075] Please refer to Figure 5 In one specific embodiment, before inputting the target image into the gesture tracking network model for processing in S101 and outputting the gesture tracking result, the following steps S101D-S101F are further included, as detailed below:

[0076] S101D: If the current frame image does not meet the gesture detection conditions, the terminal device obtains the third gesture box corresponding to the previous frame image of the current frame image.

[0077] In application, the gesture detection conditions have already been explained in S101A above, and will not be described again. The third gesture box corresponding to the previous frame image adjacent to the current frame image is the same as the second gesture box corresponding to the previous frame image adjacent to the current frame image in S1022 above; the difference lies only in the name used after determining whether the current frame image meets the gesture detection conditions.

[0078] S101E and terminal devices expand the third gesture box to obtain the third predicted gesture box.

[0079] In application, the specific process of expanding the third gesture box can be found in the description in S101B above, and will not be explained further.

[0080] S101F: The terminal device performs cropping processing on the current frame image based on the third predicted gesture box to obtain the target image.

[0081] In the application, after determining the third predicted gesture bounding box, its coordinate information (the coordinates of the top-left corner, length, and width of the third gesture bounding box) can be determined simultaneously. Then, the terminal device can establish image coordinates using the top-left corner of the current frame image as the origin. Based on the top-left corner coordinates, length, and width of the third gesture bounding box, it can determine the coverage area of ​​the third predicted gesture bounding box in the current frame image and crop this coverage area to obtain the target image.

[0082] It should be noted that the terminal device crops the current frame image based on the coordinate information of the third predicted gesture bounding box. Therefore, in other examples, after determining the third gesture bounding box corresponding to the previous frame image adjacent to the current frame image, the terminal device can only obtain the coordinate information of the third gesture bounding box in the adjacent previous frame image (only the coordinate information is obtained, not the actual third gesture bounding box). Then, based on the aforementioned coordinate information, a gesture bounding box matching the coordinate information is determined from the current frame image, and this gesture bounding box is expanded to obtain the third predicted gesture bounding box. Finally, the third predicted gesture bounding box is truncated to obtain the target image.

[0083] Because some video frames contain rapidly moving gestures, the gesture bounding boxes determined by the terminal device without using a gesture detection network model may be inaccurate (not fully capturing all gesture information). Therefore, the terminal device expands the gesture bounding box to ensure that the expanded third predicted gesture bounding box contains all the gesture information from the current frame. Thus, even for frames with rapidly moving gestures, the terminal device can correct the errors using the above steps to obtain an accurate first gesture bounding box, ensuring the accuracy of gesture tracking in the video.

[0084] It is understandable that for a video containing multiple consecutive frames, the positional changes of the gesture bounding boxes in adjacent frames may be relatively small. Therefore, the terminal device does not need to determine the first predicted gesture bounding box for each frame using a gesture detection model. Furthermore, the processing time for the terminal device to determine the third predicted gesture bounding box in the current frame using the coordinate information of the third gesture bounding box in the previous frame is shorter than the processing time required to input the current frame image into the gesture detection network for feature processing to obtain the first predicted gesture bounding box. Thus, by using the coordinate information of the third gesture bounding box in the previous frame to define the third predicted gesture bounding box in the current frame, the terminal device can improve the processing efficiency of video images while also meeting the requirement of continuous and accurate tracking of gestures in the video.

[0085] It should be noted that for the two methods of acquiring target images S101A-S101C and S101D-S101F mentioned above, the terminal device can use either one of the above methods or both of the above methods to acquire the image, and there is no limitation on this.

[0086] In one specific embodiment, there are multiple target images, and the gesture tracking result also includes the confidence score of each target image. Determining the first gesture box corresponding to the target image includes the following steps, detailed below:

[0087] The terminal device uses a non-maximum suppression algorithm to process the confidence level in order to determine the corresponding first gesture box from each target image.

[0088] In application, as explained in S101B4 above, there may be multiple target images. Similarly, it can be understood that if multiple third gesture boxes are obtained after processing the previous frame image, then after processing each third gesture box through S101D-S101F, the number of target images and the number of third gesture boxes should be the same. Based on this, when there are multiple target images, after processing each target image through the S101 gesture tracking network model, the terminal device can obtain the gesture tracking result for each target image.

[0089] In application, the aforementioned non-maximum suppression algorithm can be considered as suppressing all values ​​except for local maxima (the target image with the highest confidence). That is, it suppresses target images with lower confidence and amplifies target images with higher confidence. The preset values ​​mentioned above can be set by the user according to the actual situation. Here, confidence refers to the degree to which the gesture tracking network determines that the target image contains a gesture. In other words, the terminal device can more easily determine the first gesture bounding box containing the gesture from the target image with the highest confidence.

[0090] In one specific embodiment, a non-maximum suppression algorithm is used to process the confidence level in order to determine the corresponding first gesture box from each target image. The specific steps include the following detailed description:

[0091] The terminal device sorts each target image according to its confidence level to obtain a candidate image list.

[0092] In application, the above sorting process can specifically involve sorting the target images in descending order of their confidence scores. The candidate image list contains the target images to be processed. It should be noted that the descending order of confidence score represents the sequence in which the terminal device performs the following processing on the multiple target images.

[0093] A1. The terminal device removes the target image with the highest confidence from the candidate image list and determines the corresponding first gesture box from the target image with the highest confidence.

[0094] In the application, removing the target image with the highest confidence level from the candidate image list can be considered as distinguishing the target image with the highest confidence level from the other target images and processing them separately. Specifically, after removing the target image with the highest confidence level from the candidate image list, the terminal device can determine the corresponding first gesture box from the target image with the highest confidence level and directly store the first gesture box. Then, the remaining target images are processed using steps A2-A4.

[0095] A2. The terminal device calculates the target image with the highest confidence level and the target ratio of each target image in the candidate image list.

[0096] In application, the aforementioned target ratio includes the ratio of the intersection area to the union area between the target image and each other target image. That is, the intersection area and the union area between the target image and any other image are calculated, and the ratio of the intersection area to the union area is used as one of the target ratios. For details on calculating the ratio between the target image and any other image, please refer to the explanations in S101B1-S101B4 above.

[0097] In application, the above target ratio also includes: for any other target image, calculating the intersection area between the target image with the highest confidence and any other target image, and then calculating the ratio between the intersection area and the target image with the highest confidence; and calculating the ratio between the intersection area and the corresponding other target image.

[0098] Understandably, since the target image is obtained by expanding the bounding box of the current frame image, and each target image has different top-left corner coordinates, length, and width, the expanded target images in the current frame may overlap with each other, or there may be cases where one target image completely contains another target image (a box within a box). Therefore, for these various situations, the terminal device needs to calculate the target ratio between the target box with the highest confidence and the remaining target boxes.

[0099] For example, given the target image C with the highest confidence level and any other target image D, if target image C contains target image D, then the area of ​​the intersection between target image C and target image D is the area of ​​target image B. In this case, the ratio of the intersection area to the area of ​​target image D calculated by the terminal device should be 1.

[0100] It should be added that the example in A2 above assumes that the target image C, with the highest confidence level, completely contains the target image D. However, in reality, there are also cases where the target image C with the highest confidence level highly overlaps with the target image D, which will not be illustrated with examples in these cases.

[0101] Understandably, when target image C completely contains target image D, the confidence level of the gesture tracking network model outputting target image D is usually lower than the confidence level in target image C. Therefore, in the candidate image list, target images with high confidence typically completely contain target images with low confidence.

[0102] A3. Target images with a target ratio greater than the second preset ratio are deleted from the candidate image list to obtain a candidate image list that includes the target images that were not deleted.

[0103] In application, the aforementioned second preset ratio can be a value set by the user according to the actual situation, and there is no limitation thereto. It is understood that if any target ratio calculated in step A2 is greater than the second preset value, the terminal device can consider that the gesture information contained in the target image with the highest confidence level already includes the gesture information of the corresponding target image in the remaining candidate image list. See the example in A2 for details. Therefore, the terminal device can delete the target image from the candidate image list and then not perform the cyclical processing on that target image. In this way, the terminal device can avoid capturing multiple first gesture frames containing the same gesture information from the video image.

[0104] The terminal device executes steps A1-A3 repeatedly until the last target image remains in the candidate image list, and then determines the corresponding first gesture box from the last target image.

[0105] In applications, when only one target image remains in the candidate image list, it can be assumed that, after calculating the ratio between this target image and the previous target images, the gesture information contained in this target image is inconsistent with the gesture information contained in the previous target images. Therefore, the terminal device can directly determine the first gesture box from the last target image. Based on this, after determining multiple target images containing gestures from the current frame image, the terminal does not need to store the first gesture box contained in each target image, thus avoiding the storage of a large amount of duplicate gesture information in the stored first gesture boxes.

[0106] Please see Figure 6 , Figure 6 This is a structural block diagram of a gesture tracking device provided in an embodiment of this application. The gesture tracking device in this embodiment includes modules used for performing... Figures 1 to 5 The steps in the corresponding embodiments. Please refer to the details. Figures 1 to 5 as well as Figures 1 to 5 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 6The gesture tracking device 600 includes: a first processing module 610 and a first determining module 620, wherein:

[0107] The first processing module 610 is used to input the target image into the gesture tracking network model for processing and output the gesture tracking result.

[0108] The first determining module 620 is used to determine the first gesture box corresponding to the target image if the gesture tracking result is that the target image contains a gesture, and the interior of the first gesture box contains the target gesture content.

[0109] In one embodiment, the gesture tracking device 600 further includes:

[0110] The second processing module is used to input the current frame image into the gesture detection network for processing and output the first predicted gesture box if the current frame image meets the gesture detection conditions.

[0111] The second determining module is used to determine a second predicted gesture box based on the first predicted gesture box, wherein the second predicted gesture box surrounds the first predicted gesture box.

[0112] The third determining module is used to determine the target image based on the second predicted gesture box, and the size of the target image is a preset image size.

[0113] In one embodiment, in determining the second predicted gesture box based on the first predicted gesture box, the second determining module is specifically used for:

[0114] If the first predicted gesture box is square, then obtain the coordinates of the calibration point, length, and width of the first predicted gesture box;

[0115] Based on the coordinates, length, width, and expansion coefficient of the first predicted gesture frame, determine the coordinates, length, and width of the expanded frame.

[0116] The second predicted gesture frame is determined based on the coordinates, length, and width of the expanded calibration point.

[0117] In one embodiment, in determining the first gesture box corresponding to the target image, the first determining module 620 is specifically used for:

[0118] If the current frame image is not the first frame image, then obtain the gesture box in the target image;

[0119] Obtain the second gesture box corresponding to the previous frame image of the current frame image;

[0120] Calculate the intersection area and the union area of ​​the gesture bounding box and the second gesture bounding box in the target image, respectively;

[0121] If the ratio of the intersection area to the union area is less than a first preset ratio, then the gesture box in the target image is determined as the first gesture box corresponding to the target image.

[0122] In one embodiment, the gesture tracking device 600 further includes:

[0123] The acquisition module is used to obtain the third gesture box corresponding to the previous frame image if the current frame image does not meet the gesture detection conditions.

[0124] The bounding box expansion module is used to expand the bounding box of the third gesture to obtain the third predicted gesture box.

[0125] The cropping module is used to crop the current frame image based on the third predicted gesture box to obtain the target image.

[0126] In one embodiment, the number of target images is multiple, and the gesture tracking result also includes the confidence level of each target image. In determining the first gesture box corresponding to the target image, the first determining module 620 is specifically used for:

[0127] The confidence level is processed using a non-maximum suppression algorithm to determine the corresponding first gesture box from each target image.

[0128] In one embodiment, the first determining module 620 is specifically used for:

[0129] Each target image is sorted according to its confidence level to obtain a candidate image list;

[0130] A1. Remove the target image with the highest confidence from the candidate image list, and determine the corresponding first gesture box from the target image with the highest confidence;

[0131] A2. Calculate the target ratio between the target image with the highest confidence and each target image in the candidate image list;

[0132] A3. Remove target images with a target ratio greater than the second preset ratio from the candidate image list to obtain a candidate image list that includes the target images that were not deleted.

[0133] Repeat steps A1-A3 until the last target image remains in the candidate image list, and determine the corresponding first gesture box from the last target image.

[0134] When it is understood that, Figure 6 In the block diagram of the gesture tracking device shown, each unit / module is used to perform... Figures 1 to 5 The steps in the corresponding embodiments, and for Figures 1 to 5 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figures 1 to 5 as well as Figures 1 to 5 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0135] Figure 7 This is a structural block diagram of a terminal device provided in another embodiment of this application. For example... Figure 7 As shown, the terminal device 700 of this embodiment includes: a processor 710, a memory 720, and a computer program 730 stored in the memory 720 and executable by the processor 710, such as a gesture tracking method program. When the processor 710 executes the computer program 730, it implements the steps of each of the above-described gesture tracking method embodiments, for example... Figure 1 S101 to S102 are shown. Alternatively, the processor 710 implements the above when executing the computer program 730. Figure 6 The functions of each module in the corresponding embodiments, for example, Figure 6 For details on the functions of modules 610 to 620 shown, please refer to [link / reference needed]. Figure 6 The relevant descriptions in the corresponding embodiments.

[0136] For example, the computer program 730 may be divided into one or more units, one or more of which are stored in the memory 720 and executed by the processor 710 to complete the present application. The one or more units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 730 in the terminal device 700.

[0137] The terminal device may include, but is not limited to, processor 710 and memory 720. Those skilled in the art will understand that... Figure 7 This is merely an example of terminal device 700 and does not constitute a limitation on terminal device 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device may also include input / output devices, network access devices, buses, etc.

[0138] The processor 710 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0139] The memory 720 can be an internal storage unit of the terminal device 700, such as a hard disk or memory of the terminal device 700. The memory 720 can also be an external storage device of the terminal device 700, such as a plug-in hard disk, smart memory card, flash memory card, etc., equipped on the terminal device 700. Furthermore, the memory 720 can include both internal storage units and external storage devices of the terminal device 700.

[0140] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A gesture tracking method, characterized in that, include: If the current frame image meets the gesture detection conditions, the current frame image is input into the gesture detection network for processing, and the first predicted gesture box is output. A second predicted gesture box is determined based on the first predicted gesture box, and the second predicted gesture box surrounds the first predicted gesture box; The target image is determined based on the second predicted gesture box, and the size of the target image is a preset image size; If the current frame image does not meet the gesture detection conditions, then the third gesture box corresponding to the previous frame image of the current frame image is obtained; The third gesture box is expanded to obtain the third predicted gesture box; The target image is obtained by cropping the current frame image based on the third predicted gesture box; The target image is input into the gesture tracking network model for processing, and the gesture tracking result is output. If the gesture tracking result indicates that the target image contains a gesture, then a first gesture box corresponding to the target image is determined, and the interior of the first gesture box contains the target gesture content; Determining the first gesture box corresponding to the target image includes: If the current frame image is not the first frame image, then the gesture box in the target image is obtained; Obtain the second gesture box corresponding to the previous frame image of the current frame image; the previous frame image is an image that has been processed by the terminal device; Calculate the intersection area and the union area of ​​the gesture box in the target image and the second gesture box, respectively; If the ratio of the intersection area to the union area is less than a first preset ratio, then the gesture box in the target image is determined as the first gesture box corresponding to the target image.

2. The method according to claim 1, characterized in that, Determining the second predicted gesture box based on the first predicted gesture box includes: If the first predicted gesture box is square, then obtain the coordinates of the calibration point, length, and width of the first predicted gesture box; Based on the coordinates, length, width, and expansion coefficient of the first predicted gesture frame, determine the coordinates, length, and width of the expanded frame. The second predicted gesture frame is determined based on the coordinates, length, and width of the expanded calibration point.

3. The method according to claim 1, characterized in that, The number of target images is multiple, and the gesture tracking result also includes the confidence score of each target image. Determining the first gesture bounding box corresponding to the target image includes: The confidence score is processed using a non-maximum suppression algorithm to determine the corresponding first gesture box from each target image.

4. The method according to claim 3, characterized in that, The step of processing the confidence score using a non-maximum suppression algorithm to determine the corresponding first gesture box from each target image includes: The target images are sorted according to the confidence level to obtain a candidate image list; A1. Remove the target image with the highest confidence from the candidate image list, and determine the corresponding first gesture box from the target image with the highest confidence; A2. Calculate the target ratio between the target image with the highest confidence and each target image in the candidate image list; A3. Delete the target images whose target ratio is greater than the second preset ratio from the candidate image list to obtain a candidate image list that includes the target images that were not deleted. Steps A1-A3 are executed repeatedly until the last target image remains in the candidate image list, and the corresponding first gesture box is determined from the last target image.

5. A gesture tracking device, characterized in that, include: The second processing module is used to input the current frame image into the gesture detection network for processing and output the first predicted gesture box if the current frame image meets the gesture detection conditions. The second determining module is used to determine a second predicted gesture box based on the first predicted gesture box, wherein the second predicted gesture box surrounds the first predicted gesture box. The third determining module is used to determine the target image based on the second predicted gesture box, and the size of the target image is a preset image size; The acquisition module is used to obtain the third gesture box corresponding to the previous frame image if the current frame image does not meet the gesture detection conditions. The bounding box expansion module is used to expand the bounding box of the third gesture box to obtain the third predicted gesture box. The cropping module is used to crop the current frame image based on the third predicted gesture box to obtain the target image; The first processing module is used to input the target image into the gesture tracking network model for processing and output the gesture tracking result. The first determining module is used to determine a first gesture box corresponding to the target image if the gesture tracking result is that the target image contains a gesture, wherein the interior of the first gesture box contains the target gesture content; The first determining module is specifically used for: If the current frame image is not the first frame image, then obtain the gesture box in the target image; Obtain the second gesture box corresponding to the previous frame image of the current frame image; the previous frame image is an image that has been processed by the terminal device; Calculate the intersection area and the union area of ​​the gesture bounding box and the second gesture bounding box in the target image, respectively; If the ratio of the intersection area to the union area is less than a first preset ratio, then the gesture box in the target image is determined as the first gesture box corresponding to the target image.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images