Hole correction method, electronic equipment and computer readable medium

By correcting the void position in the depth image, the problem of abnormal detection of hand joint nodes in gesture recognition is solved, and the accuracy and user experience of gesture recognition are improved.

CN118918160APending Publication Date: 2024-11-08ZTE CORP
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310507857.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the existing gesture recognition technology, abnormal detection of hand joint nodes results in inaccurate interaction location and poor user experience.

Method used

By acquiring visible light image sequences and depth image sequences with the same frame frequency, the hole position information in the depth image is determined, and the depth information of the previous frame is corrected to improve the accuracy of the depth information, and to extract and gesture the subsequent node position information and gestures Identification provides the basis.

Benefits of technology

Improve the accuracy of the location information of the joint node, improve the accuracy and user experience of gesture recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918160A_ABST
    Figure CN118918160A_ABST
Patent Text Reader

Abstract

The invention provides a hole correction method, electronic equipment and a computer readable medium. The hole correction method comprises the following steps: acquiring a visible light image sequence and a first depth image sequence with the same acquisition frame frequency; for each frame of first depth image in the first depth image sequence, determining whether the first depth image is a depth image with a hole according to the first depth image and a visible light image meeting a preset condition in the visible light image sequence; wherein the visible light image meeting the preset condition is a visible light image corresponding to the first depth image in position; when it is determined that the first depth image is the depth image with a hole, hole position information in the first depth image is determined; correcting the depth information of the hole corresponding to the hole position information according to N frames of first depth images before the first depth image; wherein N is 1 or 2.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a hole correction method, an electronic device, and a computer-readable medium. Background Art

[0002] In some scenarios, gesture interaction with objects is required, such as viewing artifacts in a museum. However, the biggest problem with current gesture recognition is abnormal detection of hand joints, which can lead to inaccurate interaction positions and poor user experience. Summary of the invention

[0003] Embodiments of the present application provide a hole correction method, an electronic device, and a computer-readable medium.

[0004] In a first aspect, an embodiment of the present application provides a hole correction method, comprising: acquiring a visible light image sequence and a first depth image sequence with the same acquisition frame frequency; for each first depth image frame in the first depth image sequence, determining whether the first depth image is a depth image with a hole based on the first depth image and the visible light image in the visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, determining the hole position information in the first depth image; and correcting the depth information of the hole corresponding to the hole position information based on N frames of first depth images located before the first depth image; wherein N is 1 or 2.

[0005] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; a memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, any one of the above-mentioned hole correction methods is implemented.

[0006] In a third aspect, an embodiment of the present application provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned hole correction methods is implemented.

[0007] The hole correction method provided in the embodiment of the present application corrects the depth information of the hole in the depth image, improves the accuracy of the depth information of the hole, and lays a foundation for the subsequent extraction of joint point position information and gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a gesture recognition method provided by an embodiment of the present application;

[0009] Figure 2A flowchart of a human-computer interaction method provided by another embodiment of the present application;

[0010] Figure 3 A flowchart of a hand joint point detection method provided by another embodiment of the present application;

[0011] Figure 4 A flowchart of a hole correction method provided in another embodiment of the present application;

[0012] Figure 5 A flowchart of a hand region acquisition model training method provided in another embodiment of the present application;

[0013] Figure 6 A flowchart of a hand joint point detection model training method provided by another embodiment of the present application;

[0014] Figure 7 A flowchart of a dynamic gesture model training method provided by another embodiment of the present application;

[0015] Figure 8 A block diagram of an electronic device according to another embodiment of the present application;

[0016] Fig. 9 It is a schematic diagram of the finger joint points in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present application, the gesture recognition method, electronic device, and computer-readable medium provided by the present application are described in detail below in conjunction with the accompanying drawings.

[0018] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make this application thorough and complete and to enable those skilled in the art to fully understand the scope of this application.

[0019] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0020] As used herein, the term "and / or" includes any and all combinations of at least one of the associated listed items.

[0021] The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of at least one other feature, whole, step, operation, element, component and / or its group is not excluded.

[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0023] Figure 1 A flowchart of a gesture recognition method provided by one embodiment of the present application.

[0024] First, refer to Figure 1 An embodiment of the present application provides a gesture recognition method, including:

[0025] Step 100: Acquire a visible light image sequence and a first depth image sequence having the same acquisition frame frequency.

[0026] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0027] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0028] In some exemplary embodiments, a visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the portion of the visible light image frame sequence placed in the first queue is the visible light image sequence that needs to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the portion of the first depth image frame sequence placed in the second queue is the first depth image sequence that needs to be acquired; the first queue and the second queue have the same size, both M; wherein M is an integer greater than or equal to 1.

[0029] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0030] In some exemplary embodiments, the visible light video stream and the first depth video stream may be obtained by correspondingly processing the originally acquired 3D data.

[0031] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0032] In some exemplary embodiments, visible light images in the visible light image frame sequence are put into a first queue after first preprocessing, and first depth images in the first depth image frame sequence are put into a second queue after second preprocessing.

[0033] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0034] In some exemplary embodiments, the second pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0035] In some exemplary embodiments, the visible light images obtained after first preprocessing the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0036] In some exemplary embodiments, the first depth image obtained after the second preprocessing is performed on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0037] Step 101, for each first depth image frame in a first depth image sequence, determine whether the first depth image is a depth image with a hole based on the first depth image and a visible light image in a visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, determine the hole position information in the first depth image; and correct the depth information of the hole corresponding to the hole position information based on N frames of first depth images located before the first depth image; wherein N is 1 or 2.

[0038] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information according to the first depth images of N frames located before the first depth image, the step of determining whether the first depth image is a depth image with a hole according to the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence for each frame of the first depth image in the first depth image sequence is continued until there are no depth images with holes in the first depth image sequence.

[0039] In some exemplary embodiments, determining whether the first depth image is a depth image with holes based on the first depth image and the visible light image corresponding to the first depth image in the visible light image sequence includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining a difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with holes.

[0040] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated in other ways can be determined.

[0041] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0042] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames located before the first depth image includes: obtaining the joint point position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the depth information of the hole after correction as a weighted average of a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames located before the first depth image, and prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the depth information of the hole after correction based on the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames located before the first depth image.

[0043] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand region acquisition model to obtain fifth hand region information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand region information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand region information.

[0044] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0045] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0046] In some exemplary embodiments, determining the depth information after hole correction as the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refi is the weight of the hole depth information corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the jth prior information of the joint point, D avg is the j-th prior information of the joint point.

[0047] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method, etc.) of OpenCV.

[0048] In some exemplary embodiments, the prior information of the joint point includes: length prior information and angle prior information of the joint point.

[0049] In some exemplary embodiments, the length prior information of the joint points is the average value of the length information of the joint points closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; wherein B is an integer greater than or equal to 1.

[0050] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0051] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0052] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0053] In some exemplary embodiments, the offset angle information of the joint point closest to the hole relative to the wrist joint point refers to the angle information between the straight line between the joint point closest to the hole and the proximal joint point of the finger and the straight line between the proximal joint point of the finger and the wrist joint point, wherein the joint point closest to the hole and the proximal joint point of the finger belong to the joint points of the same finger.

[0054] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0055] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0056] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0057] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0058] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained from the last gesture recognition and the dynamic gesture category information obtained from the last gesture recognition may be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the last gesture recognition.

[0059] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) may be used to calculate prior information of joint points corresponding to the crowd category.

[0060] In some exemplary embodiments, determining the depth information of the hole after correction based on the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: determining the depth information of the hole corresponding to the hole position information in the first depth image based on the depth information of the two joint points closest to the hole in the first depth image; determining the depth information of the hole after correction as a weighted average of the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0061] In some exemplary embodiments, the depth information of the two joint points closest to the hole in the first depth image can be interpolated to obtain the depth information of the hole corresponding to the hole position information in the first depth image; or the depth information of the two joint points closest to the hole in the first depth image can be calculated using a regional growing algorithm to obtain the depth information of the hole corresponding to the hole position information in the first depth image.

[0062] Step 102 , respectively obtain the joint point position information in each frame of the second depth image sequence; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

[0063] In some exemplary embodiments, respectively obtaining joint point position information in each frame of the second depth image sequence includes: for each frame of the second depth image, inputting a visible light image corresponding to the position of the second depth image in the visible light image sequence into a hand region acquisition model to obtain first hand region information of the visible light image; obtaining a first target sub-image corresponding to the first hand region information in the second depth image; inputting the first target sub-image into a hand joint point detection model to obtain joint point information corresponding to the first hand region information, wherein the joint point information corresponding to the first hand region information is the joint point position information in the second depth image.

[0064] In some exemplary embodiments, the position index of the visible light image corresponding to the second depth image in the visible light image sequence is the same as the position index of the second depth image in the second depth image sequence.

[0065] In some exemplary embodiments, the first hand region information refers to information of an image region including a hand and a part or all of a forearm.

[0066] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0067] In some exemplary embodiments, the first hand region information may be coordinate information of four corners of the image region.

[0068] In some exemplary embodiments, the first hand region information may be coordinate information of one corner of the image region, width information and length information of the image region.

[0069] In some exemplary embodiments, the visible light image at a position corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain at least one of palm and back category information and crowd category information.

[0070] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image area is the palm or the back of the hand.

[0071] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image area belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0072] In some exemplary embodiments, the visible light image corresponding to the position of the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain the confidence corresponding to the first hand region information; the first hand region information with a confidence greater than the second preset threshold is selected for subsequent processing, that is, the step of obtaining the first target sub-image corresponding to the first hand region information in the second depth image is continued. Here, the first hand region information with a confidence greater than the second preset threshold is considered to be valid first hand region information; the first hand region information with a confidence less than the second preset threshold is invalid first hand region information. Screening out the valid first hand region information through the confidence narrows the range for the subsequent acquisition of joint point information, thereby improving the subsequent gesture recognition accuracy.

[0073] In some exemplary embodiments, the first target sub-image corresponding to the first hand region information in the second depth image refers to a position of the first target sub-image in the second depth image, and the position of the image region corresponding to the first hand region information in the visible light image is the same.

[0074] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0075] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0076] Step 103: Determine dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image.

[0077] In some exemplary embodiments, the gesture category information may be at least one of finger forward click, finger pinch, rotation, zoom, move, static gesture, etc.

[0078] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: inputting the joint point information in the second depth image sequence and the second depth image into a dynamic gesture model to obtain the dynamic gesture category information.

[0079] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the fingertip joint points in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain the dynamic gesture category information.

[0080] In some exemplary embodiments, correcting the joint point information of the fingertip joint points in the joint point information in the second depth image includes: obtaining a fingertip sub-image from a visible light image in the visible light image sequence whose position corresponds to the second depth image based on the joint point information of the fingertip joint points; extracting the contour information of the fingertip in the fingertip sub-image; calculating the curvature of the points on the contour of the fingertip based on the contour information of the fingertip, and taking the position information of the convex point with the largest curvature value as the corrected joint point information of the fingertip joint points.

[0081] In some exemplary embodiments, the fingertip sub-image refers to a sub-image including the fingertip.

[0082] In some exemplary embodiments, in order to correct the joint point information of each fingertip joint, it is necessary to obtain the fingertip sub-image of each fingertip respectively and extract the contour information of the fingertip in the fingertip sub-image; calculate the curvature of the points on the contour of the fingertip according to the contour information of the fingertip, and use the position information of the convex point with the largest curvature value as the corrected joint point information of the fingertip joint.

[0083] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the finger joint points in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain the dynamic gesture category information.

[0084] In some exemplary embodiments, correcting the joint point information of the finger joint points in the joint point information in the second depth image includes: obtaining a finger joint sub-image from a visible light image in a visible light image sequence whose position corresponds to the second depth image based on the joint point information of the finger joint points; extracting the contour information of the finger joints in the finger joint sub-image; calculating the curvature of the points on the contour of the finger joints based on the contour information of the finger joints, and taking the position information of the concave point with the largest curvature value as the corrected joint point information of the finger joint points.

[0085] In some exemplary embodiments, the finger gap sub-image refers to a sub-image including a finger gap.

[0086] In some exemplary embodiments, in order to correct the joint point information of each finger joint, it is necessary to obtain the fingertip sub-image of each finger joint respectively, and extract the contour information of the fingertip in the finger joint sub-image; calculate the curvature of the points on the contour of the finger joint according to the contour information of the finger joint, and use the position information of the concave point with the largest curvature value as the corrected joint point information of the finger joint.

[0087] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the fingertip joints and the joint point information of the finger joints in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain the dynamic gesture category information.

[0088] In some exemplary embodiments, determining dynamic gesture category information based on the second depth image sequence and the joint point information in the second depth image includes: for each frame of the second depth image, performing kinematic constraint processing on the joint point information in the second depth image; and inputting the kinematic constraint processed joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain the dynamic gesture category information.

[0089] In some exemplary embodiments, performing kinematic constraint processing on the joint point information in the second depth image includes at least one of the following:

[0090] Correcting the joint point information of two adjacent joint points according to a value range of a first ratio of a length between two adjacent joint points to the hand region in which they are located and a first ratio in a second depth image of N frames before the second depth image;

[0091] Correcting the joint point information of other joint points according to the value range of the offset angles of other joint points relative to the wrist joint point and the offset angles in the second depth image of N frames before the second depth image;

[0092] Correcting the joint point information of the fingertip joint point and the wrist joint point according to the value range of the finger length and the finger length in the second depth image N frames before the second depth image; wherein the finger length is the straight-line distance between the fingertip joint point and the wrist joint point;

[0093] Correcting the joint point information of the wrist joint point and the proximal joint point of the finger according to the angle information of the forearm joint point and the hand and the angle information in the second depth image of N frames before the second depth image; wherein the angle information of the forearm joint point and the hand is the angle information of the angle between the straight line determined by the forearm joint point and the wrist joint point and the straight line determined by the wrist joint point and the proximal joint point of the finger;

[0094] According to the finger joints (such as Fig. 9The joint point information of the two finger joint points closest to the finger joint point is corrected according to the value range of the distance between the two finger joint points closest to the finger joint point and the distance between the finger joint point and the two finger joint points closest to the finger joint point in the N frames of the second depth image before the second depth image;

[0095] The joint point information of the finger joint points is corrected according to the value range of the bending degree of the finger relative to the palm and the back of the hand and the bending degree in the second depth image N frames before the second depth image.

[0096] In some exemplary embodiments, the hand region refers to a hand region corresponding to hand region information obtained by inputting a visible light image into a hand region acquisition model.

[0097] In some exemplary embodiments, a first ratio of the length between two adjacent joint points to the hand region in which they are located is a first ratio of the length between two adjacent joint points to the length of the hand region in which they are located.

[0098] In some exemplary embodiments, a first ratio of the length between two adjacent joint points to the hand region in which they are located is a first ratio of the length between two adjacent joint points to the width of the hand region in which they are located.

[0099] In some exemplary embodiments, the value range of the first ratio can be determined based on the maximum and minimum values ​​of the first ratios corresponding to all historical visible light images and historical depth images in C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category; where C is an integer greater than or equal to 1.

[0100] In some exemplary embodiments, when calculating the value range of the first ratio, it can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition, or it can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.).

[0101] In some exemplary embodiments, when calculating the value range of the first ratio, C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the same population category can be used to calculate the value range of the first ratio corresponding to the population category.

[0102] In some exemplary embodiments, the value range of the first ratio corresponding to the crowd category may be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0103] In some exemplary embodiments, the value range of the first ratio corresponding to a default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category.

[0104] In some exemplary embodiments, the value range of the first ratio corresponding to the crowd category may be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained from the last gesture recognition and the crowd category information obtained from the last gesture recognition.

[0105] In some exemplary embodiments, the value range of the first ratio corresponding to a default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to a default dynamic gesture category of the default crowd category.

[0106] In some exemplary embodiments, correcting the joint point information of two adjacent joint points according to a value range of a first ratio of a length between two adjacent joint points to a hand region in which the length of the two adjacent joint points is located and a first ratio in a second depth image N frames before the second depth image includes:

[0107] When a first ratio corresponding to two adjacent joint points in the second depth image is greater than a maximum value in a value range of the first ratio, and a first intermediate value between the first ratio corresponding to two adjacent joint points in the second depth image and a first ratio in a second depth image of N frames before the second depth image is greater than a maximum value in a value range of the first ratio, the maximum value in the value range of the first ratio is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the maximum value in the value range of the first ratio.

[0108] When a first ratio corresponding to two adjacent joint points in the second depth image is greater than a maximum value in a value range of the first ratio, and a first intermediate value between the first ratio corresponding to two adjacent joint points in the second depth image and a first ratio in a second depth image of N frames before the second depth image is less than a maximum value in a value range of the first ratio, the first intermediate value is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the first intermediate value.

[0109] When a first ratio corresponding to two adjacent joint points in the second depth image is less than a minimum value in a value range of the first ratio, and a second intermediate value between the first ratio corresponding to two adjacent joint points in the second depth image and the first ratio in the second depth image of N frames before the second depth image is less than a minimum value in a value range of the first ratio, the maximum value in the value range of the first ratio is taken as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the minimum value in the value range of the first ratio.

[0110] When a first ratio corresponding to two adjacent joint points in the second depth image is less than a minimum value in a value range of the first ratio, and a second intermediate value between the first ratio corresponding to two adjacent joint points in the second depth image and a first ratio in a second depth image of N frames before the second depth image is greater than a minimum value in a value range of the first ratio, the second intermediate value is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the second intermediate value.

[0111] In some exemplary embodiments, the offset angle of other joints other than the wrist joint relative to the wrist joint refers to the angle information between the straight line between the other joints and the proximal joint of the finger and the straight line between the proximal joint of the finger and the wrist joint, wherein the other joints and the proximal joint of the finger belong to the joints of the same finger.

[0112] In some exemplary embodiments, the range of values ​​of the offset angle can be determined based on the maximum and minimum values ​​of the offset angles corresponding to all historical visible light images and historical depth images in C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category; where C is an integer greater than or equal to 1.

[0113] In some exemplary embodiments, when calculating the value range of the offset angle, it can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition, or it can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.).

[0114] In some exemplary embodiments, when calculating the value range of the offset angle, C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the value range of the first ratio corresponding to the crowd category.

[0115] In some exemplary embodiments, the value range of the offset angle corresponding to the crowd category can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0116] In some exemplary embodiments, the value range of the offset angle corresponding to a default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category.

[0117] In some exemplary embodiments, the value range of the offset angle corresponding to the crowd category information obtained from the last gesture recognition can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained from the last gesture recognition.

[0118] In some exemplary embodiments, the value range of the offset angle corresponding to a default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to a default dynamic gesture category of the default crowd category.

[0119] In some exemplary embodiments, correcting the joint point information of other joint points except the wrist joint point according to the value range of the offset angle of other joint points relative to the wrist joint point and the offset angle in the second depth image of N frames before the second depth image includes:

[0120] When the offset angle in the second depth image is greater than the maximum value in the value range of the offset angle, and a third intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is greater than the maximum value in the value range of the offset angle, the maximum value in the value range of the offset angle is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the maximum value in the value range of the offset angle.

[0121] When the offset angle in the second depth image is greater than the maximum value in the range of values ​​of the offset angle, and a fourth intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is less than the maximum value in the range of values ​​of the offset angle, the fourth intermediate value is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the fourth intermediate value.

[0122] When the offset angle in the second depth image is smaller than the minimum value in the value range of the offset angle, and a third intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is smaller than the minimum value in the value range of the offset angle, the minimum value in the value range of the offset angle is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the minimum value in the value range of the offset angle.

[0123] When the offset angle in the second depth image is smaller than the minimum value in the value range of the offset angle, and a fourth intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is larger than the minimum value in the value range of the offset angle, the fourth intermediate value is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the fourth intermediate value.

[0124] In some exemplary embodiments, the finger length refers to the straight-line distance between the fingertip joint point and the wrist joint point.

[0125] In some exemplary embodiments, the process of determining the value range of finger length and correcting the joint point information of fingertip joints and wrist joints is similar to the process of determining the value range of the aforementioned first ratio and correcting the joint point information of two adjacent joints, and will not be repeated here.

[0126] In some exemplary embodiments, the process of determining the value range of the angle information of the forearm joints and the hand, and correcting the joint point information of the wrist joints and the proximal finger joints is similar to the process of determining the value range of the aforementioned first ratio, and correcting the joint point information of two adjacent joints, and will not be repeated here.

[0127] In some exemplary embodiments, the value range of the angle information of the forearm joint and the hand can also be determined by using a spherical simulation method. Specifically, the distance between the forearm joint and the wrist joint is used as the diameter of the first sphere, and the distance between the wrist joint and the proximal finger joint is used as the diameter of the second sphere. The value range of the angle information of the forearm joint and the hand is determined by the intersection range of the first sphere and the second sphere.

[0128] In some exemplary embodiments, the process of determining the value range of the distance between a finger joint point and the two finger joint points closest to the finger joint point, and correcting the joint point information of the two finger joint points closest to the finger joint point is similar to the process of determining the value range of the aforementioned first ratio, and correcting the joint point information of two adjacent joint points, and will not be repeated here.

[0129] In some exemplary embodiments, the full extent of a finger relative to the palm and back of the hand is the angle between a straight line between a proximal finger joint and a middle finger joint, and a straight line between a proximal finger joint and a wrist joint.

[0130] In some exemplary embodiments, the process of determining the value range of the degree of bending of the finger relative to the palm and the back of the hand, and correcting the joint point information of the finger joints is similar to the process of determining the value range of the aforementioned first ratio, and correcting the joint point information of two adjacent joints, and will not be repeated here.

[0131] In some exemplary embodiments, after acquiring a visible light image sequence and a first depth image sequence with the same acquisition frame frequency, and before determining the hole position information in the first depth image, the method further includes: determining the number of depth images with holes in the first depth image sequence; when the number of depth images with holes in the first depth image sequence is greater than or equal to a third preset threshold, performing data stream cleaning processing on the visible light image sequence and the first depth image sequence, and continuing to execute the step of determining the number of depth images with holes in the first depth image sequence after the data stream cleaning processing until the number of depth images with holes in the first depth image sequence after the data stream cleaning processing is less than the third preset threshold.

[0132] In some exemplary embodiments, when the number of depth images with holes in the first depth image sequence is less than a third preset threshold, the step of determining the hole position information in the first depth image is continued.

[0133] In some exemplary embodiments, data stream cleaning processing of the visible light image sequence and the first depth image sequence includes: deleting the A-frame visible light image with the earliest acquisition time in the visible light image sequence, obtaining the A-frame visible light image with the earliest acquisition time after the visible light image sequence, and adding it to the visible light image sequence; wherein A is an integer greater than or equal to 1; deleting the A-frame first depth image with the earliest acquisition time in the first depth image sequence, obtaining the A-frame first depth image with the earliest acquisition time after the first depth image sequence, and adding it to the first depth image sequence.

[0134] In some exemplary embodiments, after determining the dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image, the method further includes: acquiring original hand contour data in the visible light image in the visible light image sequence; preprocessing the original hand contour data to obtain intermediate hand contour data with continuous contour; generating a G2 continuous hand contour smooth curve based on the intermediate hand contour data; and performing filling rendering based on the hand contour smooth curve.

[0135] In some exemplary embodiments, the OpenCV method may be used to obtain the original hand contour data.

[0136] In some exemplary embodiments, the preprocessing includes at least one of algorithms such as denoising, filtering, and region growing.

[0137] In some exemplary embodiments, a computer graphics method may be used to generate a G2 continuous smooth curve of the hand contour.

[0138] In some exemplary embodiments, filling and rendering according to the smooth curve of the hand contour includes: transforming the original acquired 3D data into 2D data corresponding to the 2D coordinate system; mapping the parameter information of the 3D data in space to the 2D plane; and using a 2D renderer to render the inside of the hand contour at the pixel level according to the smooth curve of the hand contour to obtain a 2D mesh image.

[0139] In some exemplary embodiments, the parameter information may be position, direction, size, etc.

[0140] In some exemplary embodiments, when mapping parameter information of 3D data in space to a 2D plane, depth information is not considered, which greatly reduces the amount of calculation.

[0141] In some exemplary embodiments, when a 2D renderer is used to perform pixel-level rendering of the inside of the hand contour according to the smooth curve of the hand contour to obtain a 2D mesh image, only the inside of the hand contour is rendered, and only pixel-level rendering is performed, and the number of vertices of the triangular mesh of the mesh is reduced, thereby reducing the amount of calculation.

[0142] In some exemplary embodiments, after determining the dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image, the method further includes: performing data stream cleaning processing on the visible light image sequence and the first depth image sequence.

[0143] In some exemplary embodiments, after determining the dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image, the method further includes: performing an operation corresponding to the gesture category information.

[0144] In some exemplary embodiments, the operation corresponding to the gesture category information is, for example, opening or closing a corresponding window.

[0145] The gesture recognition method provided in the embodiment of the present application corrects the depth information of the holes in the depth image before obtaining the joint point position information, thereby improving the accuracy of the joint point position information and thus improving the accuracy of gesture recognition.

[0146] In some exemplary embodiments, kinematic constraints are applied to joint point information before determining the dynamic gesture category, thereby improving the joint point detection accuracy and further improving the accuracy of gesture recognition.

[0147] In some exemplary embodiments, corresponding kinematic constraint processing is performed on the joint point information of people of different age groups, such as the elderly, adults, and children, to further improve the accuracy of joint point detection.

[0148] In some exemplary embodiments, after the hand contour information is extracted, the curve G2 corresponding to the hand contour information is made continuous for hand rendering, which greatly reduces the amount of calculation in the entire rendering process and makes the entire gesture rendering process efficient and accurate.

[0149] In some exemplary embodiments, the method of obtaining the joint point information of the fingertip joints by obtaining the curvature of a point on the contour of the fingertip based on the contour information of the fingertip improves the accuracy of the joint point information of the fingertip joints.

[0150] Figure 2 A flowchart of a human-computer interaction method provided in accordance with another embodiment of the present application.

[0151] Second, refer to Figure 2 Another embodiment of the present application provides a human-computer interaction method, comprising:

[0152] Step 200, obtain a visible light image sequence and a first depth image sequence with the same acquisition frame frequency; perform gesture recognition based on the visible light image sequence and the first depth image sequence to obtain dynamic gesture category information, and execute an operation corresponding to the dynamic gesture category information; or obtain head ray direction information, and execute an operation corresponding to the head ray direction information.

[0153] In some exemplary embodiments, when dynamic gesture category information is obtained by performing gesture recognition based on the visible light image sequence and the first depth image sequence, operations corresponding to the dynamic gesture category information are performed; when dynamic gesture category information cannot be obtained by performing gesture recognition based on the visible light image sequence and the first depth image sequence, head ray direction information is obtained, and operations corresponding to the head ray direction information are performed.

[0154] In some exemplary embodiments, the gesture recognition method of the aforementioned embodiment may be used to perform gesture recognition, which will not be described in detail here.

[0155] In some exemplary embodiments, a positioning device such as an inertial measurement unit (IMU) may be used to obtain head ray direction information.

[0156] In some exemplary embodiments, the head ray direction information may be direction information of rays emitted by a sensor located on the head.

[0157] In some exemplary embodiments, the operation corresponding to the dynamic gesture category information or the operation corresponding to the head ray direction information may be an interactive operation with exhibits in a museum, as described in Application Example 1 and Application Example 2 below.

[0158] In some exemplary embodiments, the operation corresponding to the dynamic gesture category information or the operation corresponding to the head ray direction information may be a click operation, a double-click operation, a slide operation, or the like.

[0159] Figure 3 A flowchart of a hand joint detection method provided in accordance with another embodiment of the present application.

[0160] Thirdly, refer to Figure 3 Another embodiment of the present application provides a hand joint point detection method, comprising:

[0161] Step 300: Acquire a visible light image sequence and a first depth image sequence having the same acquisition frame frequency.

[0162] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0163] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0164] In some exemplary embodiments, a visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the portion of the visible light image frame sequence placed in the first queue is the visible light image sequence that needs to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the portion of the first depth image frame sequence placed in the second queue is the first depth image sequence that needs to be acquired; the first queue and the second queue have the same size, both M; wherein M is an integer greater than or equal to 1.

[0165] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0166] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0167] In some exemplary embodiments, visible light images in the visible light image frame sequence are put into a first queue after first preprocessing, and first depth images in the first depth image frame sequence are put into a second queue after second preprocessing.

[0168] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0169] In some exemplary embodiments, the second pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0170] In some exemplary embodiments, the visible light images obtained after first preprocessing the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0171] In some exemplary embodiments, the first depth image obtained after the second preprocessing is performed on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0172] Step 301, for each first depth image frame in the first depth image sequence, determine whether the first depth image is a depth image with a hole according to the first depth image and the visible light image in the visible light image sequence that meets the preset conditions; wherein the visible light image that meets the preset conditions is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, determine the hole position information in the first depth image; and correct the depth information of the hole corresponding to the hole position information according to the first depth images of N frames located before the first depth image; wherein N is 1 or 2.

[0173] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information according to the first depth images of N frames located before the first depth image, the step of determining whether the first depth image is a depth image with a hole according to the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence for each frame of the first depth image in the first depth image sequence is continued until there are no depth images with holes in the first depth image sequence.

[0174] In some exemplary embodiments, determining whether the first depth image is a depth image with a hole problem based on the first depth image and the visible light image whose position corresponds to the first depth image in the visible light image sequence includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining the difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with holes.

[0175] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated in other ways can be determined.

[0176] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0177] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames located before the first depth image includes: obtaining the joint point position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the depth information of the hole after correction as a weighted average of a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames located before the first depth image, and prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the depth information of the hole after correction according to the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames located before the first depth image.

[0178] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand region acquisition model to obtain fifth hand region information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand region information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand region information.

[0179] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0180] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0181] In some exemplary embodiments, determining the depth information after hole correction as the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refi is the weight of the hole depth information corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the jth prior information of the joint point, D avgj is the j-th prior information of the joint point.

[0182] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method, etc.) of OpenCV.

[0183] In some exemplary embodiments, the prior information of the joint point includes: length prior information and angle prior information of the joint point.

[0184] In some exemplary embodiments, the length prior information of the joint points is the average value of the length information of the joint points closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; wherein B is an integer greater than or equal to 1.

[0185] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0186] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0187] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0188] In some exemplary embodiments, the offset angle information of the joint point closest to the hole relative to the wrist joint point refers to the angle information between the straight line between the joint point closest to the hole and the proximal joint point of the finger and the straight line between the proximal joint point of the finger and the wrist joint point, wherein the joint point closest to the hole and the proximal joint point of the finger belong to the joint points of the same finger.

[0189] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0190] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0191] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0192] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0193] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained from the last gesture recognition and the dynamic gesture category information obtained from the last gesture recognition may be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the last gesture recognition.

[0194] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) may be used to calculate prior information of joint points corresponding to the crowd category.

[0195] In some exemplary embodiments, determining the depth information of the hole after correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: performing an interpolation operation on the depth information of the two joint points closest to the hole in the first depth image to obtain the depth information of the hole corresponding to the hole position information in the first depth image; determining the depth information of the hole after correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the weighted average of the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0196] Step 302 , obtaining joint point position information in a second depth image in a second depth image sequence; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

[0197] In some exemplary embodiments, respectively obtaining joint point position information in each frame of the second depth image sequence includes: for each frame of the second depth image, inputting a visible light image corresponding to the position of the second depth image in the visible light image sequence into a hand region acquisition model to obtain first hand region information of the visible light image; obtaining a first target sub-image corresponding to the first hand region information in the second depth image; inputting the first target sub-image into a hand joint point detection model to obtain joint point information corresponding to the first hand region information, wherein the joint point information corresponding to the first hand region information is the joint point position information in the second depth image.

[0198] In some exemplary embodiments, a position index of a visible light image corresponding to the second depth image in the visible light image sequence is the same as a position index of the second depth image in the second depth image sequence.

[0199] In some exemplary embodiments, the first hand region information refers to information of an image region including a hand and a part or all of a forearm.

[0200] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0201] In some exemplary embodiments, the first hand region information may be coordinate information of four corners of the image region.

[0202] In some exemplary embodiments, the first hand region information may be coordinate information of one corner of the image region, width information and length information of the image region.

[0203] In some exemplary embodiments, the visible light image at a position corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain at least one of palm and back category information and crowd category information.

[0204] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image area is the palm or the back of the hand.

[0205] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image area belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0206] In some exemplary embodiments, the visible light image corresponding to the position of the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain the confidence corresponding to the first hand region information; the first hand region information with a confidence greater than the second preset threshold is selected for subsequent processing, that is, the step of obtaining the first target sub-image corresponding to the first hand region information in the second depth image is continued. Here, the first hand region information with a confidence greater than the second preset threshold is considered to be valid first hand region information; the first hand region information with a confidence less than the second preset threshold is invalid first hand region information. Screening out the valid first hand region information through the confidence narrows the range for the subsequent acquisition of joint point information, thereby improving the subsequent gesture recognition accuracy.

[0207] In some exemplary embodiments, the first target sub-image corresponding to the first hand region information in the second depth image refers to a position of the first target sub-image in the second depth image, and the position of the image region corresponding to the first hand region information in the visible light image is the same.

[0208] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0209] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0210] The hand joint point detection method provided in the embodiment of the present application corrects the depth information of the holes in the depth image before obtaining the joint point position information, thereby improving the accuracy of the joint point position information.

[0211] Figure 4 A flowchart of a hole correction method provided in another embodiment of the present application.

[0212] Fourthly, refer to Figure 4 Another embodiment of the present application provides a hole correction method, comprising:

[0213] Step 400: Acquire a visible light image sequence and a first depth image sequence having the same acquisition frame frequency.

[0214] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0215] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0216] In some exemplary embodiments, a visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the portion of the visible light image frame sequence placed in the first queue is the visible light image sequence that needs to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the portion of the first depth image frame sequence placed in the second queue is the first depth image sequence that needs to be acquired; the first queue and the second queue have the same size, both M; wherein M is an integer greater than or equal to 1.

[0217] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0218] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0219] In some exemplary embodiments, visible light images in the visible light image frame sequence are put into a first queue after first preprocessing, and first depth images in the first depth image frame sequence are put into a second queue after second preprocessing.

[0220] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0221] In some exemplary embodiments, the second pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0222] In some exemplary embodiments, the visible light images obtained after first preprocessing the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0223] In some exemplary embodiments, the first depth image obtained after the second preprocessing is performed on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0224] Step 401, for each first depth image frame in the first depth image sequence, determine whether the first depth image is a depth image with a hole according to the first depth image and a visible light image that meets a preset condition in the visible light image sequence; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image.

[0225] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information according to the first depth images of N frames located before the first depth image, the step of determining whether the first depth image is a depth image with a hole according to the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence for each frame of the first depth image in the first depth image sequence is continued until there are no depth images with holes in the first depth image sequence.

[0226] In some exemplary embodiments, determining whether the first depth image is a depth image with a hole problem based on the first depth image and the visible light image whose position corresponds to the first depth image in the visible light image sequence includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining the difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with a hole problem.

[0227] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated in other ways can be determined.

[0228] Step 402, when it is determined that the first depth image is a depth image with a hole, determine the hole position information in the first depth image; and correct the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image; wherein N is 1 or 2.

[0229] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0230] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames located before the first depth image includes: obtaining the joint point position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the depth information of the hole after correction as a weighted average of a basic filling value, the depth information of the hole corresponding to the hole position information in the depth images of N frames located before the first depth image, and the prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the depth information of the hole after correction according to the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the depth images of N frames located before the first depth image.

[0231] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand region acquisition model to obtain fifth hand region information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand region information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand region information.

[0232] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0233] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0234] In some exemplary embodiments, determining the depth information after hole correction as the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refi is the weight of the hole depth information corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the jth prior information of the joint point, D avgj is the j-th prior information of the joint point.

[0235] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method, etc.) of OpenCV.

[0236] In some exemplary embodiments, the prior information of the joint point includes: length prior information and angle prior information of the joint point.

[0237] In some exemplary embodiments, the length prior information of the joint points is the average value of the length information of the joint points closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; wherein B is an integer greater than or equal to 1.

[0238] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0239] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0240] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0241] In some exemplary embodiments, the offset angle information of the joint point closest to the hole relative to the wrist joint point refers to the angle information between the straight line between the joint point closest to the hole and the proximal joint point of the finger and the straight line between the proximal joint point of the finger and the wrist joint point, wherein the joint point closest to the hole and the proximal joint point of the finger belong to the joint points of the same finger.

[0242] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0243] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0244] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0245] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0246] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained from the last gesture recognition and the dynamic gesture category information obtained from the last gesture recognition may be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the last gesture recognition.

[0247] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) may be used to calculate prior information of joint points corresponding to the crowd category.

[0248] In some exemplary embodiments, determining the depth information of the hole after correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: performing an interpolation operation on the depth information of the two joint points closest to the hole in the first depth image to obtain the depth information of the hole corresponding to the hole position information in the first depth image; determining the depth information of the hole after correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the weighted average of the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0249] The hand joint point detection method provided in the embodiment of the present application corrects the depth information of the holes in the depth image, improves the accuracy of the depth information of the holes, and lays a foundation for the subsequent extraction of joint point position information and gesture recognition.

[0250] Figure 5 A flowchart of a hand region acquisition model training method provided in another embodiment of the present application.

[0251] Fifth, refer to Figure 5 Another embodiment of the present application provides a hand region acquisition model training method, comprising:

[0252] Step 500, obtaining a first data sample for model training; wherein the first data sample includes: a first historical visible light image sequence.

[0253] In some exemplary embodiments, the first historical visible light image sequence may be a first historical RGB image sequence or a first historical grayscale image sequence.

[0254] In some exemplary embodiments, the first historical visible light image sequence may be acquired from a first historical visible light video stream.

[0255] In some exemplary embodiments, the first historical visible light video stream may be decoded to obtain a first historical visible light image frame sequence, and the first historical visible light image sequence may be acquired from the first historical visible light image frame sequence.

[0256] In some exemplary embodiments, after first preprocessing is performed on the first historical visible light images in the first historical visible light image frame sequence, the first historical visible light image sequence is acquired from the first preprocessed first historical visible light image frame sequence.

[0257] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0258] In some exemplary embodiments, the first historical visible light image obtained by performing a first preprocessing on the first historical visible light image in the first historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0259] Step 501, adding annotation information to the first data sample; wherein the annotation information includes: second hand region information.

[0260] In some exemplary embodiments, the second hand region information refers to information of a first image region including a hand and part or all of a forearm in a first historical visible light image in a first historical visible light image sequence, where the image region is the hand region.

[0261] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0262] In some exemplary embodiments, the second hand region information may be coordinate information of four corners of the first image region.

[0263] In some exemplary embodiments, the second hand region information may be coordinate information of one corner of the first image region, width information and length information of the image region.

[0264] In some exemplary embodiments, the annotation information may further include: at least one of palm and back of hand category information and crowd category information.

[0265] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image area is the palm or the back of the hand.

[0266] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image area belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0267] In some exemplary embodiments, when adding annotation information to the first data sample, a target detection method may be used to detect the first image region from the first data sample to obtain the second hand region information, or the annotation information may be added manually.

[0268] Step 502: Use the first data sample as the input of the hand region acquisition model, and use the annotation information as the output of the hand region acquisition model to perform model training to obtain the hand region acquisition model.

[0269] In some exemplary embodiments, after adding the annotation information to the first data sample and before performing the model training, the method further includes: performing cross-cleaning processing on the annotation information.

[0270] Using the first data sample as the input of the hand area acquisition model, and using the annotation information as the output of the hand area acquisition model for model training to obtain the hand area acquisition model includes: using the first data sample as the input of the hand area acquisition model, and using the annotation information after cross-cleaning processing as the output of the hand area acquisition model for model training to obtain the hand area acquisition model.

[0271] In some exemplary embodiments, when adding annotation information to the first data sample, multiple people can add corresponding annotation information to the same first data sample, and the annotation information added by different people to the same first data sample can be cross-cleaned to obtain the final annotation information of the first data sample.

[0272] In some exemplary embodiments, a first data sample for model training is obtained, and data enhancement may be performed on the first data sample, and annotation information may be added to the first data sample after data enhancement; the first data sample after data enhancement is used as input of the hand region acquisition model, and the annotation information is used as output of the hand region acquisition model for model training to obtain the hand region acquisition model. The generalization of the hand region acquisition model is improved by the data enhancement method.

[0273] In some exemplary embodiments, a quantization operation may be performed on the model parameters in the trained hand region acquisition model to increase the speed of subsequently obtaining the first region information based on the hand region acquisition model.

[0274] Figure 6 A flowchart of a hand joint point detection model training method provided in another embodiment of the present application.

[0275] Sixth, refer to Figure 6 Another embodiment of the present application provides a hand joint point detection model training method, comprising:

[0276] Step 600, obtaining a second data sample for model training; wherein the second data sample includes: a second historical visible light image sequence and a first historical depth image sequence having the same acquisition frame frequency.

[0277] In some exemplary embodiments, the second historical visible light image sequence may be a second historical RGB image sequence or a second historical grayscale image sequence.

[0278] In some exemplary embodiments, the second historical visible light image sequence and the first historical depth image sequence may be acquired from a second historical visible light video stream and a first historical depth video stream having the same acquisition frame frequency.

[0279] In some exemplary embodiments, the second historical visible light video stream can be decoded to obtain a second historical visible light image frame sequence, and the second historical visible light image sequence can be obtained from the second historical visible light image frame sequence; the first historical depth video stream can be decoded to obtain a first historical depth image frame sequence, and the first historical depth image sequence can be obtained from the first historical depth image frame sequence; the second historical visible light image sequence and the first historical depth image sequence have the same size, both M; where M is an integer greater than or equal to 1.

[0280] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0281] In some exemplary embodiments, the second historical visible light image sequence and the first historical depth image sequence are aligned, that is, the second historical visible light image and the first historical depth image at the same position are acquired simultaneously.

[0282] In some exemplary embodiments, the second historical visible light images in the second historical visible light image frame sequence are subjected to a first preprocessing and then to a subsequent processing, and the first historical depth images in the first historical depth image frame sequence are subjected to a second preprocessing and then to a subsequent processing.

[0283] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0284] In some exemplary embodiments, the second pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0285] In some exemplary embodiments, the second historical visible light image obtained by performing a first preprocessing on the second historical visible light image in the second historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0286] In some exemplary embodiments, the first historical depth image obtained after performing a second preprocessing on the first historical depth image in the first historical depth image frame sequence meets the input requirement of the hand region acquisition model.

[0287] Step 601: input a second historical visible light image in a second historical visible light image sequence into a hand region acquisition model to obtain third hand region information of the second historical visible light image.

[0288] Step 602: Acquire a second target sub-image corresponding to the third hand region information of the second historical visible light image from the first historical depth image in the first historical depth image sequence whose position corresponds to the second historical visible light image; and acquire joint point information in the second target sub-image.

[0289] In some exemplary embodiments, the position index of the first historical depth image whose position in the first historical depth image sequence corresponds to the second historical visible light image in the first historical depth image sequence is the same as the position index of the second historical visible light image in the second historical visible light image sequence.

[0290] In some exemplary embodiments, in the first historical depth image whose position in the first historical depth image sequence corresponds to the second historical visible light image, the second target sub-image corresponding to the third hand region information of the second historical visible light image refers to the position of the second target sub-image in the first historical depth image sequence that corresponds to the second historical visible light image, and the image region corresponding to the third hand region information is the same as the position in the second historical visible light image.

[0291] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0292] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0293] In some exemplary embodiments, when the model is trained for the first time, other methods in the related art may be used to obtain the joint point information in the second target sub-image. Starting from the second model training, the joint point information in the second target sub-image may be obtained based on the hand joint point detection model obtained in the previous training.

[0294] Step 603: Use the first historical depth image in the first historical depth image sequence as the input of the hand joint point detection model, and use the joint point information in the second target sub-image as the output of the hand joint point detection model to perform model training to obtain the hand joint point detection model.

[0295] In some exemplary embodiments, before performing model training or obtaining joint point information in the second target sub-image, the method also includes: for each frame of the first historical depth image in the first historical depth image sequence, determining whether the first historical depth image is a historical depth image with a hole according to the first historical depth image and the second historical visible light image sequence whose position corresponds to the first historical depth image; when it is determined that the first historical depth image is a historical depth image with a hole, determining the hole position information in the first historical depth image; and correcting the depth information of the hole corresponding to the hole position information according to the N frames of the first historical depth image located before the first historical depth image; wherein N is 1 or 2.

[0296] The first historical depth image after hole correction is used as the input of the hand joint point detection model, and the joint point information in the second target sub-image is used as the output of the hand joint point detection model for model training to obtain the hand joint point detection model.

[0297] In some exemplary embodiments, determining whether the first historical depth image is a historical depth image with a hole based on a second historical visible light image whose position corresponds to the first historical depth image in a sequence of first historical depth images and second historical visible light images includes: determining a binarized image of the second historical visible light image and a binarized image of the first historical depth image; determining the difference between the binarized image of the second historical visible light image and the binarized image of the first historical depth image; when the difference is less than a first preset threshold, determining that the first historical depth image is a historical depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first historical depth image is a historical depth image with a hole.

[0298] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated in other ways can be determined.

[0299] In some exemplary embodiments, determining the hole position information in the first historical depth image includes: determining a binarized image of the first historical depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0300] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first historical depth image N frames before the first historical depth image includes: obtaining the joint point position information in the first historical depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the corrected depth information of the hole as a weighted average of a basic filling value, the depth information of the hole corresponding to the hole position information in the first historical depth image N frames before the first historical depth image, and prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the corrected depth information of the hole based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first historical depth image N frames before the first historical depth image.

[0301] In some exemplary embodiments, a second data sample for model training is obtained, and data enhancement may be performed on the second data sample, and a hand joint point detection model is obtained by model training based on the second data sample after data enhancement. The generalization of the hand joint point detection model is improved by the data enhancement method.

[0302] In some exemplary embodiments, the model parameters in the trained hand joint point detection model may be quantized to increase the speed of subsequently obtaining joint point information based on the hand joint point detection model.

[0303] Figure 7 A flowchart of a dynamic gesture model training method provided in another embodiment of the present application.

[0304] Seventh, refer to Figure 7 Another embodiment of the present application provides a dynamic gesture model training method, comprising:

[0305] Step 700, obtaining a third data sample for model training; wherein the third data sample includes: a third historical visible light image sequence and a second historical depth image sequence having the same acquisition frame frequency.

[0306] In some exemplary embodiments, the third historical visible light image sequence may be a third historical RGB image sequence or a third historical grayscale image sequence.

[0307] In some exemplary embodiments, the third historical visible light image sequence and the second historical depth image sequence may be acquired from a third historical visible light video stream and a second historical depth video stream having the same acquisition frame frequency.

[0308] In some exemplary embodiments, the third historical visible light video stream can be decoded to obtain a third historical visible light image frame sequence, and a third historical visible light image sequence can be obtained from the third historical visible light image frame sequence; the second historical depth video stream can be decoded to obtain a second historical depth image frame sequence, and a second historical depth image sequence can be obtained from the second historical depth image frame sequence; the third historical visible light image sequence and the second historical depth image sequence have the same size, both M; wherein M is an integer greater than or equal to 1.

[0309] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0310] In some exemplary embodiments, the third historical visible light image sequence and the second historical depth image sequence are aligned, that is, the third historical visible light image and the second historical depth image at the same position are acquired simultaneously.

[0311] In some exemplary embodiments, a third historical visible light image in a third historical visible light image frame sequence is subjected to a first preprocessing and then to a subsequent processing, and a second historical depth image in a second historical depth image frame sequence is subjected to a second preprocessing and then to a subsequent processing.

[0312] In some exemplary embodiments, the first pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0313] In some exemplary embodiments, the second pre-processing may be at least one of a resizing process, a pixel value normalization process, and the like.

[0314] In some exemplary embodiments, the third historical visible light image obtained after performing a first preprocessing on the third historical visible light image in the third historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0315] In some exemplary embodiments, the second historical depth image obtained after performing a second preprocessing on the second historical depth image in the second historical depth image frame sequence meets the input requirement of the hand region acquisition model.

[0316] Step 701: input a third historical visible light image in a third historical visible light image sequence into a hand region acquisition model to obtain fourth hand region information of the third historical visible light image.

[0317] Step 702: Obtain a third target sub-image corresponding to the fourth hand region information of the third historical visible light image from the second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image; input the third target sub-image into the hand joint point detection model to obtain the joint point information in the third target sub-image.

[0318] In some exemplary embodiments, the position index of the second historical depth image in the second historical depth image sequence whose position corresponds to the third historical visible light image in the second historical depth image sequence is the same as the position index of the third historical visible light image in the third historical visible light image sequence.

[0319] In some exemplary embodiments, in the second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image, the third target sub-image corresponding to the fourth hand region information of the third historical visible light image refers to the position of the third target sub-image in the second historical depth image sequence that corresponds to the third historical visible light image, and the image region corresponding to the fourth hand region information is the same as the position in the third historical visible light image.

[0320] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0321] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0322] Step 703: annotate the second historical depth image sequence with dynamic gesture category information.

[0323] In some exemplary embodiments, the gesture category information may be at least one of finger forward click, finger pinch, rotation, zoom, move, static gesture, etc.

[0324] Step 704 , taking the second historical depth image sequence and the corresponding joint point information as input of the dynamic gesture model, taking the dynamic gesture category information corresponding to the second historical depth image sequence as output of the dynamic gesture model to perform model training to obtain the dynamic gesture model.

[0325] In some exemplary embodiments, after the third target sub-image is input into the hand joint detection model to obtain the joint information in the third target sub-image, before model training, the method further includes: drawing the joints corresponding to the joint information in the second historical depth image in the third historical visible light image whose position corresponds to the second historical depth image, and when the number of third historical visible light images with abnormal joints in the third historical visible light image sequence is greater than half of the number of third historical visible light images in the third historical visible light image sequence, deleting the third data sample from the data samples for model training; and When the number is less than half of the number of third historical visible light images in the third historical visible light image sequence, for each frame of the second historical depth image in the second historical depth image sequence, determine whether the second historical depth image is a historical depth image with holes according to the second historical depth image and the third historical visible light image corresponding to the second historical depth image in the third historical visible light image sequence; when it is determined that the second historical depth image is a historical depth image with holes, determine the hole position information in the second historical depth image; and correct the depth information of the hole corresponding to the hole position information according to the N frames of second historical depth images before the second historical depth image; wherein N is 1 or 2.

[0326] Obtaining a third target sub-image corresponding to the fourth hand region information of the third historical visible light image from a second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image includes: obtaining a third target sub-image corresponding to the fourth hand region information of the third historical visible light image from a hole-corrected second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image.

[0327] In some exemplary embodiments, when the number of joint points drawn in the third historical visible light image and the corresponding joint points in the third historical visible light image are greater than or equal to a fourth preset threshold, the joint points of the third historical visible light image are determined to be normal; when the number of joint points drawn in the third historical visible light image and the corresponding joint points in the third historical visible light image are less than the fourth preset threshold, the joint points of the third historical visible light image are determined to be abnormal.

[0328] In some exemplary embodiments, before obtaining the third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image, the method also includes: for each frame of the second historical depth image in the second historical depth image sequence, determining whether the second historical depth image is a historical depth image with a hole based on the second historical depth image and the third historical visible light image whose position in the third historical visible light image sequence corresponds to the second historical depth image; when it is determined that the second historical depth image is a historical depth image with a hole, determining the hole position information in the second historical depth image; and correcting the depth information of the hole corresponding to the hole position information based on N frames of second historical depth images located before the second historical depth image; wherein N is 1 or 2.

[0329] Obtaining a third target sub-image corresponding to the fourth hand region information of the third historical visible light image from a second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image includes: obtaining a third target sub-image corresponding to the fourth hand region information of the third historical visible light image from a hole-corrected second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image.

[0330] In some exemplary embodiments, determining whether the second historical depth image is a historical depth image with a hole based on the third historical visible light image whose position corresponds to the second historical depth image in the sequence of the second historical depth image and the third historical visible light image includes: determining a binarized image of the third historical visible light image and a binarized image of the second historical depth image; determining the difference between the binarized image of the third historical visible light image and the binarized image of the second historical depth image; when the difference is less than a first preset threshold, determining that the second historical depth image is a historical depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the second historical depth image is a historical depth image with a hole.

[0331] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated in other ways can be determined.

[0332] In some exemplary embodiments, determining the hole position information in the second historical depth image includes: determining a binarized image of the second historical depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0333] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the second historical depth image N frames before the second historical depth image includes: obtaining the joint point position information in the second historical depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the corrected depth information of the hole as a weighted average of a basic filling value, the depth information of the hole corresponding to the hole position information in the second historical depth image N frames before the second historical depth image, and prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the corrected depth information of the hole based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the second historical depth image N frames before the second historical depth image.

[0334] In some exemplary embodiments, a third data sample is obtained for model training, data enhancement may be performed on the third data sample, and model training is performed based on the data-enhanced third data sample to obtain a dynamic gesture model. The generalization of the dynamic gesture model is improved by the data enhancement method.

[0335] In some exemplary embodiments, the model parameters in the trained hand joint point detection model may be quantized to increase the speed of subsequently obtaining joint point information based on the hand joint point detection model.

[0336] Two application examples are listed below to illustrate the application of the gesture recognition method of the embodiment of the present application. The listed application examples are not intended to limit the protection scope of the embodiment of the present application.

[0337] Application Example 1

[0338] This application example describes wearing a product that implements gesture recognition using the gesture recognition method of the embodiment of this application. The camera captures the exhibits in the user's field of view, which may be more than one. Through gesture clicks or head rays, a certain exhibit is selected for relevant introduction. Among them, gestures and head movements are used to interact with virtual objects. The general process of the user's field of view display process is as follows:

[0339] The main process of gesture clicking is that the camera captures the image of the exhibit and renders it, clicks the exhibit to be appreciated through gestures, and retrieves the voice information and 3D rendering information of the exhibit from the exhibit database through the exhibit ID or exhibit features.

[0340] The main process of head ray is that the camera captures the image of the exhibit and renders it. Through head ray, which lasts for 2-3 seconds, click on the exhibit you want to appreciate, and retrieve the voice information and 3D rendering information of the exhibit from the exhibit database through the exhibit ID or exhibit features.

[0341] After receiving the voice information and 3D rendering information, the voice information of the exhibit can be automatically played, and the 3D rendering information of the exhibit can be presented at the same time. The 3D rendering information can be a single exhibit. At this time, visitors can raise their hands to operate the 3D rendering exhibits, such as clicking on the exhibits with gestures, pinching the exhibits with gestures and rotating them up, down, left, and right. Visitors can view the inside and outside information of the exhibits in the device. The 3D rendering information can be an exhibit video, including its main features, development process, excavation process, etc. At this time, visitors can raise their hands to operate the playback progress of the video, etc.

[0342] When using gestures to interact with exhibits or operate page information, the rendering information of the user's hand also appears in the user's field of view to enhance the user experience and accurately control the fine position information.

[0343] The specific internal processing flow is as follows:

[0344] When the electronic device using the embodiment of the present application is turned on, an inference initialization process is performed to start loading the hand area acquisition model, the hand joint point detection model, and the dynamic gesture model.

[0345] After the power-on is completed, the camera starts to collect video data streams, decode them and put them into the RGB video queue and depth image queue respectively. After basic image preprocessing of the video stream, the RGB image is used to detect the hand area acquisition model. After obtaining the hand area information, it corresponds to the depth image and corrects the holes. Among them, if most of the depth images have holes, the video sequence of the queue will be discarded from the beginning. After correcting the holes in the depth image, the hand joint detection model is used to obtain the sequence of hand joints. At the same time, the RGB image is used to obtain the fine position information of the fingertips and update it to the joint sequence. The joints are constrained by human kinematic constraints to obtain a joint sequence that meets the constraints. Dynamic gesture model detection is performed to obtain dynamic gesture category information.

[0346] If the user clicks on an exhibit in the field of view at this time, the dynamic gesture category information obtained through the dynamic gesture model is click category information. After the application layer obtains the dynamic gesture category information, it magnifies the exhibit accordingly and starts playing relevant voice or video information about the exhibit.

[0347] At the same time, the bottom layer processes the video sequence in the queue from the beginning and collects new video streams and adds them to the end of the queue.

[0348] Application Example 2

[0349] Wearing the product designed in the embodiment of the present application, when appreciating calligraphy and painting, you can get the voice information and 3D rendering information of the calligraphy and painting through gesture clicks or head rays. At this time, the voice information and 3D rendering video information are automatically played. You can also experience it yourself.

[0350] The main process of the self-experience activity is to write and draw with the master according to the 3D rendering video. The fingertips of the gesture are used as pens, which requires a high position. And according to the depth and tracking of the gesture, the "lightness" and "speed" of writing or drawing can be determined from time to time.

[0351] The specific internal processing flow is as follows:

[0352] Basically the same as the previous application example, after receiving the user's click gesture, the application layer is in the writing and painting application and is in the experience state. The user continues the click gesture for a few seconds and tracks the changes in the gesture trajectory to obtain the handwriting of the writing and painting. When the user's pen is lifted from the scroll, the depth value of the gesture decreases, indicating that the stroke has ended. If you need to continue writing and painting, the user's pen is pressed on the scroll again. At this time, the depth value of the gesture increases, and the same gesture is continued for a few seconds. Tracking the changes in the gesture trajectory, you can get another handwriting.

[0353] By using dynamic gestures in this way, the user can feel immersed in the experience and have a better sense of experience.

[0354] Eighth aspect, refer to Figure 8 Another embodiment of the present application provides an electronic device, including: at least one processor 801; a memory 802, wherein at least one program is stored in the memory 802, and when the at least one program is executed by the at least one processor 801, any one of the above-mentioned gesture recognition methods, or any one of the above-mentioned human-computer interaction methods, or any one of the above-mentioned hand joint point detection methods, or any one of the above-mentioned hole correction methods are implemented.

[0355] In some exemplary embodiments, the electronic device may be, but is not limited to, any one or more of augmented reality (AR) / extended reality (XR) glasses, a smart helmet, and a smart watch.

[0356] In some exemplary embodiments, the electronic device further includes: a video input module 803, configured to capture a visible light video stream and a first depth video stream having the same frame frequency.

[0357] In some exemplary embodiments, the video input module 803 may be a camera.

[0358] In some exemplary embodiments, the electronic device further includes: a voice output module 804, configured to output voice information.

[0359] In some exemplary embodiments, the voice output module 804 may be a speaker.

[0360] In some exemplary embodiments, the electronic device further includes: a voice input module 805 for collecting sound information.

[0361] In some exemplary embodiments, the voice input module 805 may be a microphone.

[0362] In some exemplary embodiments, the electronic device further includes: one or more I / O interfaces 806 connected between the processor 801 and the memory 802 , and configured to implement information interaction between the processor 801 and the memory 802 .

[0363] Among them, the processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 702 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 806 is connected between the processor 801 and the memory 802, and can realize information interaction between the processor 801 and the memory 802, including but not limited to a data bus (Bus), etc.

[0364] In some embodiments, the processor 801 , the memory 802 , and the I / O interface 806 are connected to each other via a bus 807 , and further connected to other components of the computing device.

[0365] In the ninth aspect, another embodiment of the present application provides a computer-readable medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any one of the above-mentioned gesture recognition methods, or implements any one of the above-mentioned human-computer interaction methods, or implements any one of the above-mentioned hand joint detection methods, or implements any one of the above-mentioned hole correction methods.

[0366] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0367] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A void correction method, comprising: Acquire a visible light image sequence and a first depth image sequence having the same acquisition frame frequency; For each frame of the first depth image in the first depth image sequence, determine whether the first depth image is a depth image with a hole based on the first depth image and the visible light image in the visible light image sequence that meets the preset conditions; wherein the visible light image that meets the preset conditions is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, determine the hole position information in the first depth image; and correct the depth information of the hole corresponding to the hole position information based on the N frames of first depth images located before the first depth image; wherein N is 1 or 2.

2. The hole correction method according to claim 1, after correcting the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image, the method further comprises: The joint point position information in each frame of the second depth image in the second depth image sequence is obtained respectively; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

3. The hole correction method according to claim 2, wherein: The step of respectively acquiring the joint point position information in each frame of the second depth image sequence comprises: For each frame of the second depth image, inputting a visible light image at a position corresponding to the second depth image in the visible light image sequence into a hand region acquisition model to obtain first hand region information of the visible light image; Acquire a first target sub-image corresponding to the first hand region information in the second depth image; The first target sub-image is input into the hand joint point detection model to obtain the joint point information corresponding to the first hand area information, and the joint point information corresponding to the first hand area information is the joint point position information in the second depth image.

4. The hole correction method according to claim 2, wherein: The joint point information includes: Joint point information of finger joints, wrist joints, and forearm joints; Among them, the finger joints include: proximal finger joints, middle finger joints, distal finger joints, and fingertip joints.

5. The hole correction method according to claim 4, wherein: The determining of dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image comprises: For each frame of the second depth image, correcting the joint point information of the fingertip joint points in the joint point information in the second depth image; The second depth image sequence and the corrected joint point information in the second depth image are input into the dynamic gesture model to obtain the dynamic gesture category information.

6. The hole correction method according to claim 5, wherein: The correcting the joint point information of the fingertip joint points in the joint point information in the second depth image includes: Acquire a fingertip sub-image from a visible light image in the visible light image sequence whose position corresponds to the second depth image according to the joint point information of the fingertip joint point; Extracting contour information of the fingertip in the fingertip sub-image; The curvature of the points on the contour of the fingertip is calculated according to the contour information of the fingertip, and the position information of the convex point with the largest curvature value is used as the corrected joint point information of the fingertip joint point.

7. The hole correction method according to claim 2, after respectively acquiring the joint point position information in each frame of the second depth image sequence, the method further comprises: Dynamic gesture category information is determined according to the second depth image sequence and joint point information in the second depth image.

8. The hole correction method according to claim 7, wherein: The determining of dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image comprises: For each frame of the second depth image in the second depth image sequence, performing kinematic constraint processing on the joint point information in the second depth image; The joint point information after kinematic constraint processing in the second depth image sequence and the second depth image is input into the dynamic gesture model to obtain the dynamic gesture category information.

9. The hole correction method according to claim 8, wherein: The kinematic constraint processing of the joint point information in the second depth image comprises at least one of the following: Correcting the joint point information of the two adjacent joint points according to a value range of a first ratio of a length between two adjacent joint points to the hand region in which they are located and a first ratio in the second depth image of N frames before the second depth image; Correcting the joint point information of the other joint points except the wrist joint point according to the value range of the offset angle of the other joint points relative to the wrist joint point and the offset angle in the second depth image of N frames before the second depth image; Correcting the joint point information of the fingertip joint point and the wrist joint point according to the value range of the finger length and the finger length in the second depth image of N frames before the second depth image; wherein the finger length is the straight-line distance between the fingertip joint point and the wrist joint point; The joint point information of the wrist joint point and the proximal joint point of the finger is corrected according to the angle information of the forearm joint point and the hand and the angle information in the second depth image of N frames before the second depth image; wherein the angle information of the forearm joint point and the hand is the angle information of the angle between the straight line determined by the forearm joint point and the wrist joint point and the straight line determined by the wrist joint point and the proximal joint point of the finger; Correcting the joint point information of the two finger joint points closest to the finger joint point according to the value range of the distance between the finger joint point and the two finger joint points closest to the finger joint point and the distance between the finger joint point and the two finger joint points closest to the finger joint point in the N frames of second depth images before the second depth image; The joint point information of the finger joint points is corrected according to the value range of the bending degree of the finger relative to the palm and the back of the hand and the bending degree in the second depth image of N frames before the second depth image.

10. The hole correction method according to claim 7, after determining the dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image, the method further comprises: Acquiring original data of hand contours in visible light images in the visible light image sequence; Preprocessing the original hand contour data to obtain intermediate hand contour data with continuous contour; Generating a G2 continuous hand contour smooth curve according to the hand contour intermediate data; Filling and rendering are performed according to the smooth curve of the hand outline.

11. The hole correction method according to any one of claims 1 to 10, wherein: When the dynamic gesture category information is obtained, performing an operation corresponding to the dynamic gesture category information; When the dynamic gesture category information cannot be obtained, the head ray direction information is obtained, and an operation corresponding to the head ray direction information is performed.

12. The hole correction method according to claim 11, wherein: The operations include: interactive operations with exhibits in the museum.

13. The hole correction method according to any one of claims 1 to 10, wherein: The determining whether the first depth image is a depth image with a hole according to the first depth image and the visible light image in the visible light image sequence that meets a preset condition comprises: Determine a binarized image of the visible light image and a binarized image of the first depth image that meet a preset condition; Determine a difference between the binarized image of the visible light image and the binarized image of the first depth image; When the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; When the difference is greater than the first preset threshold, it is determined that the first depth image is a depth image with a hole.

14. The hole correction method according to any one of claims 1 to 10, wherein: The determining the hole position information in the first depth image comprises: Determining a binarized image of the first depth image; The hole position information is determined to be position information of pixels whose pixel values ​​are 0 in the binary image.

15. The hole correction method according to any one of claims 1 to 10, wherein: The correcting the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image includes: Acquire joint point position information in the first depth image; Determine, according to the hole position information and the joint point position information, whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information; In the case where the hole is located in the preset area corresponding to the joint point, determine the depth information after the hole correction as the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point; When the hole is located outside the preset area corresponding to the joint point, the depth information of the hole after correction is determined according to the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

16. An electronic device comprising: at least one processor; A memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, the hole correction method according to any one of claims 1 to 15 is implemented.

17. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hole correction method according to any one of claims 1 to 15.

Citation Information

Cited By

  • Void correction method, electronic device, and computer-readable medium

    EP4697266A1