Void correction method, electronic device, and computer-readable medium

By correcting the hollow information in the depth image, the joint detection accuracy in gesture recognition technology is improved, the problem of inaccurate gesture recognition in the prior art is solved, and the user experience is improved.

WO2024230407A1PCT designated stage expired Publication Date: 2024-11-14ZTE CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/087484
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-06
Filing Date
2024-04-12
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In the existing gesture recognition technology, abnormal detection of hand joint nodes results in inaccurate interaction location and poor user experience.

Method used

By acquiring visible light image sequences and depth image sequences with the same frame frequency, determine whether there are holes in the depth image, and correct the hole location based on the depth information of the previous frame to improve the accuracy of the depth image, thereby improving the accuracy of the node detection and gesture recognition.

Benefits of technology

Improves the accuracy and user experience of gesture recognition, ensuring the accuracy of interactive location.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024087484_14112024_PF_FP_ABST
    Figure CN2024087484_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present application are a void correction method, an electronic device, and a computer-readable medium. The void correction method comprises: acquiring a visible image sequence and a first depth image sequence having the same acquisition frame rate; for each first depth image in the first depth image sequence, on the basis of the first depth image and a visible image in the visible image sequence meeting a preset condition, determining whether the first depth image is a depth image having a void, wherein the visible image meeting the preset condition is a visible image of which the position corresponds to that of the first depth image; when it is determined that the first depth image is a depth image having a void, determining void position information of the first depth image; and, on the basis of N first depth images ahead of the first depth image, correcting the depth information of the void corresponding to the void position information, N being 1 or 2.
Need to check novelty before this filing date? Find Prior Art

Description

Void correction method, electronic device, and computer-readable medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to patent application No. 202310507857.1 filed with the China Patent Office on May 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to, but is not limited to, the field of image processing technology. Background Art

[0004] In some scenarios, gesture interaction with objects is necessary, such as viewing artifacts in a museum. However, a major problem with current gesture recognition is abnormal detection of hand joints, which leads to inaccurate interaction positions and a poor user experience.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a hole correction method, an electronic device, and a computer-readable medium.

[0007] In a first aspect, an embodiment of the present application provides a hole correction method, comprising: acquiring a visible light image sequence and a first depth image sequence with the same acquisition frame frequency; for each first depth image frame in the first depth image sequence, determining whether the first depth image is a depth image with a hole based on the first depth image and the visible light image in the visible light image sequence that meets preset conditions; wherein the visible light image that meets the preset conditions is a visible light image with a position corresponding to the first depth image; when it is determined that the first depth image is a depth image with a hole, determining the hole position information in the first depth image; and correcting the depth information of the hole corresponding to the hole position information based on the N frames of first depth images located before the first depth image; wherein N is 1 or 2.

[0008] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; a memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, any one of the hole correction methods described herein is implemented.

[0009] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, any one of the hole correction methods described herein is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 is a flowchart of a gesture recognition method provided by one embodiment of the present application;

[0011] FIG2 is a flow chart of a human-computer interaction method provided by another embodiment of the present application;

[0012] FIG3 is a flow chart of a hand joint detection method provided by another embodiment of the present application;

[0013] FIG4 is a flow chart of a hole correction method provided by another embodiment of the present application;

[0014] FIG5 is a flowchart of a hand region acquisition model training method provided by another embodiment of the present application;

[0015] FIG6 is a flow chart of a hand joint detection model training method provided by another embodiment of the present application;

[0016] FIG7 is a flow chart of a dynamic gesture model training method provided by another embodiment of the present application;

[0017] FIG8 is a block diagram of an electronic device according to another embodiment of the present application;

[0018] FIG9 is a schematic diagram of the finger joints in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To enable those skilled in the art to better understand the technical solution of the present application, the hole correction method, gesture recognition method, electronic device, and computer-readable medium provided in the present application are described in detail below with reference to the accompanying drawings.

[0020] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the scope of this application to those skilled in the art.

[0021] In the absence of conflict, the various embodiments of the present application and the various features therein may be combined with each other.

[0022] As used herein, the term "and / or" includes any and all combinations of at least one of the associated listed items.

[0023] The terms used herein are used only to describe specific embodiments and are not intended to limit this application. As used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements, and / or components is specified, but the presence or addition of at least one other feature, whole, step, operation, element, component, and / or group thereof is not excluded.

[0024] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0025] FIG1 is a flowchart of a gesture recognition method provided by one embodiment of the present application.

[0026] In a first aspect, referring to FIG. 1 , an embodiment of the present application provides a gesture recognition method, which may include steps 100 to 103 .

[0027] In step 100 , a visible light image sequence and a first depth image sequence having the same acquisition frame frequency are acquired.

[0028] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0029] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0030] In some exemplary embodiments, the visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the part of the visible light image frame sequence placed in the first queue is the visible light image sequence to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the part of the first depth image frame sequence placed in the second queue is the first depth image sequence to be acquired; the first queue and the second queue have the same size, both M; where M is an integer greater than or equal to 1.

[0031] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0032] In some exemplary embodiments, the visible light video stream and the first depth video stream may be obtained by correspondingly processing the originally acquired 3D data.

[0033] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0034] In some exemplary embodiments, visible light images in the visible light image frame sequence are first pre-processed and then placed in a first queue, and first depth images in the first depth image frame sequence are second pre-processed and then placed in a second queue.

[0035] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0036] In some exemplary embodiments, the second pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0037] In some exemplary embodiments, the visible light images obtained after performing the first preprocessing on the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0038] In some exemplary embodiments, the first depth image obtained after performing the second preprocessing on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0039] In step 101, for each first depth image in a first depth image sequence, it is determined whether the first depth image is a depth image with a hole based on the first depth image and a visible light image in a visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, the hole position information in the first depth image is determined; and the depth information of the hole corresponding to the hole position information is corrected based on the first depth images N frames before the first depth image; wherein N is 1 or 2.

[0040] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information based on the N frames of first depth images located before the first depth image, the step of determining whether the first depth image is a depth image with a hole based on the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence is continued for each frame of the first depth image in the first depth image sequence until there are no depth images with holes in the first depth image sequence.

[0041] In some exemplary embodiments, determining whether the first depth image is a depth image with holes based on the first depth image and the visible light image whose position in the visible light image sequence corresponds to the first depth image includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining a difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with holes.

[0042] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference can be calculated by other means.

[0043] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0044] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames before the first depth image includes: obtaining the joint position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint position information based on the hole position information and the joint position information; when the hole is located in the preset area corresponding to the joint point, determining the depth information after the hole correction as a weighted average of the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the depth information after the hole correction based on the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0045] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand area acquisition model to obtain fifth hand area information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand area information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand area information.

[0046] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0047] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0048] In some exemplary embodiments, determining the depth information after hole correction as the base filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refi is the weight of the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the j-th prior information of the joint point, D avgj is the j-th prior information of the joint point.

[0049] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method) of OpenCV.

[0050] In some exemplary embodiments, the prior information of the joint point includes: prior information of the length and prior information of the angle of the joint point.

[0051] In some exemplary embodiments, the length prior information of the joint point is the average value of the length information of the joint point closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; where B is an integer greater than or equal to 1.

[0052] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0053] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0054] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0055] In some exemplary embodiments, the offset angle information of the joint closest to the hole relative to the wrist joint refers to the angle information between a straight line between the joint closest to the hole and the proximal finger joint, and a straight line between the proximal finger joint and the wrist joint. The joint closest to the hole and the proximal finger joint belong to the same finger.

[0056] In some exemplary embodiments, when calculating the prior information of the joint points, the B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or the B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0057] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0058] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0059] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0060] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained by the previous gesture recognition and the dynamic gesture category information obtained by the previous gesture recognition can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition.

[0061] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0062] In some exemplary embodiments, determining the depth information of the hole after correction based on the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: determining the depth information of the hole corresponding to the hole position information in the first depth image based on the depth information of the two joint points closest to the hole in the first depth image; determining the depth information of the hole after correction as a weighted average of the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0063] In some exemplary embodiments, the depth information of the two joint points closest to the hole in the first depth image can be interpolated to obtain the depth information of the hole corresponding to the hole position information in the first depth image; or the depth information of the two joint points closest to the hole in the first depth image can be calculated using a region growing algorithm to obtain the depth information of the hole corresponding to the hole position information in the first depth image.

[0064] In step 102 , joint point position information in each second depth image frame in a second depth image sequence is obtained respectively; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

[0065] In some exemplary embodiments, respectively obtaining the joint point position information in each frame of the second depth image sequence includes: for each frame of the second depth image, inputting the visible light image corresponding to the position of the second depth image in the visible light image sequence into the hand area acquisition model to obtain the first hand area information of the visible light image; obtaining the first target sub-image corresponding to the first hand area information in the second depth image; inputting the first target sub-image into the hand joint point detection model to obtain the joint point information corresponding to the first hand area information, and the joint point information corresponding to the first hand area information is the joint point position information in the second depth image.

[0066] In some exemplary embodiments, the position index of the visible light image corresponding to the second depth image in the visible light image sequence is the same as the position index of the second depth image in the second depth image sequence.

[0067] In some exemplary embodiments, the first hand region information refers to information of an image region including a hand and part or all of a forearm.

[0068] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0069] In some exemplary embodiments, the first hand region information may be coordinate information of four corners of the image region.

[0070] In some exemplary embodiments, the first hand region information may be coordinate information of one corner of the image region, width information, and length information of the image region.

[0071] In some exemplary embodiments, the visible light image at a position corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to further obtain at least one of palm and back category information and crowd category information.

[0072] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image region is the palm or the back of the hand.

[0073] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image region belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0074] In some exemplary embodiments, the visible light image corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain the confidence level corresponding to the first hand region information. The first hand region information with a confidence level greater than a second preset threshold is selected for subsequent processing, i.e., the step of obtaining the first target sub-image corresponding to the first hand region information in the second depth image is continued. Here, first hand region information with a confidence level greater than the second preset threshold is considered valid first hand region information; first hand region information with a confidence level less than the second preset threshold is considered invalid first hand region information. Screening out valid first hand region information using the confidence level narrows the scope for subsequent acquisition of joint point information, thereby improving the accuracy of subsequent gesture recognition.

[0075] In some exemplary embodiments, the first target sub-image corresponding to the first hand region information in the second depth image refers to the position of the first target sub-image in the second depth image, and the position of the image region corresponding to the first hand region information in the visible light image is the same.

[0076] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0077] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0078] In step 103 , dynamic gesture category information is determined according to the second depth image sequence and the joint point information in the second depth image.

[0079] In some exemplary embodiments, the gesture category information may be at least one of finger forward click, finger pinch, rotation, zoom, move, static gesture, etc.

[0080] In some exemplary embodiments, determining dynamic gesture category information based on joint point information in the second depth image sequence and the second depth image includes: inputting the joint point information in the second depth image sequence and the second depth image into a dynamic gesture model to obtain dynamic gesture category information.

[0081] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the fingertip joints in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain dynamic gesture category information.

[0082] In some exemplary embodiments, correcting the joint point information of the fingertip joint points in the joint point information in the second depth image includes: obtaining a fingertip sub-image from a visible light image in the visible light image sequence whose position corresponds to the second depth image based on the joint point information of the fingertip joint points; extracting the contour information of the fingertip in the fingertip sub-image; calculating the curvature of the points on the contour of the fingertip based on the contour information of the fingertip, and taking the position information of the convex point with the largest curvature value as the corrected joint point information of the fingertip joint points.

[0083] In some exemplary embodiments, the fingertip sub-image refers to a sub-image including a fingertip.

[0084] In some exemplary embodiments, in order to correct the joint point information of each fingertip joint, it is necessary to obtain the fingertip sub-image of each fingertip separately and extract the contour information of the fingertip in the fingertip sub-image; calculate the curvature of the points on the contour of the fingertip based on the contour information of the fingertip, and use the position information of the convex point with the largest curvature value as the corrected joint point information of the fingertip joint.

[0085] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the finger joints in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain dynamic gesture category information.

[0086] In some exemplary embodiments, correcting the joint point information of the finger joint points in the joint point information in the second depth image includes: obtaining a finger joint sub-image from a visible light image in the visible light image sequence whose position corresponds to the second depth image based on the joint point information of the finger joint points; extracting the contour information of the finger joints in the finger joint sub-image; calculating the curvature of the points on the contour of the finger joints based on the contour information of the finger joints, and taking the position information of the concave point with the largest curvature value as the corrected joint point information of the finger joint points.

[0087] In some exemplary embodiments, the finger gap sub-image refers to a sub-image including a finger gap.

[0088] In some exemplary embodiments, in order to correct the joint point information of each finger joint, it is necessary to obtain the fingertip sub-image of each finger joint respectively and extract the contour information of the fingertip in the finger joint sub-image; calculate the curvature of the points on the contour of the finger joint based on the contour information of the finger joint, and use the position information of the concave point with the largest curvature value as the corrected joint point information of the finger joint.

[0089] In some exemplary embodiments, determining dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image includes: for each frame of the second depth image, correcting the joint point information of the fingertip joints and the joint point information of the finger joints in the joint point information in the second depth image; and inputting the corrected joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain dynamic gesture category information.

[0090] In some exemplary embodiments, determining dynamic gesture category information based on the second depth image sequence and the joint point information in the second depth image includes: for each frame of the second depth image, performing kinematic constraint processing on the joint point information in the second depth image; and inputting the kinematic constraint-processed joint point information in the second depth image sequence and the second depth image into the dynamic gesture model to obtain dynamic gesture category information.

[0091] In some exemplary embodiments, performing kinematic constraint processing on the joint point information in the second depth image includes at least one of the following:

[0092] Correcting the joint point information of two adjacent joint points according to a value range of a first ratio of a length between the two adjacent joint points to the hand region in which they are located and a first ratio in the second depth image N frames before the second depth image;

[0093] Correcting the joint point information of the other joint points according to the value range of the offset angles of the other joint points relative to the wrist joint point and the offset angles in the second depth image N frames before the second depth image;

[0094] Correcting the joint point information of the fingertip joint and the wrist joint according to the value range of the finger length and the finger length in the second depth image N frames before the second depth image; wherein the finger length is the straight-line distance between the fingertip joint and the wrist joint;

[0095] Correcting the joint point information of the wrist joint and the proximal finger joints based on the angle information of the forearm joint and the hand and the angle information in the second depth image N frames before the second depth image; wherein the angle information of the forearm joint and the hand is the angle information between the straight line defined by the forearm joint and the wrist joint and the straight line defined by the wrist joint and the proximal finger joints;

[0096] Correcting the joint point information of the two finger joints closest to the interdigital joint point according to a value range of the distance between the interdigital joint point (as shown in FIG9 ) and the two finger joints closest to the interdigital joint point and the distance between the interdigital joint point and the two finger joints closest to the interdigital joint point in the second depth image N frames before the second depth image;

[0097] The joint point information of the finger joints is corrected according to a value range of the bending degree of the finger relative to the palm and the back of the hand and the bending degree in the second depth image N frames before the second depth image.

[0098] In some exemplary embodiments, the hand region refers to a hand region corresponding to hand region information obtained by inputting a visible light image into a hand region acquisition model.

[0099] In some exemplary embodiments, the first ratio of the length between two adjacent joint points to the hand region in which they are located is the first ratio of the length between two adjacent joint points to the length of the hand region in which they are located.

[0100] In some exemplary embodiments, the first ratio of the length between two adjacent joint points to the hand region in which they are located is the first ratio of the length between two adjacent joint points to the width of the hand region in which they are located.

[0101] In some exemplary embodiments, the value range of the first ratio can be determined based on the maximum and minimum values ​​of the first ratios corresponding to all historical visible light images and historical depth images in C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category; where C is an integer greater than or equal to 1.

[0102] In some exemplary embodiments, when calculating the value range of the first ratio, the C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or the C historical visible light image sequences and C historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0103] In some exemplary embodiments, when calculating the value range of the first ratio, C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the same population category can be used to calculate the value range of the first ratio corresponding to the population category.

[0104] In some exemplary embodiments, the value range of the first ratio corresponding to the crowd category can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0105] In some exemplary embodiments, the value range of the first ratio corresponding to a default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the crowd category.

[0106] In some exemplary embodiments, the value range of the first ratio corresponding to the crowd category information obtained by the previous gesture recognition can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition.

[0107] In some exemplary embodiments, the value range of the first ratio corresponding to a default crowd category (such as adults) may be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to a default dynamic gesture category of the default crowd category.

[0108] In some exemplary embodiments, correcting the joint point information of two adjacent joint points based on a value range of a first ratio of a length between the two adjacent joint points to the hand region in which they are located and a first ratio in a second depth image N frames prior to the second depth image includes:

[0109] When the first ratio corresponding to two adjacent joint points in the second depth image is greater than the maximum value in the value range of the first ratio, and the first intermediate value between the first ratio corresponding to the two adjacent joint points in the second depth image and the first ratio in the second depth image of N frames before the second depth image is greater than the maximum value in the value range of the first ratio, the maximum value in the value range of the first ratio is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the maximum value in the value range of the first ratio.

[0110] When the first ratio corresponding to two adjacent joint points in the second depth image is greater than the maximum value in the value range of the first ratio, and the first intermediate value between the first ratio corresponding to the two adjacent joint points in the second depth image and the first ratio in the second depth image N frames before the second depth image is less than the maximum value in the value range of the first ratio, the first intermediate value is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the first intermediate value.

[0111] When the first ratio corresponding to two adjacent joint points in the second depth image is less than the minimum value in the value range of the first ratio, and the second intermediate value between the first ratio corresponding to the two adjacent joint points in the second depth image and the first ratio in the second depth image of N frames before the second depth image is less than the minimum value in the value range of the first ratio, the minimum value in the value range of the first ratio is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the minimum value in the value range of the first ratio.

[0112] When a first ratio corresponding to two adjacent joint points in the second depth image is less than a minimum value in a value range of the first ratio, and a second intermediate value between the first ratio corresponding to the two adjacent joint points in the second depth image and the first ratio in the second depth image N frames before the second depth image is greater than the minimum value in the value range of the first ratio, the second intermediate value is used as the first ratio corresponding to the two adjacent joint points in the second depth image, and the joint point information of the two adjacent joint points is corrected according to the second intermediate value.

[0113] In some exemplary embodiments, the offset angle of joints other than the wrist joint relative to the wrist joint refers to the angle between a straight line from the other joint to the proximal finger joint and a straight line from the proximal finger joint to the wrist joint, where the other joint and the proximal finger joint belong to the same finger.

[0114] In some exemplary embodiments, the range of values ​​of the offset angle can be determined based on the maximum and minimum values ​​of the offset angles corresponding to all historical visible light images and historical depth images in C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category; where C is an integer greater than or equal to 1.

[0115] In some exemplary embodiments, when calculating the value range of the offset angle, the C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition can be used for calculation, or the C historical visible light image sequences and C historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0116] In some exemplary embodiments, when calculating the value range of the offset angle, C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the value range of the first ratio corresponding to the crowd category.

[0117] In some exemplary embodiments, the value range of the offset angle corresponding to the crowd category can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0118] In some exemplary embodiments, the value range of the offset angle corresponding to the default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the same type of dynamic gesture category.

[0119] In some exemplary embodiments, the value range of the offset angle corresponding to the crowd category information obtained by the previous gesture recognition and the dynamic gesture category information obtained by the previous gesture recognition can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition.

[0120] In some exemplary embodiments, the value range of the offset angle corresponding to the default crowd category (such as adults, etc.) can be calculated using C historical visible light image sequences and C historical depth image sequences corresponding to the default dynamic gesture category of the default crowd category.

[0121] In some exemplary embodiments, correcting the joint point information of the other joint points except the wrist joint point according to the value range of the offset angle of the other joint points relative to the wrist joint point and the offset angle in the second depth image N frames before the second depth image includes:

[0122] When the offset angle in the second depth image is greater than the maximum value in the offset angle value range, and the third intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is greater than the maximum value in the offset angle value range, the maximum value in the offset angle value range is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the maximum value in the offset angle value range.

[0123] When the offset angle in the second depth image is greater than the maximum value in the range of values ​​of the offset angle, and a fourth intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is less than the maximum value in the range of values ​​of the offset angle, the fourth intermediate value is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the fourth intermediate value.

[0124] When the offset angle in the second depth image is less than the minimum value in the offset angle value range, and the third intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is less than the minimum value in the offset angle value range, the minimum value in the offset angle value range is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the minimum value in the offset angle value range.

[0125] When the offset angle in the second depth image is less than the minimum value in the value range of the offset angle, and a fourth intermediate value between the offset angle in the second depth image and the offset angle in the second depth image of N frames before the second depth image is greater than the minimum value in the value range of the offset angle, the fourth intermediate value is used as the offset angle in the second depth image, and the joint point information of other joint points is corrected according to the fourth intermediate value.

[0126] In some exemplary embodiments, the finger length refers to the straight-line distance between the fingertip joint point and the wrist joint point.

[0127] In some exemplary embodiments, the process of determining the value range of finger length and correcting the joint point information of the fingertip joints and wrist joints is similar to the process of determining the value range of the aforementioned first ratio and correcting the joint point information of two adjacent joints, and will not be repeated here.

[0128] In some exemplary embodiments, the process of determining the value range of the angle information of the forearm joint and the hand, and correcting the joint point information of the wrist joint and the proximal finger joint is similar to the process of determining the value range of the aforementioned first ratio and correcting the joint point information of two adjacent joints, and will not be repeated here.

[0129] In some exemplary embodiments, the value range of the angle information of the forearm joint and the hand can also be determined using a spherical simulation method. For example, the distance between the forearm joint and the wrist joint is used as the diameter of the first sphere, and the distance between the wrist joint and the proximal finger joint is used as the diameter of the second sphere. The value range of the angle information of the forearm joint and the hand is determined by the intersection range of the first sphere and the second sphere.

[0130] In some exemplary embodiments, the process of determining the value range of the distance between the interdigital joint point and the two finger joint points closest to the interdigital joint point, and correcting the joint point information of the two finger joint points closest to the interdigital joint point are similar to the process of determining the value range of the aforementioned first ratio and correcting the joint point information of two adjacent joint points, and will not be repeated here.

[0131] In some exemplary embodiments, the degree of bending of the finger relative to the palm and back of the hand refers to the angle between the straight line between the proximal finger joint and the middle finger joint, and the straight line between the proximal finger joint and the wrist joint.

[0132] In some exemplary embodiments, the process of determining the value range of the degree of bending of the finger relative to the palm and the back of the hand, and correcting the joint point information of the finger joint points is similar to the process of determining the value range of the aforementioned first ratio and correcting the joint point information of two adjacent joint points, and will not be repeated here.

[0133] In some exemplary embodiments, after acquiring a visible light image sequence and a first depth image sequence with the same acquisition frame frequency, and before determining the hole position information in the first depth image, the method further includes: determining the number of depth images with holes in the first depth image sequence; when the number of depth images with holes in the first depth image sequence is greater than or equal to a third preset threshold, performing data stream cleaning processing on the visible light image sequence and the first depth image sequence, and continuing to execute the step of determining the number of depth images with holes in the first depth image sequence after the data stream cleaning processing until the number of depth images with holes in the first depth image sequence after the data stream cleaning processing is less than the third preset threshold.

[0134] In some exemplary embodiments, when the number of depth images containing holes in the first depth image sequence is less than a third preset threshold, the step of determining the hole position information in the first depth image is continued.

[0135] In some exemplary embodiments, data stream cleaning processing of the visible light image sequence and the first depth image sequence includes: deleting the A-frame visible light image with the earliest acquisition time in the visible light image sequence, obtaining the A-frame visible light image with the earliest acquisition time after the visible light image sequence, and adding it to the visible light image sequence; wherein A is an integer greater than or equal to 1; deleting the A-frame first depth image with the earliest acquisition time in the first depth image sequence, obtaining the A-frame first depth image with the earliest acquisition time after the first depth image sequence, and adding it to the first depth image sequence.

[0136] In some exemplary embodiments, after determining the dynamic gesture category information based on the joint point information in the second depth image sequence and the second depth image, the method further includes: obtaining original hand contour data in the visible light image in the visible light image sequence; preprocessing the original hand contour data to obtain intermediate hand contour data with continuous contour; generating a G2 continuous hand contour smooth curve based on the intermediate hand contour data; and performing filling rendering based on the hand contour smooth curve.

[0137] In some exemplary embodiments, the OpenCV method may be used to obtain the original hand contour data.

[0138] In some exemplary embodiments, the pre-processing includes at least one of algorithms such as denoising, filtering, and region growing.

[0139] In some exemplary embodiments, a computer graphics method may be used to generate a G2 continuous smooth curve of the hand contour.

[0140] In some exemplary embodiments, filling and rendering according to the smooth curve of the hand contour includes: transforming the original collected 3D data into 2D data corresponding to the 2D coordinate system; mapping the parameter information of the 3D data in space to the 2D plane; using a 2D renderer to perform pixel-level rendering on the inside of the hand contour according to the smooth curve of the hand contour to obtain a 2D mesh image.

[0141] In some exemplary embodiments, the parameter information may be position, direction, size, etc.

[0142] In some exemplary embodiments, when mapping parameter information of 3D data in space to a 2D plane, depth information is not considered, which greatly reduces the amount of calculation.

[0143] In some exemplary embodiments, when a 2D renderer is used to perform pixel-level rendering of the inside of the hand contour according to the smooth curve of the hand contour to obtain a 2D mesh image, only the inside of the hand contour is rendered, and only pixel-level rendering is performed, and the number of vertices of the triangular mesh of the mesh is reduced, thereby reducing the amount of calculation.

[0144] In some exemplary embodiments, after determining the dynamic gesture category information based on the second depth image sequence and the joint point information in the second depth image, the method further includes: performing data stream cleaning processing on the visible light image sequence and the first depth image sequence.

[0145] In some exemplary embodiments, after determining the dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image, the method further includes: performing an operation corresponding to the gesture category information.

[0146] In some exemplary embodiments, the operation corresponding to the gesture category information is, for example, opening or closing a corresponding window.

[0147] The gesture recognition method provided in the embodiment of the present application corrects the depth information of the holes in the depth image before obtaining the joint point position information, thereby improving the accuracy of the joint point position information and thus improving the accuracy of gesture recognition.

[0148] In some exemplary embodiments, kinematic constraint processing is performed on joint point information before determining the dynamic gesture category, thereby improving the accuracy of joint point detection and further improving the accuracy of gesture recognition.

[0149] In some exemplary embodiments, corresponding kinematic constraint processing is performed on the joint point information of people of different age groups, such as the elderly, adults, and children, to further improve the accuracy of joint point detection.

[0150] In some exemplary embodiments, after the hand contour information is extracted, the curve G2 corresponding to the hand contour information is made continuous for hand rendering, which greatly reduces the computational complexity of the entire rendering process and makes the entire gesture rendering process efficient and accurate.

[0151] In some exemplary embodiments, the method of obtaining the joint point information of the fingertip joints by obtaining the curvature of a point on the outline of the fingertip based on the outline information of the fingertip improves the accuracy of the joint point information of the fingertip joints.

[0152] FIG2 is a flow chart of a human-computer interaction method provided by another embodiment of the present application.

[0153] In a second aspect, referring to Figure 2, another embodiment of the present application provides a human-computer interaction method, including: step 200, obtaining a visible light image sequence and a first depth image sequence with the same acquisition frame frequency; performing gesture recognition based on the visible light image sequence and the first depth image sequence to obtain dynamic gesture category information, and executing an operation corresponding to the dynamic gesture category information; or obtaining head ray direction information, and executing an operation corresponding to the head ray direction information.

[0154] In some exemplary embodiments, when dynamic gesture category information is obtained by performing gesture recognition based on the visible light image sequence and the first depth image sequence, an operation corresponding to the dynamic gesture category information is performed; when dynamic gesture category information cannot be obtained by performing gesture recognition based on the visible light image sequence and the first depth image sequence, head ray direction information is obtained and an operation corresponding to the head ray direction information is performed.

[0155] In some exemplary embodiments, the gesture recognition method of the aforementioned embodiment may be used to perform gesture recognition, which will not be described in detail here.

[0156] In some exemplary embodiments, a positioning device such as an inertial measurement unit (IMU) may be used to obtain head ray direction information.

[0157] In some exemplary embodiments, the head ray direction information may be direction information of rays emitted by a sensor located on the head.

[0158] In some exemplary embodiments, the operation corresponding to the dynamic gesture category information or the operation corresponding to the head ray direction information may be an interactive operation with exhibits in a museum, as described in Application Example 1 and Application Example 2 below.

[0159] In some exemplary embodiments, the operation corresponding to the dynamic gesture category information or the operation corresponding to the head ray direction information may be a click operation, a double-click operation, a sliding operation, etc.

[0160] FIG3 is a flow chart of a hand joint detection method provided in another embodiment of the present application.

[0161] In a third aspect, referring to FIG. 3 , another embodiment of the present application provides a hand joint detection method, which may include steps 300 to 302 .

[0162] In step 300 , a visible light image sequence and a first depth image sequence having the same acquisition frame frequency are acquired.

[0163] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0164] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0165] In some exemplary embodiments, the visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the part of the visible light image frame sequence placed in the first queue is the visible light image sequence to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the part of the first depth image frame sequence placed in the second queue is the first depth image sequence to be acquired; the first queue and the second queue have the same size, both M; where M is an integer greater than or equal to 1.

[0166] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0167] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0168] In some exemplary embodiments, visible light images in the visible light image frame sequence are first pre-processed and then placed in a first queue, and first depth images in the first depth image frame sequence are second pre-processed and then placed in a second queue.

[0169] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0170] In some exemplary embodiments, the second pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0171] In some exemplary embodiments, the visible light images obtained after performing the first preprocessing on the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0172] In some exemplary embodiments, the first depth image obtained after performing the second preprocessing on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0173] In step 301, for each first depth image frame in the first depth image sequence, it is determined whether the first depth image is a depth image with a hole based on the first depth image and a visible light image in the visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image; when it is determined that the first depth image is a depth image with a hole, the hole position information in the first depth image is determined; and the depth information of the hole corresponding to the hole position information is corrected based on the N frames of first depth images located before the first depth image; wherein N is 1 or 2.

[0174] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information based on the N frames of first depth images located before the first depth image, the step of determining whether the first depth image is a depth image with a hole based on the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence is continued for each frame of the first depth image in the first depth image sequence until there are no depth images with holes in the first depth image sequence.

[0175] In some exemplary embodiments, determining whether the first depth image is a depth image with holes based on the first depth image and the visible light image whose position in the visible light image sequence corresponds to the first depth image includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining a difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with holes.

[0176] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference can be calculated by other means.

[0177] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0178] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames before the first depth image includes: obtaining the joint position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint position information based on the hole position information and the joint position information; in a case where the hole is located in the preset area corresponding to the joint point, determining the depth information after the hole correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point; in a case where the hole is located outside the preset area corresponding to the joint point, determining the depth information after the hole correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0179] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand area acquisition model to obtain fifth hand area information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand area information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand area information.

[0180] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0181] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0182] In some exemplary embodiments, determining the depth information after hole correction as the base filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refiis the weight of the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the j-th prior information of the joint point, D avgj is the j-th prior information of the joint point.

[0183] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method) of OpenCV.

[0184] In some exemplary embodiments, the prior information of the joint point includes: prior information of the length and prior information of the angle of the joint point.

[0185] In some exemplary embodiments, the length prior information of the joint point is the average value of the length information of the joint point closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; where B is an integer greater than or equal to 1.

[0186] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0187] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0188] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0189] In some exemplary embodiments, the offset angle information of the joint closest to the hole relative to the wrist joint refers to the angle information between a straight line between the joint closest to the hole and the proximal finger joint, and a straight line between the proximal finger joint and the wrist joint. The joint closest to the hole and the proximal finger joint belong to the same finger.

[0190] In some exemplary embodiments, when calculating the prior information of the joint points, the B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or the B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0191] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0192] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0193] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0194] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained by the previous gesture recognition and the dynamic gesture category information obtained by the previous gesture recognition can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition.

[0195] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0196] In some exemplary embodiments, determining the depth information of the hole after correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: performing an interpolation operation on the depth information of the two joint points closest to the hole in the first depth image to obtain the depth information of the hole corresponding to the hole position information in the first depth image; determining the depth information of the hole after correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the weighted average of the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0197] In step 302 , joint point position information in a second depth image in a second depth image sequence is obtained; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

[0198] In some exemplary embodiments, respectively obtaining the joint point position information in each frame of the second depth image sequence includes: for each frame of the second depth image, inputting the visible light image corresponding to the position of the second depth image in the visible light image sequence into the hand area acquisition model to obtain the first hand area information of the visible light image; obtaining the first target sub-image corresponding to the first hand area information in the second depth image; inputting the first target sub-image into the hand joint point detection model to obtain the joint point information corresponding to the first hand area information, and the joint point information corresponding to the first hand area information is the joint point position information in the second depth image.

[0199] In some exemplary embodiments, the position index of the visible light image corresponding to the second depth image in the visible light image sequence is the same as the position index of the second depth image in the second depth image sequence.

[0200] In some exemplary embodiments, the first hand region information refers to information of an image region including a hand and part or all of a forearm.

[0201] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0202] In some exemplary embodiments, the first hand region information may be coordinate information of four corners of the image region.

[0203] In some exemplary embodiments, the first hand region information may be coordinate information of one corner of the image region, width information, and length information of the image region.

[0204] In some exemplary embodiments, the visible light image at a position corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to further obtain at least one of palm and back category information and crowd category information.

[0205] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image region is the palm or the back of the hand.

[0206] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image region belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0207] In some exemplary embodiments, the visible light image corresponding to the second depth image in the visible light image sequence is input into the hand region acquisition model to obtain the confidence level corresponding to the first hand region information. The first hand region information with a confidence level greater than a second preset threshold is selected for subsequent processing, i.e., the step of obtaining the first target sub-image corresponding to the first hand region information in the second depth image is continued. Here, first hand region information with a confidence level greater than the second preset threshold is considered valid first hand region information; first hand region information with a confidence level less than the second preset threshold is considered invalid first hand region information. Screening out valid first hand region information using the confidence level narrows the scope for subsequent acquisition of joint point information, thereby improving the accuracy of subsequent gesture recognition.

[0208] In some exemplary embodiments, the first target sub-image corresponding to the first hand region information in the second depth image refers to the position of the first target sub-image in the second depth image, and the position of the image region corresponding to the first hand region information in the visible light image is the same.

[0209] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0210] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0211] The hand joint detection method provided in the embodiment of the present application corrects the depth information of the holes in the depth image before obtaining the joint position information, thereby improving the accuracy of the joint position information.

[0212] FIG4 is a flow chart of a hole correction method provided by another embodiment of the present application.

[0213] In a fourth aspect, referring to FIG. 4 , another embodiment of the present application provides a hole correction method, comprising steps 400 to 402 .

[0214] In step 400 , a visible light image sequence and a first depth image sequence having the same acquisition frame frequency are acquired.

[0215] In some exemplary embodiments, the visible light image sequence may be an RGB image sequence or a grayscale image sequence.

[0216] In some exemplary embodiments, the visible light image sequence and the first depth image sequence may be acquired from a visible light video stream and a first depth video stream having the same acquisition frame frequency.

[0217] In some exemplary embodiments, the visible light video stream can be decoded to obtain a visible light image frame sequence, and the visible light image frame sequence is placed in a first queue, and the part of the visible light image frame sequence placed in the first queue is the visible light image sequence to be acquired; the first depth video stream is decoded to obtain a first depth image frame sequence, and the first depth image frame sequence is placed in a second queue, and the part of the first depth image frame sequence placed in the second queue is the first depth image sequence to be acquired; the first queue and the second queue have the same size, both M; where M is an integer greater than or equal to 1.

[0218] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0219] In some exemplary embodiments, the visible light image sequence in the first queue and the first depth image sequence in the second queue are aligned, that is, the visible light image and the first depth image at the same position are acquired simultaneously.

[0220] In some exemplary embodiments, visible light images in the visible light image frame sequence are first pre-processed and then placed in a first queue, and first depth images in the first depth image frame sequence are second pre-processed and then placed in a second queue.

[0221] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0222] In some exemplary embodiments, the second pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0223] In some exemplary embodiments, the visible light images obtained after performing the first preprocessing on the visible light images in the visible light image frame sequence meet the input requirements of the hand region acquisition model.

[0224] In some exemplary embodiments, the first depth image obtained after performing the second preprocessing on the first depth image in the first depth image frame sequence meets the input requirement of the hand region acquisition model.

[0225] In step 401, for each first depth image in a first depth image sequence, it is determined whether the first depth image is a depth image with a hole based on the first depth image and a visible light image in a visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image.

[0226] In some exemplary embodiments, after correcting the depth information of the hole corresponding to the hole position information based on the N frames of first depth images located before the first depth image, the step of determining whether the first depth image is a depth image with a hole based on the first depth image and the visible light image corresponding to the position of the first depth image in the visible light image sequence is continued for each frame of the first depth image in the first depth image sequence until there are no depth images with holes in the first depth image sequence.

[0227] In some exemplary embodiments, determining whether the first depth image is a depth image with holes based on the first depth image and the visible light image whose position in the visible light image sequence corresponds to the first depth image includes: determining a binarized image of the visible light image and a binarized image of the first depth image; determining a difference between the binarized image of the visible light image and the binarized image of the first depth image; when the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first depth image is a depth image with holes.

[0228] In some exemplary embodiments, when determining the difference between the binarized image of the visible light image and the binarized image of the first depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the visible light image and the binarized image of the first depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference can be calculated by other means.

[0229] In step 402, when it is determined that the first depth image is a depth image with a hole, the hole position information in the first depth image is determined; and the depth information of the hole corresponding to the hole position information is corrected based on the N frames of first depth images located before the first depth image; wherein N is 1 or 2.

[0230] In some exemplary embodiments, determining the hole position information in the first depth image includes: determining a binarized image of the first depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0231] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first depth image of N frames before the first depth image includes: obtaining the joint position information in the first depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint position information based on the hole position information and the joint position information; in a case where the hole is located in the preset area corresponding to the joint point, determining the depth information after the hole correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point; in a case where the hole is located outside the preset area corresponding to the joint point, determining the depth information after the hole correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0232] In some exemplary embodiments, obtaining joint point position information in the first depth image includes: inputting a visible light image whose position in the visible light image sequence corresponds to the first depth image into a hand area acquisition model to obtain fifth hand area information of the visible light image; obtaining a fourth target sub-image corresponding to the fifth hand area information in the first depth image; and inputting the fourth target sub-image into a hand joint point detection model to obtain joint point information corresponding to the fifth hand area information.

[0233] In some exemplary embodiments, in addition to obtaining fifth hand region information of the visible light image, the visible light image corresponding to the first depth image in the visible light image sequence is input into the hand region acquisition model to obtain: at least one of palm and back of hand category information and crowd category information.

[0234] In some exemplary embodiments, determining whether the hole corresponding to the hole position information is located within the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information includes: calculating the distance between the hole corresponding to the hole position information and the joint point corresponding to the joint point position information, and when the distance is less than or equal to the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is within the preset area corresponding to the joint point; when the distance is greater than the radius of the inscribed circle or the circumscribed circle of the preset area, determining that the hole is outside the preset area corresponding to the joint point.

[0235] In some exemplary embodiments, determining the depth information after hole correction as the base filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point includes: according to the formula Determine the depth information after hole correction; where D is the depth information after hole correction, λ refiis the weight of the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, D refi is the depth information of the hole corresponding to the hole position information in the first depth image of the first i frames before the first depth image, λ basic is the weight of the basic filling value, D basic is the basic filling value, λ avgj is the weight of the j-th prior information of the joint point, D avgj is the j-th prior information of the joint point.

[0236] In some exemplary embodiments, the basic filling value may be obtained by using a linear hole filling method or a morphological operation method (such as a dilation operation method) of OpenCV.

[0237] In some exemplary embodiments, the prior information of the joint point includes: prior information of the length and prior information of the angle of the joint point.

[0238] In some exemplary embodiments, the length prior information of the joint point is the average value of the length information of the joint point closest to the hole corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category; where B is an integer greater than or equal to 1.

[0239] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the width information of the hand region corresponding to the hand region information in the visible light image.

[0240] In some exemplary embodiments, the length information of the joint point closest to the hole is the ratio of the length information between adjacent joint points to the length information of the hand region corresponding to the hand region information in the visible light image.

[0241] In some exemplary embodiments, the angle prior information of the joint point is the average of the offset angle information of the joint point closest to the hole relative to the wrist joint point, corresponding to all historical visible light images and historical depth images in B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category.

[0242] In some exemplary embodiments, the offset angle information of the joint closest to the hole relative to the wrist joint refers to the angle information between a straight line between the joint closest to the hole and the proximal finger joint, and a straight line between the proximal finger joint and the wrist joint. The joint closest to the hole and the proximal finger joint belong to the same finger.

[0243] In some exemplary embodiments, when calculating the prior information of the joint points, the B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained from the previous gesture recognition can be used for calculation, or the B historical visible light image sequences and B historical depth image sequences corresponding to the default dynamic gesture category (such as a click gesture, etc.) can be used for calculation.

[0244] In some exemplary embodiments, when calculating the prior information of joint points, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of the same crowd category can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0245] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the same dynamic gesture category of the crowd category information obtained in the previous gesture recognition.

[0246] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to the same type of dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0247] In some exemplary embodiments, the prior information of the joint points corresponding to the crowd category information obtained by the previous gesture recognition and the dynamic gesture category information obtained by the previous gesture recognition can be calculated using B historical visible light image sequences and B historical depth image sequences corresponding to the dynamic gesture category information obtained by the previous gesture recognition.

[0248] In some exemplary embodiments, B historical visible light image sequences and B historical depth image sequences corresponding to a default dynamic gesture category of a default crowd category (such as adults, etc.) can be used to calculate the prior information of the joint points corresponding to the crowd category.

[0249] In some exemplary embodiments, determining the depth information of the hole after correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image includes: performing an interpolation operation on the depth information of the two joint points closest to the hole in the first depth image to obtain the depth information of the hole corresponding to the hole position information in the first depth image; determining the depth information of the hole after correction as a basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image, and the weighted average of the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

[0250] The hand joint point detection method provided in the embodiment of the present application corrects the depth information of the holes in the depth image, improves the accuracy of the depth information of the holes, and lays the foundation for the subsequent extraction of joint point position information and gesture recognition.

[0251] FIG5 is a flowchart of a hand region acquisition model training method provided in another embodiment of the present application.

[0252] In the fifth aspect, referring to Figure 5, another embodiment of the present application provides a hand area acquisition model training method, including steps 500 to 502.

[0253] In step 500 , a first data sample for model training is obtained; wherein the first data sample includes: a first historical visible light image sequence.

[0254] In some exemplary embodiments, the first historical visible light image sequence may be a first historical RGB image sequence or a first historical grayscale image sequence.

[0255] In some exemplary embodiments, the first historical visible light image sequence may be acquired from a first historical visible light video stream.

[0256] In some exemplary embodiments, the first historical visible light video stream may be decoded to obtain a first historical visible light image frame sequence, and the first historical visible light image sequence may be acquired from the first historical visible light image frame sequence.

[0257] In some exemplary embodiments, after first preprocessing is performed on the first historical visible light images in the first historical visible light image frame sequence, the first historical visible light image sequence is acquired from the first preprocessed first historical visible light image frame sequence.

[0258] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0259] In some exemplary embodiments, the first historical visible light image obtained by performing a first preprocessing on the first historical visible light image in the first historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0260] In step 501, annotation information is added to the first data sample; wherein the annotation information includes: second hand region information.

[0261] In some exemplary embodiments, the second hand region information refers to information of a first image region including a hand and part or all of a forearm in a first historical visible light image in a first historical visible light image sequence, where the image region is the hand region.

[0262] In some exemplary embodiments, the hand includes fingers, a palm, and a wrist.

[0263] In some exemplary embodiments, the second hand region information may be coordinate information of four corners of the first image region.

[0264] In some exemplary embodiments, the second hand region information may be coordinate information of one corner of the first image region, width information, and length information of the image region.

[0265] In some exemplary embodiments, the annotation information may further include: at least one of palm and back category information and crowd category information.

[0266] In some exemplary embodiments, the palm / back category information is used to indicate whether the hand in the image region is the palm or the back of the hand.

[0267] In some exemplary embodiments, the crowd category information is used to indicate the crowd category to which the hand in the image region belongs. The crowd category may be, for example, the elderly, adults, children, etc.

[0268] In some exemplary embodiments, when adding annotation information to the first data sample, a target detection method may be used to detect the first image region from the first data sample to obtain the second hand region information, or the annotation information may be added manually.

[0269] In step 502, the first data sample is used as the input of the hand region acquisition model, and the annotation information is used as the output of the hand region acquisition model to perform model training to obtain the hand region acquisition model.

[0270] In some exemplary embodiments, after adding the annotation information to the first data sample and before performing the model training, the method further includes: performing a cross-cleaning process on the annotation information.

[0271] The first data sample is used as the input of the hand area acquisition model, and the annotation information is used as the output of the hand area acquisition model for model training to obtain the hand area acquisition model, which includes: the first data sample is used as the input of the hand area acquisition model, and the annotation information after cross-cleaning processing is used as the output of the hand area acquisition model for model training to obtain the hand area acquisition model.

[0272] In some exemplary embodiments, when adding annotation information to the first data sample, multiple people can add corresponding annotation information to the same first data sample, and the annotation information added by different people to the same first data sample can be cross-cleaned to obtain the final annotation information of the first data sample.

[0273] In some exemplary embodiments, after obtaining a first data sample for model training, data augmentation can be performed on the first data sample, and annotation information can be added to the augmented first data sample. The augmented first data sample is used as input to the hand region acquisition model, and the annotation information is used as output of the hand region acquisition model for model training to obtain the hand region acquisition model. The data augmentation method improves the generalization of the hand region acquisition model.

[0274] In some exemplary embodiments, a quantization operation may be performed on the model parameters in the trained hand region acquisition model to increase the speed of subsequently obtaining the first region information based on the hand region acquisition model.

[0275] FIG6 is a flowchart of a hand joint detection model training method provided in another embodiment of the present application.

[0276] In the sixth aspect, referring to Figure 6, another embodiment of the present application provides a hand joint point detection model training method, which may include steps 600 to 603.

[0277] In step 600 , a second data sample is obtained for model training; wherein the second data sample includes: a second historical visible light image sequence and a first historical depth image sequence having the same acquisition frame frequency.

[0278] In some exemplary embodiments, the second historical visible light image sequence may be a second historical RGB image sequence or a second historical grayscale image sequence.

[0279] In some exemplary embodiments, the second historical visible light image sequence and the first historical depth image sequence may be acquired from a second historical visible light video stream and a first historical depth video stream having the same acquisition frame frequency.

[0280] In some exemplary embodiments, the second historical visible light video stream can be decoded to obtain a second historical visible light image frame sequence, and the second historical visible light image sequence can be obtained from the second historical visible light image frame sequence; the first historical depth video stream can be decoded to obtain a first historical depth image frame sequence, and the first historical depth image sequence can be obtained from the first historical depth image frame sequence; the second historical visible light image sequence and the first historical depth image sequence have the same size, both M; where M is an integer greater than or equal to 1.

[0281] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0282] In some exemplary embodiments, the second historical visible light image sequence and the first historical depth image sequence are aligned, that is, the second historical visible light image and the first historical depth image at the same position are acquired simultaneously.

[0283] In some exemplary embodiments, the second historical visible light image in the second historical visible light image frame sequence is subjected to a first preprocessing and then to a subsequent processing, and the first historical depth image in the first historical depth image frame sequence is subjected to a second preprocessing and then to a subsequent processing.

[0284] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0285] In some exemplary embodiments, the second pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0286] In some exemplary embodiments, the second historical visible light image obtained by performing a first preprocessing on the second historical visible light image in the second historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0287] In some exemplary embodiments, the first historical depth image obtained after performing the second preprocessing on the first historical depth image in the first historical depth image frame sequence meets the input requirements of the hand region acquisition model.

[0288] In step 601 , a second historical visible light image in a second historical visible light image sequence is input into a hand region acquisition model to obtain third hand region information of the second historical visible light image.

[0289] In step 602, a second target sub-image corresponding to the third hand region information of the second historical visible light image is obtained from the first historical depth image in the first historical depth image sequence whose position corresponds to the second historical visible light image; and joint point information in the second target sub-image is obtained.

[0290] In some exemplary embodiments, the position index of the first historical depth image whose position in the first historical depth image sequence corresponds to the second historical visible light image in the first historical depth image sequence is the same as the position index of the second historical visible light image in the second historical visible light image sequence.

[0291] In some exemplary embodiments, in the first historical depth image whose position in the first historical depth image sequence corresponds to the second historical visible light image, the second target sub-image corresponding to the third hand region information of the second historical visible light image refers to the position of the second target sub-image in the first historical depth image sequence that corresponds to the second historical visible light image, and the image region corresponding to the third hand region information is the same as the position in the second historical visible light image.

[0292] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0293] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0294] In some exemplary embodiments, during the first model training, other methods known in the related art may be used to obtain the joint point information in the second target sub-image. Starting from the second model training, the joint point information in the second target sub-image may be obtained based on the hand joint point detection model obtained from the previous training.

[0295] In step 603, the first historical depth image in the first historical depth image sequence is used as the input of the hand joint detection model, and the joint information in the second target sub-image is used as the output of the hand joint detection model for model training to obtain the hand joint detection model.

[0296] In some exemplary embodiments, before performing model training or obtaining joint point information in the second target sub-image, the method further includes: for each frame of the first historical depth image in the first historical depth image sequence, determining whether the first historical depth image is a historical depth image with a hole based on the first historical depth image and the second historical visible light image in the second historical visible light image sequence whose position corresponds to the first historical depth image; in the case where it is determined that the first historical depth image is a historical depth image with a hole, determining the hole position information in the first historical depth image; and correcting the depth information of the hole corresponding to the hole position information based on the N frames of the first historical depth image located before the first historical depth image; wherein N is 1 or 2.

[0297] The first historical depth image after hole correction is used as the input of the hand joint detection model, and the joint information in the second target sub-image is used as the output of the hand joint detection model for model training to obtain the hand joint detection model.

[0298] In some exemplary embodiments, determining whether the first historical depth image is a historical depth image with a hole based on the second historical visible light image whose position corresponds to the first historical depth image in the sequence of the first historical depth image and the second historical visible light image includes: determining a binarized image of the second historical visible light image and a binarized image of the first historical depth image; determining the difference between the binarized image of the second historical visible light image and the binarized image of the first historical depth image; when the difference is less than a first preset threshold, determining that the first historical depth image is a historical depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the first historical depth image is a historical depth image with a hole.

[0299] In some exemplary embodiments, when determining the difference between the binarized image of the second historical visible light image and the binarized image of the first historical depth image, the sum of the absolute values ​​of the differences of all pixels of the binarized image of the second historical visible light image and the binarized image of the first historical depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences of all pixels can be determined; or, the average value of the absolute values ​​of the differences of all pixels can be determined; or, the difference calculated by other means can be determined.

[0300] In some exemplary embodiments, determining the hole position information in the first historical depth image includes: determining a binarized image of the first historical depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0301] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the first historical depth image N frames before the first historical depth image includes: obtaining the joint point position information in the first historical depth image; determining whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; when the hole is located in the preset area corresponding to the joint point, determining the depth information after the hole correction as a weighted average of the basic filling value, the depth information of the hole corresponding to the hole position information in the first historical depth image N frames before the first historical depth image, and the prior information of the joint point; when the hole is located outside the preset area corresponding to the joint point, determining the depth information after the hole correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the first historical depth image N frames before the first historical depth image.

[0302] In some exemplary embodiments, a second data sample is obtained for model training, and data enhancement may be performed on the second data sample. A hand joint detection model is then trained based on the enhanced second data sample to improve the generalization of the hand joint detection model.

[0303] In some exemplary embodiments, the model parameters in the trained hand joint detection model may be quantized to increase the speed of subsequently obtaining joint information based on the hand joint detection model.

[0304] FIG7 is a flowchart of a dynamic gesture model training method provided by another embodiment of the present application.

[0305] In the seventh aspect, referring to Figure 7, another embodiment of the present application provides a dynamic gesture model training method, including steps 700 to 704.

[0306] In step 700 , a third data sample is obtained for model training; wherein the third data sample includes: a third historical visible light image sequence and a second historical depth image sequence having the same acquisition frame frequency.

[0307] In some exemplary embodiments, the third historical visible light image sequence may be a third historical RGB image sequence or a third historical grayscale image sequence.

[0308] In some exemplary embodiments, the third historical visible light image sequence and the second historical depth image sequence may be acquired from a third historical visible light video stream and a second historical depth video stream having the same acquisition frame frequency.

[0309] In some exemplary embodiments, the third historical visible light video stream can be decoded to obtain a third historical visible light image frame sequence, and a third historical visible light image sequence can be obtained from the third historical visible light image frame sequence; the second historical depth video stream can be decoded to obtain a second historical depth image frame sequence, and a second historical depth image sequence can be obtained from the second historical depth image frame sequence; the third historical visible light image sequence and the second historical depth image sequence have the same size, both M; where M is an integer greater than or equal to 1.

[0310] In some exemplary embodiments, M is an integer greater than or equal to 8 and less than or equal to 16.

[0311] In some exemplary embodiments, the third historical visible light image sequence and the second historical depth image sequence are aligned, that is, the third historical visible light image and the second historical depth image at the same position are acquired simultaneously.

[0312] In some exemplary embodiments, the third historical visible light image in the third historical visible light image frame sequence is subjected to a first preprocessing and then to a subsequent processing, and the second historical depth image in the second historical depth image frame sequence is subjected to a second preprocessing and then to a subsequent processing.

[0313] In some exemplary embodiments, the first pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0314] In some exemplary embodiments, the second pre-processing may be at least one of a size scaling process, a pixel value normalization process, and the like.

[0315] In some exemplary embodiments, the third historical visible light image obtained after performing the first preprocessing on the third historical visible light image in the third historical visible light image frame sequence meets the input requirement of the hand region acquisition model.

[0316] In some exemplary embodiments, the second historical depth image obtained after performing a second preprocessing on the second historical depth image in the second historical depth image frame sequence meets the input requirement of the hand region acquisition model.

[0317] In step 701 , a third historical visible light image in a third historical visible light image sequence is input into a hand region acquisition model to obtain fourth hand region information of the third historical visible light image.

[0318] In step 702, a third target sub-image corresponding to the fourth hand region information of the third historical visible light image is obtained from the second historical depth image in the second historical depth image sequence whose position corresponds to the third historical visible light image; the third target sub-image is input into the hand joint point detection model to obtain the joint point information in the third target sub-image.

[0319] In some exemplary embodiments, the position index of the second historical depth image in the second historical depth image sequence whose position corresponds to the third historical visible light image in the second historical depth image sequence is the same as the position index of the third historical visible light image in the third historical visible light image sequence.

[0320] In some exemplary embodiments, in the second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image, the third target sub-image corresponding to the fourth hand region information of the third historical visible light image refers to the position of the third target sub-image in the second historical depth image sequence corresponding to the third historical visible light image, and the image area corresponding to the fourth hand region information is the same as the position in the third historical visible light image.

[0321] In some exemplary embodiments, the joint point information includes: joint point information of finger joint points, joint point information of wrist joint points, and joint point information of forearm joint points; wherein, the finger joint points include: proximal finger joint points, middle finger joint points, distal finger joint points, and fingertip joint points.

[0322] In some exemplary embodiments, the finger joints further include: interdigital joints.

[0323] In step 703 , dynamic gesture category information is annotated on the second historical depth image sequence.

[0324] In some exemplary embodiments, the gesture category information may be at least one of finger forward click, finger pinch, rotation, zoom, move, static gesture, etc.

[0325] In step 704, the second historical depth image sequence and the corresponding joint point information are used as inputs of the dynamic gesture model, and the dynamic gesture category information corresponding to the second historical depth image sequence is used as output of the dynamic gesture model to perform model training to obtain the dynamic gesture model.

[0326] In some exemplary embodiments, after the third target sub-image is input into the hand joint detection model to obtain the joint information in the third target sub-image, before the model training is performed, the method further includes: drawing the joints corresponding to the joint information in the second historical depth image in the third historical visible light image whose position corresponds to the second historical depth image, and when the number of third historical visible light images with abnormal joints in the third historical visible light image sequence is greater than half of the number of third historical visible light images in the third historical visible light image sequence, deleting the third data sample from the data samples for model training; and When the number is less than half of the number of third historical visible light images in the third historical visible light image sequence, for each frame of the second historical depth image in the second historical depth image sequence, determine whether the second historical depth image is a historical depth image with a hole based on the second historical depth image and the third historical visible light image whose position corresponds to the second historical depth image in the third historical visible light image sequence; when it is determined that the second historical depth image is a historical depth image with a hole, determine the hole position information in the second historical depth image; and correct the depth information of the hole corresponding to the hole position information based on the N frames of second historical depth images located before the second historical depth image; where N is 1 or 2.

[0327] Obtaining a third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position corresponds to the third historical visible light image in the second historical depth image sequence includes: obtaining a third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position corresponds to the third historical visible light image in the hole-corrected second historical depth image sequence.

[0328] In some exemplary embodiments, when the number of joint points drawn in the third historical visible light image and the corresponding joint points in the third historical visible light image are consistent in position is greater than or equal to a fourth preset threshold, the joint points of the third historical visible light image are determined to be normal; when the number of joint points drawn in the third historical visible light image and the corresponding joint points in the third historical visible light image are consistent in position is less than the fourth preset threshold, the joint points of the third historical visible light image are determined to be abnormal.

[0329] In some exemplary embodiments, before obtaining the third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position in the second historical depth image sequence corresponds to the third historical visible light image, the method further includes: for each frame of the second historical depth image in the second historical depth image sequence, determining whether the second historical depth image is a historical depth image with a hole based on the second historical depth image and the third historical visible light image whose position in the third historical visible light image sequence corresponds to the second historical depth image; in the case where it is determined that the second historical depth image is a historical depth image with a hole, determining the hole position information in the second historical depth image; and correcting the depth information of the hole corresponding to the hole position information based on N frames of second historical depth images located before the second historical depth image; wherein N is 1 or 2.

[0330] Obtaining a third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position corresponds to the third historical visible light image in the second historical depth image sequence includes: obtaining a third target sub-image corresponding to the fourth hand area information of the third historical visible light image from the second historical depth image whose position corresponds to the third historical visible light image in the hole-corrected second historical depth image sequence.

[0331] In some exemplary embodiments, determining whether the second historical depth image is a historical depth image with a hole based on the third historical visible light image whose position corresponds to the second historical depth image in the sequence of the second historical depth image and the third historical visible light image includes: determining a binarized image of the third historical visible light image and a binarized image of the second historical depth image; determining the difference between the binarized image of the third historical visible light image and the binarized image of the second historical depth image; when the difference is less than a first preset threshold, determining that the second historical depth image is a historical depth image that can be used as a reference frame; when the difference is greater than the first preset threshold, determining that the second historical depth image is a historical depth image with a hole.

[0332] In some exemplary embodiments, when determining the difference between the binarized image of the third historical visible light image and the binarized image of the second historical depth image, the sum of the absolute values ​​of the differences between all pixels of the binarized image of the third historical visible light image and the binarized image of the second historical depth image can be determined; or, the normalized value of the sum of the absolute values ​​of the differences between all pixels can be determined; or, the average value of the absolute values ​​of the differences between all pixels can be determined; or, the difference calculated by other means can be determined.

[0333] In some exemplary embodiments, determining the hole position information in the second historical depth image includes: determining a binarized image of the second historical depth image; and determining the hole position information as position information of pixels having a pixel value of 0 in the binarized image.

[0334] In some exemplary embodiments, correcting the depth information of the hole corresponding to the hole position information based on the second historical depth image N frames before the second historical depth image includes: obtaining the joint point position information in the second historical depth image; determining whether the hole corresponding to the hole position information is located in the preset area corresponding to the joint point corresponding to the joint point position information based on the hole position information and the joint point position information; in the case where the hole is located in the preset area corresponding to the joint point, determining the depth information after the hole correction as a weighted average of the basic filling value, the depth information of the hole corresponding to the hole position information in the second historical depth image N frames before the second historical depth image, and the prior information of the joint point; in the case where the hole is located outside the preset area corresponding to the joint point, determining the depth information after the hole correction based on the depth change trend between the two joint points closest to the hole and the depth information of the hole corresponding to the hole position information in the second historical depth image N frames before the second historical depth image.

[0335] In some exemplary embodiments, a third data sample is obtained for model training, and data enhancement may be performed on the third data sample. Model training is then performed based on the data-enhanced third data sample to obtain a dynamic gesture model. The data enhancement method improves the generalization of the dynamic gesture model.

[0336] In some exemplary embodiments, the model parameters in the trained hand joint detection model may be quantized to increase the speed of subsequently obtaining joint information based on the hand joint detection model.

[0337] Two application examples are listed below to illustrate the application of the gesture recognition method according to the embodiment of the present application. The listed application examples are not intended to limit the protection scope of the embodiment of the present application.

[0338] Application Example 1

[0339] This application example describes wearing a product that implements gesture recognition using the gesture recognition method of an embodiment of this application. A camera captures an exhibit in the user's field of view, which may be more than one. By using gestures or head movements, the user selects a specific exhibit for introduction. Both gestures and head movements are used to interact with virtual objects. The general flow of the user's field of view display process is as follows.

[0340] The main process of gesture clicking is that the camera captures the exhibit image and renders it, clicks on the exhibit to be appreciated through gestures, and retrieves the exhibit's voice information and 3D rendering information from the exhibit database through the exhibit's ID or exhibit features.

[0341] The main process of head ray is that the camera captures the image of the exhibit and renders it. Through head ray, which lasts for 2-3 seconds, click on the exhibit you want to appreciate, and retrieve the exhibit's voice information and 3D rendering information from the exhibit database through the exhibit's ID or exhibit features.

[0342] After receiving the audio and 3D rendering information, the device automatically plays the audio information of the exhibit and simultaneously displays the 3D rendering of the exhibit. The 3D rendering information can be a single exhibit. Visitors can raise their hands to manipulate the 3D rendering, such as clicking on the exhibit, pinching the exhibit and rotating it up, down, left, and right. Visitors can view the inside, outside, and front-facing information of the exhibit on the device. The 3D rendering information can also be a video of the exhibit, including its main features, development history, and excavation process. Visitors can also raise their hands to control the video playback progress, etc.

[0343] When using gestures to interact with exhibits or operate page information, the rendering information of the user's hand also appears in the user's field of view to enhance the user experience and accurately control fine position information.

[0344] An exemplary internal processing flow is as follows.

[0345] When the electronic device using the embodiment of the present application is turned on, an inference initialization process is performed to start loading the hand area acquisition model, the hand joint point detection model, and the dynamic gesture model.

[0346] After the power-on is complete, the camera starts to collect video data streams, which are decoded and placed into the RGB video queue and depth image queue respectively. After basic image preprocessing of the video stream, the RGB image is used to detect the hand area acquisition model. After obtaining the hand area information, it corresponds to the depth image and performs a process of correcting holes. Among them, if most of the depth images have holes, the video sequence of the queue will be discarded from the beginning. After correcting the holes in the depth image, the hand joint detection model is used to obtain the sequence of hand joints. At the same time, the RGB image is used to obtain the fine position information of the fingertips and update it to the joint sequence. The joints are constrained by human kinematic constraints to obtain a joint sequence that meets the constraints. Dynamic gesture model detection is performed to obtain dynamic gesture category information.

[0347] If the user clicks on an exhibit in the field of view at this time, the dynamic gesture category information obtained by the dynamic gesture model is click category information. After the application layer obtains the dynamic gesture category information, it zooms in on the exhibit accordingly and starts playing relevant voice or video information about the exhibit.

[0348] At the same time, the bottom layer processes the video sequence in the queue from the beginning and collects new video streams and adds them to the end of the queue.

[0349] Application Example 2

[0350] Wearing the product designed in the embodiment of this application, while appreciating calligraphy and paintings, you can obtain voice information and 3D rendering information of the calligraphy and paintings through gestures or head rays. At this time, the voice information and 3D rendering video information will be automatically played. You can also experience it yourself.

[0351] The main part of the hands-on experience is writing and drawing with a master based on a 3D rendered video. Using your fingertips as a pen requires precise positioning. The depth and tracking of your gestures allow you to adjust the weight and speed of your writing or drawing.

[0352] An exemplary internal processing flow is as follows.

[0353] Basically the same as the previous application example, after receiving the user's tap gesture, the application layer is now in the writing and drawing application and in the experience state. The user continues the tap gesture for a few seconds, and the changes in the gesture trajectory are tracked to obtain the writing and drawing handwriting. When the user's pen is lifted from the scroll, the depth value of the gesture decreases, indicating that the stroke has ended. To continue writing or drawing, the user presses the pen on the scroll again, and the depth value of the gesture increases. The same gesture is continued for a few seconds, and the changes in the gesture trajectory are tracked to obtain another handwriting.

[0354] By using dynamic gestures in this way, the user can be immersed in the experience and have a better sense of experience.

[0355] In the eighth aspect, referring to Figure 8, another embodiment of the present application provides an electronic device, including: at least one processor 801; a memory 802, and at least one program is stored on the memory 802. When the at least one program is executed by the at least one processor 801, any one of the above-mentioned gesture recognition methods, or any one of the above-mentioned human-computer interaction methods, or any one of the above-mentioned hand joint detection methods, or any one of the above-mentioned hole correction methods is implemented.

[0356] In some exemplary embodiments, the electronic device may be, but is not limited to, any one or more of augmented reality (AR) / extended reality (XR) glasses, a smart helmet, and a smart watch.

[0357] In some exemplary embodiments, the electronic device further includes: a video input module 803 configured to capture a visible light video stream and a first depth video stream having the same frame frequency.

[0358] In some exemplary embodiments, the video input module 803 may be a camera.

[0359] In some exemplary embodiments, the electronic device further includes: a voice output module 804 configured to output voice information.

[0360] In some exemplary embodiments, the voice output module 804 may be a speaker.

[0361] In some exemplary embodiments, the electronic device further includes: a voice input module 805 configured to collect sound information.

[0362] In some exemplary embodiments, the voice input module 805 may be a microphone.

[0363] In some exemplary embodiments, the electronic device further includes: one or more I / O interfaces 806 connected between the processor 801 and the memory 802 , and configured to implement information exchange between the processor 801 and the memory 802 .

[0364] Among them, the processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 702 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 806 is connected between the processor 801 and the memory 802, and can realize information interaction between the processor 801 and the memory 802, including but not limited to a data bus (Bus), etc.

[0365] In some embodiments, the processor 801 , the memory 802 , and the I / O interface 806 are connected to each other via a bus 807 , and further connected to other components of the computing device.

[0366] In the ninth aspect, another embodiment of the present application provides a computer-readable medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any one of the above-mentioned gesture recognition methods, or implements any one of the above-mentioned human-computer interaction methods, or implements any one of the above-mentioned hand joint detection methods, or implements any one of the above-mentioned hole correction methods.

[0367] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0368] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A void correction method, comprising: Acquire a visible light image sequence and a first depth image sequence having the same acquisition frame frequency; For each first depth image in the first depth image sequence, determining whether the first depth image is a depth image with a hole according to the first depth image and a visible light image in the visible light image sequence that meets a preset condition; wherein the visible light image that meets the preset condition is a visible light image whose position corresponds to the first depth image; When it is determined that the first depth image is a depth image with a hole, determine the hole position information in the first depth image; and correct the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image; wherein N is 1 or 2.

2. The hole correction method according to claim 1, after correcting the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image, the method further comprises: The joint point position information in each frame of the second depth image in the second depth image sequence is obtained respectively; wherein the second depth image sequence is a depth image sequence obtained by correcting all depth images with holes in the first depth image sequence.

3. The hole correction method according to claim 2, wherein: The step of respectively acquiring the joint point position information in each frame of the second depth image sequence comprises: For each frame of the second depth image, inputting a visible light image at a position corresponding to the second depth image in the visible light image sequence into a hand region acquisition model to obtain first hand region information of the visible light image; Acquire a first target sub-image corresponding to the first hand region information in the second depth image; The first target sub-image is input into the hand joint point detection model to obtain the joint point information corresponding to the first hand area information, and the joint point information corresponding to the first hand area information is the joint point position information in the second depth image.

4. The method according to claim 2, after respectively acquiring the joint point position information in each second depth image frame in the second depth image sequence, the method further comprises: Dynamic gesture category information is determined according to the second depth image sequence and joint point information in the second depth image.

5. The hole correction method according to claim 4, wherein: The joint point information includes: Joint point information of finger joints, wrist joints, and forearm joints; Among them, the finger joints include: proximal finger joints, middle finger joints, distal finger joints, and fingertip joints.

6. The hole correction method according to claim 5, wherein: The determining of dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image comprises: For each frame of the second depth image, correcting the joint point information of the fingertip joint points in the joint point information in the second depth image; The second depth image sequence and the corrected joint point information in the second depth image are input into a dynamic gesture model to obtain the dynamic gesture category information.

7. The hole correction method according to claim 6, wherein: The correcting the joint point information of the fingertip joint points in the joint point information in the second depth image includes: Acquire a fingertip sub-image from a visible light image in the visible light image sequence whose position corresponds to the second depth image according to the joint point information of the fingertip joint point; Extracting contour information of the fingertip in the fingertip sub-image; The curvature of the points on the contour of the fingertip is calculated according to the contour information of the fingertip, and the position information of the convex point with the largest curvature value is used as the corrected joint point information of the fingertip joint point.

8. The hole correction method according to claim 4, wherein: The determining of dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image comprises: For each frame of the second depth image in the second depth image sequence, performing kinematic constraint processing on the joint point information in the second depth image; The second depth image sequence and the kinematic constraints in the second depth image are processed The joint point information is input into the dynamic gesture model to obtain the dynamic gesture category information.

9. The hole correction method according to claim 8, wherein: The kinematic constraint processing of the joint point information in the second depth image comprises at least one of the following: Correcting the joint point information of the two adjacent joint points according to a value range of a first ratio of a length between two adjacent joint points to the hand region in which they are located and a first ratio in the second depth image of N frames before the second depth image; Correcting the joint point information of the other joint points except the wrist joint point according to the value range of the offset angle of the other joint points relative to the wrist joint point and the offset angle in the second depth image of N frames before the second depth image; Correcting the joint point information of the fingertip joint point and the wrist joint point according to the value range of the finger length and the finger length in the second depth image of N frames before the second depth image; wherein the finger length is the straight-line distance between the fingertip joint point and the wrist joint point; The joint point information of the wrist joint point and the proximal joint point of the finger is corrected according to the angle information of the forearm joint point and the hand and the angle information in the second depth image of N frames before the second depth image; wherein the angle information of the forearm joint point and the hand is the angle information of the angle between the straight line determined by the forearm joint point and the wrist joint point and the straight line determined by the wrist joint point and the proximal joint point of the finger; Correcting the joint point information of the two finger joint points closest to the finger joint point according to the value range of the distance between the finger joint point and the two finger joint points closest to the finger joint point and the distance between the finger joint point and the two finger joint points closest to the finger joint point in the N frames of second depth images before the second depth image; The joint point information of the finger joint points is corrected according to the value range of the bending degree of the finger relative to the palm and the back of the hand and the bending degree in the second depth image of N frames before the second depth image.

10. The hole correction method according to claim 4, after determining the dynamic gesture category information according to the second depth image sequence and the joint point information in the second depth image, the method further comprises: Acquiring original data of hand contours in visible light images in the visible light image sequence; Preprocessing the original hand contour data to obtain intermediate hand contour data with continuous contour; Generating a G2 continuous hand contour smooth curve according to the hand contour intermediate data; Filling and rendering are performed according to the smooth curve of the hand outline.

11. The hole correction method according to any one of claims 1 to 10, wherein: When the dynamic gesture category information is obtained, performing an operation corresponding to the dynamic gesture category information; When the dynamic gesture category information cannot be obtained, the head ray direction information is obtained, and an operation corresponding to the head ray direction information is performed.

12. The hole correction method according to claim 11, wherein: The operations include: interactive operations with exhibits in the museum.

13. The hole correction method according to any one of claims 1 to 10, wherein: The determining whether the first depth image is a depth image with a hole according to the first depth image and the visible light image in the visible light image sequence that meets a preset condition comprises: Determine a binarized image of the visible light image and a binarized image of the first depth image that meet a preset condition; Determine a difference between the binarized image of the visible light image and the binarized image of the first depth image; When the difference is less than a first preset threshold, determining that the first depth image is a depth image that can be used as a reference frame; When the difference is greater than the first preset threshold, it is determined that the first depth image is a depth image with a hole.

14. The hole correction method according to any one of claims 1 to 10, wherein: The determining the hole position information in the first depth image comprises: Determining a binarized image of the first depth image; The hole position information is determined to be position information of pixels whose pixel values ​​are 0 in the binary image.

15. The hole correction method according to any one of claims 1 to 10, wherein: The correcting the hole depth information corresponding to the hole position information according to the N frames of first depth images before the first depth image includes: Acquire joint point position information in the first depth image; Determine, according to the hole position information and the joint point position information, whether the hole corresponding to the hole position information is located in a preset area corresponding to the joint point corresponding to the joint point position information; In the case where the hole is located in the preset area corresponding to the joint point, determine the depth information after the hole correction as the basic filling value, the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image, and the weighted average of the prior information of the joint point; When the hole is located outside the preset area corresponding to the joint point, the depth information of the hole after correction is determined according to the basic filling value, the depth information between the two joint points closest to the hole, and the depth information of the hole corresponding to the hole position information in the first depth image of N frames before the first depth image.

16. An electronic device comprising: at least one processor; A memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, the hole correction method according to any one of claims 1 to 15 is implemented.

17. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hole correction method according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • Method for filling cavity in Kinect depth image in real time

    CN104299220A

  • Kinect deep image remediation method based on colorful image segmentation

    CN104680496A

  • 3D hand posture estimation method based on point cloud

    CN110135340A

  • Depth information completion method and device, monitoring system and storage medium

    CN111107337A

  • Real-time gesture recognition method and system based on deep learning

    CN113269089A