Dynamic gesture recognition method, apparatus, readable storage medium and computer device

The dynamic gesture recognition method uses a deep learning model and skin detection algorithm to determine gesture movement direction and magnitude in three-dimensional space, addressing the limitations of two-dimensional recognition with complex calculations and low real-time performance.

JP7590041B2Active Publication Date: 2024-11-26NANCHANG VIRTUAL REALITY RES INST CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023576238
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-03-15
Filing Date
2021-06-15
Publication Date
2024-11-26
Estimated Expiration
2041-06-15

AI Technical Summary

Technical Problem

Conventional gesture recognition methods are limited to determining movement direction on a two-dimensional plane, with complex calculations leading to low real-time performance.

Method used

A dynamic gesture recognition method using a trained hand detection deep learning model to obtain graphic information of a minimum circumscribed rectangle, combined with a skin detection algorithm to calculate center distance, slope rate, and average depth value, enabling determination of gesture movement direction and magnitude in three-dimensional space.

Benefits of technology

The method simplifies the process and enhances real-time performance by determining gesture movements qualitatively and quantitatively in three-dimensional space, overcoming limitations of two-dimensional recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590041000012
    Figure 0007590041000012
  • Figure 0007590041000013
    Figure 0007590041000013
  • Figure 0007590041000014
    Figure 0007590041000014
Patent Text Reader

Abstract

According to the present invention, a dynamic gesture recognition method, a system, a readable storage medium and a computer device are provided. The method includes: performing hand object detection on a target image by a trained hand detection deep learning model to obtain graphic information of a minimum bounding rectangle in a hand region; calculating a center distance and a slope rate between the minimum bounding rectangles corresponding to two times based on the graphic information of the minimum bounding rectangles corresponding to a second time and a first time; selecting a hand skin region from the target image by a skin detection algorithm, linking the hand skin region with a depth map to calculate hand skin region average depth values ​​corresponding to the second time and the first time, respectively; and determining a gesture movement direction and a magnitude of the movement amount in the corresponding direction based on the center distance, the slope rate and the hand skin region average depth values ​​corresponding to the two times. According to the present invention, the existing problems of the prior art that only the movement direction on a two-dimensional plane can be determined, the calculation process is complicated, and the real-time performance of gesture recognition is low can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims priority to an earlier application, Application No. 202110273657.5, filed on March 15, 2021, entitled "Dynamic Gesture Recognition Method, Apparatus, Readable Storage Medium, and Computer Device," the contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of computer technology, and in particular to a dynamic gesture recognition method, apparatus, readable storage medium and computer device. [Background technology]

[0003] Gesture recognition is an important tool in human-machine interaction. In human-machine interaction using VR (Virtual Reality) technology, virtual gesture recognition allows users to adjust the volume or control other virtual movement keys.

[0004] In the conventional technology, the image is gridded, and the presence or absence of the hand in each grid is marked by a skin detection algorithm, and the presence or absence of the binary image is marked, and the direction of the hand movement is obtained by performing logical operations on the binary image. This method can only determine the movement direction on a two-dimensional plane. There is also a gesture recognition method based on the Tower algorithm, but the calculation process is very complicated and the amount of calculation is enormous, which reduces the real-time performance of gesture recognition. Summary of the Invention

[0005] In view of the above, an object of the present invention is to provide a dynamic gesture recognition method for solving the problems existing in the prior art, that is, the movement direction can only be determined on a two-dimensional plane, the calculation process is complicated, and the real-time performance of gesture recognition is low.

[0006] The present invention provides a dynamic gesture recognition method, comprising: Performing hand object detection on the target image using the trained hand detection deep learning model to obtain graphic information of a minimum bounding rectangle in the hand region; calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of the minimum bounding rectangle corresponding to the second time and graphic information of the minimum bounding rectangle corresponding to the first time, wherein the first time and the second time are adjacent times; selecting a hand skin region from the target image using a skin detection algorithm, linking the hand skin region with a depth map, and calculating an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time, respectively; determining a gesture movement direction and a magnitude of movement in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time.

[0007] In the dynamic gesture recognition method provided by the present invention, the skin detection algorithm is linked to a depth map, and the hand skin area average depth value at two adjacent times is obtained, so that the hand movement direction and the magnitude of the movement can be determined from a three-dimensional space, and the gesture movement can be analyzed qualitatively and quantitatively. In the present invention, the gesture is judged based on the center distance between the minimum bounding rectangles corresponding to the first and second times, the slope rate, and the hand skin area average depth value at two adjacent times, so that the calculation process is simpler and the real-time performance is higher.

[0008] In addition, based on the above dynamic gesture recognition method in the present invention, the following additional configuration requirements may be included.

[0009] Furthermore, by performing hand object detection on a target image using a trained hand detection deep learning model, graphic information of a minimum bounding rectangle in a hand region can be obtained. Obtaining an RGB image including a hand captured by an RGB camera; inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; obtaining graphic information of a minimum bounding rectangle in a hand region from a detection result of the hand object detection; The graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle.

[0010] Further, calculating a center distance and a slope ratio between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of the minimum bounding rectangle corresponding to the second time and graphic information of the minimum bounding rectangle corresponding to the first time, Calculate the center distance and the slope between the minimum bounding rectangle corresponding to the second time and the minimum bounding rectangle corresponding to the first time by the following formula:

number

[0011] Further, by using a skin detection algorithm, a hand skin region is selected from the target image, and the hand skin region is linked to a depth map to calculate an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time, respectively. converting the RGB image into a YCrCb space and detecting skin within a minimum bounding rectangle using an ellipse skin detection algorithm to select a hand skin region from the target image; and calculating an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time by associating the hand skin region with a corresponding depth map according to the following formula:

number

[0012] Further, determining a gesture movement direction and a magnitude of a movement amount in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time, If d is equal to or smaller than a threshold thr1, it is determined that there is no horizontal movement of the hand in the uvz coordinate system; d is greater than the threshold thr1, and p cx1 =p cx2 In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v =p cy2 -p cy1 determining that d is greater than the threshold thr1, and p cy1=p cy2 In this case, the hand moves only in the u direction in the uvz coordinate system, and the momentum x v =p cx2 -p cx1 determining that d is greater than the threshold thr1, and p cx1 ≠p cx2 , p cy1 ≠p cy2 In this case, the motion component x of the hand along the u direction in the uvz coordinate system is v =p cx2 -p cx1 and the component of motion along the v direction is y v =p cy2 -p cy1 and determining that the direction of motion along the u-v plane is the slope direction k; The motion component z of the hand along the z direction in the uvz coordinate system v =d v2 -d v1 It is.

[0013] Another object of the present invention is to provide a dynamic gesture recognition device for solving the problems existing in the prior art, such as the fact that only the direction of movement on a two-dimensional plane can be determined, the calculation process is complicated, and the real-time performance of gesture recognition is low.

[0014] The present invention provides a dynamic gesture recognition device, the device comprising: a detection module for detecting hand objects in a target image using a trained hand detection deep learning model, thereby obtaining graphic information of a minimum bounding rectangle in a hand region; a first calculation module for calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of the minimum bounding rectangle corresponding to the second time and graphic information of the minimum bounding rectangle corresponding to the first time, the first calculation module being adjacent to the second time; a second calculation module for selecting a hand skin area from the target image by a skin detection algorithm, linking the hand skin area with a depth map, and calculating an average depth value of the hand skin area corresponding to the second time and an average depth value of the hand skin area corresponding to the first time, respectively; and a determination module for determining a gesture movement direction and a magnitude of movement in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time.

[0015] In the dynamic gesture recognition device provided by the present invention, the skin detection algorithm is linked to a depth map, and the hand skin area average depth value at two adjacent times is obtained, so that the hand movement direction and the magnitude of the movement can be determined from a three-dimensional space, and the gesture movement can be analyzed qualitatively and quantitatively. In the present invention, the gesture is judged based on the center distance between the minimum circumscribed rectangles corresponding to the first and second times, the slope rate, and the hand skin area average depth value at two adjacent times, so the calculation process is simpler and the real-time performance is higher.

[0016] In addition, based on the above dynamic gesture recognition device in the present invention, the following additional configuration requirements may be included.

[0017] Furthermore, the detection module Acquire an RGB image including the hand taken by an RGB camera, inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; From the detection result of the hand object detection, graphic information of a minimum bounding rectangle in the hand region is obtained, and the graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle.

[0018] Furthermore, the first calculation module calculates a center distance and a slope rate between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time by the following formula:

number

[0019] Furthermore, the second calculation module: converting the RGB image into a YCrCb space and detecting skin within a minimum bounding rectangle using an ellipse skin detection algorithm to select a hand skin region from the target image; linking the hand skin area with the corresponding depth map, and calculating an average depth value of the hand skin area corresponding to the second time and an average depth value of the hand skin area corresponding to the first time according to the following formula:

number

[0020] Furthermore, the determination module If d is equal to or smaller than a threshold thr1, it is determined that there is no horizontal movement of the hand in the uvz coordinate system; d is greater than the threshold thr1, and p cx1 =p cx2 In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v =p cy2 -p cy1 It is determined that d is greater than the threshold thr1, and p cy1 =p cy2 In this case, the hand moves only in the u direction in the uvz coordinate system, and the momentum x v =p cx2 -p cx1 It is determined that d is greater than the threshold thr1, and p cx1 ≠p cx2 , p cy1 ≠p cy2 In this case, the motion component x of the hand along the u direction in the uvz coordinate system is v =p cx2 -p cx1 and the component of motion along the v direction is y v =p cy2 -p cy1 and the direction of motion along the u-v plane is determined to be the slope direction k; The motion component z of the hand along the z direction in the uvz coordinate system v =d v2 -d v1 It is.

[0021] The invention further proposes a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method described above.

[0022] The invention further proposes a computer device including a memory, a processor and a computer program stored in the memory and executed by the processor, the steps of the method being implemented when the computer program is executed by the processor.

[0023] Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. [Brief description of the drawings]

[0024] The above and / or additional aspects and advantages of the embodiments of the present invention will become apparent and will be more readily understood by the following detailed description of the embodiments taken in conjunction with the accompanying drawings. [Figure 1] 2 is a flowchart of a dynamic gesture recognition method according to an embodiment of the present invention. [Diagram 2] 2 is a detailed flowchart of step S101 in FIG. 1. [Diagram 3] FIG. 4 is a structural block diagram of a dynamic gesture recognition device according to another embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and are not all of the embodiments. All other embodiments obtained based on the embodiments of the present invention under the premise that those skilled in the art do not exert creative efforts are all within the protection scope of the present invention.

[0026] Referring to FIG. 1, the dynamic gesture recognition method proposed in one embodiment of the present invention includes the following steps S101 to S104.

[0027] S101: Obtaining graphic information of a minimum bounding rectangle in a hand region by performing hand object detection on a target image using a trained hand detection deep learning model.

[0028] Here, referring to FIG. 2, step S101 is S1011: Acquiring an RGB image including a hand captured by an RGB camera; S1012: Inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; S1013: acquiring graphic information of a minimum bounding rectangle in a hand region from a detection result of the hand object detection; The graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle.

[0029] Here, the coordinates of the upper left corner of the minimum bounding rectangle are (P x ,P y ), and the width and height of the rectangle are denoted by w and h, respectively, and may be measured in pix.

[0030] S102: Calculating the center distance and the slope rate between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time based on the graphic information of the minimum circumscribing rectangle corresponding to the second time and the graphic information of the minimum circumscribing rectangle corresponding to the first time, wherein the first time and the second time are adjacent times.

[0031] However, for example, if the second time is time t and the first time is time t-1, the first time may be one time earlier than the second time.

[0032] Here, the center distance and the slope ratio between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time are calculated by the following formula.

number

[0033] S103: Using a skin detection algorithm, select a hand skin area from the target image, link the hand skin area with a depth map, and calculate an average depth value of the hand skin area corresponding to the second time and an average depth value of the hand skin area corresponding to the first time, respectively.

[0034] Here, the RGB image is transformed into YCrCb space, and an elliptical skin detection algorithm is used to detect skin within a minimum bounding rectangle to select the hand skin region from the target image.

[0035] The hand skin region and the corresponding depth map are linked together to calculate the hand skin region average depth value corresponding to the second time and the hand skin region average depth value corresponding to the first time according to the following formula.

number

[0036] S104: Determining a gesture movement direction and a magnitude of movement in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time.

[0037] Specifically, when d is equal to or smaller than a threshold thr1, it is determined that there is no movement of the hand in the horizontal direction in the uvz coordinate system (i.e., the uv direction in the uvz coordinate system); d is greater than the threshold thr1, and p cx1 =p cx2 In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v =p cy2 -p cy1 (unit: pix), and the positive or negative result of the subtraction indicates the direction of movement. d is greater than the threshold thr1, and p cy1 =p cy2 In this case, the hand moves only in the u direction in the uvz coordinate system, and the momentum x v =p cx2 -p cx1 (unit: pix), and the positive or negative result of the subtraction indicates the direction of movement. d is greater than the threshold thr1, and p cx1 ≠p cx2 , p cy1 ≠p cy2 In this case, the motion component x of the hand along the u direction in the uvz coordinate system is v =p cx2 -pcx1 (unit: pix), and the motion component y of the motion along the v direction v =p cy2 -p cy1 (unit: pix), and the direction of motion along the uv plane is determined to be the gradient direction k, Here, the motion component z of the hand in the z direction in the uvz coordinate system is v =d v2 -d v1 (Unit: mm). v If is less than or equal to the threshold thr2, it indicates that there is no motion in the z direction, but if z v If is greater than the threshold thr2, the motion component is d v2 -d v1 The positive or negative result of the subtraction indicates the direction of hand movement in the z-axis direction.

[0038] As described above, in the dynamic gesture recognition method provided by this embodiment, the skin detection algorithm is linked to a depth map to obtain the average depth value of the hand skin region at two adjacent times, thereby making it possible to determine the hand movement direction and the magnitude of the movement from a three-dimensional space, and to analyze the gesture movement qualitatively and quantitatively. In the present invention, the gesture is judged based on the center distance between the minimum circumscribing rectangles corresponding to the first and second times, the slope rate, and the average depth value of the hand skin region at two adjacent times, so that the calculation process is simpler and the real-time performance is higher.

[0039] Referring to FIG. 3, a dynamic gesture recognition device is proposed in another embodiment of the present invention, a detection module for detecting hand objects in a target image using a trained hand detection deep learning model, thereby obtaining graphic information of a minimum bounding rectangle in a hand region; a first calculation module for calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of the minimum bounding rectangle corresponding to the second time and graphic information of the minimum bounding rectangle corresponding to the first time, the first calculation module being adjacent to the second time; a second calculation module for selecting a hand skin area from the target image by a skin detection algorithm, linking the hand skin area with a depth map, and calculating an average depth value of the hand skin area corresponding to the second time and an average depth value of the hand skin area corresponding to the first time, respectively; and a determination module for determining a gesture movement direction and a magnitude of movement in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time.

[0040] In this embodiment, the detection module includes: Acquire an RGB image including the hand taken by an RGB camera, inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; From the detection result of the hand object detection, graphic information of a minimum bounding rectangle in the hand region is obtained, and the graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle.

[0041] In this embodiment, the first calculation module calculates a center distance and a slope ratio between the minimum bounding rectangle corresponding to the second time and the minimum bounding rectangle corresponding to the first time according to the following formula:

number

[0042] In this embodiment, the second calculation module: converting the RGB image into a YCrCb space and detecting skin within a minimum bounding rectangle using an ellipse skin detection algorithm to select a hand skin region from the target image; The hand skin region and the corresponding depth map are linked together to calculate the hand skin region average depth value corresponding to the second time and the hand skin region average depth value corresponding to the first time according to the following formula.

number

[0043] In this embodiment, the determination module: If d is equal to or smaller than a threshold thr1, it is determined that there is no horizontal movement of the hand in the uvz coordinate system; d is greater than the threshold thr1, and p cx1 =p cx2In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v =p cy2 -p cy1 It is determined that d is greater than the threshold thr1, and p cy1 =p cy2 In this case, the hand moves in the u direction in the uvz coordinate system, and the momentum x v =p cx2 -p cx1 It is determined that d is greater than the threshold thr1, and p cx1 ≠p cx2 , p cy1 ≠p cy2 Then, the motion component x of the hand in the u direction in the uvz coordinate system is v =p cx2 -p cx1 and the component of motion along the v direction is y v =p cy2 -p cy1 and the direction of motion along the u-v plane is determined to be the slope direction k; Here, the motion component z of the hand in the z direction in the uvz coordinate system is v =d v2 -d v1 It is.

[0044] In the dynamic gesture recognition device provided by this embodiment, the skin detection algorithm is linked to a depth map, and the hand skin area average depth value at two adjacent times is obtained, so that the hand movement direction and the magnitude of the movement can be determined from a three-dimensional space, and the gesture movement can be analyzed qualitatively and quantitatively. In the present invention, the gesture is judged based on the center distance between the minimum circumscribed rectangles corresponding to the first and second times, the slope rate, and the hand skin area average depth value at two adjacent times, so that the calculation process is simpler and the real-time performance is higher.

[0045] Moreover, an embodiment of the present invention further proposes a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0046] Moreover, an embodiment of the present invention further proposes a computer device including a memory, a processor, and a computer program stored in the memory and executed by the processor, the steps of the method being realized when the computer program is executed by the processor.

[0047] The logic and / or steps shown in the flowcharts or otherwise described herein may be thought of, for example, as a sequence of executable commands to implement logical functions, and may be embodied in any computer-readable medium to be provided for use with or in combination with a command execution system, device, or appliance (e.g., a computer-based system, a system including a processor, or other system that can obtain and execute commands from the command execution system, device, or appliance). As used herein, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use with or in combination with a command execution system, device, or appliance.

[0048] More specific examples of computer readable media (not an exhaustive list) include an electrical connection having one or more wires (electronic device), a portable computer disk (magnetic device), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read only memory (CDROM). The computer readable medium may also be paper or other suitable medium on which the program may be printed, for example by optically scanning the paper or other medium for subsequent editing or interpretation, or may be processed in other suitable manners, if desired, to obtain the program in electronic form and then stored in computer memory.

[0049] It should be understood that each part of the present invention may be realized by hardware, software, firmware, or a combination thereof. In the above embodiment, a number of steps or methods may be realized by software or firmware stored in a memory and executed by an appropriate command execution system. For example, when realized by hardware, as in other embodiments, it may be realized by one or a combination of techniques known in the art, such as a discrete logic circuit having logic gate circuits for performing logic functions on data signals, an application specific integrated circuit having a combination of appropriate logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0050] In the description herein, the reference words "one embodiment," "several embodiments," "exemplary," "particular examples," or "several examples" are intended to indicate that the specific features, structures, materials, or features described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In the description herein, the exemplary expressions for the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or features described may be combined in any suitable manner in any one or more embodiments or examples.

[0051] Although embodiments of the present invention have been illustrated and described, it should be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to those embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is limited by the claims and their equivalents.

Claims

1. Performing hand object detection on the target image using the trained hand detection deep learning model to obtain graphic information of a minimum bounding rectangle in the hand region; calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of a minimum bounding rectangle corresponding to the second time and graphic information of a minimum bounding rectangle corresponding to the first time, the first time and the second time being adjacent to each other; selecting a hand skin region from the target image using a skin detection algorithm, linking the hand skin region with a depth map, and calculating an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time, respectively; determining a gesture movement direction and a magnitude of the movement amount in the corresponding direction based on the center distance, the slope rate, the hand skin region average depth value corresponding to the second time, and the hand skin region average depth value corresponding to the first time; Calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of the minimum bounding rectangle corresponding to the second time and graphic information of the minimum bounding rectangle corresponding to the first time, Calculate the center distance and the slope between the minimum bounding rectangle corresponding to the second time and the minimum bounding rectangle corresponding to the first time by the following formula: [0010] However, (p x1 , p y1 ) indicates the coordinates of the upper left corner vertex of the minimum circumscribing rectangle corresponding to the first time, and w 1 , h 1 respectively indicate the width and height of the minimum circumscribing rectangle corresponding to the first time, and (p cx1 , p cy1 ) indicates the coordinates of the center point of the minimum circumscribing rectangle corresponding to the first time, and (p x2 , p y2 ) indicates the coordinates of the upper left corner vertex of the minimum circumscribing rectangle corresponding to the second time, and w 2 , h 2 respectively indicate the width and height of the minimum circumscribing rectangle corresponding to the second time, and (p cx2 , p cy2 ) indicates the coordinates of the center point of the minimum circumscribing rectangle corresponding to the second time, d indicates the center distance between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time, k indicates the slope rate between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time, Determining a gesture movement direction and a magnitude of a movement amount in the corresponding direction based on the center distance, the slope rate, the hand skin area average depth value corresponding to the second time, and the hand skin area average depth value corresponding to the first time, If d is equal to or smaller than a threshold value thr1, it is determined that there is no horizontal movement of the hand in the uvz coordinate system; d is greater than the threshold thr1, and p cx1 = p cx2 In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v = p cy2 -p cy1 determining that d is greater than the threshold thr1, and p cy1 = p cy2 In this case, the hand moves only in the u direction in the uvz coordinate system, and the momentum x v = p cx2 -p cx1 determining that d is greater than the threshold thr1, and p cx1 ≠ p cx2 , p cy1 ≠ p cy2 In this case, the hand moves in the uv direction in the uvz coordinate system, and the motion component x v = p cx2 -p cx1 and the motion component y of the motion along the v direction v = p cy2 -p cy1 and determining that The motion component z of the hand along the z direction in the uvz coordinate system v = d v2 -d v1 and where d v1 indicates the average depth value of the hand skin region corresponding to the first time, and d v2 indicates the average depth value of the hand skin region corresponding to the second time. A dynamic gesture recognition method comprising:

2. By performing hand object detection on a target image using a trained hand detection deep learning model, graphic information of a minimum bounding rectangle in a hand region is obtained. Obtaining an RGB image including a hand portion captured by an RGB camera; inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; acquiring graphic information of a minimum bounding rectangle in a hand region from a detection result of the hand object detection; The graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle. The dynamic gesture recognition method according to claim 1 .

3. selecting a hand skin region from the target image using a skin detection algorithm, linking the hand skin region with a depth map, and calculating an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time, converting the RGB image to a YCrCb space and detecting skin within a minimum bounding rectangle using an elliptical skin detection algorithm to select a hand skin region from the target image; and calculating an average depth value of the hand skin region corresponding to the second time and an average depth value of the hand skin region corresponding to the first time by associating the hand skin region with a corresponding depth map according to the following formula: [0025] However, d e1 indicates a depth value corresponding to each pixel in the hand skin region corresponding to the first time, and d v1 indicates the average depth value of the hand skin region corresponding to the first time, and d e2 indicates a depth value corresponding to each pixel in the hand skin region corresponding to the second time, and d v2 indicates the average depth value of the hand skin region corresponding to the second time, and N indicates the number of hand skin pixel points; The dynamic gesture recognition method according to claim 2 .

4. a detection module for detecting hand objects in a target image using a trained hand detection deep learning model, thereby obtaining graphic information of a minimum bounding rectangle in a hand region; a first calculation module for calculating a center distance and a slope rate between a minimum bounding rectangle corresponding to a second time and a minimum bounding rectangle corresponding to a first time based on graphic information of a minimum bounding rectangle corresponding to the second time and graphic information of a minimum bounding rectangle corresponding to the first time, the first calculation module being adjacent to the second time; a second calculation module for selecting a hand skin area from the target image by a skin detection algorithm, linking the hand skin area with a depth map, and calculating an average depth value of the hand skin area corresponding to the second time and an average depth value of the hand skin area corresponding to the first time, respectively; a determination module for determining a gesture movement direction and a magnitude of a movement amount in the corresponding direction based on the center distance, the slope rate, the hand skin area average depth value corresponding to the second time, and the hand skin area average depth value corresponding to the first time, The first calculation module calculates a center distance and a slope rate between a minimum bounding rectangle corresponding to the second time and a minimum bounding rectangle corresponding to the first time by the following formula: [0030] However, (p x1 , p y1 ) indicates the coordinates of the upper left corner vertex of the minimum circumscribing rectangle corresponding to the first time, and w 1 , h 1 respectively indicate the width and height of the minimum circumscribing rectangle corresponding to the first time, and (p cx1 , p cy1 ) indicates the coordinates of the center point of the minimum circumscribing rectangle corresponding to the first time, and (p x2 , p y2 ) indicates the coordinates of the upper left corner vertex of the minimum circumscribing rectangle corresponding to the second time, and w 2 , h 2 respectively indicate the width and height of the minimum circumscribing rectangle corresponding to the second time, and (p cx2 , p cy2 ) indicates the coordinates of the center point of the minimum circumscribing rectangle corresponding to the second time, d indicates the center distance between the minimum circumscribing rectangle corresponding to the second time and the minimum circumscribing rectangle corresponding to the first time, k indicates the slope rate between the center point of the minimum circumscribing rectangle corresponding to the second time and the center point of the minimum circumscribing rectangle corresponding to the first time, The determination module includes: If d is equal to or smaller than a threshold value thr1, it is determined that there is no horizontal movement of the hand in the uvz coordinate system; d is greater than the threshold thr1, and p cx1 = p cx2 In this case, the hand moves only in the v direction in the uvz coordinate system, and the momentum y v = p cy2 -p cy1 It is determined that d is greater than the threshold thr1, and p cy1 = p cy2 In this case, the hand moves only in the u direction in the uvz coordinate system, and the momentum x v = p cx2 -p cx1 It is determined that d is greater than the threshold thr1, and p cx1 ≠ p cx2 , p cy1 ≠ p cy2 In this case, the hand moves in the uv direction in the uvz coordinate system, and the motion component x v = p cx2 -p cx1 and the motion component y of the motion along the v direction v = p cy2 -p cy1 It is determined that The motion component z of the hand along the z direction in the uvz coordinate system v = d v2 -d v1 and where d v1 indicates the average depth value of the hand skin region corresponding to the first time, and d v2 indicates the average depth value of the hand skin region corresponding to the second time. A dynamic gesture recognition device comprising:

5. The detection module includes: Acquire an RGB image including a hand captured by an RGB camera; inputting the RGB image into the trained hand detection deep learning model to perform hand object detection; From the detection result of the hand object detection, graphic information of a minimum bounding rectangle in the hand region is obtained; The graphic information includes the coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the rectangle. The dynamic gesture recognition device according to claim 4 .

6. A readable storage medium having a computer program stored thereon, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 3. A readable storage medium.

7. A computer device including a memory, a processor, and a computer program stored in the memory and executed by the processor, When the computer program is executed by the processor, the method according to any one of claims 1 to 3 is realized.

1. A computer device comprising:

Citation Information

Patent Citations

  • Gesture recognition apparatus, method for controlling gesture recognition apparatus, and control programprogram

    JP2011192090A

  • Dynamic gesture recognition method and device, gesture interaction control method and device

    JP2020508511A