Multi-view gesture recognition method and device, computer device and storage medium

Abstract

The application discloses a multi-view gesture recognition method, which comprises the following steps: installing at least two TOF sensors for collecting 3D information of hands in a gesture interaction space; collecting infrared IR images and depth images under each view angle in the gesture interaction space in real time through each TOF sensor; detecting and processing the collected infrared IR images; correcting and calculating hand key point coordinates under each view angle to obtain a hand key point 3D coordinate set under all view angles; performing calibration and mapping processing on the hand key point 3D coordinate set under all view angles; calculating and processing to obtain a hand feature map, and performing gesture recognition processing on the obtained hand feature map to obtain a gesture recognition result. The application also discloses a device, a computer device and a storage medium for implementing the multi-view gesture recognition method. The application improves the recognition accuracy and can recognize more complex gestures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture recognition method, and particularly relates to a multi-view gesture recognition method, device, computer equipment and storage medium. BACKGROUND

[0002] In the field of human-computer interaction, gesture interaction as the most widely used and efficient interaction mode is widely used in robot control, industrial production, sign language recognition, multimedia video control and many other fields. The accuracy and efficiency of gesture interaction depend on reliable static and dynamic gesture recognition.

[0003] At present, the mainstream gesture recognition depends on the 2D image recognition result under a single view, uses an RGB camera to obtain a video stream, detects a hand, and then performs gesture recognition on the hand. Some methods also model video frames to obtain hand 3D information and improve the accuracy of gesture recognition. However, due to the limitations of field of view angle, environmental space size and other factors, it is difficult to accurately recognize the information under a single view, and it is also difficult to solve the situation of incomplete hand caused by some complex gestures and partial occlusion, which ultimately reduces the accuracy of gesture recognition and cannot recognize more complex gestures.

[0004] Therefore, the present applicant finds a method to solve the above problems through beneficial exploration and research, and the technical solution to be introduced below is generated in this background. SUMMARY

[0005] One of the technical problems to be solved by the present application is to provide a multi-view gesture recognition method that improves recognition accuracy and can recognize more complex gestures, in view of the deficiencies of the prior art.

[0006] The second technical problem to be solved by the present application is to provide a multi-view gesture recognition device for implementing the above multi-view gesture recognition method.

[0007] The third technical problem to be solved by the present application is to provide a computer equipment for implementing the above multi-view gesture recognition method.

[0008] The fourth technical problem to be solved by the present application is to provide a computer readable storage medium for implementing the above multi-view gesture recognition method.

[0009] As a multi-view gesture recognition method of the first aspect of the present application, comprising:

[0010] install at least two TOF sensors for collecting 3D information of the hand in the gesture interaction space, and calibrate each TOF sensor to obtain an intrinsic matrix and lens distortion parameters of each TOF sensor and a relative position transformation matrix between the multiple TOF sensors;

[0011] During operation, infrared IR images and depth images in each view angle in the gesture interaction space are collected in real time by each TOF sensor;

[0012] The collected infrared IR images are detected to obtain hand key point coordinates in each view angle;

[0013] The hand key point coordinates in each view angle are corrected and calculated according to the intrinsic matrix and lens distortion parameters obtained through calibration and in combination with the collected depth images, to obtain a hand key point 3D coordinate set in all view angles;

[0014] The hand key point 3D coordinate set in all view angles is calibrated and mapped according to the relative position transformation matrix between the multiple TOF sensors obtained through calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with a wrist key point as a coordinate origin; and

[0015] The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated to obtain a hand feature map, and the hand feature map is subjected to gesture recognition processing to obtain a gesture recognition result.

[0016] In a preferred embodiment of the present application, the detection processing of the collected infrared IR images to obtain hand key point coordinates in each view angle comprises:

[0017] The collected infrared IR images are detected in real time by a hand detection model based on IR images;

[0018] When a human hand is detected in the infrared IR images, the hand region in the collected infrared IR images is cropped to obtain a local IR image of the human hand; and

[0019] The cropped local IR image of the human hand is input into a hand key point detection model based on IR images for detection to obtain hand key point coordinates in each view angle;

[0020] In a preferred embodiment of the present application, the correction and calculation of the hand key point coordinates in each view angle according to the intrinsic matrix and lens distortion parameters obtained through calibration and in combination with the collected depth images to obtain a hand key point 3D coordinate set in all view angles comprises:

[0021] According to the intrinsic matrix and the lens distortion matrix obtained through calibration, the hand key point coordinates and the collected depth images under each view angle are corrected;

[0022] The corrected hand key point coordinates are remapped to the corrected depth images to obtain two-dimensional coordinates and depth data of each hand key point under each view angle; and

[0023] According to the intrinsic matrix obtained through calibration, the two-dimensional coordinates and depth data of each hand key point under each view angle are calculated and processed to obtain the hand key point 3D coordinates under each view angle; and

[0024] The hand key point 3D coordinates under each view angle are collectively processed to obtain a hand key point 3D coordinate set under all view angles.

[0025] In a preferred embodiment of the present application, the hand key point 3D coordinate set under all view angles is calibrated and mapped according to the relative position transformation matrix between the multiple TOF sensors obtained through calibration to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as the coordinate origin, including:

[0026] The hand key point 3D coordinate set under all view angles is fused and calibrated according to the relative position transformation matrix between the multiple TOF sensors obtained through calibration; and

[0027] The hand key point 3D coordinate set under all view angles after the fusion and calibration is remapped to the three-dimensional coordinate system with the wrist key point as the coordinate origin to obtain the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin.

[0028] In a preferred embodiment of the present application, the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed to obtain a hand feature map, including:

[0029] The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is normalized to obtain a normalized hand key point 3D coordinate set;

[0030] The Euclidean distances between each normalized hand key point are calculated to generate a plurality of normalized distance maps between hand key points;

[0031] According to each key point, a node connection map of a plurality of hand key points far from the wrist is generated by selecting the point farthest from the wrist key point adjacent to the key point;

[0032] According to each key point, a node connection map of a plurality of hand key points close to the wrist is generated by selecting the point closest to the wrist key point adjacent to the key point;

[0033] The normalized distance map between the plurality of hand key points, the node connection map of the plurality of hand key points far from the wrist, and the node connection map of the plurality of hand key points close to the wrist are synthesized to obtain a hand feature map.

[0034] In a preferred embodiment of the present application, the hand feature map is subjected to gesture recognition by a gesture recognition model based on hand key point 3D coordinate information fusion feature map, and a gesture recognition result is generated.

[0035] As an implementation of the above multi-view gesture recognition method according to the second aspect of the present application, a multi-view gesture recognition device comprises:

[0036] At least two TOF sensors installed in the gesture interaction space for collecting hand 3D information;

[0037] A calibration processing module, configured to calibrate each TOF sensor to obtain an intrinsic matrix and a lens distortion parameter of each TOF sensor and a relative position transformation matrix between the plurality of TOF sensors;

[0038] An image acquisition module, configured to acquire infrared (IR) images and depth images in each view in the gesture interaction space in real time by each TOF sensor when working;

[0039] A hand key point coordinate detection and acquisition module, configured to detect and process the acquired IR images to obtain hand key point coordinates in each view;

[0040] A correction calculation processing module, configured to correct and calculate the hand key point coordinates in each view according to the intrinsic matrix and the lens distortion parameter obtained by calibration and in combination with the acquired depth images to obtain a hand key point 3D coordinate set in all views;

[0041] A calibration mapping processing module, configured to calibrate and map the hand key point 3D coordinate set in all views according to the relative position transformation matrix between the plurality of TOF sensors obtained by calibration to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with a wrist key point as a coordinate origin; and

[0042] A gesture recognition calculation processing module, configured to calculate the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin to obtain a hand feature map, and to perform gesture recognition processing on the obtained hand feature map to obtain a gesture recognition result.

[0043] As a third aspect of the application, a computer device for implementing the multi-view gesture recognition method comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0044] At least two TOF sensors for collecting 3D information of the hand are installed in the gesture interaction space, and each TOF sensor is calibrated to obtain the intrinsic matrix and lens distortion parameters of each TOF sensor and the relative position transformation matrix between the multiple TOF sensors;

[0045] In operation, each TOF sensor collects infrared IR images and depth images in each view of the gesture interaction space in real time;

[0046] The collected infrared IR images are detected to obtain the hand key point coordinates in each view;

[0047] The hand key point coordinates in each view are calculated and corrected according to the intrinsic matrix and lens distortion parameters obtained by calibration and in combination with the collected depth images, to obtain a hand key point 3D coordinate set in all views;

[0048] The hand key point 3D coordinate set in all views is calibrated and mapped according to the relative position transformation matrix between the multiple TOF sensors obtained by calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as the coordinate origin; and

[0049] The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated to obtain a hand feature map, and the hand feature map is subjected to gesture recognition processing to obtain a gesture recognition result.

[0050] As a fourth aspect of the application, a computer readable storage medium for implementing the multi-view gesture recognition method described above stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0051] At least two TOF sensors for collecting 3D information of the hand are installed in the gesture interaction space, and each TOF sensor is calibrated to obtain the intrinsic matrix and lens distortion parameters of each TOF sensor and the relative position transformation matrix between the multiple TOF sensors;

[0052] In operation, each TOF sensor collects infrared IR images and depth images in each view of the gesture interaction space in real time;

[0053] The collected infrared IR images are detected to obtain the hand key point coordinates in each view;

[0054] According to the intrinsic matrix and the lens distortion parameter obtained by calibration and in combination with the collected depth image, the hand key point coordinates under each view angle are corrected and calculated, and a hand key point 3D coordinate set under all view angles is obtained;

[0055] According to the relative position transformation matrix between the multiple TOF sensors obtained by calibration, the hand key point 3D coordinate set under all view angles is calibrated and mapped, and a hand key point 3D coordinate set under a three-dimensional coordinate system with the wrist key point as a coordinate origin is obtained; and

[0056] The hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed, a hand feature map is obtained, and gesture recognition processing is performed on the obtained hand feature map, so that a gesture recognition result is obtained.

[0057] Due to the adoption of the above technical solutions, the present application has the following beneficial effects:

[0058] 1. The present application reprojects the hand key point coordinates with the wrist key point as the 3D coordinate origin, which can restore the real state of the hand to the greatest extent, and through the extraction of the hand feature map, the extracted gesture feature has rotation and translation invariance, so that gestures in various postures can be recognized;

[0059] 2. Compared with the traditional single or dual binocular RGB sensor, the present application can obtain a depth map and an IR map through one frame of data, has small calculation amount and high real-time performance; at the same time, the present application adopts an active light source, has less interference from ambient light and can be used under complex ambient light conditions, and has low hardware requirement and is stable and reliable;

[0060] 3. Compared with the generally adopted single view angle scheme, the present application adopts a multi-view angle scheme, can obtain more comprehensive state information of the hand, and fully utilizes the technical characteristics of the TOF to restore the geometric relationship of the fingers of the hand in space, can more accurately recognize gestures and has better robustness in the face of shielding and other scenes. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0062] Figure 1 is a flow chart of the multi-view angle gesture recognition method of the present application.

[0063] Figure 2 is a structural schematic diagram of a multi-view gesture recognition device of the present application.

[0064] Figure 3 is an internal structure diagram of a computer device of the present application. DETAILED DESCRIPTION

[0065] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the present application is further described below in combination with specific drawings.

[0066] Referring to Figure 1 , the multi-view gesture recognition method of the present application is shown in the figure and includes the following steps:

[0067] Step S10, two TOF sensors for collecting 3D information of a hand are installed in a gesture interaction space, and each TOF sensor is calibrated to obtain an intrinsic matrix I_M and a lens distortion parameter D_M of each TOF sensor and a relative position transformation matrix T_M between the multiple TOF sensors. In the present embodiment, the number of TOF sensors is not limited to the number in the present embodiment, and should be set according to the gesture interaction space and the gesture recognition accuracy. When installing, the gesture interaction space should be covered as comprehensively as possible under the condition of ensuring that part of the fields of view coincide.

[0068] Step S20, during work, an infrared IR image IR_M and a depth image DEP_M under each view in the gesture interaction space are collected in real time by each TOF sensor.

[0069] Step S30, the collected infrared IR image IR_M is detected to obtain hand key point coordinates under each view.

[0070] Step S40, the hand key point coordinates under each view are calculated and processed for correction according to the intrinsic matrix I_M and the lens distortion parameter D_M obtained by calibration and in combination with the collected depth image DEP_M, to obtain a hand key point 3D coordinate set under all views.

[0071] Step S50, the hand key point 3D coordinate set under all views is processed for calibration mapping according to the relative position transformation matrix T_M between the multiple TOF sensors obtained by calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with a wrist key point as a coordinate origin.

[0072] Step S60, the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed to obtain a hand feature map, and the obtained hand feature map is processed for gesture recognition to obtain a gesture recognition result.

[0073] In step S30, the captured infrared IR image IR_M is detected to obtain the hand key point coordinates under each view, including the following steps:

[0074] In step S31, the captured infrared IR image IR_M is detected in real time by the hand detection model HD_MODEL based on the IR image; wherein the hand detection model HD_MODEL based on the IR image is a conventional detection model in the art, using a general detector model-yolov5s framework, which first manually labels the hand positions in a large number of IR images under different views to obtain the labeling information, and then sends it to the model for training to obtain HD_MODEL.

[0075] In step S32, when the human hand in the infrared IR image IR_M is detected, the hand region in the captured infrared IR image IR_M is cropped to obtain the local IR image HD_ROI of the human hand;

[0076] In step S33, the cropped local IR image HD_ROI of the human hand is sent to the hand key point detection model KPD_MODEL based on the IR image for detection to obtain the hand key point coordinates under each view. Wherein the hand key point detection model KPD_MODEL based on the IR image is a conventional detection model in the art, using a general detector model-yolov5s framework, which will be cropped and processed after the head ROI region detected by HD_MODEL is cropped, and 21 key points of the hand are labeled to obtain the labeling information of the key points, and then sent to the model for training to obtain the hand key point detection model KPD_MODEL.

[0077] In step S40, the hand key point coordinates under each view are calculated and processed according to the internal parameter matrix I_M and the lens distortion parameter D_M obtained by calibration and combined with the captured depth image DEP_M to obtain the hand key point 3D coordinate set under all views, including the following steps:

[0078] In step S41, the hand key point coordinates under each view and the captured depth image DEP_M are corrected according to the internal parameter matrix I_M and the lens distortion parameter D_M obtained by calibration;

[0079] In step S42, the corrected hand key point coordinates are remapped to the corrected depth image DEP_M to obtain the two-dimensional coordinates and depth data of each hand key point under each view;

[0080] In step S43, the two-dimensional coordinates and depth data of each hand key point under each view are calculated and processed according to the internal parameter matrix I_M obtained by calibration to obtain the hand key point 3D coordinates under each view.

[0081] Step S44, the hand key point 3D coordinates under each view are collected to obtain the hand key point 3D coordinate set under all views.

[0082] In step S50, the hand key point 3D coordinate set under all views is calibrated and mapped according to the relative position transformation matrix T_M between the multiple TOF sensors obtained by calibration to obtain the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin, including the following steps:

[0083] Step S51, the hand key point 3D coordinate set under all views is fused and calibrated according to the relative position transformation matrix T_M between the multiple TOF sensors obtained by calibration;

[0084] Step S52, the hand key point 3D coordinate set under all views after the fusion and calibration is remapped into the three-dimensional coordinate system with the wrist key point as the coordinate origin to obtain the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin.

[0085] In step S60, the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated to obtain the hand feature map, including the following steps:

[0086] Step S61, the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is normalized to obtain the normalized hand key point 3D coordinate set;

[0087] Step S62, the Euclidean distance between each hand key point after normalization is calculated to generate a normalized distance map DISTANCE_MAP between multiple hand key points; specifically, the coordinates between the key points are calculated through the 21 key point coordinates in the coordinate system with the wrist key point as the coordinate origin obtained in step S61, and the maximum distance is normalized to obtain a feature map DISTANCE_MAP with distance as the characteristic value, which is 21*21 in size;

[0088] Step S63, a node connection map F_DISTANCE_MAP between multiple hand key points and the wrist is generated by selecting the most distant point from the wrist key point from the adjacent key nodes according to each key point; specifically, a 21*21 distance matrix is generated through the 21 key point coordinates in the coordinate system with the wrist key point as the coordinate origin obtained in step S61, and the most distant point from the wrist is selected as the center of each key point, and the value of the distance matrix is assigned as 1 and the other values are assigned as 0 to generate the node connection map F_DISTANCE_MAP;

[0089] In step S64, a plurality of node connection graphs N_DISTANCE_MAP in which the hand points are closer to the wrist point are generated according to the selection of the adjacent points closest to the wrist point of each key point. Specifically, 21 key point coordinates in the coordinate system with the wrist key point as the coordinate origin are obtained through step S61, a 21*21 distance matrix is generated, and the points closest to the wrist point are selected from the adjacent key nodes with each key point as the center. The value of the distance matrix is assigned as 1, and the other values are assigned as 0, and the node connection graph N_DISTANCE_MAP is generated.

[0090] In step S65, the normalized distance graph DISTANCE_MAP between the plurality of hand key points, the node connection graph F_DISTANCE_MAP in which the plurality of hand key points are farther from the wrist, and the node connection graph N_DISTANCE_MAP in which the plurality of hand key points are closer to the wrist are synthesized to obtain a hand feature map.

[0091] The wrist key point is used as the 3D coordinate origin for the re-projection of the hand key point coordinates, which can maximize the restoration of the true state of the hand, and the feature map of the hand is extracted, so that the extracted gesture feature has rotation and translation invariance, thereby enabling the recognition of gestures in various postures.

[0092] In step S60, the hand gesture recognition model KPGR_MODEL based on the 3D coordinate information of the hand key points is used to fuse the feature map to recognize the gesture of the obtained hand feature map, and a gesture recognition result is generated. The gesture recognition model KPGR_MODEL based on the 3D coordinate information of the hand key points is a conventional recognition model in the art, which is realized based on the ResNet18 framework model. The labeled multi-gesture hand feature map after fusion is input into the network for training to obtain the gesture recognition result.

[0093] Referring to Figure 2 , the multi-view gesture recognition device of the present application is shown in the figure, which includes two TOF sensors 110, a calibration processing module 120, an image acquisition module 130, a hand key point coordinate detection and acquisition module 140, a correction calculation processing module 150, a calibration mapping processing module 160, and a gesture recognition calculation processing module 170.

[0094] The two TOF sensors 110 are installed in the gesture interaction space and are used to collect hand 3D information. In this embodiment, the number of TOF sensors 110 is not limited to the number in this embodiment, and should be set according to the gesture interaction space and the gesture recognition accuracy. When installed, the gesture interaction space should be covered as comprehensively as possible under the condition that part of the field of view is overlapped.

[0095] The calibration processing module 160 is configured to perform calibration mapping processing on the hand key point 3D coordinate set in all views according to the relative position transformation matrix T_M between the plurality of TOF sensors obtained through calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as a coordinate origin.

[0096] The image acquisition module 130 is configured to acquire infrared IR images IR_M and depth images DEP_M in each view in the gesture interaction space in real time through each TOF sensor 110 in operation.

[0097] The hand key point coordinate detection and acquisition module 140 is configured to perform detection processing on the acquired infrared IR images IR_M, to obtain hand key point coordinates in each view.

[0098] The correction calculation processing module 150 is configured to perform correction calculation processing on the hand key point coordinates in each view according to the intrinsic matrix I_M and the lens distortion parameters D_M obtained through calibration and in combination with the acquired depth images DEP_M, to obtain a hand key point 3D coordinate set in all views.

[0099] The calibration mapping processing module 160 is configured to perform calibration mapping processing on the hand key point 3D coordinate set in all views according to the relative position transformation matrix T_M between the plurality of TOF sensors obtained through calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as a coordinate origin.

[0100] The gesture recognition calculation processing module 170 is configured to perform calculation processing on the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as a coordinate origin, to obtain a hand feature map, and perform gesture recognition processing on the obtained hand feature map, to obtain a gesture recognition result.

[0101] The various modules in the multi-view gesture recognition device can be realized by software, hardware and combinations thereof in whole or in part. The various modules described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the various modules.

[0102] The application further provides a computer device for implementing the multi-view gesture recognition method, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 3The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store user information, record information and files. The network interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the multi-view gesture recognition method.

[0103] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the technical solution, and does not constitute a limitation on the computer device to which the technical solution is applied. The specific computer device can include more or fewer components than those shown in the figure, or some components can be combined, or have a different component arrangement.

[0104] Specifically, the computer device of the present application includes a memory and a processor. The memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0105] Step S10, two TOF sensors for collecting 3D information of the hand are installed in the gesture interaction space, and each TOF sensor is calibrated to obtain an intrinsic matrix I_M and a lens distortion parameter D_M of each TOF sensor and a relative position transformation matrix T_M between the multiple TOF sensors;

[0106] Step S20, during operation, the infrared IR image IR_M and the depth image DEP_M under each view in the gesture interaction space are collected in real time by each TOF sensor;

[0107] Step S30, the collected infrared IR image IR_M is detected to obtain the hand key point coordinates under each view;

[0108] Step S40, the hand key point coordinates under each view are corrected and calculated according to the intrinsic matrix I_M and the lens distortion parameter D_M obtained by calibration and in combination with the collected depth image DEP_M, to obtain a hand key point 3D coordinate set under all views;

[0109] Step S50, the hand key point 3D coordinate set under all views is calibrated and mapped according to the relative position transformation matrix T_M between the multiple TOF sensors obtained by calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as the coordinate origin.

[0110] Step S60, the hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed to obtain a hand feature map, and the hand gesture recognition result is obtained by performing hand gesture recognition processing on the obtained hand feature map.

[0111] The application further provides a computer readable storage medium for implementing the multi-view gesture recognition method, and a computer program is stored on the computer readable storage medium.

[0112] Step S10, two TOF sensors for collecting hand 3D information are installed in the gesture interaction space, and each TOF sensor is calibrated to obtain the intrinsic matrix I_M and the lens distortion parameter D_M of each TOF sensor and the relative position transformation matrix T_M between the multiple TOF sensors.

[0113] Step S20, during work, the infrared IR image IR_M and the depth image DEP_M under each view in the gesture interaction space are collected in real time by each TOF sensor.

[0114] Step S30, the collected infrared IR image IR_M is detected to obtain the hand key point coordinates under each view.

[0115] Step S40, the hand key point coordinates under each view are corrected and calculated according to the intrinsic matrix I_M and the lens distortion parameter D_M obtained by calibration and in combination with the collected depth image DEP_M to obtain the hand key point 3D coordinate set under all views.

[0116] Step S50, the hand key point 3D coordinate set under all views is calibrated and mapped according to the relative position transformation matrix T_M between the multiple TOF sensors obtained by calibration to obtain the hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin.

[0117] Step S60, the hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed to obtain a hand feature map, and the hand gesture recognition result is obtained by performing hand gesture recognition processing on the obtained hand feature map.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The basic principles and main features of the present application and the advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-view gesture recognition method, characterized in that, The application relates to a gesture recognition method and device. At least two TOF sensors for collecting 3D information of a hand are installed in a gesture interaction space, and each TOF sensor is calibrated to obtain an intrinsic matrix and a lens distortion parameter of each TOF sensor and a relative position transformation matrix among the TOF sensors; During operation, infrared IR images and depth images in each view angle in the gesture interaction space are collected by each TOF sensor in real time; The collected IR images are detected to obtain hand key point coordinates in each view angle; The hand key point coordinates in each view angle are corrected and calculated according to the intrinsic matrix and the lens distortion parameter obtained through calibration and in combination with the collected depth images, so that a hand key point 3D coordinate set in all view angles is obtained; the hand key point coordinates in each view angle and the collected depth images are corrected according to the intrinsic matrix and the lens distortion matrix obtained through calibration; the corrected hand key point coordinates are remapped to the corrected depth images, so that two-dimensional coordinates and depth data of each hand key point in each view angle are obtained; the two-dimensional coordinates and the depth data of each hand key point in each view angle are calculated according to the intrinsic matrix, so that hand key point 3D coordinates in each view angle are obtained; and the hand key point 3D coordinates in each view angle are collected, so that the hand key point 3D coordinate set in all view angles is obtained; The hand key point 3D coordinate set in all view angles is calibrated and mapped according to the relative position transformation matrix among the TOF sensors obtained through calibration, so that a hand key point 3D coordinate set in a three-dimensional coordinate system with a wrist key point as a coordinate origin is obtained; and The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated, so that a hand feature map is obtained, and gesture recognition is performed on the hand feature map, so that a gesture recognition result is obtained. 2.The multi-view gesture recognition method of claim 1, wherein, The collected IR images are detected to obtain hand key point coordinates in each view angle, including: The collected IR images are detected in real time by an IR image-based hand detection model; When a human hand is detected in the IR images, a hand region in the collected IR images is cut, so that a local IR image of the human hand is obtained; and The cut local IR image of the human hand is input into an IR image-based hand key point detection model for detection, so that hand key point coordinates in each view angle are obtained. 3.The multi-view gesture recognition method of claim 1, wherein, The hand key point 3D coordinate set in all view angles is calibrated and mapped according to the relative position transformation matrix among the TOF sensors obtained through calibration, so that a hand key point 3D coordinate set in a three-dimensional coordinate system with a wrist key point as a coordinate origin is obtained, including: The hand key point 3D coordinate set in all view angles is fused and calibrated according to the relative position transformation matrix among the TOF sensors obtained through calibration; and Map the hand key point 3D coordinate set after the fusion calibration processing to a three-dimensional coordinate system with the wrist key point as the coordinate origin, to obtain a hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin. 4.The multi-view gesture recognition method of claim 1, wherein, The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is calculated and processed to obtain a hand feature map, including: The hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin is normalized to obtain a normalized hand key point 3D coordinate set; The Euclidean distance between each hand key point after normalization is calculated to generate a plurality of hand key point normalized distance maps; According to each key point, the most distant point from the distance wrist key point is connected to generate a plurality of hand key point distance wrist far node connection maps; According to each key point, the nearest point from the distance wrist key point is connected to generate a plurality of hand key point distance wrist near node connection maps; The plurality of hand key point normalized distance maps, the plurality of hand key point distance wrist far node connection maps, and the plurality of hand key point distance wrist near node connection maps are synthesized to obtain a hand feature map. 5.The multi-view gesture recognition method of claim 1, wherein, The hand gesture recognition model based on the hand key point 3D coordinate information fusion feature map is used to recognize the hand feature map to generate a hand gesture recognition result.

6. A multi-view gesture recognition apparatus, characterized by comprising: It includes: At least two TOF sensors installed in the gesture interaction space for collecting hand 3D information; A calibration processing module, the calibration processing module is used for calibrating each TOF sensor to obtain the intrinsic matrix and lens distortion parameters of each TOF sensor and the relative position transformation matrix between the plurality of TOF sensors; An image acquisition module, the image acquisition module is used for acquiring infrared IR images and depth images in each view of the gesture interaction space in real time through each TOF sensor when working; A hand key point coordinate detection and acquisition module, the hand key point coordinate detection and acquisition module is used for detecting and processing the collected infrared IR images to obtain the hand key point coordinates in each view; A correction calculation processing module, the correction calculation processing module is used for correcting and calculating the hand key point coordinates in each view according to the intrinsic matrix and lens distortion parameters obtained by calibration and combining the collected depth images to obtain a hand key point 3D coordinate set in all views; including: correcting the hand key point coordinates in each view and the collected depth images according to the intrinsic matrix and lens distortion matrix obtained by calibration; remapping the corrected hand key point coordinates to the corrected depth images to obtain two-dimensional coordinates and depth data of each hand key point in each view; calculating the two-dimensional coordinates and depth data of each hand key point in each view according to the intrinsic matrix obtained by calibration to obtain the hand key point 3D coordinates in each view; and collecting the hand key point 3D coordinates in each view to obtain the hand key point 3D coordinate set in all views; a calibration mapping processing module, configured to perform calibration mapping processing on the hand key point 3D coordinate set in all views according to a relative position transformation matrix between the plurality of TOF sensors obtained through calibration, to obtain a hand key point 3D coordinate set in a three-dimensional coordinate system with the wrist key point as a coordinate origin; and a gesture recognition computing processing module, configured to perform computing processing on the hand key point 3D coordinate set in the three-dimensional coordinate system with the wrist key point as the coordinate origin, to obtain a hand feature map, and perform gesture recognition processing on the obtained hand feature map, to obtain a gesture recognition result. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the multi-view gesture recognition method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-view gesture recognition method in any one of claims 1 to 5.

Images