Gesture dataset generation method and client device

Abstract

The application relates to the technical field of virtual reality interaction, and provides a gesture dataset generation method and a client device, a backpack type gesture data acquisition device is built based on a depth camera and multiple industrial cameras, the whole process of generating a gesture dataset is realized from gesture data acquisition to gesture data labeling and then to gesture data verification, the coupling between the processes is reduced, the generation efficiency and quantity of the gesture dataset are improved, the whole generation process does not need the intervention of manual work and a third party, and the generation cost is saved; and in the data labeling process, a gesture estimation algorithm based on deep learning and a matching tracking algorithm based on a hand model are used to label 3D gestures, a sliding window is used to optimize the 3D gestures labeled by the two algorithms, so that more accurate 3D gesture labeling results are obtained, the automatic labeling of 3D gestures is realized, and the generation efficiency and quality of the gesture dataset are further improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality interaction technology, and provides a method for generating gesture datasets and a client device. Background Technology

[0002] With the arrival of the metaverse era, human society has entered a stage of rapid development of virtualization. The development of virtualization is inseparable from Virtual Reality (VR) and Augmented Reality (AR) technologies, and 3D gesture interaction is a crucial interaction method in VR and AR technologies.

[0003] Currently, the core technology of 3D gesture interaction is based on deep learning algorithms to detect hand joints. Since the essence of deep learning algorithms is to fully train on a dataset, the foundation for accurate and real-time 3D gesture interaction technology lies in providing a large number of high-quality gesture datasets with diverse application scenarios, rather than a small number of single or low-precision gesture datasets. This is because the quantity and quality of gesture data directly affect the accuracy and generalization of gesture joint estimation; therefore, the performance of a gesture project depends on the quality of the gesture dataset collected for its application scenarios. Furthermore, the development of a gesture project has time constraints, which also necessitates high efficiency in generating gesture datasets.

[0004] Therefore, generating a high-quality gesture dataset within a limited timeframe is of significant research value for 3D gesture interaction. Summary of the Invention

[0005] This application provides a method and client device for generating gesture datasets, which can improve the quality and generation efficiency of gesture datasets.

[0006] On one hand, embodiments of this application provide a method for generating a gesture dataset, including:

[0007] In response to a data acquisition command, a single-hand gesture image is acquired simultaneously by a depth camera and multiple industrial cameras. The single-hand gesture image acquired by the depth camera includes a gesture depth image and a gesture RGB image, while the single-hand gesture image acquired by the industrial cameras is a gesture RGB image.

[0008] In response to the data annotation instruction, based on the gesture depth image and gesture RGB image captured by the depth camera, the initial 3D coordinates of each hand joint are determined by a deep learning-based gesture estimation algorithm and a hand model-based matching and tracking algorithm, respectively; and for each hand joint, the target 3D coordinates of the hand joint are determined by the mean or variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms, respectively.

[0009] In response to the data verification instruction, the hand joint corresponding to each target 3D coordinate is projected onto a gesture RGB image collected by a corresponding industrial camera according to a calibration matrix between the depth camera and each industrial camera; and the gesture RGB images collected by each industrial camera after projection are screened to generate a gesture dataset for gesture interaction.

[0010] In another aspect, an embodiment of the present application provides a client device, comprising a processor, a memory, a display screen and a plurality of communication interfaces, the communication interfaces, the display screen, the memory and the processor being connected through a bus;

[0011] The display screen is used for displaying a user interface and an image, and the user interface is used for receiving an operation instruction.

[0012] The memory stores a computer program, and the processor executes the following operations according to the computer program:

[0013] In response to a data collection instruction, a single-hand gesture image collected by a depth camera and a plurality of industrial cameras within a set time period is acquired through the communication interfaces, the single-hand gesture image collected by the depth camera comprising a gesture depth image and a gesture RGB image;

[0014] In response to a data labeling instruction, initial 3D coordinates of each hand joint are determined according to the gesture depth image and the gesture RGB image collected by the depth camera by using a gesture estimation algorithm based on deep learning and a matching tracking algorithm based on a hand model respectively; and for each hand joint, a target 3D coordinate of the hand joint is determined according to a mean value or a variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms respectively.

[0015] In response to the data verification instruction, the hand joint corresponding to each target 3D coordinate is projected onto a gesture RGB image collected by a corresponding industrial camera according to a calibration matrix between the depth camera and each industrial camera; and the gesture RGB images collected by each industrial camera after projection are screened to generate a gesture dataset for gesture interaction.

[0016] In another aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer executable instructions, the computer executable instructions being used for causing a computer device to execute the gesture dataset generation method provided by an embodiment of the present application.

[0017] The gesture dataset generation method and the client device provided by the embodiment of the application pass through the whole process from gesture data collection to gesture data labeling and then to gesture data verification to generate a gesture dataset, realize pipeline operation, improve the generation efficiency and quantity of the gesture dataset, and save the generation cost without the intervention of manual work and a third party. In the data collection process, the gesture images collected by one depth camera and multiple industrial cameras within a set time period are obtained by responding to a data collection instruction, the fast collection of multiple data streams is realized, and the efficiency is higher. In the data labeling process, the gesture depth images and the gesture RGB images collected by the depth camera are processed by using a gesture estimation algorithm based on deep learning and a matching tracking algorithm based on a hand model respectively by responding to a data labeling instruction, the initial 3D coordinates of each hand joint are determined, the initial 3D coordinates determined by the two algorithms are optimized, the target 3D coordinates are obtained, the target 3D coordinates of each hand joint are determined by the two algorithms, the limitation of a single algorithm is overcome, the accuracy of 3D gesture labeling is improved, and then the quality of the gesture dataset is improved. In the data verification process, the hand joints corresponding to the target 3D coordinates are projected onto the gesture RGB images collected by the corresponding industrial cameras according to the calibration matrix between the depth camera and each industrial camera by using a data verification instruction, and the quality of the gesture dataset is further improved after data screening. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1A An effect picture of restoring a 3D gesture from an image provided by the embodiment of the application;

[0020] Figure 1A Another effect picture of restoring a 3D gesture from an image provided by the embodiment of the application;

[0021] Figure 2 A schematic diagram of a fixed gesture data collection device provided by the embodiment of the application;

[0022] Figure 3 A schematic diagram of a backpack type gesture data collection device provided by the embodiment of the application;

[0023] Figure 4 A schematic diagram of a virtual display device provided by the embodiment of the application;

[0024] Figure 5 Architecture diagram of gesture dataset development software provided for an embodiment of the present application;

[0025] Figure 6A Gesture data collection interface provided for an embodiment of the present application;

[0026] Figure 6B Gesture data annotation interface provided for an embodiment of the present application;

[0027] Figure 6C Gesture data verification interface provided for an embodiment of the present application;

[0028] Figure 7 Overall architecture diagram of generating gesture dataset provided for an embodiment of the present application;

[0029] Figure 8 Timing flowchart of three modules of gesture data collection, annotation and verification provided for an embodiment of the present application;

[0030] Figure 9 Flowchart of gesture dataset generation method provided for an embodiment of the present application;

[0031] Figure 10 Flowchart of 3D gesture annotation based on gesture estimation algorithm of deep learning provided for an embodiment of the present application;

[0032] Figure 11 Structure diagram of deep learning model based on Hourglass network provided for an embodiment of the present application;

[0033] Figure 12 Kinematic structure schematic diagram of 26-degree-of-freedom left hand model provided for an embodiment of the present application;

[0034] Figure 13 Flowchart of 3D gesture annotation based on matching tracking algorithm of hand model provided for an embodiment of the present application; Figure 14 Framework structure diagram of matching tracking algorithm of hand model provided for an embodiment of the present application;

[0035] Figure 15 Principle diagram of sliding window provided for an embodiment of the present application;

[0036] Figure 16 Method flowchart of optimal fusion of 3D coordinates of each hand joint annotated by two algorithms provided for an embodiment of the present application;

[0037] Figure 17 Client device structure diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0038] To clearly describe the embodiments of the present application, the following gives an explanatory description of the terms of the present application.

[0039] Depth camera: contains an RGB sensor and a depth sensor, which can synchronously collect RGB images and depth images. Generally, it does not have a hardware synchronization interface.

[0040] Industrial camera: contains an RGB sensor, which can collect RGB images, and has a hardware synchronization interface.

[0041] 3D gesture estimation: refers to the estimation of three-dimensional coordinates representing the position of hand joint points using RGB images or depth images or both.

[0042] The design idea of the embodiments of the present application is summarized as follows.

[0043] 3D gesture interaction is a new type of human-computer interaction. Accurate and stable 3D gesture estimation determines the user experience of AR / VR products. 3D gesture estimation is mainly realized based on deep learning. Since the performance of deep learning depends on the quantity and quality of 3D gesture datasets, providing a large number of high-quality and different application scenario gesture datasets for deep learning training is a key prerequisite to ensure good performance of deep learning, which can improve the accuracy and generalization of 3D gesture estimation.

[0044] Due to the difficulty and complexity of 3D gesture data acquisition, there is still a large gap between the currently public 3D gesture datasets and real application scenarios in China and internationally. Neither in terms of quality nor in terms of quantity is it conducive to the training of deep learning for 3D gesture estimation, which seriously affects the application of 3D gesture interaction in AR / VR products.

[0045] However, there are still many problems in the generation of gesture datasets. For example, the problems existing in the hardware level are: the acquisition device is heavy and has poor mobility, the acquisition scene is single, and the acquisition data quality is poor; the problems existing in the software level are: the annotation process is tedious, the data precision is low, and the generation efficiency is low. In this way, the deep learning model trained based on a small amount of low-quality and single-scene 3D gesture dataset will reduce the effect of 3D gesture interaction when performing 3D gesture estimation.

[0046] For example, in the process of virtual meeting interaction, when the VR device incorrectly recognizes gestures or does not recognize gestures synchronously based on the images collected by the multi-camera, it will cause the VR device to be unable to accurately restore the hand movements of the interactive object. It may appear that for a hand image collected from a certain perspective, the VR device can accurately restore the 3D gesture, as shown in (a) of Figure 1A , but for a hand image collected from another perspective, the VR device cannot accurately restore the 3D gesture, as shown in (b) of Figure 1A , fromFigure 1A It can be clearly seen that the user's hand image can accurately reproduce the gesture in the viewpoint of (a). However, in the viewpoint of (b), the reproduced gesture deviates from the position of the user's hand. This is due to the problem caused by the multi-view camera failing to expose synchronously. This situation will seriously damage the user's VR experience and may even cause problems such as the inability to trigger virtual items or reproduce the user's actual hand movements in virtual meeting scenarios.

[0047] For example, based on hand images captured from multiple perspectives, the reconstructed 3D gestures can accurately reflect the true position of the hand, such as... Figure 1B As shown in (c), the lack of hand images captured from multiple perspectives can lead to a certain deviation between the reconstructed 3D gesture and the actual hand position, resulting in a poor user experience. Figure 1B As shown in (d) in the figure.

[0048] like Figure 1A (a) and Figure 1B As shown in (c), a realistic and natural immersive experience will only be achieved when the reconstructed 3D gestures fit the actual hand position.

[0049] To ensure that deep learning models can accurately and in real-time estimate 3D gestures in virtual reality interactive scenarios, it is necessary to improve the accuracy of the network parameters they learn, which requires providing deep learning models with high-precision, high-quality gesture datasets for training.

[0050] In view of this, this application provides a gesture dataset generation method and client device, which greatly promotes the advancement of 3D gesture estimation algorithms and the development of AR / VR technology. On the one hand, it streamlines the entire process from gesture data acquisition to gesture data annotation and then to gesture data verification to generate a gesture dataset, reducing the coupling between processes and realizing a factory-like assembly line operation for gesture data streams. This improves the generation efficiency and quantity of gesture datasets, and the entire generation process does not require manual or third-party intervention, saving generation costs. On the other hand, in the data annotation process, two algorithms are used to annotate 3D gestures: a deep learning-based gesture estimation algorithm and a hand model-based matching and tracking algorithm. A sliding window is used to calculate the mean and variance of the 3D gesture annotations of each algorithm in the continuous data stream. The 3D gestures are optimized based on the mean and variance of the two algorithms to obtain more accurate 3D gesture annotation results, realizing automated 3D gesture annotation and further improving the generation efficiency and quality of gesture datasets.

[0051] Meanwhile, the embodiment of the present application improves the structure of the gesture data set collection device, and designs a backpack type collection device architecture. One person can conveniently and portably move the collection device to various application scenarios for gesture data collection, and the collection method is more flexible and diverse.

[0052] The embodiments of the present application will be described in detail below with reference to the drawings.

[0053] Referring to Figure 2 The fixed gesture data collection device provided by the embodiment of the present application is shown in FIG. 1. Figure 2 The collection device contains multiple cameras, which are arranged around the collection frame and face the inside of the collection frame. The collection device ensures the diversity of the gesture data set through multi-view capture, but due to the bulkiness of the collection device, multiple people are needed to move it, and the angles of the cameras are difficult to adjust, resulting in low efficiency and quality of the captured gesture data, increasing the difficulty of capturing gesture data in various scenes and different lighting environments, and making it difficult to meet the requirements of 3D gesture projects for various application scenario data.

[0054] Based on the shortcomings of the existing gesture data collection device, the embodiment of the present application improves the structure of the collection device and builds a portable backpack type collection device, which can efficiently generate a large amount of high-quality gesture data.

[0055] As shown in FIG. 2, the backpack type gesture data collection device provided by the embodiment of the present application is shown in FIG. 2. Figure 3 The collection device includes a depth camera and multiple industrial cameras, and the depth camera and the industrial cameras are deployed in a backpack type architecture. Considering that 3D gesture estimation is mainly applied in AR, VR and other virtual scenarios, and AR, VR and other virtual display devices are generally worn on the head of the user, and multiple cameras are installed on the virtual display device, the same row of cameras are aligned left and right, and the lowermost row of cameras is slightly tilted downward, as shown in FIG. 3. Figure 4 Therefore, the angles of the depth camera and the industrial cameras used for gesture data collection can be set according to the angles of the downward tilted binocular cameras on the AR, VR and other virtual display devices, so that the collected gesture data is more suitable for AR, VR and other virtual scenarios.

[0056] The backpack type gesture data collection device built by the embodiment of the present application is convenient to carry, and one person can easily carry the collection device to various scenes and lighting environments. The moving process frees the hands, and gesture data collection can be performed anytime and anywhere, thereby solving the problem of single collection scene of the fixed gesture collection device. Meanwhile, while ensuring portability, the synchronous collection of multiple gesture data streams is still guaranteed, and in combination with a portable notebook computer with three usb3.0 ports and a graphics card, a large amount of high-quality gesture data sets in different scenes can be generated.

[0057] In some embodiments, Figure 3 In the backpack-type gesture data acquisition device shown, the camera lens can be replaced according to requirements without changing the structure of the acquisition device, so as to meet the requirements of different field angles and focal lengths and other parameters, and realize the rapid completion of gesture algorithm transplantation optimization work on other products using different models of cameras.

[0058] It should be noted that, Figure 3 In the backpack-type gesture data acquisition device shown, the number of industrial cameras is not limited. For example, 6 industrial cameras and one depth camera are used in the embodiment of the application to realize the synchronous acquisition of 7-camera 8-way gesture data (one depth data and 7 RGB data).

[0059] At present, the development software used to generate gesture data sets usually only has a data acquisition interface and corresponding acquisition functions. If a complete gesture data set is to be obtained, the collected data needs to be labeled and verified by a third party and manually, and the generation cost and efficiency are low.

[0060] To solve the above problems, the embodiment of the application develops a software for generating gesture data sets, which integrates a gesture data acquisition interface, a gesture data labeling interface and a gesture data verification interface. Each interface realizes corresponding functions, and connects the whole process of generating gesture data sets, solves the coupling between data acquisition, data labeling and data verification, can efficiently and automatically label the collected gesture data, and generate a large number of high-precision gesture data sets.

[0061] As Figure 5 As shown in the architectural diagram of the gesture data set development software provided by the embodiment of the application, the software includes three interfaces, namely a gesture data acquisition interface, a gesture data labeling interface and a gesture data verification interface. In the gesture data acquisition interface, the main functions realized include multi-camera synchronization, multi-camera calibration, gesture data acquisition and gesture data display. In the gesture data labeling interface, the main functions realized include gesture data display, gesture data labeling, gesture data encryption and gesture data export. In the gesture data verification interface, the main functions realized include gesture data display, gesture data projection, gesture data deletion and gesture data rollback.

[0062] The gesture data acquisition interface is as shown in Figure 6AAs shown, before gesture data acquisition, the depth camera and multiple industrial cameras can be synchronized and calibrated; during gesture data acquisition, the start and end function keys are used to control the multiple cameras (including the depth camera and industrial cameras) to start and end the acquisition of multi-channel gesture data simultaneously, and auxiliary information (such as user ID, gesture category, gesture serial number, etc.) can also be input during gesture data acquisition to facilitate the management of gesture data collected by multiple cameras. Among them, the user ID can distinguish the gesture data of different users, the gesture category can distinguish the gesture state (such as five fingers open, fist, etc.), and the gesture serial number can get the number of gestures.

[0063] As shown in the gesture data annotation interface Figure 6B , single-frame 3D gesture annotation can be performed on the single-hand gesture images (including gesture depth images and gesture RGB images) collected by the depth camera, and batch 3D gesture annotation can also be performed by inputting the start frame and end frame of the single-hand gesture images collected by the depth camera. During annotation, you can choose (OPT) whether to annotate the left hand or the right hand. After 3D gesture annotation is completed, the annotation results can also be encrypted and exported.

[0064] As shown in the gesture data verification interface Figure 6C , since deep learning training generally uses 2D images, after obtaining the annotated 3D gesture, it is projected into the gesture RGB images collected by multiple industrial cameras, thereby obtaining a large amount of high-precision gesture data sets for deep learning training. Among them, the gesture data verification interface can perform batch verification on the results of 3D gesture projection by inputting the start frame and end frame. If the verification result is biased, it will be deleted in batches. At the same time, single-frame verification can be performed on the results of 3D gesture projection by inputting the jump frame. If the verification result is biased, the frame will be deleted. At the same time, the verification process can also display the deleted gesture RGB images. In order to avoid accidental deletion, the deleted gesture RGB images can be stored in the recycle bin, and the deleted images can be restored by reverting.

[0065] As shown in the interface Figure 6A to Figure 6C , each interface contains gesture data display function, which can display single-hand gesture images collected by different cameras in real time on the same interface.

[0066] When using the above software to generate gesture data sets, the application embodiment adopts multi-process technology to avoid data acquisition and data processing process jam, and allocates two processes for multi-camera gesture data acquisition and interface function implementation. The processes interact with each other through shared memory.

[0067] The software integrating the gesture data collection interface, the gesture data labeling interface and the gesture data verification interface is installed on a client device, can jointly act on the underlying data stream, and realizes the full process of automatic collection, labeling and verification.

[0068] The software in the above embodiments of the application can be compatible with various systems (such as Windows, Mac and Linux, etc.), and thus can be installed on various client devices (such as notebook computers, desktop computers, tablets, mobile phones, etc.), combined with the above backpack type gesture data collection device, and can punch through the full process of gesture data set generation, realize pipeline operation of gesture data stream, and generate multi-scene, large batch and high-precision gesture data sets through automatic labeling and data verification.

[0069] Referring to Figure 7 The overall architecture diagram for generating gesture data sets provided by the embodiments of the application mainly includes an acquisition device building module, a data collection module, a data labeling module and a data verification module, and the data collection module, the data labeling module and the data verification module can access the same gesture database.

[0070] In the data collection module, the depth camera and the plurality of industrial cameras included in the acquisition device are calibrated to obtain a calibration matrix between the cameras, and the depth camera and the plurality of industrial cameras included in the acquisition device are synchronously exposed, if the synchronization is successful, the depth camera and the plurality of industrial cameras synchronously collect single-hand gesture images, and the collected single-hand gesture images are stored in the gesture database for use by the data labeling module and the data verification module.

[0071] In the data labeling module, the single-hand gesture images collected by the depth camera are obtained from the gesture database, 3D gesture estimation is performed using a plurality of algorithms to obtain 3D coordinates of hand joint points, and the 3D coordinates are stored in the gesture database for use by the data verification module.

[0072] In the data verification module, based on the calibration matrix between the cameras, the 3D coordinates labeled according to the single-hand gesture images collected by the depth camera are projected into the single-hand gesture images synchronously collected by the plurality of industrial cameras, and the single-hand gesture images collected by the plurality of industrial cameras are screened according to the projection results to generate a high-precision gesture data set, which is stored in the gesture database.

[0073] Figure 7 The timing flowchart of each module is shown in Figure 8 The user issues an operation instruction through a user interface to control the depth camera and the plurality of industrial cameras in the acquisition device to collect images matching the current operation instruction, and the corresponding images are processed by a bottom algorithm to generate a gesture data set.

[0074] In the embodiment of the present application, before collecting gesture data, the depth camera and the plurality of industrial cameras included in the collection device need to be operated synchronously. The user sends a synchronization instruction to the client device installed with the software by clicking the "synchronization" option in the gesture data collection interface shown in Figure 6A After receiving the synchronization instruction, the client device sets the depth camera and the plurality of industrial cameras to be synchronous. In the specific implementation, the exposure parameters of the depth camera are obtained through the LED lamp image, and the exposure parameters of the depth camera are set to the plurality of industrial cameras through the hardware synchronization interface of the plurality of industrial cameras by the single-chip microcomputer or the like controller, so as to realize the synchronization between the depth camera and the plurality of industrial cameras.

[0075] In the embodiment of the present application, before collecting gesture data, the depth camera and the plurality of industrial cameras included in the collection device also need to be calibrated. In the specific implementation, the user controls the depth camera and the plurality of industrial cameras to synchronously collect calibration images by clicking the "collect calibration images" option in the gesture data collection interface shown in Figure 6A After the calibration image collection is completed, the user clicks the "generate calibration file" option in the gesture data collection interface shown in Figure 6A After the calibration image collection is completed, the user clicks the "generate calibration file" option in the gesture data collection interface shown in

[0076] After the synchronization and calibration of the depth camera and the plurality of industrial cameras in the portable and mobile backpack-type gesture data collection device are completed, in combination with the client device installed with the above software, a large number of high-precision gesture data sets in multiple scenes can be efficiently generated.

[0077] The method flow of generating the gesture data set is shown in Figure 9 The method flow of generating the gesture data set is shown in

[0078] S901: In response to a data collection instruction, a single-hand gesture image synchronously collected by a depth camera and a plurality of industrial cameras is acquired.

[0079] An alternative embodiment is that when the user clicks the "start" option in the gesture data collection interface shown in Figure 6A When the user clicks the "end" option in the gesture data collection interface shown in Figure 6A When the user clicks the "end" option in the gesture data collection interface shown in

[0080] Another optional implementation is that when S901 is performed, the user inputs a collection duration in the gesture data collection interface shown in FIG. 8A, and then clicks the "Start" option in the gesture data collection interface shown in FIG. 8A to send a data collection instruction to the client device. The client device controls the depth camera and the multiple industrial cameras in the collection device to synchronously collect single-hand gesture images according to the received data collection instruction. When the input collection duration is reached, the client automatically controls the depth camera and the multiple industrial cameras in the collection device to stop collecting single-hand gesture images. Figure 6A Figure 6A The depth camera includes a depth sensor and an RGB sensor, so the single-hand gesture images collected by the depth camera include a gesture depth image and a gesture RGB image. The industrial camera only includes an RGB sensor, so the single-hand gesture images collected by the industrial camera are gesture RGB images.

[0081] S902: In response to the data labeling instruction, the initial 3D coordinates of each hand joint are determined according to the gesture depth image and the gesture RGB image collected by the depth camera by using a gesture estimation algorithm based on deep learning and a matching tracking algorithm based on a hand model respectively; and for each hand joint, the target 3D coordinates of the hand joint are determined according to the mean or variance of the initial 3D coordinates of the hand joint in a set number of frames determined by the two algorithms respectively.

[0082] In the embodiments of the present application, in order to realize automatic labeling of the collected single-hand gesture images, reduce labeling cost and improve labeling quality, in S902, the gesture estimation algorithm based on deep learning and the matching tracking algorithm based on the hand model are used to simultaneously perform 3D gesture labeling, two groups of 3D coordinates of hand joints are obtained, then the sliding window algorithm is used to respectively calculate the continuous mean and variance of the 3D coordinates of the hand joints in each group, and the continuous mean and variance of the 3D coordinates of the hand joints in each group are used for optimization and fusion, so as to realize automatic labeling of 3D gestures, and thus high-precision 3D coordinates of hand joints are obtained.

[0083] In the embodiments of the present application, in order to realize automatic labeling of the collected single-hand gesture images, reduce labeling cost and improve labeling quality, in S902, the gesture estimation algorithm based on deep learning and the matching tracking algorithm based on the hand model are used to simultaneously perform 3D gesture labeling, two groups of 3D coordinates of hand joints are obtained, then the sliding window algorithm is used to respectively calculate the continuous mean and variance of the 3D coordinates of the hand joints in each group, and the continuous mean and variance of the 3D coordinates of the hand joints in each group are used for optimization and fusion, so as to realize automatic labeling of 3D gestures, and thus high-precision 3D coordinates of hand joints are obtained.

[0084] The process of 3D gesture labeling by using the gesture estimation algorithm based on deep learning is described in detail in the following. Figure 10 The process mainly includes the following steps.

[0085] S9021: The 3D point cloud of each hand joint is determined according to the gesture depth image collected by the depth camera.

[0086] ​In S9021, after inputting the gesture depth image and the gesture RGB image collected by the depth camera into the trained deep learning model, joint point detection is performed on the gesture depth image, 3D point cloud of each hand joint point is extracted, and the gravity center of the gesture depth image is extracted.

[0087] Optionally, the number of the detected hand joint points is 21.

[0088] S9022: For each 3D point, the pixel depth coordinates of the 3D point are determined according to the weight and the bias of each pixel point in the gesture RGB image collected by the depth camera with respect to the 3D point.

[0089] In S9022, the calculation formula of the pixel depth coordinates of each 3D point of the hand joint point is as follows:

[0090]

[0091] wherein, xi and yi respectively represent the horizontal coordinate and the vertical coordinate of the ith pixel point p in the gesture RGB image collected by the depth camera, represents the depth value of the ith pixel point p in the gesture depth image collected by the depth camera, ω ki represents the weight of the ith pixel point p with respect to the kth 3D point, Δu ki , Δv ki , Δz ki represents the bias of the kth 3D point with respect to the horizontal coordinate, the vertical coordinate and the depth value of the ith pixel point p respectively. The weight and the bias between the pixel point and the 3D point are determined by the training of the deep learning model.

[0092] S9023: The pixel depth coordinates are converted into the depth camera coordinates according to the gravity center of the gesture depth image and the internal parameters of the depth camera, and the initial 3D coordinates of the 3D point are obtained.

[0093] In S9023, the conversion between the coordinates is a known mathematical calculation process, which will not be described in detail.

[0094] In the embodiments of the present application, the network structure used by the deep learning algorithm is not limited. For example, Hourglass, VGG-19 and ResNet-50 network models can be used.

[0095] Optionally, the embodiment of the present application uses the Hourglass network model for 3D gesture labeling. The Hourglass network model retains the spatial information of each resolution to meet the needs of feature extraction at different scales. Therefore, the Hourglass network model is a full convolutional network structure that first down-samples and then up-samples, the entire network does not change the scale of the features, only the depth of the features is changed, which can effectively improve the accuracy in 3D gesture estimation. Moreover, when training the Hourglass network model, the Adam optimizer is used. The Adam optimizer calculates the first moment estimate and the second moment estimate of the gradient, and independently designs an adaptive learning rate for different network parameters, which combines the advantages of adaptive gradient (AdaGrad) and root mean square propagation (RMSProp) in stochastic gradient descent, and has the characteristics of easy implementation, high computational efficiency and low memory occupation.

[0096] Referring to Figure 11 The structural diagram of the deep learning model implemented based on the Hourglass network provided by the embodiment of the present application uses the Hourglass as the backbone network to extract complex features of the hand joint, and estimates the offset of all pixel points relative to each joint (Offset maps u, Offset maps v and Offset maps z) and the influence weight of all pixel points on each joint (weights ω) according to the planar image (u, v) and the 3D point cloud of the hand joint, and directly calculates the initial 3D coordinates of each hand joint according to the weight and offset of all pixel points, without complex post-processing operations.

[0097] Considering that the estimation of the depth information of gestures with occlusions between hands will be biased when only using the gesture estimation algorithm based on deep learning for 3D gesture labeling, in S902, the matching tracking algorithm based on the hand model is also used for 3D gesture labeling to make up for the above defects.

[0098] When the matching tracking algorithm based on the hand model is used for 3D gesture labeling, the 26-degree-of-freedom parameters of the skeleton of the preset hand model are optimized to obtain a target hand model that best matches the hand contour in the gesture depth image and the gesture RGB image, and then the initial 3D coordinates of each hand joint are read from the target hand model to complete the 3D gesture labeling.

[0099] Referring to Figure 12 The kinematic structure diagram of the 26-degree-of-freedom left hand model is shown, L, R, M, I and T represent the little finger, ring finger, middle finger, index finger and thumb respectively, and each joint of each finger occupies 4-degree-of-freedom parameters in the 26-degree-of-freedom parameters, including two 1-degree-of-freedom parameters (in the x and y directions) and two 2-degree-of-freedom parameters (in the x and y directions). Figure 12and a 2-DOF parameter (indicated by a hollow circle) and a 6-DOF parameter (indicated by a solid circle) at the wrist. Figure 12 and a 2-DOF parameter (indicated by a hollow circle) and a 6-DOF parameter (indicated by a solid circle) at the wrist. Figure 12 and a 2-DOF parameter (indicated by a hollow circle) and a 6-DOF parameter (indicated by a solid circle) at the wrist. Adjusting the 20-DOF parameters outside the wrist can drive the hand model to different poses, and adjusting the 6-DOF parameters of the wrist can adjust the position of the hand model in space. Therefore, the 26-DOF parameters of the hand model skeleton can accurately capture and recognize the poses of the hand joint points.

[0100] Specifically, the process of 3D gesture labeling based on the matching tracking algorithm of the hand model is as follows: Figure 13 , mainly including the following steps:

[0101] S9024: According to the gesture RGB image collected by the depth camera, the 2D points of each hand joint point and the hand center point are extracted.

[0102] An optional implementation is that when S9024 is performed, the MediaPipe can be used to extract 21 2D hand joint points and a 2D hand center point from the gesture RGB image collected by the depth camera. Among them, MediaPipe Hands is a high-fidelity hand and finger tracking algorithm that can accurately infer 21 hand joint points and a hand center point from a single image through machine learning (Machine Learning, ML).

[0103] S9025: The depth value of the hand center point is obtained from the gesture depth image collected by the depth camera, and the 3D coordinates of the hand center point are determined in combination with the internal parameters of the depth camera.

[0104] In S9025, the relationship between the gesture depth image and the gesture RGB image collected by the depth camera is used, the depth value of the hand center point is obtained from the gesture depth image according to the 2D coordinates of the hand center point, and then the 3D coordinates of the hand center point are determined according to the 2D coordinates and the depth value of the hand center point, and the internal parameters of the depth camera.

[0105] S9026: According to the 3D coordinates of the hand center point, the hand 3D point cloud within the preset center range is extracted from the gesture depth image.

[0106] In S9026, the hand 3D point cloud within the preset center range of the hand center point is extracted by performing point cloud segmentation on the gesture depth image according to the 3D coordinates of the hand center point, and the hand contour is obtained.

[0107] S9027: The 3D points of each hand joint point are extracted from the hand 3D point cloud, and the 2D points of each hand joint point are taken as the constraint condition for matching the corresponding 3D points with the preset hand model.

[0108] In S9027, the fingers are identified from the segmented hand 3D point cloud, the 3D points of each hand joint are obtained, and the preset hand model is initialized to obtain the pose of each hand joint. The Iterative Closest Point (ICP) method is used to match each joint of the preset hand model with each hand joint in the 3D point cloud. In the matching process, the 2D points of each hand joint are used as the constraint condition for matching the corresponding 3D points and the preset hand model, and the parameters of the hand model optimized in the last frame are used as prior knowledge. The constraint conditions include point cloud alignment, contour alignment, wrist alignment, and joint alignment, and the prior knowledge includes spatial pose prior, temporal prior, and kinematic prior.

[0109] In the matching process, if the matching is successful, the parameters of each subsequent frame of the hand model are initialized based on the optimized results of the last frame of the hand model, and then the hand model is optimized to meet the constraint conditions. If the matching fails, the initial matching step is re-executed.

[0110] S9028: According to the matching result, the initial 3D coordinates of the 3D points of each hand joint in the preset hand model are determined.

[0111] In S9028, according to the matching result of each hand joint in the 3D point cloud and each hand joint in the hand model, the initial 3D coordinates of each hand joint matched in the preset hand model are obtained.

[0112] The framework structure of the matching tracking algorithm based on the hand model in S9025-S9028 is described in detail in Figure 14 .

[0113] It should be noted that the order of the gesture estimation algorithm based on deep learning and the matching tracking algorithm based on the hand model for 3D gesture labeling in the embodiments of the present application is not limited, and they can be performed simultaneously or separately.

[0114] In the process of 3D gesture labeling, gesture estimation algorithms based on deep learning and matching tracking algorithms based on hand models have certain limitations in different application scenarios. For the matching tracking algorithm based on the hand model, the hand edge profile in the gesture depth image provided as a prerequisite must be extremely clear and stable, and the depth error of the gesture depth image must not exceed 1 cm. However, most depth cameras on the market are difficult to meet the above two points. However, in scenes where light is relatively dark and does not affect the capture of gesture depth images, when a stable and clear hand profile can be provided, this method can provide accurate and stable 3D coordinates of each hand joint. For the gesture estimation algorithm based on deep learning, the 3D gesture joint is obtained by identifying and estimating the input image, and the requirement for the hand profile is lower. However, the estimation of the depth information of the gesture with occlusion between hands will be biased, and the deep learning model itself has certain requirements for the image. In most cases, the prediction result of the deep learning model is unstable. By combining the two algorithms, the limitations of a single algorithm can be overcome, thereby improving the accuracy of 3D gesture labeling.

[0115] An optional embodiment is that when the labeling results obtained by the two algorithms are combined, a sliding window algorithm can be used to optimize and fuse the 3D results of each hand joint labeled by the two algorithms. The sliding window algorithm can frame the data arranged in time sequence according to the specified unit length (similar to a window of a specified length), so as to calculate the statistical indicators (such as mean, variance, etc.) in the frame. This is equivalent to a length-specified slider sliding on a scale, and the data in the slider can be fed back every time the slider moves by one unit.

[0116] Referring to Figure 15 The principle diagram of the sliding window provided by the embodiment of the application is shown in the following figure. The length of the sliding window is 3, indicated by a dashed line. Each time it moves to the right by 1 step, and the sum in the sliding window is calculated.

[0117] In specific implementation, after obtaining the 3D gesture results labeled by each algorithm, for each hand joint in each set of labeled hand joints, a sliding window algorithm is used to determine the mean or variance of the initial 3D coordinates of the hand joint in a specified number of frames, and the target 3D coordinates of the hand joint are determined by optimizing and fusing the continuous mean and variance of the initial 3D coordinates of the hand joint, so as to obtain high-precision 3D coordinates of the hand joint and complete automatic labeling.

[0118] The 3D coordinate optimization and fusion process of each hand joint is shown in Figure 16 , which mainly includes the following steps:

[0119] S9029: Determine whether the difference between the mean values of the initial 3D coordinates of the hand joints within the set number of frames determined by the two algorithms respectively exceeds a preset mean value threshold, or determine whether at least one of the variances of the initial 3D coordinates of the hand joints within the set number of frames determined by the two algorithms respectively exceeds a preset variance threshold.

[0120] Since the gesture data is collected, a gesture action is continuous and there is no 3D gesture mutation. The mean value and variance calculated in the sliding window can be used to determine the mutation.

[0121] An optional implementation is that for each hand joint, the first mean value of the initial 3D coordinates of the hand joint within the set number of frames determined by the gesture estimation algorithm based on deep learning is determined, and the second mean value of the initial 3D coordinates of the hand joint within the set number of frames determined by the matching tracking algorithm based on the hand model is determined. The difference between the first mean value and the second mean value is calculated, and the difference is compared with a preset mean value threshold. If the difference exceeds the preset mean value threshold, it indicates that the hand action is discontinuous within the set number of frames, and the 3D coordinates have a mutation, and S90210 should be performed. If the difference does not exceed the preset mean value threshold, it indicates that the hand action is continuous within the set number of frames, and the 3D coordinates have no mutation, and S90211 should be performed.

[0122] Another optional implementation is that for each hand joint, the first variance of the initial 3D coordinates of the hand joint within the set number of frames determined by the gesture estimation algorithm based on deep learning is determined, and the second variance of the initial 3D coordinates of the hand joint within the set number of frames determined by the matching tracking algorithm based on the hand model is determined. The two variances are compared with a preset variance threshold respectively. If at least one of the two variances exceeds the preset variance threshold, it indicates that the hand action is discontinuous within the set number of frames, and the 3D coordinates have a mutation, and S90210 should be performed. If both variances do not exceed the preset variance threshold, it indicates that the hand action is continuous within the set number of frames, and the 3D coordinates have no mutation, and S90211 should be performed.

[0123] Optionally, the set number of frames is 10 frames.

[0124] In some embodiments, the mean value and the variance of each hand joint within the set number of frames can also be combined to determine the continuity of the gesture action.

[0125] S90210: Discard the initial 3D coordinates of the hand joints within the set number of frames.

[0126] When the difference between the first mean value and the second mean value exceeds the preset mean value threshold, or when at least one of the first variance and the second variance exceeds the preset variance threshold, it is indicated that the initial 3D coordinates of the hand joint node are incorrectly labeled, and therefore, the initial 3D coordinates of the hand joint node in a set number of frames are removed.

[0127] S90211: The initial 3D coordinates of the hand joint node determined by the two algorithms are weighted according to the preset algorithm weight to obtain the target 3D coordinates of the hand joint node.

[0128] When the difference between the first mean value and the second mean value does not exceed the preset mean value threshold, or when the first variance and the second variance do not exceed the preset variance threshold, it is indicated that the initial 3D coordinates of the hand joint node are accurately labeled, and the initial 3D coordinates of the hand joint node determined by the two algorithms can be weighted according to the preset algorithm weight to obtain the target 3D coordinates of the hand joint node.

[0129] For example, assuming that the preset algorithm weight ratio of the gesture estimation algorithm based on deep learning and the matching tracking algorithm based on the hand model is 3:7, the initial 3D coordinates of a hand joint node determined by the gesture estimation algorithm based on deep learning are P1, and the initial 3D coordinates of a hand joint node determined by the matching tracking algorithm based on the hand model are P2, and the formula of the target 3D coordinates of the hand joint node is: 0.3*P1+0.7*P2.

[0130] In the embodiments of the present application, the 3D coordinates of each hand joint node are labeled based on the gesture depth image and the gesture RGB image collected by the depth camera, and the depth camera and the plurality of industrial cameras are synchronized, so that the 3D labeling result of the single-hand gesture image collected by the depth camera can be used as the 3D labeling result of the single-hand gesture image collected by the industrial camera, thereby obtaining a large amount of high-precision gesture data set. Through the backpack-type gesture data collection device and the client device, the single-hand RGB image (including the single-hand RGB image collected by the plurality of industrial cameras and the single-hand RGB image collected by the depth camera) and the single-hand depth image (the single-hand depth image collected by the depth camera) are used to solve the single-hand occlusion problem in 3D gesture labeling (i.e., determining the 3D coordinates of each hand joint node), and the gesture data set meets the research and application requirements of 3D gesture estimation and hand region detection algorithms.

[0131] To further improve the precision of the gesture data set, the labeling result of the 3D gesture is verified in the embodiments of the present application, which is specifically described with reference to S903.

[0132] S903: In response to the data verification instruction, project the hand joint corresponding to each target 3D coordinate to the gesture RGB image collected by the corresponding industrial camera according to the calibration matrix between the depth camera and each industrial camera; and screen the gesture RGB images collected by each industrial camera after projection to generate a gesture dataset for gesture interaction.

[0133] When verifying the 3D gesture annotation result, single-frame verification can be performed, or multi-frame verification can be performed.

[0134] An optional implementation is that the user clicks the "open image" option in the gesture data verification interface, selects a gesture RGB image collected by the industrial camera which needs to be verified, and then checks the "2D coordinate projection" in the gesture data verification interface. Figure 6C An optional implementation is that the user clicks the "open image" option in the gesture data verification interface, selects a gesture RGB image collected by the industrial camera which needs to be verified, and then checks the "2D coordinate projection" in the gesture data verification interface. Figure 6C The client device projects the target 3D coordinates of each hand joint obtained from the single-hand gesture image synchronized with the current gesture RGB image to the current gesture RGB image according to the calibration matrix between the depth camera and the industrial camera corresponding to the current gesture RGB image, and if the target 3D coordinates of each hand joint can correctly restore the gesture in the current gesture RGB image, the current gesture RGB image is retained, and if the target 3D coordinates of each hand joint cannot correctly restore the gesture in the current gesture RGB image, the user clicks the "delete image" in the gesture data verification interface to delete the current gesture RGB image from the gesture dataset to obtain the screened gesture dataset. Figure 6C The client device projects the target 3D coordinates of each hand joint obtained from the single-hand gesture image synchronized with the current gesture RGB image to the current gesture RGB image according to the calibration matrix between the depth camera and the industrial camera corresponding to the current gesture RGB image, and if the target 3D coordinates of each hand joint can correctly restore the gesture in the current gesture RGB image, the current gesture RGB image is retained, and if the target 3D coordinates of each hand joint cannot correctly restore the gesture in the current gesture RGB image, the user clicks the "delete image" in the gesture data verification interface to delete the current gesture RGB image from the gesture dataset to obtain the screened gesture dataset.

[0135] Another optional implementation is that the user inputs the start frame and the end frame in the gesture data verification interface, or clicks the "select all images" option to select multiple gesture RGB images collected by the industrial camera which need to be verified, and then checks the "2D coordinate projection" in the gesture data verification interface. Figure 6C Another optional implementation is that the user inputs the start frame and the end frame in the gesture data verification interface, or clicks the "select all images" option to select multiple gesture RGB images collected by the industrial camera which need to be verified, and then checks the "2D coordinate projection" in the gesture data verification interface. Figure 6C The client device projects the target 3D coordinates of each hand joint obtained from the single-hand gesture image synchronized with each gesture RGB image to the corresponding gesture RGB image according to the calibration matrix between the depth camera and the industrial camera corresponding to each gesture RGB image, removes the check mark at the lower right corner of the image for the gesture RGB image that can be restored, retains the check mark at the lower right corner of the image for the gesture RGB image that cannot be restored, and then clicks the "delete image" option in the gesture data verification interface to delete the multiple gesture RGB images from the gesture dataset to obtain the screened gesture dataset. Figure 6C The client device projects the target 3D coordinates of each hand joint obtained from the single-hand gesture image synchronized with each gesture RGB image to the corresponding gesture RGB image according to the calibration matrix between the depth camera and the industrial camera corresponding to each gesture RGB image, removes the check mark at the lower right corner of the image for the gesture RGB image that can be restored, retains the check mark at the lower right corner of the image for the gesture RGB image that cannot be restored, and then clicks the "delete image" option in the gesture data verification interface to delete the multiple gesture RGB images from the gesture dataset to obtain the screened gesture dataset.

[0136] For the gesture RGB image whose verification process is deleted, it is stored in the recycle bin, and when the user mistakenly deletes it, the user can click Figure 6C The "recall deleted image" option in the gesture data verification interface is used to recall the mistakenly deleted gesture RGB image.

[0137] The embodiment of the present application provides a gesture data set generation method, which connects the whole process from gesture data collection to gesture data labeling and then to gesture data verification to generate the gesture data set, realizes pipeline operation, improves the generation efficiency and quantity of the gesture data set, and saves the generation cost without the intervention of manual and third parties. In the data collection process, the gesture images collected by one depth camera and multiple industrial cameras within a set time period are obtained by responding to the data collection instruction, so that the multi-path data stream is quickly collected, and the efficiency is higher; in the data labeling process, the gesture depth image and the gesture RGB image collected by the depth camera are processed by using the gesture estimation algorithm based on deep learning and the matching tracking algorithm based on the hand model respectively according to the corresponding data labeling instruction, the initial 3D coordinates of each hand joint are determined, and the initial 3D coordinates determined by the two algorithms are optimized to obtain more accurate target 3D coordinates, so that the target 3D coordinates of each hand joint are determined by the two algorithms, the limitation of the single algorithm is overcome, the accuracy of 3D gesture labeling is improved, and then the quality of the gesture data set is improved; in the data verification process, the hand joints corresponding to the target 3D coordinates are projected onto the gesture RGB image collected by the corresponding industrial camera according to the calibration matrix between the depth camera and each industrial camera by using the data verification instruction, and the quality of the gesture data set is further improved after data screening.

[0138] Based on the same technical concept, the embodiment of the present application provides a client device which is equipped with software integrating a gesture data collection interface, a gesture data labeling interface and a gesture data verification interface, and can realize the gesture data set generation method provided in the above embodiment.

[0139] Referring to Figure 17 The client device comprises a processor 1701, a memory 1702, a display screen 1703 and multiple communication interfaces 1704, and the communication interfaces 1704, the display screen 1703, the memory 1702 and the processor 1701 are connected through a bus 1705.

[0140] The display screen 1703 is used for the display of a user interface and an image, and the user interface is used for receiving an operation instruction.

[0141] The memory 1702 stores a computer program, and the processor 1701 executes the following operations according to the computer program:

[0142] In response to the data collection instruction, a single-hand gesture image collected by a depth camera and a plurality of industrial cameras within a set time period is acquired through the communication interface 1704. The single-hand gesture image collected by the depth camera includes a gesture depth image and a gesture RGB image.

[0143] In response to the data labeling instruction, initial 3D coordinates of each hand joint are determined according to the gesture depth image and the gesture RGB image collected by the depth camera by using a gesture estimation algorithm based on deep learning and a matching tracking algorithm based on a hand model respectively. For each hand joint, a target 3D coordinate of the hand joint is determined according to a mean value or a variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms respectively.

[0144] In response to the data verification instruction, the hand joint corresponding to each target 3D coordinate is projected onto a gesture RGB image collected by a corresponding industrial camera according to a calibration matrix between the depth camera and each industrial camera. The gesture RGB images collected by the industrial cameras after projection are filtered to generate a gesture dataset for gesture interaction.

[0145] Optionally, the processor 1701 determines the initial 3D coordinates of each hand joint according to the gesture depth image and the gesture RGB image collected by the depth camera by using the gesture estimation algorithm based on deep learning. The specific operation is as follows:

[0146] The 3D point cloud of each hand joint is determined according to the gesture depth image collected by the depth camera.

[0147] For each 3D point, the pixel depth coordinate of the 3D point is determined according to the weight and the bias amount of all pixel points in the gesture RGB image collected by the depth camera with respect to the 3D point.

[0148] The pixel depth coordinate is converted to the depth camera coordinate according to the center of gravity of the gesture depth image and the calibrated internal parameter of the depth camera to obtain the initial 3D coordinate of the 3D point.

[0149] Optionally, the calculation formula of the pixel depth coordinate of each 3D point of each hand joint is as follows:

[0150]

[0151] wherein, Xi and Y1 represent the horizontal coordinate and the vertical coordinate of the i-th pixel point p in the gesture RGB image collected by the depth camera, respectively. D1 represents the depth value of the i-th pixel point p in the gesture depth image collected by the depth camera, ω ki ωk represents the weight of the i-th pixel point p with respect to the k-th 3D point, Δuki Δx ki Δy ki represents the offset amount of the kth 3D point relative to the horizontal coordinate, the vertical coordinate and the depth value of the ith pixel point p respectively.

[0152] Optionally, the processor 1701 determines the initial 3D coordinates of each hand joint according to the gesture depth image and the gesture RGB image collected by the depth camera by using a matching tracking algorithm based on a hand model, and the specific operation is as follows:

[0153] extracting 2D points of each hand joint and a hand center point from the gesture RGB image collected by the depth camera;

[0154] obtaining a depth value of the hand center point from the gesture depth image collected by the depth camera, and determining a 3D coordinate of the hand center point in combination with an internal parameter of the depth camera which has been calibrated;

[0155] extracting a hand 3D point cloud within a preset center range from the gesture depth image according to the 3D coordinate of the hand center point;

[0156] extracting 3D points of each hand joint from the hand 3D point cloud, and taking 2D points of each hand joint as prior knowledge of matching of the corresponding 3D points with a preset hand model;

[0157] determining initial 3D coordinates of 3D points of each hand joint from the preset hand model according to a matching result.

[0158] Optionally, for each hand joint, the processor 1701 determines a target 3D coordinate of the hand joint according to a mean value or a variance of the initial 3D coordinates of the hand joint within a set number of frames determined by two algorithms respectively, and the specific operation is as follows:

[0159] determining whether a difference between the mean values of the initial 3D coordinates of the hand joint within the set number of frames determined by the two algorithms respectively exceeds a preset mean value threshold, or determining whether at least one of the variances of the initial 3D coordinates of the hand joint within the set number of frames determined by the two algorithms respectively exceeds a preset variance threshold;

[0160] if yes, the initial 3D coordinates of the hand joint within the set number of frames are eliminated;

[0161] if no, the initial 3D coordinates of the hand joint determined by the two algorithms respectively are weighted according to a preset algorithm weight, so as to obtain a target 3D coordinate of the hand joint.

[0162] Optionally, before acquiring the gesture images synchronously captured by the depth camera and the plurality of industrial cameras within the set time length, the processor 1701 further performs:

[0163] in response to the synchronization instruction, setting the depth camera and each industrial camera to be synchronized;

[0164] in response to the calibration instruction, controlling the depth camera and each industrial camera to synchronously capture calibration images, and aligning coordinate systems of the depth camera and each industrial camera according to the calibration images to obtain a calibration matrix between the depth camera and each industrial camera.

[0165] Optionally, the angles at which the depth camera and each industrial camera capture images are set according to the angles of the downwardly inclined binocular camera on the virtual display device, and the deployment manner of the depth camera and each industrial camera is a backpack-style architecture.

[0166] It should be noted that, Figure 17 is only an example, and the client device necessary for the gesture dataset generation method steps provided by the embodiments of the present application is not shown, and the client device further includes terminal common devices such as a keyboard, a power supply, and the like.

[0167] The embodiments of the present application Figure 17 The processor involved in the embodiments of the present application can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0168] The embodiments of the present application also provide a computer-readable storage medium for storing some instructions, which when executed, can complete the gesture dataset generation method in the foregoing embodiments.

[0169] The embodiments of the present application also provide a computer program product for storing a computer program, the computer program being used to execute the gesture dataset generation method in the foregoing embodiments.

[0170] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the present application are stored in a memory such as a computer memory or disk storage for use by, or in connection with, the software on the computer system. The software can provide for programs to be transferred to another computer readable medium (e.g., a removable medium, or a medium conveyed through a computer network) for use in a different system.

[0171] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts described above.

[0172] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in conjunction with the flowcharts described above.

[0173] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts described above.

[0174] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for generating a gesture dataset, characterized in that, include: In response to a data acquisition command, a single-hand gesture image is acquired simultaneously by a depth camera and multiple industrial cameras. The single-hand gesture image acquired by the depth camera includes a gesture depth image and a gesture RGB image, while the single-hand gesture image acquired by the industrial cameras is a gesture RGB image. In response to the data annotation instruction, based on the gesture depth image and gesture RGB image captured by the depth camera, the initial 3D coordinates of each hand joint are determined by a deep learning-based gesture estimation algorithm and a hand model-based matching and tracking algorithm, respectively; and for each hand joint, the target 3D coordinates of the hand joint are determined by the mean or variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms, respectively. In response to the data verification command, based on the calibration matrix between the depth camera and each industrial camera, the hand joints corresponding to the 3D coordinates of each target are projected onto the RGB image of the gesture acquired by the corresponding industrial camera; and from the RGB image of the gesture acquired by the multiple industrial cameras, the RGB image of the gesture that correctly restores the gesture in the image is selected to generate a gesture dataset for gesture interaction. Specifically, for each hand joint, the target 3D coordinates of the hand joint are determined based on the mean or variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms, including: For each of the two algorithms, a sliding window is applied to calculate the mean of the initial 3D coordinates of the hand joints within a set number of frames, or the variance of the initial 3D coordinates of the hand joints within a set number of frames. Determine whether the difference between the means of the two algorithms within the set number of frames exceeds a preset mean threshold, or determine whether there is a variance among the variances of the two algorithms within the set number of frames that exceeds a preset variance threshold; If the number of frames exceeds the limit, the initial 3D coordinates of the hand joints within the set frame number will be discarded. If the target 3D coordinates of the hand joint are not exceeded, the initial 3D coordinates of the hand joint determined by the two algorithms are weighted according to the preset algorithm weights to obtain the target 3D coordinates of the hand joint.

2. The method as described in claim 1, characterized in that, Based on the gesture depth image and gesture RGB image captured by the depth camera, a deep learning-based gesture estimation algorithm is used to determine the initial 3D coordinates of each hand joint, including: Based on the gesture depth image captured by the depth camera, determine the 3D point cloud of each hand joint; For each 3D point, the pixel depth coordinates of the 3D point are determined based on the weights and biases of all pixels in the gesture RGB image captured by the depth camera and the 3D point. Based on the centroid of the gesture depth image and the calibrated intrinsic parameters of the depth camera, the pixel depth coordinates are transformed into depth camera coordinates to obtain the initial 3D coordinates of the 3D point.

3. The method as described in claim 2, characterized in that, The formula for calculating the pixel depth coordinates of the 3D points of each hand joint is as follows: in, Let x and y represent the x and y coordinates of the i-th pixel p in the RGB image of the gesture captured by the depth camera, respectively. ω represents the depth value of the i-th pixel p in the gesture depth image captured by the depth camera. ki Δu represents the weight of the i-th pixel p relative to the k-th 3D point. ki Δv ki Δz ki This represents the offset of the k-th 3D point relative to the x-coordinate, y-coordinate, and depth value of the i-th pixel point p.

4. The method as described in claim 1, characterized in that, Based on the gesture depth image and gesture RGB image captured by the depth camera, a matching and tracking algorithm based on a hand model is used to determine the initial 3D coordinates of each hand joint, including: Based on the RGB image of the gesture captured by the depth camera, extract the 2D points of each hand joint and the center point of the hand; The depth value of the center point of the hand is obtained from the gesture depth image captured by the depth camera, and the 3D coordinates of the center point of the hand are determined by combining the intrinsic parameters of the depth camera that have been calibrated. Based on the 3D coordinates of the hand's center point, extract the hand's 3D point cloud within a preset central range from the gesture depth image; Extract the 3D points of each hand joint from the 3D point cloud of the hand, and use the 2D points of each hand joint as the constraint condition for matching the corresponding 3D points with the preset hand model. Based on the matching results, the initial 3D coordinates of the 3D points of each hand joint are determined from the preset hand model.

5. The method as described in claim 1, characterized in that, Before acquiring gesture images simultaneously captured by one depth camera and multiple industrial cameras within a set time period, the method further includes: In response to the synchronization command, the depth camera is set to synchronize with each industrial camera; In response to a calibration command, the depth camera and each of the industrial cameras are controlled to synchronously acquire calibration images, and the coordinate systems of the depth camera and each of the industrial cameras are aligned according to the calibration images to obtain the calibration matrix between the depth camera and each of the industrial cameras.

6. The method according to any one of claims 1-5, characterized in that, The angle at which the depth camera and each of the industrial cameras acquire images is set according to the angle of the downward-tilted binocular camera on the virtual display device, and the depth camera and each of the industrial cameras are deployed in a backpack-style architecture.

7. A client device, characterized in that, It includes a processor, a memory, a display screen, and multiple communication interfaces, wherein the communication interfaces, the display screen, the memory, and the processor are connected via a bus; The display screen is used for displaying the user interface and images, and the user interface is used for receiving operation commands; The memory stores a computer program, and the processor performs the following operations according to the computer program: In response to a data acquisition command, the communication interface is used to acquire single-hand gesture images simultaneously acquired by one depth camera and multiple industrial cameras within a set time period. The single-hand gesture images acquired by the depth camera include gesture depth images and gesture RGB images. In response to the data annotation instruction, based on the gesture depth image and gesture RGB image captured by the depth camera, the initial 3D coordinates of each hand joint are determined by a deep learning-based gesture estimation algorithm and a hand model-based matching and tracking algorithm, respectively; and for each hand joint, the target 3D coordinates of the hand joint are determined by the mean or variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms, respectively. In response to the data verification command, based on the calibration matrix between the depth camera and each industrial camera, the hand joints corresponding to the 3D coordinates of each target are projected onto the RGB image of the gesture acquired by the corresponding industrial camera; and from the RGB image of the gesture acquired by the multiple industrial cameras, the RGB image of the gesture that correctly restores the gesture in the image is selected to generate a gesture dataset for gesture interaction. Specifically, for each hand joint, the target 3D coordinates of the hand joint are determined based on the mean or variance of the initial 3D coordinates of the hand joint within a set number of frames determined by the two algorithms, including: For each of the two algorithms, a sliding window is applied to calculate the mean of the initial 3D coordinates of the hand joints within a set number of frames, or the variance of the initial 3D coordinates of the hand joints within a set number of frames. Determine whether the difference between the means of the two algorithms within the set number of frames exceeds a preset mean threshold, or determine whether there is a variance among the variances of the two algorithms within the set number of frames that exceeds a preset variance threshold; If the number of frames exceeds the limit, the initial 3D coordinates of the hand joints within the set frame number will be discarded. If the target 3D coordinates of the hand joint are not exceeded, the initial 3D coordinates of the hand joint determined by the two algorithms are weighted according to the preset algorithm weights to obtain the target 3D coordinates of the hand joint.

8. The client device as described in claim 7, characterized in that, The processor determines the initial 3D coordinates of each hand joint point based on the gesture depth image and gesture RGB image captured by the depth camera, using a deep learning-based gesture estimation algorithm. Specifically, the operation is as follows: Based on the gesture depth image captured by the depth camera, determine the 3D point cloud of each hand joint; For each 3D point, the pixel depth coordinates of the 3D point are determined based on the weights and biases of all pixels in the gesture RGB image captured by the depth camera and the 3D point. Based on the centroid of the gesture depth image and the calibrated intrinsic parameters of the depth camera, the pixel depth coordinates are transformed into depth camera coordinates to obtain the initial 3D coordinates of the 3D point.

9. The client device as described in claim 7, characterized in that, The processor determines the initial 3D coordinates of each hand joint point using a hand model-based matching and tracking algorithm based on the gesture depth image and gesture RGB image captured by the depth camera. Specifically, the operation is as follows: Based on the RGB image of the gesture captured by the depth camera, extract the 2D points of each hand joint and the center point of the hand; The depth value of the center point of the hand is obtained from the gesture depth image captured by the depth camera, and the 3D coordinates of the center point of the hand are determined by combining the intrinsic parameters of the depth camera that have been calibrated. Based on the 3D coordinates of the hand's center point, extract the hand's 3D point cloud within a preset central range from the gesture depth image; Extract the 3D points of each hand joint from the 3D point cloud of the hand, and use the 2D points of each hand joint as prior knowledge for matching the corresponding 3D points with the preset hand model; Based on the matching results, the initial 3D coordinates of the 3D points of each hand joint are determined from the preset hand model.

Images