Hand positioning method and VR device

Abstract

The present invention discloses a hand positioning method and VR device, comprising: obtaining an image to be processed captured by a binocular camera of a VR device; inputting the image to be processed into a hand detection network, performing at least one depthwise separable convolution on the image to be processed, and extracting an initial hand feature vector of the image to be processed; convolving the initial hand feature vector to generate two initial hand images of different scales; inputting the initial hand image into a hand key point estimation network, performing at least one depthwise separable convolution on the initial hand image to obtain a target hand feature vector of the initial hand image; performing a differentiable maximum index operation on a heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points; and determining the three-dimensional coordinates of the hand key points based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera. This method ensures hand positioning accuracy and improves operation speed.

Description

Technical Field

[0001] The present invention relates to the field of VR technology, and in particular to a hand positioning method and VR equipment. Background Art

[0002] Augmented reality (AR) and virtual reality (VR) technologies are increasingly being used in the new generation of human-computer interaction. Three-dimensional hand gesture estimation and tracking, a crucial component of real-time gesture interaction, can determine the position of key points on the hand, laying the foundation for subsequent interactive tasks related to hand position. Therefore, accurate positioning of the hand and its key points is crucial for VR device applications.

[0003] In related technologies, when locating the hands and key points of the hands, it is usually necessary to rely on large-scale high-quality labeled data. However, this method not only increases the development difficulty but also reduces the running speed.

[0004] The above drawbacks are particularly evident in binocular imaging scenarios. Therefore, there is an urgent need for a hand positioning method that can improve the operating speed while ensuring positioning accuracy. Summary of the Invention

[0005] In an exemplary embodiment of the present invention, a hand positioning method and a VR device are provided to improve the operating speed while ensuring the hand positioning accuracy.

[0006] According to a first aspect of an exemplary embodiment, a hand positioning method is provided, which is applied to a VR device, comprising:

[0007] Obtaining an image to be processed captured by a binocular camera of the VR device;

[0008] Inputting the image to be processed into a hand detection network, performing at least one depthwise separable convolution on the image to be processed, and extracting an initial hand feature vector of the image to be processed;

[0009] performing convolution on the initial hand feature vector to generate two initial hand images of different scales;

[0010] Inputting the initial hand image into a hand key point estimation network, performing at least one depthwise separable convolution on the initial hand image to obtain a target hand feature vector of the initial hand image;

[0011] Performing a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points;

[0012] The three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0013] According to a second aspect of the exemplary embodiment, there is provided a VR device, comprising a processor, a memory, and at least one external communication interface, wherein the processor, the memory, and the external communication interface are all connected via a bus;

[0014] The external communication interface is configured to receive the image to be processed captured by the binocular camera;

[0015] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0016] Inputting the image to be processed into a hand detection network, performing at least one depthwise separable convolution on the image to be processed, and extracting an initial hand feature vector of the image to be processed;

[0017] performing convolution on the initial hand feature vector to generate two initial hand images of different scales;

[0018] Inputting the initial hand image into a hand key point estimation network, performing at least one depthwise separable convolution on the initial hand image to obtain a target hand feature vector of the initial hand image;

[0019] Performing a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points;

[0020] The three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0021] According to a third aspect of the exemplary embodiment, there is provided a hand positioning device, the device comprising:

[0022] An image acquisition module, configured to acquire an image to be processed captured by a binocular camera of the VR device;

[0023] a first processing module, configured to input the image to be processed into a hand detection network, perform at least one depthwise separable convolution on the image to be processed, and extract an initial hand feature vector of the image to be processed;

[0024] a second processing module, configured to perform convolution on the initial hand feature vector to generate two initial hand images of different scales;

[0025] a third processing module, configured to input the initial hand image into a hand key point estimation network, perform at least one depthwise separable convolution on the initial hand image, and obtain a target hand feature vector of the initial hand image;

[0026] a first coordinate determination module, configured to perform a differentiable maximum index operation on a heat map of two-dimensional coordinates of hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points;

[0027] The second coordinate determination module is used to determine the three-dimensional coordinates of the hand key points based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0028] According to a fourth aspect of the exemplary implementation, a computer storage medium is provided, wherein the computer storage medium stores computer program instructions, and when the instructions are executed on a computer, the computer executes the hand positioning method as described in the first aspect.

[0029] The embodiments of the present application have the following beneficial effects:

[0030] The network structures of the hand detection network and the hand keypoint estimation network are improved. Specifically, in the hand detection network, at least one depthwise separable convolution is performed on the processed image to extract the initial hand feature vector. This application effectively reduces the number of parameters and computation without compromising performance. The initial hand feature vector is then convolved to generate two initial hand images of different scales, reducing the number of layers in the feature pyramid structure and, to a certain extent, reducing the computation. In the hand keypoint estimation network, the initial hand image is input into the network and subjected to at least one depthwise separable convolution to obtain the target hand feature vector of the initial hand image. A differentiable maximum indexing operation is performed on the heat map of the 2D coordinates of the hand keypoints predicted from the target hand feature vector to obtain the 2D coordinates of the hand keypoints, reducing overall processing time. Finally, the 3D coordinates of the hand keypoints are determined based on the 2D coordinates of the hand keypoints and the intrinsic and extrinsic parameters of the binocular camera. Therefore, while maintaining accuracy, the two networks were lightweighted in terms of volume and speed, which improved the running speed and thus the speed of hand positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 An example diagram of an application scenario of a hand positioning method provided by an embodiment of the present invention is shown;

[0033] Figure 2 The following is a flowchart illustrating a hand positioning method provided by an embodiment of the present invention;

[0034] Figure 3 The following is an exemplary diagram of a process for extracting an initial hand feature vector provided by an embodiment of the present invention;

[0035] Figure 4 The following is an exemplary diagram showing a partial structure of a hand detection network provided by an embodiment of the present invention;

[0036] Figure 5 The following is a schematic diagram showing the structure of a residual unit provided by an embodiment of the present invention;

[0037] Figure 6 The following is a schematic diagram showing the structure of a downsampling residual unit provided by an embodiment of the present invention;

[0038] Figure 7 The following is a flowchart illustrating a process of extracting an initial hand image provided by an embodiment of the present invention;

[0039] Figure 8 A partial schematic diagram of another hand detection network provided by an embodiment of the present invention is exemplarily shown;

[0040] Figure 9 The following is a schematic diagram showing the structure of a hand detection network provided by an embodiment of the present invention;

[0041] Figure 10 The following is a flowchart illustrating a process for extracting a target hand feature vector provided by an embodiment of the present invention;

[0042] Figure 11 The following is an exemplary diagram showing a structure of a hand key point estimation network provided by an embodiment of the present invention;

[0043] Figure 12 A schematic diagram exemplarily illustrates a flow chart of a hand positioning method provided by an embodiment of the present invention;

[0044] Figure 13 A schematic diagram exemplarily illustrates a flow chart of another hand positioning method provided by an embodiment of the present invention;

[0045] Figure 14 The following is a schematic diagram showing the structure of a hand positioning device provided by an embodiment of the present invention;

[0046] Figure 15The figure shows a schematic structural diagram of a VR device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0048] Augmented reality (AR) and virtual reality (VR) technologies are increasingly being used in the new generation of human-computer interaction. Three-dimensional hand gesture estimation and tracking, a crucial component of real-time gesture interaction, can determine the position of key points on the hand, laying the foundation for subsequent interactive tasks related to hand position. Therefore, accurate positioning of the hand and its key points is crucial for VR device applications.

[0049] Existing visual gesture tracking technologies are mainly divided into monocular and binocular solutions. Monocular solutions use a single RGB camera or depth camera to directly detect hands and estimate the positions of 3D key points. This places high demands on 3D gesture estimation algorithms, typically requiring the design of sophisticated neural network models and training with large, high-quality annotated datasets. This results in high development costs and difficulty in ensuring generalization. Binocular solutions, on the other hand, rely on the positions of hand and key points in binocular images, offering greater generalization capabilities and lower development complexity. However, due to the need to process both binocular images simultaneously, their efficiency is relatively low. Therefore, there is an urgent need for a hand localization method that can achieve both high accuracy and high speed.

[0050] To this end, an embodiment of the present application provides a hand localization method for use in VR devices. The method improves the network structures of the hand detection network and the hand key point estimation network. For example, in the hand detection network, at least one depthwise separable convolution is performed on the image to be processed to extract the initial hand feature vector of the image to be processed, and the initial hand feature vector is convolved to generate two initial hand images of different scales. In the hand key point estimation network, the initial hand image is input into the hand key point estimation network, and at least one depthwise separable convolution is performed on the initial hand image to obtain the target hand feature vector of the initial hand image. A differentiable maximum index operation is performed on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points. Finally, the three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera. Therefore, in the embodiment of the present application, while maintaining accuracy, the two networks are lightweighted in terms of both volume and speed, improving the running speed and, in turn, the speed of hand localization.

[0051] After introducing the design concepts of the embodiments of this application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of this application and are not limiting. In specific implementations, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0052] refer to Figure 1 , shows an application scenario diagram of a hand positioning method, in which a binocular camera shoots the hand to obtain each image to be processed. For clarity, Figure 1 The binocular camera is directly shown in the figure. In actual application, the binocular camera can be integrated into the VR device. This is just an example and does not constitute any limitation.

[0053] To further illustrate the technical solutions provided by the embodiments of the present application, the following is a detailed description of the technical solutions in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative work. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present application.

[0054] The following combination Figure 1 The application scenario shown, refer to Figure 2 A flowchart of a hand positioning method is shown to illustrate the technical solution provided by an embodiment of the present application.

[0055] S201: Obtain an image to be processed captured by a binocular camera of a VR device.

[0056] S202: Input the image to be processed into a hand detection network, perform at least one depthwise separable convolution on the image to be processed, and extract an initial hand feature vector of the image to be processed.

[0057] S203: Convolve the initial hand feature vector to generate two initial hand images of different scales.

[0058] S204: Input the initial hand image into a hand key point estimation network, perform at least one depthwise separable convolution on the initial hand image, and obtain a target hand feature vector of the initial hand image.

[0059] S205 , performing a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points.

[0060] S206: Determine the three-dimensional coordinates of the hand key points based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0061] In the embodiments of the present application, improvements are made to the network structures of the hand detection network and the hand key point estimation network. Specifically, in the hand detection network, at least one depthwise separable convolution is performed on the image to be processed to extract the initial hand feature vector of the image to be processed. This depthwise separable convolution is then applied, effectively reducing the number of parameters and computational complexity without compromising the performance. The initial hand feature vector is then convolved to generate two initial hand images of different scales, reducing the number of layers in the feature pyramid structure and, to a certain extent, reducing the computational complexity. In the hand key point estimation network, the initial hand image is input into the hand key point estimation network and subjected to at least one depthwise separable convolution to obtain the target hand feature vector of the initial hand image. A differentiable maximum indexing operation is performed on the heat map of the two-dimensional coordinates of the hand key points predicted from the target hand feature vector to obtain the two-dimensional coordinates of the hand key points, reducing overall processing time. Finally, the three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera. Therefore, while maintaining accuracy, the two networks were lightweighted in terms of volume and speed, which improved the running speed and thus the speed of hand positioning.

[0062] Regarding S201, for a VR device with an integrated binocular camera, while the VR device is capturing a hand image, an image to be processed captured by the binocular camera of the VR device is obtained in real time. The image to be processed is obtained by capturing the hand, but may include the entire hand, part of the hand, or no hand.

[0063] To determine the 2D coordinates of hand keypoints, we first need to identify the hand's position, crop a partial hand image, and then identify the positions of the hand keypoints based on the partial hand image to obtain their 2D coordinates. In this process, the hand detection network determines the hand's position, while the hand keypoint estimation network determines the 2D coordinates of the hand keypoints. The following describes the recognition processes of these two networks.

[0064] In S202, the image to be processed is input into the hand detection network, and at least one depth-separable convolution is performed on the image to be processed to extract the initial hand feature vector of the image to be processed. Figure 3 and Figure 4 , the process of extracting the initial hand feature vector of the image to be processed is explained, Figure 3 Flowchart showing a process of extracting an initial hand feature vector, Figure 4 A partial structural diagram of a hand detection network is shown:

[0065] S301 , performing conventional convolution on the image to be processed to obtain a first hand feature vector.

[0066] For example, an image to be processed is input to network layer D11, where it undergoes conventional convolution (Con2D) and maximum pooling (MaxPool). Furthermore, the activation function Relu6 is used in this layer to obtain the first hand feature vector. Conventional convolution and pooling in network layer D11 reduces the resolution of the processed data.

[0067] S302: Perform a depthwise separable convolution on the first hand feature vector to obtain a second hand feature vector.

[0068] For example, the first hand feature vector is input to the network layer D12, and residual processing is performed in the residual unit (Resblock) of the network layer D12, and downsampling processing is performed in the downsampling residual unit (DownSampleRes) of the network side D12. In the residual unit, the first hand feature vector is processed by depthwise separable convolution and conventional convolution (DwConv), and the activation function Relu6 is used. Figure 5 A schematic diagram of the structure of a residual unit is shown. In the down-sampling residual unit, the first hand feature vector is processed by depth-separable convolution and conventional convolution, as well as maximum pooling, and the activation function Relu6 is used. Figure 6 A schematic structural diagram of a downsampling residual unit is shown.

[0069] In a specific example, the depthwise separable convolution uses a 3x3 channel-by-channel convolution and a 1x1 ordinary convolution to replace the conventional 3x3 convolution, effectively reducing the number of parameters and the amount of computation without affecting the effect. In this way, except for the conventional convolution used for the initial input network layer D11 convolution processing, all other network layers are replaced with depthwise separable convolution. After the above optimization of the hand key point estimation network, the generalization of the network is enhanced through network pre-training technology and data set augmentation technology. Combined with the inference acceleration strategy of the neural network framework, millisecond-level inference speed is achieved under CPU resource conditions only, ensuring the real-time performance of the binocular hand positioning solution.

[0070] In addition, the result of the network layer D12 is input to the network layer D13, and a depthwise separable convolution process (Conv2D) is performed in the network layer D13 to obtain a second hand feature vector.

[0071] S303 , fusing the result of performing maximum pooling on the second hand feature vector and the result of performing depthwise separable convolution on the second hand feature vector to obtain an initial hand feature vector of the image to be processed.

[0072] Among them, the second hand feature vector is subjected to maximum pooling processing in the network layer D14, and the second hand feature vector is subjected to depthwise separable convolution processing in the network layer D15. The results of the two processings are fused to obtain the initial hand feature vector of the image to be processed.

[0073] As described above, the process of determining the initial hand feature vector of the hand detection network is described. Next, the process of generating the initial hand image by the hand detection network is described.

[0074] Involving S203, convolution is performed on the initial hand feature vector to generate two initial hand images of different scales. Figure 7 and Figure 8 ,The process of the initial hand image is explained, Figure 7 A flowchart showing the process of extracting an initial hand image is shown. Figure 8 A partial schematic diagram of a hand detection network is shown.

[0075] S701: Perform depthwise separable convolution on the initial hand feature vector to obtain a third hand feature vector.

[0076] For example, in the network layer D16 application Figure 5 The residual unit shown performs a depth-wise separable convolution on the initial hand feature vector, using a single Relu6 activation to obtain a third hand feature vector.

[0077] S702 : Perform upsampling convolution on the third hand feature vector, and then fuse it with the second hand feature vector to obtain a fourth hand feature vector.

[0078] Exemplarily, the third hand feature vector is upsampled and convolved in the network layer D17 and fused with the second hand feature vector output by the aforementioned network layer D13 to obtain a fourth hand feature vector.

[0079] S703 : Perform a depthwise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector.

[0080] For example, in the network layer D18 application Figure 5 The residual unit shown performs a depthwise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector.

[0081] S704 : Perform conventional convolution on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

[0082] For example, the third hand feature vector is subjected to conventional convolution at the network layer D19 to obtain an initial hand image of the first scale F1; the fifth hand feature vector is subjected to conventional convolution at the network layer D191 to obtain an initial hand image of the second scale F2. For the relevant data of the first scale and the second scale, see Figure 4 The initial hand image is determined based on the initial hand image. The initial hand images at two scales form a feature pyramid. This pyramid is used to classify the hand and regress the hand's rectangular region. This determines the hand's position within the entire image to be processed, allowing for cropping of the image to obtain a partial hand image, i.e., the initial hand image. Reducing the number of layers in the feature pyramid structure to two improves processing speed to a certain extent.

[0083] In order to make the structure of the hand detection network clearer, Figure 9 A schematic diagram of the structure of a hand detection network is shown. In this network, the number of residual units in the backbone network is reduced, further reducing the algorithm time consumption and improving the processing speed.

[0084] When there are multiple images to be processed, the hand key point estimation network can be used to obtain the hand position, and then the image to be processed can be cropped to obtain a partial hand image. However, in order to alleviate the problem of hand detection frame position jitter that may occur when using only the hand detection network, a hand tracking algorithm based on Kalman filtering is introduced. Kalman filtering models the changes in the hand position in consecutive frames and predicts the hand position in the next frame. Compared with other tracking algorithms, it is faster and can be used in conjunction with the hand detection network to implement a "multi-frame one detection" optimization strategy, further improving the efficiency of real-time applications. It helps accelerate the hand positioning process and optimizes the stability of hand detection, thereby improving the performance of the overall algorithm.

[0085] In this way, a preset number of hand key points can be set in advance, wherein the two-dimensional coordinates of the preset number of hand key points are predicted based on the two-dimensional coordinates of the hand key points of the previous frame to be processed using the Kalman filter algorithm. The two-dimensional coordinates of the remaining hand key points are determined using the hand key point estimation network.

[0086] After obtaining the initial hand image, we will now describe how the hand keypoint estimation network determines the two-dimensional coordinates of the hand keypoints. To ensure overall computational efficiency, the hand keypoint estimation network has also been lightweighted and improved. While maintaining accuracy, the following optimization process is provided.

[0087] In step S204, the initial hand image is input into the hand key point estimation network, and at least one depthwise separable convolution is performed on the initial hand image to obtain the target hand feature vector of the initial hand image. Specifically, Figure 10 and Figure 11 ,The process of target hand feature vector is explained, Figure 10 Flowchart showing a process of extracting target hand feature vectors, Figure 11 Shown is a structural diagram of a hand key point estimation network:

[0088] S101: Input the initial hand image into the hand key point estimation network, perform conventional convolution processing on the initial hand image, and obtain a sixth hand feature vector.

[0089] Exemplarily, the initial hand image is input into the hand key point estimation network, conventional convolution processing is performed in the network layer D21, and the activation function Relu6 is used to obtain the sixth hand feature vector.

[0090] S102: Perform depthwise separable convolution processing on the sixth hand feature vector to obtain a seventh hand feature vector.

[0091] For example, application Figure 5 The residual unit shown, and Figure 6 The downsampled residual unit shown performs depthwise separable convolution on the sixth hand feature vector at network layer D22 to obtain the seventh hand feature vector. Other forms of residual units and downsampled residual units can also be used and are not limited here. The depthwise separable convolution uses a 3x3 channel-by-channel convolution and a 1x1 normal convolution instead of a conventional 3x3 convolution, effectively reducing the number of parameters and computation without significantly affecting the performance.

[0092] S103 , performing hourglass processing on the seventh hand eigenvector to obtain an eighth hand eigenvector.

[0093] For example, the seventh hand feature vector is subjected to hourglass processing at the network layer D23 to obtain the eighth hand feature vector. The hourglass processing can improve the effect of hand posture estimation.

[0094] S104: Perform depthwise separable convolution processing on the eighth eigenvector to obtain a target hand eigenvector.

[0095] In the network layer D24, the eighth eigenvector is subjected to depthwise separable convolution, which can be applied Figure 5 The residual unit shown is implemented; the result is further subjected to depthwise separable convolution in network layer D25 to obtain the target hand feature vector. As mentioned above, D21-D24 constitute the backbone network of the hand key point estimation network.

[0096] In step S205, the hand keypoint estimation network is used to predict a heatmap of the 2D coordinates of the hand keypoints based on the target hand feature vector. In network layer D26, a differentiable maximum index operation is performed on this heatmap using Soft-argmax to find the coordinate output with the highest probability, resulting in the 2D coordinates of the hand keypoints.

[0097] In this way, the heat map post-processing process, which was originally difficult to accelerate, is embedded in the inference process of the hand key point estimation network. Combined with the neural network framework, end-to-end accelerated inference of two-dimensional estimation of hand key points can be achieved, greatly reducing the time consumption of the overall algorithm.

[0098] In step S206, the three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera. This process can be achieved by combining the projection equations of the two cameras using the principle of binocular stereo imaging and combining the internal and external parameters of the cameras to solve the three-dimensional coordinates of the corresponding hand key points.

[0099] In addition, since fisheye cameras have a larger field of view and a smaller focal length than ordinary cameras, they are more widely applicable in miniature smart wearable devices such as AR / VR glasses. However, their imaging may be distorted, and additional correction processing is required in actual applications. In this way, in order to improve the positioning accuracy, after obtaining the two-dimensional coordinates of the key points of the hand, it is determined whether the image to be processed is distorted (for example, it is taken with a fisheye camera). If so, the distortion coefficient is applied to correct the two-dimensional coordinates of the key points of the hand. Then, the corrected two-dimensional coordinates of the key points of the hand and the internal and external parameters of the binocular camera are used to determine the three-dimensional coordinates of the key points of the hand. In this way, the method of the embodiment of the present application can take into account the positioning of hands with ordinary images and distorted images, greatly improving the scope of application of the algorithm.

[0100] In order to make the process of this application clearer, two complete flowcharts are used below to illustrate the process of hand positioning. Figure 12 and Figure 13 .

[0101] Figure 12 A flow chart of a hand positioning method is shown.

[0102] S1201. Obtain an image to be processed captured by a binocular camera of a VR device.

[0103] S1202: Perform conventional convolution on the image to be processed to obtain a first hand feature vector.

[0104] S1203 . Perform a depthwise separable convolution on the first hand feature vector to obtain a second hand feature vector.

[0105] S1204 , fusing the result of performing maximum pooling on the second hand feature vector and the result of performing depthwise separable convolution on the second hand feature vector to obtain an initial hand feature vector of the image to be processed.

[0106] S1205 . Perform a depthwise separable convolution on the initial hand feature vector to obtain a third hand feature vector.

[0107] S1206 , performing upsampling convolution on the third hand feature vector, and fusing the result with the second hand feature vector to obtain a fourth hand feature vector.

[0108] S1207 . Perform a depthwise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector.

[0109] S1208 : Perform conventional convolution on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

[0110] S1209: Input the initial hand image into the hand key point estimation network, perform conventional convolution processing on the initial hand image, and obtain a sixth hand feature vector.

[0111] S1210 : Perform depthwise separable convolution processing on the sixth hand feature vector to obtain a seventh hand feature vector.

[0112] S1211. Perform hourglass processing on the seventh hand eigenvector to obtain an eighth hand eigenvector.

[0113] S1212: Perform depthwise separable convolution processing on the eighth eigenvector to obtain a target hand eigenvector.

[0114] S1213. Perform a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points.

[0115] S1214: If the image to be processed is distorted, apply the distortion coefficient to correct the two-dimensional coordinates of the key points of the hand.

[0116] S1215. Determine the three-dimensional coordinates of the hand key points using the corrected two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0117] The embodiments of the present application reduce the dependence on large-scale high-quality annotated data, and reduce the difficulty and cost of development. Through the lightweight hand key point estimation network and hand detection network, while improving the running speed, the storage resource usage is reduced, which is conducive to the lightweight deployment of portable wearable devices (such as VR devices). In addition, the hand detection network and hand key point estimation network based on deep learning can be trained using distorted images imaged by fisheye cameras, so that it supports direct positioning of hands and key points on distorted images, and then combined with the two-dimensional coordinates of the hand key points predicted by the provided camera distortion coefficient correction, the corresponding three-dimensional space coordinates can be solved in conjunction with the binocular stereo vision principle, which is compatible with the hand posture estimation and tracking of normal images and distorted images, unifying the two under the same algorithm framework, greatly enriching the scalability of the algorithm.

[0118] Figure 13 A flow chart of another hand positioning method is shown.

[0119] S131. Obtain an image to be processed captured by a binocular camera of a VR device.

[0120] After acquiring the binocular camera images, the two images are scaled and padded to a uniform 256*256 resolution, and then input into the pre-trained hand detection network.

[0121] S132: Apply a hand key point estimation network to identify the position of the hand in the image to be processed.

[0122] After post-processing the hand detection network's output, the hand's position in the image is determined, and the hand can be identified as left or right. This hand detection network has been lightweighted and uses a specific neural network framework for inference acceleration, ensuring efficient operation. Furthermore, by introducing a specially designed hand tracking algorithm (the Kalman Green Wave algorithm), the hand detection network can be run only once across several consecutive frames, further improving the overall efficiency of hand localization.

[0123] S133 , trimming the image to be processed to obtain a hand layout image (initial hand image).

[0124] After obtaining the hand positions in the two images, the correspondence between the hand targets in the two images can be obtained by combining the left and right hand categories obtained at the same time, thereby determining which targets are the same hand, and performing partial image cropping on each of the two original images to obtain a partial image containing only the hand.

[0125] S134. Apply a hand detection network to identify the two-dimensional coordinates of the hand key points in the hand partial image.

[0126] By feeding the resulting partial hand image into a pre-trained hand keypoint estimation network, the positions of 21 hand keypoints (16 joints and 5 fingertips) in the image are determined, and their positions in the original image before cropping can be restored. This hand keypoint estimation network uses a lightweight tail network and accelerates inference using a specific neural network framework, ensuring real-time performance.

[0127] S135: If the binocular camera is a fisheye camera, apply the distortion coefficient to correct the two-dimensional coordinates of the key points of the hand.

[0128] S136. Determine the three-dimensional coordinates of the hand key points using the corrected two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0129] Before calculating the 3D coordinates of the hand key points, it's necessary to determine whether the input original image was taken with a fisheye camera. This is done by determining whether the distortion coefficient in the input binocular camera parameters is zero. If the distortion coefficient is not zero, the obtained 2D coordinates of the hand key points are corrected to obtain the corrected 2D coordinates. If the distortion coefficient is zero, no correction is required and the 3D coordinates are calculated directly. Using the principle of binocular stereo imaging, the projection equations of the two cameras are combined, and the camera internal and external parameters are combined to solve for the corresponding 3D coordinates of the hand key points.

[0130] like Figure 14 As shown, based on the same inventive concept, an embodiment of the present invention provides a hand positioning device, including an image acquisition module 141, a first processing module 142, a second processing module 143, a third processing module 144, a first coordinate determination module 145 and a second coordinate determination module 146.

[0131] The image acquisition module 141 is used to acquire the image to be processed captured by the binocular camera of the VR device;

[0132] A first processing module 142 is configured to input the image to be processed into a hand detection network, perform at least one depthwise separable convolution on the image to be processed, and extract an initial hand feature vector of the image to be processed;

[0133] The second processing module 143 is used to perform convolution on the initial hand feature vector to generate two initial hand images of different scales;

[0134] a third processing module 144, configured to input the initial hand image into a hand key point estimation network, perform at least one depthwise separable convolution on the initial hand image, and obtain a target hand feature vector of the initial hand image;

[0135] A first coordinate determination module 145 is configured to perform a differentiable maximum index operation on a heat map of two-dimensional coordinates of hand key points predicted by a target hand feature vector to obtain two-dimensional coordinates of the hand key points;

[0136] The second coordinate determination module 146 is used to determine the three-dimensional coordinates of the hand key points based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0137] In some exemplary embodiments, the first processing module 142 is specifically configured to:

[0138] Perform conventional convolution on the image to be processed to obtain the first hand feature vector;

[0139] Performing depth-wise separable convolution on the first hand feature vector to obtain a second hand feature vector;

[0140] The result of maximum pooling on the second hand feature vector is fused with the result of depthwise separable convolution on the second hand feature vector to obtain the initial hand feature vector of the image to be processed.

[0141] In some exemplary embodiments, the second processing module 143 is specifically configured to:

[0142] Perform depth-wise separable convolution on the initial hand feature vector to obtain the third hand feature vector;

[0143] After upsampling and convolving the third hand feature vector, it is fused with the second hand feature vector to obtain the fourth hand feature vector;

[0144] Performing depth-wise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector;

[0145] Conventional convolution is performed on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

[0146] In some exemplary embodiments, the third processing module 144 is specifically configured to:

[0147] Input the initial hand image into the hand key point estimation network, perform conventional convolution processing on the initial hand image, and obtain the sixth hand feature vector;

[0148] Performing depthwise separable convolution on the sixth hand feature vector to obtain a seventh hand feature vector;

[0149] Performing hourglass processing on the seventh hand eigenvector to obtain the eighth hand eigenvector;

[0150] Perform depth-wise separable convolution on the eighth eigenvector to obtain the target hand eigenvector.

[0151] In some exemplary embodiments, the two-dimensional coordinates of a preset number of hand key points are determined by applying a Kalman filter algorithm based on the two-dimensional coordinates of the hand key points in the previous frame of the image to be processed.

[0152] In some exemplary embodiments, the system further includes a judgment module configured to, after obtaining the two-dimensional coordinates of the key points of the hand, judge whether the image to be processed is distorted, and if so, apply the distortion coefficient to correct the two-dimensional coordinates of the key points of the hand;

[0153] The second coordinate determination module 146 is specifically configured to:

[0154] The 3D coordinates of the hand key points are determined by applying the corrected 2D coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0155] Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will be omitted.

[0156] like Figure 15 As shown, based on the same inventive concept, an embodiment of the present invention provides a VR device, which includes a processor 151, a memory 152 and at least one external communication interface 153, and the processor 151, the memory 152 and the external communication interface 153 are all connected through a bus 154;

[0157] The external communication interface 153 is configured to receive the image to be processed captured by the binocular camera of the VR device;

[0158] The memory 152 stores a computer program, and the processor 151 is configured to perform the following operations based on the computer program:

[0159] Input the image to be processed into the hand detection network, perform at least one depth-separable convolution on the image to be processed, and extract the initial hand feature vector of the image to be processed;

[0160] Convolve the initial hand feature vector to generate two initial hand images of different scales;

[0161] Input the initial hand image into the hand key point estimation network, perform at least one depth-separable convolution on the initial hand image, and obtain the target hand feature vector of the initial hand image;

[0162] Perform a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points;

[0163] Based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera, the three-dimensional coordinates of the hand key points are determined.

[0164] In some exemplary embodiments, the processor 151 is configured to:

[0165] Perform conventional convolution on the image to be processed to obtain the first hand feature vector;

[0166] Performing depth-wise separable convolution on the first hand feature vector to obtain a second hand feature vector;

[0167] The result of maximum pooling on the second hand feature vector is fused with the result of depthwise separable convolution on the second hand feature vector to obtain the initial hand feature vector of the image to be processed.

[0168] In some exemplary embodiments, the processor 151 is further configured to:

[0169] Perform depth-wise separable convolution on the initial hand feature vector to obtain the third hand feature vector;

[0170] After upsampling and convolving the third hand feature vector, it is fused with the second hand feature vector to obtain the fourth hand feature vector;

[0171] Performing depth-wise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector;

[0172] Conventional convolution is performed on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

[0173] In some exemplary embodiments, the processor 151 is further configured to:

[0174] Input the initial hand image into the hand key point estimation network, perform conventional convolution processing on the initial hand image, and obtain the sixth hand feature vector;

[0175] Performing depthwise separable convolution on the sixth hand feature vector to obtain a seventh hand feature vector;

[0176] Performing hourglass processing on the seventh hand eigenvector to obtain the eighth hand eigenvector;

[0177] Perform depth-wise separable convolution on the eighth eigenvector to obtain the target hand eigenvector.

[0178] In some exemplary embodiments, the two-dimensional coordinates of a preset number of hand key points are predicted based on the two-dimensional coordinates of the hand key points of the previous frame of the image to be processed by applying a Kalman filter algorithm.

[0179] In some exemplary embodiments, the processor 151 is configured to: after obtaining the two-dimensional coordinates of the key points of the hand, determine whether the image to be processed is distorted, and if so, apply the distortion coefficient to correct the two-dimensional coordinates of the key points of the hand;

[0180] The processor 151 is configured to execute:

[0181] The 3D coordinates of the hand key points are determined by applying the corrected 2D coordinates of the hand key points and the internal and external parameters of the binocular camera.

[0182] An embodiment of the present invention further provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned hand positioning method.

[0183] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0187] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A hand positioning method, characterized in that: Applied to VR devices, including: Obtaining an image to be processed captured by a binocular camera of the VR device; Inputting the image to be processed into a hand detection network, performing at least one depthwise separable convolution on the image to be processed, and extracting an initial hand feature vector of the image to be processed; performing convolution on the initial hand feature vector to generate two initial hand images of different scales; Inputting a partial hand image into a hand key point estimation network, performing at least one depthwise separable convolution on the partial hand image to obtain a target hand feature vector of the partial hand image; wherein the partial hand image is determined by constructing a feature pyramid based on the two initial hand images of different scales, performing hand classification and hand rectangular region regression processing through the feature pyramid to determine the position of the hand in the image to be processed, and cropping the image to be processed according to the position to obtain the partial hand image; Performing a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points; The three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

2. The method according to claim 1, wherein The step of performing at least one depthwise separable convolution on the image to be processed to extract an initial hand feature vector of the image to be processed includes: Performing conventional convolution on the image to be processed to obtain a first hand feature vector; Performing a depthwise separable convolution on the first hand feature vector to obtain a second hand feature vector; A result of performing maximum pooling on the second hand feature vector and a result of performing depthwise separable convolution on the second hand feature vector are fused to obtain an initial hand feature vector of the image to be processed.

3. The method according to claim 2, characterized in that The convolution of the initial hand feature vector to generate two initial hand images of different scales includes: Performing a depthwise separable convolution on the initial hand feature vector to obtain a third hand feature vector; performing upsampling convolution on the third hand feature vector and fusing the resultant vector with the second hand feature vector to obtain a fourth hand feature vector; performing a depthwise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector; Conventional convolution is performed on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

4. The method according to claim 1, wherein The step of inputting the hand partial image into a hand key point estimation network and performing at least one depthwise separable convolution on the hand partial image to obtain a target hand feature vector of the hand partial image includes: Inputting the hand partial image into a hand key point estimation network, performing conventional convolution processing on the hand partial image to obtain a sixth hand feature vector; performing depthwise separable convolution processing on the sixth hand feature vector to obtain a seventh hand feature vector; performing hourglass processing on the seventh hand eigenvector to obtain an eighth hand eigenvector; Perform depthwise separable convolution processing on the eighth hand feature vector to obtain a target hand feature vector.

5. The method according to claim 1, wherein The two-dimensional coordinates of the preset number of hand key points are predicted based on the two-dimensional coordinates of the hand key points of the previous frame of image to be processed by applying the Kalman filter algorithm.

6. The method according to claim 1, characterized in that After obtaining the two-dimensional coordinates of the key points of the hand, determining whether the image to be processed is distorted, and if so, applying the distortion coefficient to correct the two-dimensional coordinates of the key points of the hand; The determining of the three-dimensional coordinates of the hand key points based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera includes: The three-dimensional coordinates of the hand key points are determined by applying the corrected two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

7. A VR device, characterized in that: comprising a processor, a memory, and at least one external communication interface, wherein the processor, the memory, and the external communication interface are all connected via a bus; The external communication interface is configured to receive the image to be processed captured by the binocular camera of the VR device; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Inputting the image to be processed into a hand detection network, performing at least one depthwise separable convolution on the image to be processed, and extracting an initial hand feature vector of the image to be processed; performing convolution on the initial hand feature vector to generate two initial hand images of different scales; Inputting a partial hand image into a hand key point estimation network, performing at least one depthwise separable convolution on the partial hand image to obtain a target hand feature vector of the partial hand image; wherein the partial hand image is determined by constructing a feature pyramid based on the two initial hand images of different scales, performing hand classification and hand rectangular region regression processing through the feature pyramid to determine the position of the hand in the image to be processed, and cropping the image to be processed according to the position to obtain the partial hand image; Performing a differentiable maximum index operation on the heat map of the two-dimensional coordinates of the hand key points predicted by the target hand feature vector to obtain the two-dimensional coordinates of the hand key points; The three-dimensional coordinates of the hand key points are determined based on the two-dimensional coordinates of the hand key points and the internal and external parameters of the binocular camera.

8. The VR device according to claim 7, wherein: The processor is configured to: Performing conventional convolution on the image to be processed to obtain a first hand feature vector; Performing a depthwise separable convolution on the first hand feature vector to obtain a second hand feature vector; A result of performing maximum pooling on the second hand feature vector and a result of performing depthwise separable convolution on the second hand feature vector are fused to obtain an initial hand feature vector of the image to be processed.

9. The VR device according to claim 8, wherein: The processor is further configured to: Perform depth-wise separable convolution on the initial hand feature vector to obtain a third hand feature vector performing upsampling convolution on the third hand feature vector and fusing the resultant vector with the second hand feature vector to obtain a fourth hand feature vector; performing a depthwise separable convolution on the fourth hand feature vector to obtain a fifth hand feature vector; Conventional convolution is performed on the third hand feature vector and the fifth hand feature vector respectively to obtain two initial hand images of different scales.

10. The VR device according to claim 7, wherein: The processor is further configured to: Inputting the hand partial image into a hand key point estimation network, performing conventional convolution processing on the hand partial image to obtain a sixth hand feature vector; performing depthwise separable convolution processing on the sixth hand feature vector to obtain a seventh hand feature vector; performing hourglass processing on the seventh hand eigenvector to obtain an eighth hand eigenvector; Perform depthwise separable convolution processing on the eighth hand feature vector to obtain a target hand feature vector.

Images