A gesture 3D key point detection method for AR device and electronic device

Abstract

The present application relates to a method for detecting 3D key points of gestures of an AR device, the method comprising: acquiring a first image and a second image through a binocular camera, acquiring rough 3D key points of the first image through a detection network or a previous frame image, further, obtaining more accurate optimized 2D key point information through an optimization network, and calculating monocular 3D key points based on the optimized 2D key points. The monocular 3D key points are projected onto the second image, and the relative depth information and 2D key point coordinates of the second image are obtained through the optimization network. Finally, the 2D key point coordinates of the first image and the second image are fused to calculate the binocular gesture 3D key points and palm size. Through this method, the problem of poor accuracy of gesture 3D key point prediction results in the related art is solved, while taking into account both accuracy and efficiency.

Description

Technical Field

[0001] The present application relates to the field of augmented reality, and in particular to a gesture 3D key point detection method and electronic device for an AR device. Background Art

[0002] With the development of augmented reality (AR) technology, gesture interaction has become an indispensable part of it. Users can use some natural gestures that are in line with their habits to realize interactive operations such as clicking, selecting, moving, and scaling 3D objects in augmented reality scenes.

[0003] In the related technology, for AR devices, the common gesture 3D key point detection method is as follows:

[0004] 1. Use a depth camera module with a large field of view to predict gesture key points. However, AR devices using this solution generally have the problem of high power consumption and high cost.

[0005] 2. Obtain images through a monocular camera, and estimate 3D key points based on the monocular image combined with a preset human hand model or human hand size. In this method, if interactive operations are to be achieved, the size of the user's palm needs to be measured in advance, which goes against the original intention of natural interaction and the steps are also relatively cumbersome.

[0006] 3. Use a binocular camera to obtain two images, run a 2D algorithm on each of the two images to obtain 2D key points, and then calculate the 3D key points based on the internal and external parameters of the binocular camera. However, when complex situations such as occlusion occur, this method will cause a certain degree of 2D error in the two images obtained, which will lead to a larger error in the final output 3D key points. Therefore, it is easy to cause hand deformation and sudden changes in the position of the operated virtual object. In addition, since this method needs to run the algorithm twice for each frame of the image, it cannot take into account both efficiency and accuracy.

[0007] Currently, no effective solution has been proposed for the problem of poor accuracy of gesture 3D key point prediction results in related technologies. Summary of the invention

[0008] The embodiments of the present application provide a method for detecting 3D key points of gestures of an AR device and an electronic device, so as to at least solve the problem of poor accuracy of 3D key point prediction results of gestures in the related art.

[0009] In a first aspect, an embodiment of the present application provides a method for detecting 3D key points of gestures of an AR device, the method comprising:

[0010] Step 101: Acquire a first image and a second image through a binocular camera, wherein the first image and the second image are respectively taken by a first camera module and a second camera module of the binocular camera at the same time;

[0011] Step 102: query whether there is a previous frame image containing palm information, if so, obtain the rough 2D key point information of the palm in the first image based on the previous frame image, if not, obtain the rough 2D key point information of the palm in the first image through a key point detection network;

[0012] Step 103: performing a preprocessing step on the rough 2D key point information to obtain an input image and input key point information for an optimization network, and obtaining relative depth information of the first image and optimized 2D key point coordinates through the optimization network;

[0013] Step 104: Calculate the monocular 3D key point coordinates of the first image according to the palm size information, the relative depth information of the first image and the optimized 2D key point coordinates;

[0014] Step 105: Determine whether the frame sequence of the first image is an integer multiple of a preset frame interval, if so, execute step 106, if not, execute step 108;

[0015] Step 106: projecting the monocular 3D key point coordinates to the second image, and obtaining relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network;

[0016] Step 107: Based on the preset optimization problem, the relative depth information of the second image and the optimized 2D key point coordinates, the binocular 3D key point coordinates are calculated by the least square method, and the size information of the palm is updated according to the binocular 3D key point coordinates;

[0017] Step 108: Initialize the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, and re-execute steps 102 to 108.

[0018] In some embodiments, obtaining rough 2D key point information of the palm in the first image through a key point detection network includes:

[0019] The backbone network extracts features of the first image to obtain a downsampled feature map;

[0020] Converting the feature map into an output layer of a preset dimension;

[0021] The output layer is parsed, and the left and right hand information and the rough 2D key point information in the first image are obtained according to the confidence level.

[0022] In some embodiments, obtaining relative depth information of the first image and optimizing 2D key point coordinates through an optimization network includes:

[0023] Acquire the input image, and preliminarily extract features of the input image to obtain a first initial feature map;

[0024] Acquire the input key point information, render according to Gaussian distribution based on the input key point information to obtain a second initial feature map, and complete splicing of the second initial feature map with the first initial feature map at the channel layer to obtain a first fused feature map;

[0025] Based on the first fused feature map, feature extraction and convolution operations are performed through the backbone network to obtain relative depth information of the first image and optimized 2D key point coordinates.

[0026] In some embodiments, the step of performing feature extraction and convolution operation on the backbone network to obtain relative depth information of the first image and optimize 2D key point coordinates includes:

[0027] The backbone network extracts features from the first fused feature map to obtain a second fused feature map;

[0028] The second fused feature map is subjected to convolution processing of a first preset dimension to obtain a first key point feature map, and the optimized 2D key point coordinates of the first image are calculated according to the first key point feature map;

[0029] Further extracting features from the second fused feature map to obtain a second key point feature map, wherein the second key point feature map has the same size as the first key point feature map;

[0030] The first key point feature map is multiplied by the second key point feature map, and the result of the multiplication is average pooled to obtain relative depth information of the first image, and,

[0031] Adding a convolution of a second preset dimension to the second fused feature map to obtain a one-dimensional score of the second fused feature map;

[0032] Determine whether the one-dimensional score is greater than a preset score threshold. If so, the palm is robust and continue to calculate the optimized 2D key point coordinates. If not, the palm is lost and step 102 is executed.

[0033] In some embodiments, projecting the monocular 3D key point coordinates to the second image, and acquiring relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network includes:

[0034] Projecting the monocular 3D key point coordinates onto the second image according to the external parameters of the first camera module and the second camera module;

[0035] Acquire 2D key point coordinates in the second image, perform a preprocessing step on the 2D key point coordinates, and obtain an input image and input key point information for the optimization network;

[0036] Acquire an input image of the second image, and preliminarily extract features of the input image to obtain a first initial feature map;

[0037] Based on the input key point information of the second image, rendering is performed according to Gaussian distribution to obtain a second initial feature map, and the second initial feature map is spliced ​​with the first initial feature map at the channel layer to obtain a first fused feature map;

[0038] Based on the first fused feature map, feature extraction and convolution operations are performed through the backbone network to obtain relative depth information of the second image and optimized 2D key point coordinates.

[0039] In some embodiments, performing a preprocessing step on the rough 2D key point information to obtain an input image and input key point information for optimizing the network includes:

[0040] Calculate a key point bounding box according to the rough 2D key point information;

[0041] Determine the side length of the square according to the size of the key point bounding box and the pixel distance between the key points;

[0042] Cutting off a square area according to the side length of the square and the center of the key point bounding box to obtain a cut-off image;

[0043] Scaling the captured image to a preset size to obtain an input image for the optimization network;

[0044] The key point coordinates in the captured image are calculated to obtain input key point information for the optimization network.

[0045] In some embodiments, calculating the 3D key point coordinates of the first image according to the size information, the relative depth information of the first image and the optimized 2D key point coordinates includes:

[0046] The absolute position of the palm is calculated according to the size information, the optimized 2D key point coordinates and the internal reference information of the first camera module:

[0047] According to the absolute position, the relative depth information of the first image and the optimized 2D key point coordinates, the monocular 3D key point coordinates are calculated by the following formula:

[0048]

[0049] in, is the 3D key point coordinate, D is the absolute position, d is the relative depth information of the first image, k x , k y 、c x 、c y They are respectively the internal parameters of the first camera module.

[0050] In some embodiments, initializing the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates includes:

[0051] The previous frame image is directly initialized according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, or,

[0052] Predict change information of a next frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, and initialize the previous frame image in combination with the change information.

[0053] In some embodiments, updating the size information of the palm according to the binocular 3D key point coordinates includes:

[0054] The size information of the palm is calculated according to the binocular 3D key points, and the size information is filtered based on a preset filtering rule to update the size information of the palm.

[0055] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a gesture key point detection method for an AR device as described in the first aspect when executing the computer program.

[0056] Compared with the related art, the gesture 3D key point detection method of an AR device provided in the embodiment of the present application has at least the following beneficial effects:

[0057] 1. By optimizing the network and continuing to optimize on the basis of known rough 2D key points, more accurate optimized 2D key points can be calculated.

[0058] 2. By projecting the monocular 3D result onto another image and optimizing the network, it is possible to ensure that the binocular 2D key points are as close as possible. Furthermore, the final 3D key points will not have any abnormalities, and the hand shape will be more stable, which can meet the interaction requirements of hand shape consistency and stability.

[0059] 3. It is not necessary to run the binocular algorithm for every frame. You can flexibly set it to run the binocular algorithm every N frames according to the hardware performance, and only run the monocular algorithm in other cases. It can ensure accuracy while taking into account efficiency and reducing power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 This is a schematic diagram of an application scenario of a gesture 3D key point detection method for an AR device according to an embodiment of the present application;

[0062] Figure 2 is a flow chart of a gesture key point detection method of an AR device according to an embodiment of the present application;

[0063] Figure 3 is a flow chart of obtaining rough 2D key points through a detection network according to an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a heatmap according to an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of key points of a palm according to an embodiment of the present application;

[0066] Figure 6 is a flow chart of optimizing a network according to an embodiment of the present application;

[0067] Figure 7 is a schematic diagram of a second initial characteristic graph according to an embodiment of the present application;

[0068] Figure 8 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0070] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0071] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0072] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0073] The embodiment of the present application provides a method for detecting 3D key points of gestures of an AR device, which can be applied in the following aspects: Figure 1 In the application scenario shown, Figure 1is a schematic diagram of an application scenario of a gesture 3D key point detection method for an AR device according to an embodiment of the present application, such as Figure 1 As shown, after the AR device is initialized, the augmented reality scene is displayed in the user's field of view. The AR device obtains the user's interactive gestures through the camera device, generates operation instructions in the augmented reality scene according to the interactive gestures, and controls the operation of the AR device based on the operation instructions. In the entire interactive process, the detection of gesture key points is an important link. The embodiment of the present application is based on a binocular camera algorithm, which fully integrates the two picture information of the left and right cameras, and can obtain more accurate gesture key points, thereby improving the control stability of the AR device, while being compatible with efficiency and accuracy.

[0074] Figure 2 is a flowchart of a gesture key point detection method for an AR device according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0075] S201: Acquire a first image and a second image through a binocular camera, wherein the first image and the second image are respectively taken by a first camera module and a second camera module of the binocular camera at the same time;

[0076] The binocular camera can be set on a head-mounted display device or on a hardware device with an AR operating environment, such as a smartphone or tablet computer.

[0077] S202: Check whether there is a previous frame image containing palm information. If so, obtain rough 2D key point information of the palm in the first image based on the previous frame image. If not, obtain rough 2D key point information of the palm in the first image through a key point detection network.

[0078] It should be noted that in some cases, the binocular camera may not have a previous frame image, for example, when the camera is turned on for the first time, there is no previous frame image. In addition, in some cases, even if there is a previous frame image, it may not include palm information, for example, the user's palm leaves the camera's field of view during the process.

[0079] In view of the above situation, this embodiment can be implemented flexibly. That is, the subsequent steps are performed by judging whether there is a previous frame image containing palm information. If there is, the rough 2D key points of the palm are obtained by combining the previous frame image and the current image; if there is no current image, the rough 2D key points of the current image are obtained by detecting the detection network.

[0080] S203: performing a preprocessing step on the rough 2D key point information to obtain an input image and input key point information for optimizing the network, and obtaining relative depth information of the first image and optimized 2D key point coordinates through the optimization network;

[0081] In this embodiment, the purpose of performing preprocessing on the rough 2D key point information is to convert it into a form that can be recognized and processed by the optimization network;

[0082] The optimization network performs calculations and predictions based on the above input image and input key point information, and can obtain more accurate optimized 2D key points and relative depth information based on the original rough 2D key points.

[0083] S204: Calculating the monocular 3D key point coordinates of the first image according to the palm size information, the relative depth information of the first image and the optimized 2D key point coordinates;

[0084] The size information of the palm is determined according to the distance between key points, and can also be specified manually.

[0085] S205: Determine whether the frame sequence of the first image is an integer multiple of a preset frame interval, if so, execute step S206, if not, execute step S208;

[0086] It can be understood that when step S206 is executed sequentially, that is, based on the monocular 3D key points, the binocular 3D key points are continued to be calculated in combination with the second image, and finally the binocular 3D results are used for key point tracking; and when jumping to execute step S208, the monocular 3D results are directly used for key point tracking.

[0087] In conventional binocular solutions, each frame of an image must run a binocular 2D algorithm on the left and right images to obtain a 2D result, and then the 2D result is fused to obtain a 3D result. Therefore, conventional binocular solutions cannot reduce the frame rate and have poor efficiency.

[0088] In this embodiment, whether to further combine the second image with the 3D result of the first image to run the 3D algorithm depends on the frame sequence of the first image and the preset frame interval. In addition, the preset frame interval can be flexibly set according to the computing power of the hardware device. When the computing power is very high, it can be set to a smaller value, such as 1; otherwise, it can be set to a larger value, such as 5. Therefore, the embodiment of the present application can realize the frame reduction operation of the binocular algorithm, achieving the purpose of balancing accuracy and efficiency.

[0089] S206: Projecting the monocular 3D key point coordinates to the second image according to the external parameters between the first camera module and the second camera module, and obtaining the relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network;

[0090] It should be noted that since the 3D result of the first image is used to initialize the 2D key points of the second image, the information of the two images is fully integrated; therefore, the final 2D result will be consistent with the hand shape of the first image, and the actual calculated 3D error will be much smaller.

[0091] S207: Based on the preset optimization problem and the relative depth information of the second image and the optimized 2D key point coordinates, the binocular 3D key point coordinates are calculated by the least square method, and the size information of the palm is updated according to the binocular 3D key point coordinates;

[0092] The construction of the preset optimization problem is relatively intuitive, that is, to calculate a set of 3D points so that the pixel error projected onto the binocular is as small as possible, and the relative error of the binocular estimation is also as small as possible. The optimization goal is: E = E pix_err +αE depth_err Among them, E pix_err and E depth_err They are pixel error and depth error respectively. α is used to control the weight between pixel error and depth error, which can be weighed according to actual conditions.

[0093] In addition, since more accurate binocular 3D key points are recalculated, the palm size information obtained by updating the binocular 3D key points is also more accurate.

[0094] S208: Initialize the previous frame image in step S202 according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, and re-execute steps S202 to S208.

[0095] It should be noted that, corresponding to the above step S205, when only the monocular algorithm is run on the first image, key point tracking is performed according to the monocular 3D result (i.e., the monocular 3D key point coordinates in step S204) and the subsequent previous frame image is initialized; when the binocular algorithm continues to be run in combination with the second image, key point tracking is performed according to the binocular 3D result (i.e., the binocular 3D key point coordinates in step S207) and the subsequent previous frame image is initialized.

[0096] In some of the embodiments, if there is no previous frame image including palm information, the rough 2D key point information of the palm in the first image is obtained through a key point detection network. Figure 3 is a flow chart of obtaining rough 2D key points through a detection network according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:

[0097] S301, obtaining a first image, and scaling the first image to a uniform size, for example, 256*192*1.

[0098] S302, the backbone network extracts features of the first image to obtain a downsampled feature map; wherein the backbone network can be a common module such as Resnet, MobileNet, Pelee, or a stack of some custom convolutions, as long as it has sufficient ability to extract features. Furthermore, the downsampled feature map obtained by the backbone network has a size of W*H*C. For example, if it is downsampled by 32 times, the feature map size is 8*6*256.

[0099] S303, converting the feature map into an output layer of preset dimensions; wherein, a 1*1*C*(2*K+2) convolution may be used to convert the output layer into a desired W*H*(2+2*K) dimension, wherein K is the number of key points.

[0100] S304, parsing the output layer, and obtaining the left and right hand information and the rough 2D key point coordinates in the first image according to the confidence in each dimension.

[0101] Among them, the first two dimensions of the output layer channel represent the confidence of the hand and the confidence of the left and right hand classification respectively, and the last 2*K dimensions represent the position information of each 2D key point, which is optional. The specific parsing steps are as follows:

[0102] Step 1: Traverse the first bit on all channels of W*H and find all locations where the score is greater than the set threshold (for example, 0.5). This location is the palm area.

[0103] Step 2: Determine the left and right hands based on the second bit of the channel. If it is greater than 0, it is the right hand, otherwise it is the left hand. In this embodiment, only the set of data with the highest score is retained for the left and right hands. This processing can greatly reduce the time consumption of analysis;

[0104] Step 3: Calculate the rough 2D key points of the first image. Assume that the downsampling factor is F, and the output heatmap is denoted by H. Figure 4 is a schematic diagram of a heatmap according to an embodiment of the present application, such as Figure 4 As shown, the data in the i-th column and j-th row are processed, and the coordinates of the k-th key point are calculated according to the following formula 1:

[0105] Formula 1:

[0106] Among them, i, j, and k represent the i-th column, j-th row, and the k-th key point on the i-th column and j-th row in the Heatmap, respectively. is the coordinate of the kth key point on the i-th column and j-th row, F is the preset downsampling ratio, and H represents Heatmap.

[0107] It should be noted that, since there are only two hands of the user in the AR scene, one is the left hand. Therefore, this embodiment uses the above characteristics as a priori knowledge of the detection network. Based on this priori knowledge, the detection network only needs to find the highest score matching the left and right hands in the first image, without executing the complex NMS steps in the detection network, which can better improve efficiency and reduce time consumption.

[0108] In some embodiments, if there is currently a previous frame image including palm information, a rough 2D key point of the palm in the first image is obtained based on the previous frame image, and the specific process includes:

[0109] According to the internal parameters of the first camera module, the 3D key points in the previous frame image are projected onto the first image to obtain 21 rough 2D key points, wherein the coordinates of the rough 2D key points are calculated by the following formula 2:

[0110] Formula 2:

[0111] Among them, K r It is the internal parameter of the first camera module. X, Y, and Z represent the values ​​on each coordinate axis respectively.

[0112] In some embodiments, before inputting into the optimization network, the rough 2D key points need to be preprocessed to obtain input images and input key point information that can be recognized by the optimization network, wherein the preprocessing process includes the following steps:

[0113] Step 1: Construct a bounding box of 21 key points based on the rough 2D key point information. Optionally, the width and height of the key point bounding box are w and h respectively, and the vertex coordinates of the upper left corner are (x tl ,y tl );

[0114] Step 2: Take the center of the key point bounding box as the center, and intercept the square area with the square side length s. If it exceeds the image, fill it with pure black. The square side length s is calculated by the following formula 3:

[0115] Formula 3: s = max{w*α1, h*α1, dis 0_9 * 2}

[0116] Among them, dis 0_9 Yes Figure 5 The pixel distance between key points 0 to 9 shown, Figure 5 is a schematic diagram of the palm key points according to the embodiment of the present application; α1 and α2 are parameters for controlling the preset outward expansion range of the hand, which can be selected according to actual conditions, and can generally be selected as 1.4 and 2.6 respectively. This method can cope with various hand shapes and ensure a hand area of ​​a reasonable size.

[0117] Step 3: Scale the captured image to m*m (e.g. 128) to obtain the input image for optimizing the network, where m is the input size required for the optimization network.

[0118] Step 4: Calculate the coordinates of each 2D key point on the input image using the following formula 4. The 2D key point coordinates are used as the input key point information of the optimization network:

[0119] Formula 4:

[0120] in, represents the input key points used to optimize the network, Represents the original rough 2D key points.

[0121] In some of these embodiments, Figure 6 is a flowchart of optimizing a network according to an embodiment of the present application, such as Figure 6 As shown, obtaining the relative depth information of the first image and optimizing the 2D key point coordinates by optimizing the network includes:

[0122] Step 1: Get an input image of size m*m, preliminarily extract features to get the first initial feature map (Heatmap1). The size of Heatmap1 is m2*m2*c, where m2 can be equal to m, or 0.5 / 0.25 times of m, etc. c represents the number of channels, which can be 32.

[0123] Step 2: Get the input key point information (21 2D key points kpts), and based on the input key points, render the second initial feature map (heatmap2) on the m2*m2*21 data according to the Gaussian distribution. Figure 7 is a schematic diagram of a second initial characteristic diagram according to an embodiment of the present application, such as Figure 7 As shown, there is a larger response at the key corresponding position. In addition, other constants of the Gaussian distribution can be adjusted and set by themselves, which has no core impact on the invention of this application, so it will not be repeated.

[0124] Step 3: Heatmap1 and Heatmap2 are spliced ​​at the channel layer to form the first fused feature map (Heatmap) of m2*m2*(c+21) dimensions. It should be noted that the image information and key point information are fused through the splicing of the channel layer.

[0125] Step 4: The backbone network extracts the features of the first fused feature map to obtain the second fused feature map (Heatmapf) of m3*m3*c3 dimensions, wherein the backbone network can be Pelee, MobileNet, Hourglass, etc.

[0126] Step 5: The second fusion feature map (Heatmapf) is convolved with the first preset dimension (1*1*c3*21) to obtain the first key point feature map H of m3*m3*21 dimensions kpt Based on the first key point feature map H kpt , calculate the kth optimized 2D key point coordinates by the following formula:

[0127] Formula 5:

[0128] The coordinates calculated by the above formula 5 are more accurate and smoother than the method of directly taking the maximum value in the related art.

[0129] Step 6: Further feature extraction is performed on the second fused feature map to obtain a second key point feature map of the same size. The first key point feature map is multiplied by the second key point feature map to obtain a new key point feature map. The result of the multiplication is average pooled to obtain 1*1*21-dimensional relative depth information. Among them, the multiplication of the first key point feature map and the second key point feature map is similar to the "attention mechanism" technique, which can make the optimization network pay more attention to the characteristics of the 2D key point position.

[0130] In some of the embodiments, considering that the palm may move out of the camera's field of view and other tracking instability may occur during the user's use of the AR device, the embodiment of the present application needs to determine whether the palm is robust during the process of obtaining the relative depth information of the first image and optimizing the coordinates of the 2D key points through the backbone network, and the process specifically includes the following steps:

[0131] After the second fused feature map (heatmapf) is globally pooled, a convolution of the second preset dimension 1*1*c3*1 is added to it to obtain a one-dimensional score of the second fused feature map;

[0132] It is determined whether the one-dimensional score is greater than a preset score threshold, such as 0.5. If so, the palm is considered robust and the subsequent logic continues to run; if not, the palm is considered lost and the process jumps to step S202 to re-detect the palm key points.

[0133] In some embodiments, after obtaining the optimized 2D key points, they are further converted into monocular 3D key points, wherein the monocular 3D key point coordinates are calculated according to the palm size information, the relative depth information and the optimized 2D key point coordinates, including:

[0134] The 3D result is calculated using the size information of the palm to determine the size information of the palm, wherein the size information of the palm can be obtained by taking two key points on the palm, such as Figure 5 The length between the key points 0 and 9 is used as the hand size information lengthpalm ; In addition, for the first frame of palm image currently acquired by the camera, since it is initialized using the binocular 3D result stored in the system, it does not automatically calculate the palm size. In this case, you can also manually specify a reasonable value as the palm size information, such as 8.5cm, which will be automatically updated according to the palm image later;

[0135] Furthermore, according to the size information, the optimized 2D key point coordinates and the internal parameters of the first camera module, the absolute position D of the palm is calculated by the following formula 6:

[0136] Formula 6:

[0137]

[0138] Among them, x i ,y i is the 2D coordinate of the i-numbered column, k x , k y , c x , c y is the intrinsic parameter information of the right camera.

[0139] Finally, according to the absolute position, the relative depth information of the first image and the optimized 2D key point coordinates, the 3D key point coordinates are calculated by the following formula 7:

[0140] Formula 7:

[0141] in, is the 3D key point coordinate, D is the absolute position, d is the relative depth information of the first image, k x , k y 、c x 、c y It is the internal reference of the first camera module.

[0142] In some of the embodiments, after projecting the monocular 3D key point to the second image according to the external parameters between the first camera module and the second camera module, projecting the monocular 3D key point coordinates to the second image, and obtaining the relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network include:

[0143] According to the external parameters of the first camera module and the second camera module, the monocular 3D key point coordinates are projected to the second image; the rough 2D key point coordinates in the second image are obtained, and the 2D key point coordinates are preprocessed to obtain the input image and input key point information for optimizing the network; the input image of the second image is obtained, and the features of the input image are preliminarily extracted to obtain the first initial feature map; based on the input key point information of the second image, the second initial feature map is rendered according to the Gaussian distribution to obtain the second initial feature map, and the second initial feature map and the first initial feature map are spliced ​​at the channel layer to obtain the first fused feature map; based on the first fused feature map, the relative depth information of the second image and the optimized 2D key point coordinates are obtained by performing feature extraction and convolution operations through the backbone network;

[0144] It should be noted that the implementation process of "obtaining the relative depth information of the second image and optimizing the key point coordinates by optimizing the network" is similar to the implementation logic of "obtaining the relative depth information of the first image and optimizing the key point coordinates by optimizing the network" in the above embodiment. Those skilled in the art can refer to the above embodiment to implement "obtaining the relative depth information of the second image and optimizing the key point coordinates" without creative work. Therefore, this step will not be repeated in this application.

[0145] In some embodiments, after obtaining more accurate binocular 3D key point coordinates, it is necessary to update the palm size information according to the binocular 3D key point coordinates. The length can be directly recalculated according to the distance between the binocular 3D key point coordinates. palm ; In addition, when calculating length palm When the palm size is filtered, it is necessary to perform a larger filtering on the palm size according to the preset filtering rules to ensure that the palm size is smooth and does not affect the movement of the hand. After this filtering step, the depth estimation is more stable. After actual application, the delay of the palm change in the AR scene can be eliminated.

[0146] In some of the embodiments, initializing the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates includes:

[0147] Directly initialize the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates;

[0148] Predict the change information of the next frame image based on the binocular 3D key point coordinates or the monocular 3D key point coordinates, and then initialize the previous frame image based on the change information.

[0149] The change information may be the offset of the palm. When initializing the previous frame image, the offset is added to the existing 3D key point coordinates for prediction. In addition, the overall change of the palm may be predicted based on parameters such as the rotation speed and translation speed of the palm, and the next frame image may be initialized based on the change.

[0150] In some embodiments, an electronic device is provided, which may be a terminal. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a gesture 3D key point detection method of an AR device is implemented. The display screen of the electronic device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0151] In one embodiment, Figure 8 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 8 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 8 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a gesture 3D key point detection method for an AR device, and the database is used to store data.

[0152] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting 3D key points of gestures of an AR device, characterized in that: The method comprises: Step 101: Acquire a first image and a second image through a binocular camera, wherein the first image and the second image are respectively taken by a first camera module and a second camera module of the binocular camera at the same time; Step 102: query whether there is a previous frame image containing palm information, if so, obtain the rough 2D key point information of the palm in the first image based on the previous frame image, if not, obtain the rough 2D key point information of the palm in the first image through a key point detection network; Step 103: performing a preprocessing step on the rough 2D key point information to obtain an input image and input key point information for an optimization network, and obtaining relative depth information of the first image and optimized 2D key point coordinates through the optimization network; Step 104: Calculate the monocular 3D key point coordinates of the first image according to the palm size information, the relative depth information of the first image and the optimized 2D key point coordinates; Step 105: Determine whether the frame sequence of the first image is an integer multiple of a preset frame interval, if so, execute step 106, if not, execute step 108; Step 106: projecting the monocular 3D key point coordinates to the second image, and obtaining relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network; Step 107: Based on the preset optimization problem, the relative depth information of the second image and the optimized 2D key point coordinates, the binocular 3D key point coordinates are calculated by the least square method, and the size information of the palm is updated according to the binocular 3D key point coordinates; Step 108: Initialize the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, and re-execute steps 102 to 108.

2. The method according to claim 1, characterized in that The acquiring rough 2D key point information of the palm in the first image through the key point detection network includes: The backbone network extracts features of the first image to obtain a downsampled feature map; Converting the feature map into an output layer of a preset dimension; The output layer is parsed, and the left and right hand information and the rough 2D key point information in the first image are obtained according to the confidence level.

3. The method according to claim 2, characterized in that The obtaining the relative depth information of the first image and optimizing the 2D key point coordinates by optimizing the network includes: Acquire the input image, and preliminarily extract features of the input image to obtain a first initial feature map; Acquire the input key point information, render according to Gaussian distribution based on the input key point information to obtain a second initial feature map, and complete splicing of the second initial feature map with the first initial feature map at the channel layer to obtain a first fused feature map; Based on the first fused feature map, feature extraction and convolution operations are performed through the backbone network to obtain relative depth information of the first image and optimized 2D key point coordinates.

4. The method according to claim 3, characterized in that The step of performing feature extraction and convolution operation through the backbone network to obtain relative depth information of the first image and optimize 2D key point coordinates includes: The backbone network extracts features from the first fused feature map to obtain a second fused feature map; The second fused feature map is subjected to convolution processing of a first preset dimension to obtain a first key point feature map, and the optimized 2D key point coordinates of the first image are calculated according to the first key point feature map; Further extracting features from the second fused feature map to obtain a second key point feature map, wherein the second key point feature map has the same size as the first key point feature map; The first key point feature map is multiplied by the second key point feature map, and the result of the multiplication is average pooled to obtain relative depth information of the first image, and, Adding a convolution of a second preset dimension to the second fused feature map to obtain a one-dimensional score of the second fused feature map; Determine whether the one-dimensional score is greater than a preset score threshold. If so, the palm is robust and continue to calculate the optimized 2D key point coordinates. If not, the palm is lost and step 102 is executed.

5. The method according to claim 1, characterized in that The projecting the monocular 3D key point coordinates to the second image, and acquiring the relative depth information of the second image and optimizing the 2D key point coordinates through the optimization network includes: Projecting the 3D monocular key point coordinates onto the second image according to the external parameters of the first camera module and the second camera module; Acquire the rough 2D key point coordinates in the second image, perform a preprocessing step on the 2D key point coordinates, and obtain an input image and input key point information for the optimization network; Acquire an input image of the second image, and preliminarily extract features of the input image to obtain a first initial feature map; Based on the input key point information of the second image, rendering is performed according to Gaussian distribution to obtain a second initial feature map, and the second initial feature map is spliced ​​with the first initial feature map at the channel layer to obtain a first fused feature map; Based on the first fused feature map, feature extraction and convolution operations are performed through the backbone network to obtain relative depth information of the second image and optimized 2D key point coordinates.

6. The method according to claim 1, characterized in that The performing of a preprocessing step on the rough 2D key point information to obtain an input image and input key point information for optimizing the network comprises: Calculate a key point bounding box according to the rough 2D key point information; Determine the side length of the square according to the size of the key point bounding box and the pixel distance between the key points; Cutting off a square area according to the side length of the square and the center of the key point bounding box to obtain a cut-off image; Scaling the captured image to a preset size to obtain an input image for the optimization network; The key point coordinates in the captured image are calculated to obtain input key point information for the optimization network.

7. The method according to claim 1, characterized in that Calculating the 3D key point coordinates of the first image according to the size information, the relative depth information of the first image and the optimized 2D key point coordinates includes: The absolute position of the palm is calculated according to the size information, the optimized 2D key point coordinates and the internal reference information of the first camera module: According to the absolute position, the relative depth information of the first image and the optimized 2D key point coordinates, the monocular 3D key point coordinates are calculated by the following formula: in, is the 3D key point coordinate, D is the absolute position, d is the relative depth information of the first image, k x , k y 、c x 、c y They are respectively the internal parameters of the first camera module.

8. The method according to claim 1, characterized in that Initializing the previous frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates includes: The previous frame image is directly initialized according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, or, Predict change information of a next frame image according to the binocular 3D key point coordinates or the monocular 3D key point coordinates, and initialize the previous frame image in combination with the change information.

9. The method according to claim 1, characterized in that: The updating of the palm size information according to the binocular 3D key point coordinates comprises: The size information of the palm is calculated according to the binocular 3D key points, and the size information is filtered based on a preset filtering rule to update the size information of the palm.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, a gesture key point detection method for an AR device according to any one of claims 1 to 9 is implemented.

Images