A real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera

Abstract

The present invention discloses a real-time detection method for key points of both thumbs based on a close-range monocular grayscale camera. By taking lens distortion into consideration while adopting target detection and key point detection, the robustness-enhancing feature of target detection and the high-precision feature of key point detection are integrated to jointly solve the real-time and precision problems of thumb key point detection during device use. For the task of thumb key point detection, a network loss function is designed as an optimization target. An optimized deep network is obtained through supervised learning and statistical gradient descent to solve the problem of thumb key point detection. At the same time, according to the real-time requirement, a lightweight network structure is designed in the present invention. A small convolution kernel is adopted and a large number of parameters are reused to reduce the parameter magnitude of the network. The task of thumb feature point detection is achieved while reducing the amount of computation and memory usage.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera. Background Art

[0002] In image processing, a key point is essentially a feature. It is an abstract description of a fixed area or spatial physical relationship, describing the combination or contextual relationship within a certain neighborhood.

[0003] It is not just a point information or represents a location, but also represents the combined relationship between the context and the surrounding neighborhood.

[0004] Key point detection refers to finding points with special properties in an image. These points usually have a high ability to represent image features.

[0005] The core idea of key point detection is to find local maxima in an image, which usually correspond to features such as edges and corners in the image.

[0006] The goal of key point detection is to use computers to find the coordinates of these points in an image. As a basic task in the field of computer vision, key point detection is of vital importance for high-level tasks such as recognition and classification.

[0007] Keypoint detection, also known as keypoint localization or keypoint alignment, may have slightly different names in different tasks.

[0008] For example, facial key point positioning is called facemark alignment, and human key point detection is called pose alignment.

[0009] Typically, the input is an image containing a target, such as an image of a face, a body, or a hand, and the output is a set of predefined key point locations, such as facial features and facial contours, various joints of the body, various joints of the hand, etc.

[0010] The process of computer vision key point detection can be divided into the following stages:

[0011] First generation computer vision: based on manual engineering methods, such as Hough transform, template matching, etc.

[0012] Second-generation computer vision: machine learning-based methods, such as support vector machines and decision trees.

[0013] Third-generation computer vision: methods based on deep learning, such as convolutional neural networks, recurrent neural networks, etc.

[0014] At present, key point detection algorithms can be divided into two categories:

[0015] 1) According to the format of model output, it can be divided into regression method and heatmap method.

[0016] The regression method directly regresses the coordinates of key points. The input image is directly sent to the convolutional network, and then processed by the fully connected layer to directly output the coordinates of the key points. Common methods include DeepPose, MTCNN, and MobileNet.

[0017] The regression model is relatively simple and can be differentiated throughout the process.

[0018] In addition, the amount of calculation is small and the processing speed is relatively faster.

[0019] At the same time, because all its key points come from the last feature map, it has better continuity and stability.

[0020] However, the fully connected layer will cause the model to overfit easily, making it difficult to train, and the model lacks the ability to generalize spatially.

[0021] The solution is to use CNN layer instead of fully connected layer to optimize the problem.

[0022] The heatmap method uses a groundtruth and network output as an image. The image contains multiple channels, each of which represents a type of key point. There are as many channels as there are key points.

[0023] The observation points in each channel are distributed with point as the center and r as Gaussian. The simplest way to extract key points from an image is to extract the position of the point with the maximum pixel value, that is, the coordinates of the key points.

[0024] Its structural input graph is processed by the convolutional network and then outputs a heatmap.

[0025] Common methods include Openpose, HRnet, and Restnet. The heatmap method is widely used in practice. It can make full use of the adjacent key points and spatial information, so its accuracy can be higher.

[0026] However, key point detection is currently mainly a general need, such as facial key point detection, human key point detection, and key point detection of all hand joints. However, when wearing AR glasses for mobile gaming, it is necessary to use the camera mounted on the AR glasses to identify the user's gaming finger (usually the thumb), so that the virtual user's thumb can be displayed on the game screen displayed on the AR glasses, thereby increasing the user's entertainment. However, due to lens distortion, the finger target occupies a lot of pixels in the field of view. The lens distortion has a great impact on the detection of thumb key points. The AR glasses are devices with limited computing power, and the movement of the thumb in front of the camera makes it more difficult to accurately calibrate the key points, making it impossible to achieve real-time and accurate thumb key point detection. Summary of the Invention

[0027] The purpose of the present invention is to provide a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera, which solves the above-mentioned technical problems pointed out in the prior art.

[0028] The present invention provides a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera, comprising the following steps:

[0029] Starting the AR glasses initialization program to capture the image in the current specific usage scenario to obtain a target image; the specific usage scenario includes the current scenario of operating an electronic device with both thumbs to control a game;

[0030] The target image is detected by a feature point detection deep network model to output the thumb key points.

[0031] Preferably, the feature point detection deep network model includes an input layer, a thumb detector layer, a thumb key point detector layer, and a thumb feature point output layer;

[0032] The thumb detector layer is a thumb target detection model;

[0033] The thumb key point detector layer is a thumb key point detection model;

[0034] The thumb detector layer detects and outputs target information in the target image using a pre-trained thumb target detection model; the target information includes coordinate information of the thumb center point, thumb radius information, and thumb type information in the target image;

[0035] The thumb key point detector layer detects and outputs thumb feature points corresponding to target information using a pre-trained thumb key point detection model.

[0036] Preferably, the target image is detected and outputted by a feature point detection deep network model to obtain thumb key points, which includes the following steps:

[0037] Inputting the target image into a thumb target detection model and outputting target information;

[0038] Detecting the target information through a thumb key point detection model to obtain thumb feature points;

[0039] The thumb feature points are transmitted to an application program, and the thumb feature points are displayed in the application program.

[0040] Preferably, the thumb target detection model refers to a detection model constructed by a thumb target detector TBNet network and obtained after training and convergence; wherein the thumb target detector TBNet network training convergence condition is that the TBNet loss function TBNetLoss is less than or equal to a preset first loss function threshold;

[0041] The TBNet loss function TBNetLoss is calculated as follows: ;

[0042] Where bbd represents the thumb bounding box, i represents the left hand or right hand, confidence is the confidence of the thumb target; l represents the left hand; r represents the right hand; λ is the weight value, which can be adjusted during training; n represents the sample set;

[0043] The thumb object detection model includes multiple pooling layers, multiple convolutional layers, and at least two reverse residual block layers;

[0044] Wherein, the pooling layer includes at least one average pooling layer;

[0045] Preferably, the thumb key point detection model is constructed and trained by a thumb key point detector TKNet network; the training convergence condition of the thumb key point detector TKNet network is that the TKNet network loss function TKNetLoss is less than or equal to a preset second loss function threshold;

[0046] The TKNet network loss function TKNetLoss is calculated as follows:

[0047] ;

[0048] Where, is the heat map output by the network; is the true value of the heat map;

[0049] The thumb key point detection model includes a first reverse residual block layer group, a second reverse residual block layer group, a third reverse residual block layer group, a first convolutional layer group, a second convolutional layer group, a transposed convolutional layer and a maximum suppression (NMS) module;

[0050] The first reverse residual block layer group includes three reverse residual block layers; the second reverse residual block layer group includes four reverse residual block layers; the third reverse residual block layer group includes two reverse residual block layers; the first convolution layer group includes three convolution layers; and the second convolution layer group includes one convolution layer.

[0051] Preferably, inputting the target image into a thumb target detection model and outputting target information comprises the following steps:

[0052] Processing the target image through the average pooling layer to obtain a first feature map;

[0053] Processing the first feature map through a convolutional layer to output a second feature map;

[0054] Processing the second feature map through the first reverse residual block layer and the first convolution layer to output a third feature map; and processing the second feature map through the second reverse residual block layer and the second convolution layer to output a fourth feature map;

[0055] Processing the third feature map and the fourth feature map through a reverse residual block layer to output a fifth feature map;

[0056] Processing the fifth feature map through a convolutional layer to output a sixth feature map;

[0057] The sixth feature map is processed by the third convolution layer and the first pooling layer to obtain a seventh feature map; the sixth feature map is processed by the fourth convolution layer and the second pooling layer to obtain an eighth feature map; and the sixth feature map is processed by the fifth convolution layer and the third pooling layer to obtain a ninth feature map.

[0058] Preferably, the first feature map is a single-channel feature map with a height of 160 and a width of 120;

[0059] The second feature map is a 24-channel feature map with a height of 80 and a width of 60;

[0060] The third feature map is a feature map with a height of 20 and a width of 15 for 16 channels; the fourth feature map is a feature map with a height of 20 and a width of 15 for 32 channels;

[0061] The fifth feature map is a 64-channel feature map with a height of 10 and a width of 8;

[0062] The sixth feature map is a 128-channel feature map with a height of 5 and a width of 4;

[0063] The seventh feature map is a 4-channel feature map with a height of 1 and a width of 1, and its content represents the coordinate information of the center point of the thumb;

[0064] The eighth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb radius information;

[0065] The ninth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb type information.

[0066] Preferably, detecting the target information through a thumb key point detection model to obtain thumb feature points includes the following steps:

[0067] Processing the target information through a first reverse residual block layer group to obtain a first feature map;

[0068] Processing the first feature map through a second reverse residual block layer group to obtain a second feature map;

[0069] Processing the second feature map through a third reverse residual block layer group to output a third feature map;

[0070] Processing the III feature map through the first convolutional layer group to obtain the IV feature map;

[0071] Processing the IV-th feature map through a transposed convolutional layer to output a V-th feature map;

[0072] Processing the Vth feature map through the second convolutional layer to obtain a VIth feature map;

[0073] The VIth feature map is processed by a maximum value suppression module, and the thumb feature points are output.

[0074] Preferably, the first feature map is a 24-channel feature map with a height of 32 and a width of 32;

[0075] The second feature map is a 48-channel feature map with a height of 16 and a width of 16;

[0076] The third feature map is a 64-channel feature map with a height of 8 and a width of 8;

[0077] The IVth feature map is a feature map of 96 channels with a height of 8 and a width of 8;

[0078] The Vth feature map is a 24-channel feature map with 16 heights and 16 widths;

[0079] The VIth feature map is a heatmap feature map of 1 channel with a height of 16 and a width of 16.

[0080] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0081] Analysis of the method for real-time thumb key point detection based on a close-range monocular grayscale camera provided by the present invention reveals that, in specific applications, target images captured by the same device (such as AR glasses) are identified, analyzed, and processed. The target image, resulting from nonlinear lens distortion, is then used to detect thumb feature points using a deep network model with minimal network model depth. Both target detection (i.e., the thumb detector layer) and key point detection (i.e., the thumb key point detector layer) are simultaneously employed. The robustness of target detection is enhanced with the high precision of key point detection, jointly addressing the real-time and accuracy challenges of thumb key point detection during device use (such as when wearing AR glasses for mobile gaming). A network loss function is designed as the optimization objective for thumb key point detection. An optimized deep network is obtained through supervised learning and statistical gradient descent to address the thumb key point detection challenge. Furthermore, to meet real-time requirements, the present invention employs a lightweight network structure, employs small convolution kernels, and reuses a large number of parameters to reduce the network parameter level. This reduces computational complexity and memory usage while achieving thumb feature point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0083] Figure 1 A schematic diagram of the overall operation steps of a method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera provided by the present invention;

[0084] Figure 2 A schematic diagram of a game simulation of a method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera provided by the present invention;

[0085] Figure 3 A schematic diagram of further detailed operation steps of a method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera provided by the present invention;

[0086] Figure 4A schematic diagram of the structure simulation of TBNet, a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera, provided by the present invention;

[0087] Figure 5 A schematic diagram of the TKNet network structure simulation of a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera provided by the present invention;

[0088] Figure 6 A schematic diagram of the simulation of the key points of the left and right thumbs of the real-time detection method of the key points of both thumbs based on a close-range monocular grayscale camera provided by the present invention. DETAILED DESCRIPTION

[0089] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0090] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0091] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0092] like Figure 1 As shown, the present invention proposes a real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera, which includes the following steps:

[0093] Step S1: starting the AR glasses initialization program to capture the image in the current specific usage scenario to obtain a target image; the specific usage scenario includes the current scenario of operating an electronic device with both thumbs to control a game;

[0094] The target image refers to a target image of a thumb (the so-called target image containing a thumb means that the current image contains the thumb object);

[0095] Step S2: Detecting the target image through a feature point detection deep network model to obtain thumb key points;

[0096] The feature point detection deep network model includes an input layer, a thumb detector layer, a thumb key point detector layer, and a thumb feature point output layer; the thumb detector layer is a thumb target detection model; the thumb key point detector layer is a thumb key point detection model;

[0097] The thumb detector layer detects and outputs target information in the target image using a pre-trained thumb target detection model; the target information includes coordinate information of the thumb center point, thumb radius information, and thumb type information in the target image;

[0098] The thumb key point detector layer detects and outputs thumb feature points corresponding to target information using a pre-trained thumb key point detection model;

[0099] The parameters of the thumb target detection model and the thumb key point detection model are shared through convolution kernels, that is, the same convolution kernel is used for convolution operations at different positions of the feature point detection deep network model. This parameter sharing method enables the network to have fewer parameters, thereby reducing the risk of overfitting and more effectively learning features. Specifically, when the convolution kernel is applied to one position (i.e., the target information) of an image (i.e., the target image), the feature weights learned by it can be shared at other positions of the image. This means that during training, there is no need to learn a new convolution kernel for each position, but rather the same convolution kernel is used to perform convolution operations on the entire image. Through parameter sharing, the feature point detection deep network model can better capture local features in the image, because the same feature may be the same for different positions in the image. This shared parameter design also makes CNN more efficient when processing large-scale image data.

[0100] The above-mentioned embodiment of the present application recognizes and analyzes the target image collected by the same device (such as AR glasses), and detects the thumb feature points of the target image generated by the nonlinearity of the lens distortion through a feature point detection deep network model with a very small network model depth. At the same time, the target detection (i.e., the above-mentioned thumb detector layer) and key point detection (i.e., the above-mentioned thumb key point detector layer) are integrated to improve the robustness of target detection and the high precision of key point detection, so as to jointly solve the problems encountered during the use of the device (such as wearing AR glasses for mobile phone game entertainment, such as Figure 2 The real-time and accuracy issues of thumb key point detection are shown in Figure 2.

[0101] Specifically, if Figure 3 As shown, in step S2, the target image is detected and outputted by a feature point detection deep network model to obtain thumb key points, including the following steps:

[0102] Step S21: inputting the target image into a thumb target detection model and outputting target information;

[0103] It should be noted that the acquisition of the target image containing the thumb in the embodiment of the present application does not involve inputting the captured image into a detection model. Instead, the AR glasses directly capture the entire image of the thumb manipulating the electronic device in real time, and then identify the characteristic points of the thumb.

[0104] The above-mentioned embodiment of the present application fully utilizes the lens distortion of a camera (such as AR glasses) and directly detects and recognizes target information using the target image generated by the nonlinearity of the lens distortion, thereby improving the accuracy of subsequent thumb feature point recognition;

[0105] The thumb target detection model refers to a detection model constructed by a thumb target detector TBNet network and obtained after training and convergence; wherein, the thumb target detector TBNet network training convergence condition is that the TBNet loss function TBNetLoss is less than or equal to a preset first loss function threshold;

[0106] The TBNet loss function TBNetLoss is calculated as follows: ;

[0107] Where bbd represents the thumb bounding box, i represents the left or right hand, confidence is the confidence of the thumb target (the class layer of the network outputs a floating point number in the range of (0, 1) as the classification confidence); l represents the left hand; r represents the right hand; λ is the weight value, which can be adjusted during training; n represents the sample set;

[0108] The TBNet loss function TBNetLoss is used as a nonlinear optimization target to train TBNet (i.e., the thumb target detector TBNet network). The trained TBNet is used as the thumb target detector (i.e., the thumb target detection model).

[0109] The thumb object detection model includes multiple pooling layers, multiple convolutional layers, and at least two reverse residual block layers;

[0110] Wherein, the pooling layer includes at least one average pooling layer;

[0111] It should be noted that the structure of the above TBNet is as follows Figure 4 As shown, the TBNet in the embodiment of the present application uses the network loss function as the optimization target, obtains the optimized deep network (i.e., the above-mentioned thumb target detection model) through supervised learning and statistical gradient descent, and solves the problem of thumb key point detection.

[0112] Step S22: Detecting the target information using a thumb key point detection model to obtain thumb feature points;

[0113] The thumb key point detection model is constructed and trained by a thumb key point detector TKNet network; the training convergence condition of the thumb key point detector TKNet network is that the TKNet network loss function TKNetLoss is less than or equal to a preset second loss function threshold;

[0114] The TKNet network loss function TKNetLoss is calculated as follows:

[0115] ;

[0116] Where, is the heat map output by the network; is the groundtruth of the heatmap;

[0117] The thumb key point detection model includes a first reverse residual block layer group, a second reverse residual block layer group, a third reverse residual block layer group, a first convolutional layer group, a second convolutional layer group, a transposed convolutional layer and a maximum suppression (NMS) module;

[0118] The first reverse residual block layer group includes three reverse residual block layers; the second reverse residual block layer group includes four reverse residual block layers; the third reverse residual block layer group includes two reverse residual block layers; the first convolution layer group includes three convolution layers; and the second convolution layer group includes one convolution layer.

[0119] It should be noted that the TKNet network is a lightweight convolutional neural network that uses a lightweight network structure and a small convolution kernel to reuse a large number of parameters to reduce the parameter level of the network, thereby achieving the task of thumb feature point detection while reducing the amount of calculation and memory usage. The TKNet network structure is as follows: Figure 5 As shown;

[0120] The difference between the thumb key point detection model in the embodiment of the present application and the thumb target detection model in step S10 is that the thumb target detection model detects the thumb in the target image and identifies the target information (i.e., the center point coordinate information and thumb radius information of the left thumb and the center point coordinate information and thumb radius information of the right thumb in the target image); further, the target information is detected by the thumb key point detection model, and the key points of the left thumb and the key points of the right thumb (such as Figure 6 shown).

[0121] Step S23: transmitting the thumb feature points to an application program, and displaying the thumb feature points in the application program.

[0122] In the above-mentioned embodiment of the present application, the display processing of the thumb feature point in the application means that when playing mobile phone game entertainment using AR glasses, the position of the thumb will be simulated on the AR glasses according to the display position of the thumb feature point, thereby enhancing the user's entertainment;

[0123] It should be noted that the above-mentioned embodiment of the present application designs a network loss function as an optimization target for the task of thumb key point detection, obtains an optimized deep network through supervised learning and statistical gradient descent, and solves the problem of thumb key point detection. At the same time, according to the real-time requirements, the present invention designs a lightweight network structure, adopts a small convolution kernel, and reuses a large number of parameters to reduce the parameter level of the network, thereby achieving the task of thumb feature point detection while reducing the amount of calculation and memory usage.

[0124] Specifically, in step S21, the target image is input into the thumb target detection model, and target information is output, which includes the following steps:

[0125] Step S211: Processing the target image through the average pooling layer to obtain a first feature map;

[0126] The first feature map is a single-channel feature map with a height of 160 and a width of 120;

[0127] Step S212: Processing the first feature map through a convolutional layer to output a second feature map;

[0128] The second feature map is a 24-channel feature map with a height of 80 and a width of 60;

[0129] Step S213: Processing the second feature map through the first reverse residual block layer and the first convolution layer to output a third feature map; and processing the second feature map through the second reverse residual block layer and the second convolution layer to output a fourth feature map;

[0130] The third feature map is a feature map with a height of 20 and a width of 15 for 16 channels; the fourth feature map is a feature map with a height of 20 and a width of 15 for 32 channels;

[0131] Step S214: Processing the third feature map and the fourth feature map through a reverse residual block layer to output a fifth feature map;

[0132] The fifth feature map is a 64-channel feature map with a height of 10 and a width of 8;

[0133] Step S215: Processing the fifth feature map through a convolutional layer to obtain a sixth feature map;

[0134] The sixth feature map is a 128-channel feature map with a height of 5 and a width of 4;

[0135] Step S216: Processing the sixth feature map through the third convolution layer and the first pooling layer to obtain a seventh feature map; processing the sixth feature map through the fourth convolution layer and the second pooling layer to obtain an eighth feature map; and processing the sixth feature map through the fifth convolution layer and the third pooling layer to obtain a ninth feature map.

[0136] The seventh feature map is a 4-channel feature map with a height of 1 and a width of 1, and its content represents the coordinate information of the center point of the thumb;

[0137] The eighth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb radius information;

[0138] The ninth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb type information.

[0139] It should be noted that if Figure 4As shown, the embodiment of the present application first inputs a picture captured by the camera, and after processing including the average pooling layer, a single-channel feature map with a height of 160 and a width of 120 is generated (i.e., the first feature map mentioned above). The image captured by the input camera is processed by the average pooling layer, which can reduce the resolution of the image, thereby reducing the amount of subsequent calculations. At the same time, it can also extract the average features in the area and obtain representative information. The single-channel 160*120 feature map provides basic feature information for subsequent operations; further through the convolution layer, a 24-channel*80*60 feature map is generated (i.e., the second feature map mentioned above), and the image can be further learned through the convolution layer. The deeper features of the network generate a 24-channel 80*60 feature map. The convolution layer can further extract spatial features from the feature map obtained in the previous step; further through two reverse residual block layers and convolution layers, it becomes a 16-channel * 20*15 and 32-channel * 20*15 feature map (that is, the third feature map mentioned above). The reverse residual block is a special network structure used to solve the problem of difficult training of deep networks. It allows data on the main path to be directly passed to deeper layers. At the same time, it learns differential features through additional residual paths, enhances the model's ability to learn features, further deepens feature expression, and changes the feature map to 16*2 0*15 and 32*20*15 feature maps; further pass through the reverse residual block layer to generate a 64-channel*10*8 feature map (the fourth feature map mentioned above), further extract the deep features in the feature map, and compress the size of the feature map to generate a 64*10*8 feature map, which can reduce the amount of calculation and more concentratedly represent the image content; pass through the convolution layer again to generate a 128-channel*5*4 feature map (the fifth feature map). After the last pass through the convolution layer, a 128*5*4 feature map will be obtained. The feature map at this time contains a lot of important information for recognition, which is divided into three ways to generate prediction output: The first path passes through the convolutional layer and the pooling layer to generate a 4-channel * 1 * 1 thumb center prediction (i.e., the thumb center coordinate information of the sixth feature map); the second path passes through the convolutional layer and the pooling layer to generate a 2-channel * 1 * 1 thumb radius prediction (i.e., the thumb radius information of the seventh feature map); the third path passes through the convolutional layer and the pooling layer to generate a 2-channel * 1 * 1 thumb type prediction (i.e., the thumb type information of the eighth feature map). The above embodiment of the present application uses a hierarchical and progressive feature extraction process based on deep learning, which accurately predicts various thumb-related parameters by gradually compressing the spatial size of the image and increasing the number of channels in different dimensions;

[0140] The above-mentioned embodiment of the present application uses the network loss function as the optimization target, and obtains the optimized deep network (i.e., the above-mentioned thumb target detection model) through supervised learning and statistical gradient descent, which facilitates the subsequent detection of thumb feature points (feature points or key points) more quickly, reduces the interference of a large number of feature points, and improves the efficiency of thumb feature point detection.

[0141] Specifically, in step S22, the target information is detected by a thumb key point detection model to obtain thumb feature points, which includes the following steps:

[0142] Step S221: Processing the target information through a first reverse residual block layer group to obtain a first feature map;

[0143] The first feature map is a feature map with a height of 32 and a width of 32 for 24 channels;

[0144] Step S222: processing the first feature map through a second reverse residual block layer group to obtain a second feature map;

[0145] The second feature map is a 48-channel feature map with a height of 16 and a width of 16;

[0146] Step S223: Processing the second feature map through a third reverse residual block layer group to obtain a third feature map;

[0147] The third feature map is a 64-channel feature map with a height of 8 and a width of 8;

[0148] Step S224: processing the third feature map through the first convolutional layer group to obtain a fourth feature map;

[0149] The IVth feature map is a feature map of 96 channels with a height of 8 and a width of 8;

[0150] Step S225: Processing the IV-th feature map through a transposed convolution layer to output a V-th feature map;

[0151] The Vth feature map is a 24-channel feature map with 16 heights and 16 widths;

[0152] Step S226: Processing the Vth feature map through the second convolutional layer to obtain a VIth feature map;

[0153] The sixth feature map is a heatmap feature map of 1 channel with a height of 16 and a width of 16;

[0154] Step S227: Processing the VI-th feature map through a maximum value suppression module to output thumb feature points.

[0155] It should be noted that, in the above embodiment of the present application, the target information (target information or thumb target detection result map, which is a map containing the coordinate information of the thumb center point, the thumb radius information and the thumb type information) is first input into the thumb key point detection model, and then the thumb key point detection model is used to detect and output the thumb feature points; in the thumb key point detection model, if Figure 5As shown in the figure, the target information first passes through 3 reverse residual block layers to generate a 24-channel feature map with a height of 32 and a width of 32 (i.e., the above-mentioned feature map I). The reverse residual block can help the model avoid the problems of gradient disappearance and explosion and improve the training process; then it passes through 4 reverse residual block layers to generate a 48-channel feature map with a height of 16 and a width of 16 (i.e., the above-mentioned feature map II), further extracting features while maintaining the effective training depth of the network; further passing through 2 reverse residual block layers, more abstract high-level features are generated, generating a 64*8*8d feature map (i.e., the above-mentioned feature map III); further passing through 3 3*3 convolutional layers to capture medium-sized local features , generating a 96*8*8 feature map (i.e., the IV feature map mentioned above); further passing through the transposed convolution layer to generate a 24*16*16 feature map (i.e., the V feature map mentioned above), increasing the spatial dimension of the feature map, that is, enlarging the size of the feature map and restoring the spatial resolution of the feature points; further passing through a 1*1 convolution layer, integrating the information of each channel, reducing the number of channels, generating a 1*16*16 heatmap feature map (i.e., the VI feature map mentioned above) Finally, the VI feature map is passed through the maximum value suppression NMS module to eliminate redundant detection boxes, improve the accuracy of key point positioning, ensure accurate detection and identification of feature points, and output the feature points of the thumb;

[0156] The above-mentioned embodiment of the present application designs a lightweight network structure, adopts a small convolution kernel, reuses a large number of parameters to reduce the parameter level of the network, and realizes the task of thumb feature point detection while reducing the amount of calculation and memory usage.

[0157] In summary, the present invention proposes a real-time detection method for thumb key points of both hands based on a close-range monocular grayscale camera. This method uses target images captured by the same device (such as AR glasses) for recognition and analysis. The target image, generated by nonlinear lens distortion, is then used to detect thumb feature points using a deep network model with minimal network model depth. This method simultaneously employs target detection (i.e., the thumb detector layer) and key point detection (i.e., the thumb key point detector layer). This method integrates the robustness of target detection with the high precision of key point detection to jointly address the real-time and accuracy issues of thumb key point detection during device use (such as when wearing AR glasses for mobile gaming). For the task of thumb key point detection, a network loss function is designed as the optimization objective. An optimized deep network is obtained through supervised learning and statistical gradient descent to address the thumb key point detection problem. Furthermore, to meet real-time requirements, the present invention employs a lightweight network structure, employs small convolution kernels, and reuses a large number of parameters to reduce the network parameter level. This reduces computational complexity and memory usage while achieving thumb feature point detection.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time detection method for key points of thumbs of both hands based on a close-range monocular grayscale camera, characterized in that: The steps are as follows: Starting an AR glasses initialization program to capture images in a current specific usage scenario to obtain a target image; the current specific usage scenario includes a scenario where the thumbs of both hands are currently operating an electronic device to control a game; The target image is detected by a feature point detection deep network model to output the thumb key points; The feature point detection deep network model includes an input layer, a thumb detector layer, a thumb key point detector layer, and a thumb feature point output layer; the thumb detector layer is a thumb target detection model; The thumb key point detector layer is a thumb key point detection model; The thumb object detection model includes multiple pooling layers, multiple convolutional layers, and at least two reverse residual block layers; Wherein, the pooling layer includes at least one average pooling layer; The thumb key point detection model includes a first reverse residual block layer group, a second reverse residual block layer group, a third reverse residual block layer group, a first convolutional layer group, a second convolutional layer group, a transposed convolutional layer and a maximum suppression (NMS) module; Among them, the first reverse residual block layer group includes three reverse residual block layers; the second reverse residual block layer group includes four reverse residual block layers; the third reverse residual block layer group includes two reverse residual block layers; the first convolution layer group includes three convolution layers; and the second convolution layer group includes one convolution layer.

2. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 1, characterized in that: The thumb detector layer detects and outputs target information in the target image using a pre-trained thumb target detection model; the target information includes coordinate information of the thumb center point, thumb radius information, and thumb type information in the target image; The thumb key point detector layer detects and outputs thumb feature points corresponding to target information using a pre-trained thumb key point detection model.

3. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 2, characterized in that: The step of detecting and outputting the target image through a feature point detection deep network model to obtain thumb key points includes the following steps: Inputting the target image into a thumb target detection model and outputting target information; Detecting the target information through a thumb key point detection model to obtain thumb feature points; The thumb feature points are transmitted to an application program, and the thumb feature points are displayed in the application program.

4. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 3, characterized in that: The thumb target detection model refers to a detection model constructed by a thumb target detector TBNet network and obtained after training and convergence; wherein the thumb target detector TBNet network training convergence condition is that the TBNet loss function TBNetLoss is less than or equal to a preset first loss function threshold; The TBNet loss function TBNetLoss is calculated as follows: ； Where bbd represents the thumb bounding box, i represents the left hand or right hand, confidence is the confidence of the thumb target; l represents the left hand; r represents the right hand; λ is the weight value, which can be adjusted during training; n represents the sample set.

5. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 4, characterized in that: The thumb key point detection model is constructed and trained by a thumb key point detector TKNet network; the training convergence condition of the thumb key point detector TKNet network is that the TKNet network loss function TKNetLoss is less than or equal to a preset second loss function threshold; The TKNet network loss function TKNetLoss is calculated as follows: ； Where, is the heat map output by the network; is the true value of the heat map.

6. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 5, characterized in that: Inputting the target image into the thumb target detection model and outputting target information includes the following steps: Processing the target image through the average pooling layer to obtain a first feature map; Processing the first feature map through a convolutional layer to output a second feature map; Processing the second feature map through the first reverse residual block layer and the first convolution layer to output a third feature map; The second feature map is processed by a second reverse residual block layer and a second convolutional layer to output a fourth feature map; Processing the third feature map and the fourth feature map through a reverse residual block layer to output a fifth feature map; Processing the fifth feature map through a convolutional layer to output a sixth feature map; The sixth feature map is processed by the third convolution layer and the first pooling layer to obtain a seventh feature map; the sixth feature map is processed by the fourth convolution layer and the second pooling layer to obtain an eighth feature map; and the sixth feature map is processed by the fifth convolution layer and the third pooling layer to obtain a ninth feature map.

7. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 6, characterized in that: The first feature map is a single-channel feature map with a height of 160 and a width of 120; The second feature map is a 24-channel feature map with a height of 80 and a width of 60; The third feature map is a feature map with a height of 20 and a width of 15 for 16 channels; the fourth feature map is a feature map with a height of 20 and a width of 15 for 32 channels; The fifth feature map is a 64-channel feature map with a height of 10 and a width of 8; The sixth feature map is a 128-channel feature map with a height of 5 and a width of 4; The seventh feature map is a 4-channel feature map with a height of 1 and a width of 1, and its content represents the coordinate information of the center point of the thumb; The eighth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb radius information; The ninth feature map is a 2-channel feature map with a height of 1 and a width of 1, and its content represents thumb type information.

8. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 7, characterized in that: Detecting the target information using a thumb key point detection model to obtain thumb feature points includes the following steps: Processing the target information through a first reverse residual block layer group to obtain a first feature map; Processing the first feature map through a second reverse residual block layer group to obtain a second feature map; Processing the second feature map through a third reverse residual block layer group to output a third feature map; Processing the III feature map through the first convolutional layer group to obtain the IV feature map; Processing the IV-th feature map through a transposed convolutional layer to output a V-th feature map; Processing the Vth feature map through the second convolutional layer to obtain a VIth feature map; The VIth feature map is processed by a maximum value suppression module, and the thumb feature points are output.

9. The method for real-time detection of key points of thumbs of both hands based on a close-range monocular grayscale camera according to claim 8, characterized in that: The first feature map is a feature map with a height of 32 and a width of 32 for 24 channels; The second feature map is a 48-channel feature map with a height of 16 and a width of 16; The third feature map is a 64-channel feature map with a height of 8 and a width of 8; The IVth feature map is a feature map of 96 channels with a height of 8 and a width of 8; The Vth feature map is a 24-channel feature map with 16 heights and 16 widths; The VIth feature map is a heatmap feature map of 1 channel with a height of 16 and a width of 16.

Images