A method and system for parallel prediction of multi-regions of hand
The multi-region parallel prediction model with DarkNet53, SPP, and CBAM modules addresses the challenge of varying hand feature sizes, enhancing prediction accuracy for hand regions.
Patent Information
- Application Number
- CN202210550375.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-20
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2042-05-20
AI Technical Summary
Current lightweight real-time capture and reconstruction systems for hands struggle with handling differences in feature sizes and have poor prediction accuracy, especially for small hand areas.
A method and system for parallel prediction of multiple regions of a hand using a multi-region parallel prediction model with a backbone network of DarkNet53 structure, an SPP module with multiple convolutional kernels, and a CBAM module to handle features of different sizes, combined with a prediction module for accurate hand region detection.
The system effectively reduces differences in feature sizes and achieves high prediction accuracy for hand regions, including the whole hand and specific parts like knuckles and the palm, by leveraging the SPP and CBAM modules for feature extraction and fusion.
Smart Images
Figure CN114998927B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for parallel prediction of multiple regions of a hand. Background Art
[0002] With the rapid development of computer science and computer vision technology, the application of image processing is becoming more and more extensive. Object prediction is one of the classic problems in computer vision and an important research direction of vision-based pattern recognition applications. It requires searching any picture, effectively marking the objects in the picture, and extracting the bounding rectangle.
[0003] The key area recognition technology of the hand is mostly used in basic work such as gesture recognition and posture estimation, and is also widely used in the security field. Generally, it is combined with deep learning to extract the key area of the hand, and the deep learning model is used to perform subsequent feature extraction and target category discrimination based on the key area of the hand. Although the prediction accuracy of the key area of the hand is high, it is difficult to effectively extract targets with small hand areas. To solve the above problems, the prior art discloses a lightweight real-time capture and reconstruction system for the hand. First, an RGB camera is used for gesture recognition and capture, and then a real-time detector is used to detect the hand frame. Finally, a deep neural network is introduced to extract the coordinates of the key points of the hand, which broadens the spatial range of gesture recognition and capture, and can effectively extract small targets. However, the differences in features of different sizes in the hand area cannot be processed, and the prediction accuracy is poor. Summary of the invention
[0004] In order to solve the problem that the current lightweight real-time capture and reconstruction system of the hand cannot handle the differences in different size features of the hand area and the prediction accuracy is poor, the present invention proposes a method and system for parallel prediction of multiple regions of the hand, which can handle the differences in different size features of the hand area and can accurately predict the hand as a whole and the hand area with high prediction accuracy.
[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:
[0006] A method for parallel prediction of multiple regions of a hand, comprising the following steps:
[0007] S1. Obtain a certain number of hand image samples at different hand angles to form an image dataset;
[0008] S2. labeling each hand image sample in the image data set to obtain image labeling samples corresponding to each hand image sample;
[0009] S3. preprocessing the hand image samples;
[0010] S4. Construct a multi-region parallel prediction model for the hand. The multi-region parallel prediction model for the hand includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of several convolutional layers. The subsequent network includes an SPP module, a CBAM module and a prediction module composed of several parallel convolutional kernels of different sizes.
[0011] S5. Divide the image dataset and the image annotation samples corresponding to the hand image samples in the image dataset into a training set, a validation set, and a test set, use the training set to train the hand multi-region parallel prediction model, then use the validation set to evaluate the hand multi-region parallel prediction model during the training process, and use the test set to test the effectiveness of the hand multi-region parallel prediction model, and obtain a trained hand multi-region parallel prediction model;
[0012] S6. Collect the hand image to be predicted, preprocess the hand image to be predicted, input the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and output the prediction result of the hand image to be predicted.
[0013] In the present technical scheme, hand image samples are first obtained from different hand angles to improve the diversity of hand image samples, and then the obtained hand image samples are annotated to obtain the position and classification of the annotated parts in the hand image samples, and the hand image samples are preprocessed. The hand image samples are input into the hand multi-region parallel prediction model for parallel operation. The hand multi-region parallel prediction model includes a backbone network of DarkNet53 structure, an SPP module composed of several parallel convolution kernels of different sizes, a CBAM module and a prediction module. The backbone network is used to extract features of hand image samples. The convolution kernels of different sizes in the SPP module can calculate and extract features of different sizes and fuse features of different sizes. The CBAM module can capture images of features of different sizes. The prediction module predicts the recognition results of the hand image samples. The hand multi-region parallel prediction model reduces the differences in features of different sizes through the cooperation of the SPP module and the CBAM module, and has high prediction accuracy.
[0014] Preferably, in step S1, the hand image sample is obtained by photographing with a USB high-definition camera, and the USB high-definition camera is connected to a computer terminal to transmit the obtained hand image sample data to the computer terminal.
[0015] Preferably, in step S2, the specific steps of labeling each hand image sample in the image data set are:
[0016] S21. Mark the real bounding box of the whole hand from the wrist to the fingertips in the hand image sample to obtain an image annotation sample of the whole hand from the wrist to the fingertips;
[0017] S22. marking the true bounding box of the palm in the hand image sample to obtain an image annotation sample of the palm;
[0018] S23. Mark the real bounding boxes of the two knuckles of the thumb and the three knuckles of the remaining four fingers in the hand image sample, and obtain the image annotation samples of the hand of the two knuckles of the thumb and the three knuckles of the remaining four fingers.
[0019] Preferably, in step S3, the specific steps of the preprocessing are:
[0020] S31. Scale the width and height ratio of the hand image samples to a uniform value;
[0021] S32. Randomly horizontally flip and add noise to the scaled hand image samples;
[0022] S33. Randomly select a number of hand image samples for splicing to obtain spliced hand image samples with the same specifications.
[0023] Here, the aspect ratio of the hand image samples is scaled to a uniform value to facilitate the recognition and parallel operation of the hand image samples in the input hand multi-region parallel prediction model. The aspect ratio of the hand image samples is scaled to a uniform value, and several hand image samples are randomly selected for splicing to obtain spliced hand image samples of the same specifications, thereby realizing data enhancement, increasing the robustness of the hand multi-region parallel prediction model and preventing overfitting.
[0024] Preferably, in step S4, the backbone network, SPP module, CBAM module, and prediction module of the hand multi-region parallel prediction model are connected in sequence, the hand image sample is input into the backbone network of the hand multi-region parallel prediction model, the backbone network extracts the features of the hand image sample, and outputs a first feature map of the hand image sample, the output first feature map is first input into the SPP module, and the convolution kernels of different sizes in the SPP module are used to extract features of different sizes to obtain a second feature map, the second feature maps are added and output, and a third feature map containing information of different scales is obtained, the obtained third feature map is input into the CBAM module, and the CBAM module further extracts features of different sizes to obtain a final fourth feature map, and finally the fourth feature map is input into the prediction module, processed by the prediction module, and the prediction bounding box of the whole hand from the wrist to the fingertips, the palm, the two knuckles of the thumb, and the three knuckles of the remaining four fingers in the fourth feature map and the classification probability of the prediction bounding box are predicted and output, wherein the hand image sample with a classification probability greater than a positive threshold is defined as a positive sample, and the hand image sample with a classification probability less than a negative threshold is defined as a negative sample.
[0025] Preferably, the CBAM module is provided with a channel attention module and a spatial attention module, the spatial attention module is arranged after the channel attention module, the third feature map input to the CBAM module is provided with a channel, the channel attention module captures the relationship information between the channels of the third feature map, and the channel attention module is provided with a first maximum pooling layer Maxpool, a first average pooling layer Avgpool, and a feedforward neural network shared module Shared MLP connected in sequence, the third feature map input to the CBAM module is input into the first maximum pooling layer Maxpool and the first average pooling layer Avgpool, parallel operations are performed, the output result is input into the feedforward neural network shared module Shared MLP, weights are shared, and then the results output by the feedforward neural network shared module Shared MLP are added, and then processed with a Sigmoid function to obtain the channel attention feature map M of the channel attention module. c Finally, the third feature map and the channel attention feature map M c Multiply to get the final output feature map F′ of the channel attention module, the channel attention feature map M c The specific calculation formula is as follows:
[0026] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0027] Among them, F is the third feature map of the input CBAM module, AvgPool() is the average pooling operation, MaxPool() is the maximum pooling operation, MLP() is the feedforward neural network, and σ() is the Sigmoid function;
[0028] The specific formula for calculating the final output feature map F′ of the channel attention module is as follows:
[0029] F′=F*M c (F)
[0030] The spatial attention module is provided with a second maximum pooling layer Maxpool, a second average pooling layer Avgpool, and a convolutional layer connected in sequence. The final output feature map F′ of the channel attention module is input into the second maximum pooling layer Maxpool and the second average pooling layer Avgpool for serial calculation. The output results are connected in the channel dimension and then input into the convolutional layer for calculation. The result obtained by the convolutional layer calculation is processed by the Sigmoid function to obtain the spatial attention feature map M. s (F′), the final output feature map F′ of the last channel attention module and the spatial attention feature map M s (F′) is multiplied to obtain the fourth feature map of the final output of the CBAM module,
[0031] Spatial attention feature map M s The specific calculation formula is as follows:
[0032] M s (F′)=σ(f 7*7 ([AvgPool(F′);MaxPool(F′)]))
[0033] Among them, f 7*7 () is the convolution layer operation, [;] means to connect the results of serial calculation in the channel dimension, σ() is the Sigmoid function;
[0034] The specific formula for calculating the fourth feature map of the final output of the CBAM module is as follows:
[0035] F c =F′*M s (F′)
[0036] Among them, F c It is the fourth feature map of the final output of the CBAM module.
[0037] Preferably, both the backbone network and the subsequent networks need to be trained, and the network parameters and weights of the backbone network use the network parameters and weights obtained by pre-training on the general ImageNet image classification dataset, which can ensure the fast convergence speed of the backbone network and improve the training speed of the multi-region parallel prediction model of the hand.
[0038] Preferably, in step S5, the part of the hand image sample within the hand area is set as the foreground image sample, and the part outside the hand area is set as the background image sample. The loss function formula used when training the hand multi-area parallel prediction model is as follows:
[0039] Loss = L box +L cls +L obj
[0040] Among them, L box is the error caused by the hand multi-region parallel prediction model predicting the bounding box, L cls is the error caused by the classification of hand image samples, L obj Indicates the error caused by judging the foreground image samples and the background image samples;
[0041] L box The specific formula for solving is as follows:
[0042]
[0043] Among them, λ coord is a hyperparameter, w i ,hi ,x i ,y i are the width, height, and center coordinates of the true bounding box respectively; w′ i ,h′ i ,x′ i ,y′ i The width, height, and center point coordinates of the bounding box are predicted by the multi-region parallel prediction model of the hand, Indicates whether the jth predicted bounding box in the i-th grid is a foreground image sample. If so, then The value is 1, otherwise, The value is 0; s 2 is the total number of grids, and B represents the number of true bounding boxes;
[0044] L cls The specific formula for solving is as follows:
[0045]
[0046] Among them, λ class is a hyperparameter, p i (c) is the classification probability, p′ i (c) is 1-p i (c);
[0047] L obj The specific formula for solving is as follows:
[0048]
[0049] where λ noobj is a hyperparameter, Indicates whether the jth predicted bounding box in the i-th grid is a background image sample. If so, then The value is 1, otherwise, The value is 0; c i is the confidence of foreground image samples and background image samples; c′ i Predict the confidence of foreground image samples and background image samples for the multi-region parallel prediction model of the hand.
[0050] Preferably, during the training process, the gradient descent method is used to update the network parameters and weights of the hand multi-region parallel prediction model, the validation set is used to evaluate the hand multi-region parallel prediction model during the training process, and the test set is used to test the effectiveness of the hand multi-region parallel prediction model to obtain a trained hand multi-region parallel prediction model.
[0051] The present application also proposes a system for predicting hand multi-region parallel prediction, the system comprising:
[0052] An image data set acquisition module is used to obtain a certain number of hand image samples at different hand angles to form an image data set;
[0053] A labeling module, used to label each hand image sample in the image data set to obtain image labeling samples corresponding to each hand image sample;
[0054] A preprocessing module, used for preprocessing hand image samples;
[0055] The hand multi-region parallel prediction model construction module is used to construct the hand multi-region parallel prediction model. The hand multi-region parallel prediction model includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of several convolutional layers. The subsequent network includes an SPP module, a CBAM module and a prediction module composed of several parallel convolutional kernels of different sizes.
[0056] A training module is used to divide the image data set and the image annotation samples corresponding to the hand image samples in the image data set into a training set, a validation set and a test set, use the training set to train the hand multi-region parallel prediction model, then use the validation set to evaluate the hand multi-region parallel prediction model during the training process, and use the test set to test the effectiveness of the hand multi-region parallel prediction model, and obtain a trained hand multi-region parallel prediction model;
[0057] The processing module collects the hand image to be predicted, preprocesses the hand image to be predicted, inputs the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and outputs the prediction result of the hand image to be predicted.
[0058] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0059] The present invention proposes a method and system for multi-region parallel prediction of a hand. First, hand image samples are obtained from different hand angles to improve the diversity of the hand image samples. Then, the obtained hand image samples are labeled and preprocessed. The hand image samples are input into a multi-region parallel prediction model of the hand for parallel calculation. The multi-region parallel prediction model of the hand is provided with a backbone network of a DarkNet53 structure, an SPP module composed of several parallel convolution kernels of different sizes, and a CBAM module. The backbone network is used to extract features of the hand image samples. The convolution kernels of different sizes of the SPP module can calculate and extract features of different sizes and fuse features of different sizes. The CBAM module can capture images of features of different sizes. Through the cooperation of the SPP module and the CBAM module in the multi-region parallel prediction model of the hand, the difference of features of different sizes is reduced, and the prediction accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 A schematic diagram showing a flow chart of a method for hand multi-region parallel prediction proposed in Embodiment 1 of the present invention;
[0061] Figure 2 A structural diagram showing a hand multi-region parallel prediction model proposed in Example 2 of the present invention;
[0062] Figure 3 A structural diagram showing a backbone network proposed in Embodiment 2 of the present invention;
[0063] Figure 4 A structural diagram showing the SPP module proposed in Embodiment 2 of the present invention;
[0064] Figure 5 A structural diagram showing the CBAM channel attention module proposed in Example 2 of the present invention;
[0065] Figure 6 A structural diagram showing the CBAM spatial attention module proposed in Example 2 of the present invention;
[0066] Figure 7 A structural diagram of a system for parallel prediction of multiple hand regions proposed in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0067] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0068] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent actual sizes. The description of the directions of parts such as "upper" and "lower" does not limit the present patent;
[0069] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;
[0070] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.
[0071] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0072] Example 1
[0073] Figure 1 A flowchart of a method for hand multi-region parallel prediction proposed in an embodiment of the present invention is shown, comprising the following steps:
[0074] S1. Obtain a certain number of hand image samples at different hand angles to form an image dataset;
[0075] There are many methods for obtaining hand image samples. In this embodiment, the hand image samples are obtained by shooting with a USB high-definition camera. The shooting environment is ensured to be free of interference. The shooting objects are selected as 25 students and 25 teachers. The USB high-definition camera is used to shoot the left and right hands of each person from different angles. A total of 1,000 pictures are taken. The USB high-definition camera is connected to the computer, and the obtained hand image sample data is transmitted to the computer.
[0076] S2. Annotate each hand image sample in the image dataset. Each hand image sample needs to undergo 16 annotation steps to obtain image annotation samples corresponding to each hand image sample. The specific steps are as follows:
[0077] S21. Mark the real bounding box of the whole hand from the wrist to the fingertips in the hand image sample. The real bounding box in this step is a rectangular box, and the image annotation sample of the whole hand from the wrist to the fingertips is obtained;
[0078] S22. Mark the real bounding box of the palm in the hand image sample. The real bounding box in this step is a rectangular box, and obtain the image annotation sample of the palm;
[0079] S23. Mark the real bounding boxes of the two knuckles of the thumb and the three knuckles of the remaining four fingers in the hand image sample. The real bounding boxes of this step are rectangular boxes of different sizes, and image annotation samples of the hand of the two knuckles of the thumb and the three knuckles of the remaining four fingers are obtained.
[0080] S3. Preprocess the hand image samples. The specific steps of preprocessing are as follows:
[0081] S31. Scale the width and height ratio of the hand image samples to a uniform value;
[0082] S32. Randomly horizontally flip and add noise to the scaled hand image samples;
[0083] S33. Randomly select 4 hand image samples for splicing to obtain spliced hand image samples with the same specifications.
[0084] S4. Construct a multi-region parallel prediction model for the hand. The multi-region parallel prediction model for the hand includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of 53 convolutional layers. The subsequent network includes an SPP module composed of 3 parallel convolutional kernels of different sizes, a CBAM module, and a prediction module.
[0085] S5. Divide the image data set and the image annotation samples corresponding to the hand image samples in the image data set into a training set, a validation set and a test set in a ratio of 7:1:2, wherein the training set has 700 hand image samples and 700 image annotation samples corresponding to the hand image samples, the validation set has 100 hand image samples and 100 image annotation samples corresponding to the hand image samples, and the test set has 200 hand image samples and 200 image annotation samples corresponding to the hand image samples. The hand multi-region parallel prediction model is trained using the training set, and then the hand multi-region parallel prediction model is evaluated during the training process using the validation set, and the effectiveness of the hand multi-region parallel prediction model is tested using the test set, thereby obtaining a trained hand multi-region parallel prediction model.
[0086] S6. Collect the hand image to be predicted, preprocess the hand image to be predicted, input the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and output the prediction result of the hand image to be predicted.
[0087] Example 2
[0088] Figure 2 The structure diagram of the hand multi-region parallel prediction model is shown in Figure 2. Figure 2 The hand multi-region parallel prediction model includes a backbone network and a subsequent network. The subsequent network includes an SPP module, a CBAM module, and a prediction module. Figure 3 , the backbone network is a DarkNet53 structure consisting of 53 convolutional layers; see Figure 4 The SPP module consists of three parallel convolution kernels of different sizes. Both the backbone network and the subsequent network need to be trained. The network parameters and weights of the backbone network are obtained by pre-training on the general ImageNet image classification dataset, and the network weights of the subsequent network are obtained by training on the general MSCOCO object detection dataset to ensure fast network convergence and improve the training speed of the multi-region parallel prediction model of the hand.
[0089] See also Figure 2, the backbone network, SPP module, CBAM module, and prediction module of the hand multi-region parallel prediction model are connected in sequence, the hand image sample is input into the backbone network of the hand multi-region parallel prediction model, the backbone network extracts the features of the hand image sample, and outputs the first feature map of the hand image sample. The output first feature map is first input into the SPP module, and the convolution kernels of different sizes in the SPP module are used to extract features of different sizes to obtain the second feature map. The second feature maps are added and output to obtain the third feature map containing information of different scales. The obtained third feature map is input into the CBAM module, and the CBAM module further extracts features of different sizes to obtain the final fourth feature map. Finally, the fourth feature map is input into the SPP module. The hand image is input to the prediction module, processed by the prediction module, and the prediction output is the prediction bounding box of the whole hand from the wrist to the fingertips, the palm, the two knuckles of the thumb, and the three knuckles of the remaining four fingers in the fourth feature map, as well as the classification probability of the prediction bounding box, wherein the positive threshold is set to 0.5, the negative threshold is set to 0.3, and the hand image samples with a classification probability value higher than 0.5 are defined as positive samples, and the hand image samples with a classification probability value lower than 0.3 are defined as negative samples. During the training process, it is necessary to match the predicted bounding box with positive and negative samples, and calculate the loss of the matching results, so as to gradually optimize the network parameters. In order to increase the stability of the training, it is also necessary to add the labeled real bounding box as a positive sample to the training to increase the number of positive samples;
[0090] See also Figure 5 and Figure 6 The CBAM module is provided with a channel attention module and a spatial attention module, the spatial attention module is arranged after the channel attention module, the third feature map input to the CBAM module is provided with a channel, the channel attention module captures the relationship information between the channels of the third feature map, the channel attention module is provided with a first maximum pooling layer Maxpool, a first average pooling layer Avgpool, and a feedforward neural network shared module Shared MLP which are connected in sequence, the third feature map input to the CBAM module is input into the first maximum pooling layer Maxpool and the first average pooling layer Avgpool, parallel operations are performed, the output result is input into the feedforward neural network shared module Shared MLP, weights are shared, and then the output results of the feedforward neural network shared module Shared MLP are added, and then processed with a Sigmoid function to obtain a channel attention feature map M of the channel attention module c Finally, the third feature map and the channel attention feature map M c Multiply to get the final output feature map F′ of the channel attention module, the channel attention feature map M c The specific calculation formula is as follows:
[0091] M c(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0092] Where F is the third feature map of the input CBAM module, AvgPool() is the average pooling operation, MaxPool() is the maximum pooling operation, MLP() is the feedforward neural network, and σ() is the Sigmoid function;
[0093] The specific formula for calculating the final output feature map F′ of the channel attention module is as follows:
[0094] F′=F*M c (F)
[0095] The spatial attention module is provided with a second maximum pooling layer Maxpool, a second average pooling layer Avgpool, and a convolution layer connected in sequence. The final output feature map F′ of the channel attention module is input into the second maximum pooling layer Maxpool and the second average pooling layer Avgpool for serial calculation. The output results are connected in the channel dimension and then input into the convolution layer for calculation. The Sigmoid function is used to process the result obtained by the convolution layer calculation to obtain the spatial attention feature map M. s (F′), and finally the final output feature map F′ of the channel attention module and the spatial attention feature map M s (F′) is multiplied to obtain the fourth feature map of the final output of the CBAM module;
[0096] Spatial attention feature map M s The specific calculation formula is as follows:
[0097] M s (F′)=σ(f 7*7 ([AvgPool(F′);MaxPool(F′)]))
[0098] Among them, f 7*7 () is the convolution layer operation, [;] means to connect the results of serial calculation in the channel dimension, σ() is the Sigmoid function;
[0099] The specific formula for calculating the fourth feature map of the final output of the CBAM module is as follows:
[0100] F c =F′*M s (F′)
[0101] Among them, F c It is the fourth feature map of the final output of the CBAM module.
[0102] Example 3
[0103] See also Figure 1 In step S5, the part of the hand image sample within the hand area is set as the foreground image sample, and the part outside the hand area is set as the background image sample. The loss function formula used when training the hand multi-area parallel prediction model is as follows:
[0104] Loss = L box +L cls +L obj
[0105] Among them, L box is the error caused by the hand multi-region parallel prediction model predicting the bounding box, L cls is the error caused by the classification of hand image samples, L obj Indicates the error caused by judging the foreground image samples and the background image samples;
[0106] L box The specific formula for solving is as follows:
[0107]
[0108] Among them, λ coord is a hyperparameter, w i ,h i ,x i ,y i are the width, height, and center coordinates of the true bounding box respectively; w′ i ,h′ i ,x′ i ,y′ i The width, height, and center point coordinates of the bounding box are predicted by the multi-region parallel prediction model of the hand, Indicates whether the jth predicted bounding box in the i-th grid is a foreground image sample. If so, then The value is 1, otherwise, The value is 0; s 2 is the total number of grids, and B represents the number of true bounding boxes;
[0109] L cls The specific formula for solving is as follows:
[0110]
[0111] Among them, λ class is a hyperparameter, p i (c) is the classification probability, p′ i (c) is 1-p i (c);
[0112] L obj The specific formula for solving is as follows:
[0113]
[0114] where λ noobj is a hyperparameter, Indicates whether the jth predicted bounding box in the i-th grid is a background image sample. If so, then The value is 1, otherwise, The value is 0; c i is the confidence of foreground image samples and background image samples; c′ i The confidence of the foreground image samples and background image samples predicted by the multi-region parallel prediction model of the hand is calculated; the network parameters and weights of the multi-region parallel prediction model of the hand are updated by the gradient descent method, and the validation set is used to evaluate the multi-region parallel prediction model of the hand during the training process. First, the training set is input into the multi-region parallel prediction model of the hand during the training process, and then the loss function is used to measure the training effect of the multi-region parallel prediction model of the hand. Finally, the multi-region parallel prediction model of the hand is verified and the parameters of the multi-region parallel prediction model of the hand are optimized; the test set is input into the multi-region parallel prediction model of the hand during the training process, and the effectiveness of the multi-region parallel prediction model of the hand is tested using the test set to complete the prediction of hand image samples.
[0115] Example 4
[0116] like Figure 7 As shown, a system for predicting hand multi-region parallel prediction includes:
[0117] An image data set acquisition module 11 is used to acquire a certain number of hand image samples at different hand angles to form an image data set;
[0118] A labeling module 12 is used to label each hand image sample in the image data set to obtain image labeling samples corresponding to each hand image sample;
[0119] A preprocessing module 13, used for preprocessing the hand image samples;
[0120] The hand multi-region parallel prediction model construction module 14 is used to construct the hand multi-region parallel prediction model. The hand multi-region parallel prediction model includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of 53 convolutional layers. The subsequent network includes an SPP module composed of 3 parallel convolution kernels of different sizes, a CBAM module and a prediction module.
[0121] The training module 15 is used to divide the image data set and the image annotation samples corresponding to the hand image samples in the image data set into a training set, a validation set and a test set, use the training set to train the hand multi-region parallel prediction model, then use the validation set to evaluate the hand multi-region parallel prediction model in the training process, and use the test set to test the effectiveness of the hand multi-region parallel prediction model, so as to obtain a trained hand multi-region parallel prediction model;
[0122] The processing module 16 collects the hand image to be predicted, preprocesses the hand image to be predicted, inputs the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and outputs the prediction result of the hand image to be predicted.
[0123] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A method for parallel prediction of multiple regions of a hand, characterized in that: The method comprises the following steps: S1. Obtain a certain number of hand image samples at different hand angles to form an image dataset; S2. labeling each hand image sample in the image data set to obtain image labeling samples corresponding to each hand image sample; S3. preprocessing the hand image samples; S4. Construct a multi-region parallel prediction model for the hand. The multi-region parallel prediction model for the hand includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of several convolutional layers. The subsequent network includes an SPP module, a CBAM module and a prediction module composed of several parallel convolutional kernels of different sizes. In step S4, the backbone network, SPP module, CBAM module, and prediction module of the hand multi-region parallel prediction model are connected in sequence, the hand image sample is input into the backbone network of the hand multi-region parallel prediction model, the backbone network extracts the features of the hand image sample, and outputs the first feature map of the hand image sample, the output first feature map is first input into the SPP module, and the convolution kernels of different sizes in the SPP module are used to extract features of different sizes to obtain the second feature map, the second feature maps are added and then output, and a third feature map containing information of different scales is obtained, the obtained third feature map is input into the CBAM module, and the CBAM module further extracts features of different sizes to obtain the final fourth feature map, and finally the fourth feature map is input into the prediction module, processed by the prediction module, and the prediction boundary box of the whole hand from the wrist to the fingertips, the palm, the two knuckles of the thumb, and the three knuckles of the remaining four fingers in the fourth feature map and the classification probability of the prediction boundary box are predicted and output, wherein the hand image sample with a classification probability greater than the positive threshold is defined as a positive sample, and the hand image sample with a classification probability less than the negative threshold is defined as a negative sample; S5. Divide the image dataset and the image annotation samples corresponding to the hand image samples in the image dataset into a training set, a validation set, and a test set, use the training set to train the hand multi-region parallel prediction model, then use the validation set to evaluate the hand multi-region parallel prediction model during the training process, and use the test set to test the effectiveness of the hand multi-region parallel prediction model, and obtain a trained hand multi-region parallel prediction model; S6. Collect the hand image to be predicted, preprocess the hand image to be predicted, input the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and output the prediction result of the hand image to be predicted.
2. The method for hand multi-region parallel prediction according to claim 1, characterized in that: In step S1, the hand image sample is obtained by photographing with a USB high-definition camera, and the USB high-definition camera is connected to a computer terminal to transmit the obtained hand image sample data to the computer terminal.
3. The method for hand multi-region parallel prediction according to claim 2, characterized in that: In step S2, the specific steps of labeling each hand image sample in the image dataset are: S21. Mark the real bounding box of the whole hand from the wrist to the fingertips in the hand image sample to obtain an image annotation sample of the whole hand from the wrist to the fingertips; S22. marking the true bounding box of the palm in the hand image sample to obtain an image annotation sample of the palm; S23. Mark the real bounding boxes of the two knuckles of the thumb and the three knuckles of the remaining four fingers in the hand image sample, and obtain the image annotation samples of the hand of the two knuckles of the thumb and the three knuckles of the remaining four fingers.
4. The method for hand multi-region parallel prediction according to claim 3, characterized in that: In step S3, the specific steps of the preprocessing are: S31. Scale the width and height ratio of the hand image samples to a uniform value; S32. Randomly horizontally flip and add noise to the scaled hand image samples; S33. Randomly select a number of hand image samples for splicing to obtain spliced hand image samples with the same specifications.
5. The method for hand multi-region parallel prediction according to claim 4, characterized in that: The CBAM module is provided with a channel attention module and a spatial attention module, wherein the spatial attention module is arranged after the channel attention module, the third feature map input to the CBAM module is provided with a channel, and the channel attention module captures the relationship information between the channels of the third feature map, and the channel attention module is provided with a first maximum pooling layer Maxpool, a first average pooling layer Avgpool, and a feedforward neural network shared module Shared MLP which are connected in sequence, the third feature map input to the CBAM module is input to the first maximum pooling layer Maxpool and the first average pooling layer Avgpool, parallel operations are performed, the output results are input to the feedforward neural network shared module Shared MLP, weights are shared, and then the results output by the feedforward neural network shared module Shared MLP are added, and then processed with a Sigmoid function to obtain the channel attention feature map M of the channel attention module. c Finally, the third feature map and the channel attention feature map M c Multiply to get the final output feature map F′ of the channel attention module, the channel attention feature map M c The specific calculation formula is as follows: M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Among them, F is the third feature map of the input CBAM module, AvgPool() is the average pooling operation, MaxPool() is the maximum pooling operation, MLP() is the feedforward neural network, and σ() is the Sigmoid function; The specific formula for calculating the final output feature map F′ of the channel attention module is as follows: F′=F*M c (F) The spatial attention module is provided with a second maximum pooling layer Maxpool, a second average pooling layer Avgpool, and a convolutional layer connected in sequence. The final output feature map F′ of the channel attention module is input into the second maximum pooling layer Maxpool and the second average pooling layer Avgpool for serial calculation. The output results are connected in the channel dimension and then input into the convolutional layer for calculation. The result obtained by the convolutional layer calculation is processed by the Sigmoid function to obtain the spatial attention feature map M. s (F′), the final output feature map F′ of the last channel attention module and the spatial attention feature map M s (F′) is multiplied to obtain the fourth feature map of the final output of the CBAM module, Spatial attention feature map M s The specific calculation formula is as follows: M s (F′)=σ(f 7*7 ([AvgPool(F′);MaxPool(F′)])) Among them, f 7*7 () is the convolution layer operation, [;] means to connect the results of serial calculation in the channel dimension, σ() is the Sigmoid function; The specific formula for calculating the fourth feature map of the final output of the CBAM module is as follows: F c =F′*M s (F′) Among them, F c It is the fourth feature map of the final output of the CBAM module.
6. The method for hand multi-region parallel prediction according to claim 5, characterized in that: Both the backbone network and subsequent networks need to be trained. The network parameters and weights of the backbone network use the network parameters and weights obtained by pre-training on the general ImageNet image classification dataset.
7. The method for hand multi-region parallel prediction according to claim 6, characterized in that: In step S5, the part of the hand image sample within the hand area is set as the foreground image sample, and the part outside the hand area is set as the background image sample. The loss function formula used when training the hand multi-area parallel prediction model is as follows: Loss=L box +L cls +L obj Among them, L box is the error caused by the hand multi-region parallel prediction model predicting the bounding box, L cls is the error caused by the classification of hand image samples, L obi Indicates the error caused by judging the foreground image samples and the background image samples; L box The specific formula for solving is as follows: Among them, λ coord is a hyperparameter, w i ,h i , x i ,y i are the width, height, and center coordinates of the true bounding box respectively; w′ i , h′ i , x′ i , y′ i The width, height, and center point coordinates of the bounding box are predicted by the multi-region parallel prediction model of the hand, Indicates whether the jth predicted bounding box in the i-th grid is a foreground image sample. If so, then The value is 1, otherwise, The value is 0; s 2 is the total number of grids, and B represents the number of true bounding boxes; L cls The specific formula for solving is as follows: Among them, λ class is a hyperparameter, p i (c) is the classification probability, p′ i (c) is 1-p i (c); L obi The specific formula for solving is as follows: Among them, λ noobi is a hyperparameter, Indicates whether the jth predicted bounding box in the i-th grid is a background image sample. If so, then The value is 1, otherwise, The value is 0; c i is the confidence of foreground image samples and background image samples; c′ i Predict the confidence of foreground image samples and background image samples for the multi-region parallel prediction model of the hand.
8. The method for hand multi-region parallel prediction according to claim 7, characterized in that: During the training process, the gradient descent method is used to update the network parameters and weights of the hand multi-region parallel prediction model. The validation set is used to evaluate the hand multi-region parallel prediction model during training, and the test set is used to test the effectiveness of the hand multi-region parallel prediction model to obtain the trained hand multi-region parallel prediction model.
9. A system for parallel prediction of multiple regions of a hand, characterized in that: The system comprises: An image data set acquisition module is used to obtain a certain number of hand image samples at different hand angles to form an image data set; A labeling module, used to label each hand image sample in the image data set to obtain image labeling samples corresponding to each hand image sample; A preprocessing module, used for preprocessing hand image samples; The hand multi-region parallel prediction model construction module is used to construct the hand multi-region parallel prediction model. The hand multi-region parallel prediction model includes a backbone network and a subsequent network. The backbone network is a DarkNet53 structure composed of several convolutional layers. The subsequent network includes an SPP module, a CBAM module and a prediction module composed of several parallel convolution kernels of different sizes. The backbone network, SPP module, CBAM module and prediction module of the hand multi-region parallel prediction model are connected in sequence. The hand image sample is input into the backbone network of the hand multi-region parallel prediction model. The backbone network extracts the features of the hand image sample and outputs the first feature map of the hand image sample. The output first feature map is first input into the SPP module and is not input into the SPP module. The convolution kernel of the same size is used to calculate and extract features of different sizes to obtain a second feature map. The second feature maps are added and output to obtain a third feature map containing information of different scales. The third feature map is input into the CBAM module, and features of different sizes are further extracted by the CBAM module to obtain a final fourth feature map. Finally, the fourth feature map is input into the prediction module, and is processed by the prediction module to predict and output the predicted bounding boxes of the whole hand from the wrist to the fingertips, the palm, the two knuckles of the thumb, and the three knuckles of the remaining four fingers in the fourth feature map, as well as the classification probability of the predicted bounding box, wherein the hand image samples with a classification probability greater than a positive threshold are defined as positive samples, and the hand image samples with a classification probability less than a negative threshold are defined as negative samples; A training module is used to divide the image data set and the image annotation samples corresponding to the hand image samples in the image data set into a training set, a validation set and a test set, use the training set to train the hand multi-region parallel prediction model, then use the validation set to evaluate the hand multi-region parallel prediction model during the training process, and use the test set to test the effectiveness of the hand multi-region parallel prediction model, and obtain a trained hand multi-region parallel prediction model; The processing module collects the hand image to be predicted, preprocesses the hand image to be predicted, inputs the preprocessed hand image to be predicted into the trained hand multi-region parallel prediction model, and outputs the prediction result of the hand image to be predicted.