A gesture segmentation method, system, device and medium based on graph convolution and attention mechanism

Abstract

A method, system, device and medium for gesture segmentation based on graph convolution and attention mechanism, the method comprising: using an RGB image containing a hand as network input, preprocessing an image data set, and dividing the preprocessed image data set into a training set and a test set; segmenting the RGB image using superpixels to obtain an initial segmentation region mask of the RGB image, inputting the initial segmentation region mask of the RGB image into a graph convolution layer, constructing a graph convolution pre-trained network in combination with an attention mechanism model, training the graph convolution pre-trained network, and optimizing network parameters in combination with a cross-entropy loss function to obtain a classification model; the system, device and medium are used to implement a method for gesture segmentation based on graph convolution and attention mechanism; the present invention improves the accuracy of hand categories in images, makes gesture segmentation more accurate, and makes edge segmentation clearer, and reduces the size of model parameter files, making it easier to deploy on hardware devices and having higher operating efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision image processing, and specifically relates to a gesture segmentation method, system, device and medium based on graph convolution and attention mechanism. Background Art

[0002] In gesture recognition research, vision-based gesture recognition is the most widely used technique. Gesture segmentation is the foundation and prerequisite for gesture recognition, and the quality of its segmentation impacts the accuracy of subsequent gesture recognition results. Gesture segmentation involves separating the gesture as the foreground from the background. Traditional gesture recognition algorithms use manually designed gesture features and then employ image recognition or machine learning algorithms to segment and detect the hand. Gesture segmentation techniques can be primarily categorized into template detection, skin color detection, and motion analysis. Segmentation techniques rely on high-level gesture features, such as those of the fingertips, fingers, and joints. Their goals are to detect the presence of a hand and to define the hand's area. Common skin color model algorithms primarily utilize skin color information related to the hand and use image spatial processing techniques to filter out background interference, ultimately obtaining gesture contours and location information. Common skin color models include the elliptical skin color model, thresholding, Gaussian model, and skin color probability distribution model. However, traditional methods are susceptible to interference from complex backgrounds and environments, resulting in low gesture recognition accuracy and poor robustness. Furthermore, these methods rely heavily on expert prior knowledge and manually designed features, necessitating a massive dataset annotation task.

[0003] Secondly, gesture segmentation algorithms that rely on depth images have emerged. These algorithms convert the depth values ​​of pixels in the captured image into grayscale values ​​and then segment gestures based on the grayscale values. Although depth cameras provide more comprehensive and stereoscopic three-dimensional position information, the equipment is expensive and the technology is incomplete, making it difficult to promote.

[0004] Due to its advantages in object detection, deep learning has quickly become a research hotspot in the field of gesture segmentation and recognition. While fully convolutional residual networks (FCNs) can achieve good semantic segmentation results, they lack precision in processing edge details. U-Net, a classic network in the field of semantic segmentation, uses an encoder-decoder structure to achieve good results. Incorporating skip connections further facilitates the recovery of object boundary contours. However, because U-Net uses a deep convolutional neural network, the model has a relatively large number of parameters, making it prone to overfitting on smaller datasets. This not only requires longer training times and more computing resources, but also makes it difficult to deploy on hardware.

[0005] In real-world applications, gesture segmentation and recognition face numerous challenges and interference, including the flexibility and variability of gestures, complex recognition environments, and uneven lighting. The complexity and variability of real-world scenarios, including skin-colored backgrounds and a variety of indoor and outdoor backgrounds, can interfere with gesture recognition. Furthermore, during the gesture segmentation process, gesture contours are easily obscured by other objects, affecting the algorithm's ability to extract target features. Severe occlusion can even lead to inaccurate gesture shape extraction. Furthermore, to effectively perform in real-world applications, gesture segmentation algorithms and networks must typically be deployed on hardware devices. However, most current gesture segmentation algorithms suffer from slow processing speeds and large network sizes, creating pain points that are difficult to address in practical applications.

[0006] Chinese invention application publication number CN114926898A discloses a gesture recognition model training and gesture recognition method, apparatus, device, and medium. The method captures multiple gesture images of hands wearing red gloves, converts each gesture image from RGB color space to YCrCb space, partitions the image into foreground and background, and binarizes it to obtain a gesture segmentation image. A convolutional neural network structure is then constructed and trained using the multiple gesture segmentation images to obtain a gesture recognition model. However, the two-stage processing method employed in this invention cannot achieve end-to-end processing, and the fixed threshold segmentation method requires the user to wear red, tight-fitting rubber gloves. This limitation significantly affects the versatility and stability of the patented method. Specifically, in some application scenarios, it is difficult for subjects to wear the corresponding gloves, and in complex scenes, objects of the same color significantly affect the segmentation and classification results, and edge segmentation is disrupted. Furthermore, this model uses a classical neural network for gesture detection and recognition. Due to the nature of the network, it inevitably ignores long-range contextual relationships. The network has many layers and relatively complex steps, resulting in a large network model and inconvenient deployment on everyday hardware devices.

[0007] The Chinese invention application with application publication number CN113673313A discloses a hand gesture recognition method based on a layered convolutional neural network. In the segmentation part, a segmentation network based on the VGG19 network is adopted, and different convolution kernel branches are introduced to be connected in parallel. The final segmentation result is obtained by fusing the results of multiple branches. However, this multi-branch segmentation structure based on U-net is bound to cause redundant waste of computing power and additional burden on model storage space. Summary of the Invention

[0008] In response to the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a gesture segmentation method, system, device and medium based on graph convolution and attention mechanism, which takes an RGB image containing a hand as the network input and uses the characteristics of graph convolution to focus on the long-range contextual relationship in the input image, which can be used to solve the technical problems of inaccurate hand detection and unclear edge segmentation in the existing gesture segmentation method; by using superpixel segmentation and graph convolution processing, the network scale and number of parameters of gesture segmentation are greatly reduced, so as to solve the technical problem that the gesture segmentation method in the prior art has a large network scale and is not convenient for migration to hardware devices.

[0009] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0010] A gesture segmentation method based on graph convolution and attention mechanism includes the following steps:

[0011] Step 1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain an RGB image dataset and a hand segmentation label image dataset; preprocess the RGB images in the RGB image dataset, and divide the preprocessed RGB image dataset into a training set and a test set;

[0012] Step 2: Perform superpixel segmentation on the RGB image in the RGB image dataset obtained in step 1 based on the SLIC algorithm to obtain the initial segmentation region mask seg_index of the RGB image;

[0013] Step 3: Construct the graph convolution layer;

[0014] Step 4: Build an attention mechanism model, including spatial attention module and channel attention module;

[0015] Step 5: Based on the graph convolution layer constructed in step 3 and the attention mechanism model constructed in step 4, build a graph convolution pre-training network;

[0016] Step 6: Input the training set in step 1 into the graph convolution pre-training network constructed in step 5 for training, and then optimize the network parameters of the graph convolution pre-training network constructed in step 5 by backpropagating the cross entropy loss function value until the network converges and a trained classification model is obtained;

[0017] Step 7: Use the classification model obtained in step 6 to perform gesture segmentation on the test set in step 1 to obtain the segmentation results of the test set.

[0018] The specific process of step 1 is:

[0019] Step 1.1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain RGB image datasets and hand segmentation label image datasets; set the RGB image to a three-dimensional matrix, the three-dimensional matrix F∈R c×m×n , where R represents the entire real number domain, m represents the length of the input image, n represents the width of the input image, and c represents the number of channels of the input image. The RGB image is read in, and then the values ​​of all pixels of the RGB image are extracted as eigenvalues ​​in the three-dimensional matrix F;

[0020] Step 1.2: Resize the RGB image from step 1.1 to c × 1280 × 896.

[0021] Step 1.3: Normalize the RGB image processed in step 1.2, that is, normalize the eigenvalues ​​in step 1.1 to between [0, 1]. The normalization method is as follows:

[0022]

[0023] Where input refers to the input single RGB image data, min(input) is the minimum pixel value in the RGB image, and max(input) is the maximum pixel value in the RGB image;

[0024] Step 1.4: Divide the RGB image dataset processed in step 1.3, select 70% of the RGB image dataset as the training set, and the remaining 30% as the test set.

[0025] In step 2, the number of blocks block_num for superpixel segmentation is 20, and the compactness parameter compactness is 0.92.

[0026] The specific process of step 3 is as follows:

[0027] Step 3.1: Input the pixel-level feature matrix F of the RGB image into the initial segmentation region mask seg_index obtained in step 2, F={f 11 , f 12 ,...,f hl}, where h is the number of rows of the pixel-level feature matrix F, l is the number of columns of the pixel-level feature matrix F, and f ij is the eigenvector contained in the pixel in the i-th row (i=1,2,...,h) and j-th column (j=1,2,...,l) of the pixel-level feature matrix F, n is the number of pixels in the pixel-level feature matrix F, that is, n=h×l; the pixel-level feature matrix F is divided into N regions {R1, R2,...,R N}, R kRefers to the kth region, k = 1, 2, ..., N; uses the center vector r k Represents each region within the pixel-level feature matrix F, the center vector r k For region R k The mean of all pixel features in is calculated as:

[0028]

[0029] Among them, num k represents the number of pixels in region Rk, k = 1, 2, ..., N;

[0030] All regions in the pixel-level feature matrix F are regarded as nodes for constructing the graph structure, and the node represents the value set with F g ={r1, r2, ..., r N} represents, where ri is the value of the i-th node, and the representative values ​​of all nodes are obtained;

[0031] Step 3.2: Construct the adjacency matrix A through the connection relationship between the nodes in step 3.1, and set the constructed adjacency matrix A to the node fully connected state, that is, each position in this adjacency matrix is ​​1, and the size is block_num×block_num; in order to deeply capture the relationship between different areas, set a weight matrix W of size 3×3 d ; Use Mahalanobis distance metric to represent the similarity between nodes. The specific calculation formula is as follows:

[0032]

[0033] Among them, (r i -r j ) T Yes (r i -r j ), M is a symmetric positive semidefinite matrix, and the calculation method of M is

[0034] The Mahalanobis distance D(r i , r j ) Construct the weight matrix W of the trainable dynamic adjacency matrix. The calculation formula of the weight matrix W is as follows:

[0035]

[0036] Among them, γ is a hyperparameter, r i and r j represents two nodes, D(·) represents the Mahalanobis distance metric;

[0037] Update the adjacency matrix A through the weight matrix W to reflect the graph structure at the current time step, that is, multiply the node's adjacency matrix A by the weight matrix W element by element to obtain the dynamic adjacency matrix A at time step t t ;

[0038] Step 3.3: The dynamic adjacency matrix A obtained in step 3.2 t Use pixel-level graph nodes to update the PGNU strategy for graph convolution operations to obtain the pixel-level feature matrix F new And the updated adjacency matrix A new Specifically, the PGNU strategy is used to propagate and update the features of the nodes obtained in step 3.1. The node update method is as follows:

[0039]

[0040] Among them, r i is the eigenvalue of the node in the i-th region, r i new is the i-th region node r i Updated eigenvalue, N i For node r i The neighbor node set of node r, u is the node r i Neighbor node's node, A t is the dynamic adjacency matrix at time step t, A t The parameters in are trainable parameters, and σ is a nonlinear activation function;

[0041] During the propagation process, the characteristic information between nodes is aggregated and transmitted on the graph, and the representative value of each node is updated. Get the updated node representative value set F in step 3.1 g ; According to the column direction of the matrix, the dynamic adjacency matrix A at time step t obtained in step 3.2 is t Sum and divide by the number of non-zero elements in the adjacency matrix A to obtain the average adjacency value; add the adjacency matrix A to the average adjacency value to update the adjacency matrix A to obtain the updated adjacency matrix A new ; Through the updated node feature value set F in step 3.1 g Update each region {R1, R2, ..., R N}, update the pixel-level feature matrix F in step 3.1, and obtain the pixel-level feature matrix F new , the specific formula is as follows:

[0042]

[0043] in, and are the input and output features belonging to the jth region, A .j represents the jth row of the adjacency matrix A, r i represents the node feature of the i-th region, and D is the weight matrix of feature propagation learned by the back-propagation gradient algorithm.

[0044] The specific process of step 4 is:

[0045] (1) Constructing a spatial attention module: In the spatial attention module, the average eigenvalue and the maximum eigenvalue of the channel dimension of the initial input feature matrix X are calculated to obtain an average feature matrix X1 and a maximum feature matrix X2 of size 1×h×w, where h is the number of rows of the input feature matrix and w is the number of columns of the input feature matrix; the average feature matrix X1 and the maximum feature matrix X2 are concatenated by channels to obtain a feature matrix X with a channel number of 2. all , the feature matrix X all Input to the classic convolutional layer CNN, set the convolution kernel size of the classic convolutional layer to 5×5, and perform edge padding before convolution. The padding data is 0 and the padding size is 2, and the spatial attention mask feature map M with a size of 1×h×w is obtained. s ; Combine the initial input feature matrix X with the spatial attention mask feature map M s Multiply them to get the feature matrix X′ after spatial attention enhancement, and add the feature matrix X′ after spatial attention enhancement to the initial input feature matrix X to get the feature map of the final output of the spatial attention module;

[0046] (2) Construct a channel attention module: In the channel attention module, perform a global pooling operation on the initial input feature matrix Y, compress the eigenvalue of each channel of the initial input feature matrix Y to a value, and obtain a c×1×1 feature matrix, where c is the number of channels of the input feature matrix; reduce the dimension of the c×1×1 feature matrix through the fully connected layer linear1 to obtain a (c / r)×1×1 feature matrix, where r is the dimensionality reduction ratio parameter; increase the dimension of the (c / r)×1×1 feature matrix through the fully connected layer linear2 to obtain a c×1×1 feature matrix after the increase in dimension; finally, use the Sigmoid function to limit the value of each channel in the c×1×1 feature matrix after the increase in dimension to between [0, 1], and obtain the channel attention mask M c ; Combine the initial input feature matrix Y with the channel attention mask M c Multiply them to get the feature matrix Y' after channel attention enhancement, and add the feature matrix Y' after channel attention enhancement to the initial input feature matrix Y to get the feature map finally output by the channel attention module.

[0047] The specific process of step 5 is:

[0048] Based on the graph convolution layer constructed in step 3, four graph convolution layers for feature extraction are established, namely gcn1, gcn2, gcn3, gcn4, and the graph convolution output layer gcn all , as follows:

[0049] The first layer gcn1:

[0050] Input the initial image segmentation region mask seg_index obtained in step 2 and the RGB image preprocessed in step 1.3 into the graph convolution layer gcn1 constructed in step 3 to obtain the first-scale graph model. The first-layer pixel-level feature matrix F1 in the first-scale graph model has a size of b×c×1280×896, where b refers to the number of samples processed simultaneously during each training session and c is the number of channels.

[0051] The second layer gcn2:

[0052] Perform maximum pooling on the first-layer pixel-level feature matrix F1, that is, divide the input first-layer pixel-level feature matrix F1 into several 3×3 areas, and set the sliding step size of the pooling operation to 2; for each pooling area, take the largest eigenvalue in the area as the pooling result, and obtain the pooled pixel-level feature matrix F1′. Input the pooled pixel-level feature matrix F1′ into the attention mechanism model constructed in step 4, that is, the pooled pixel-level feature matrix F1′ is processed by the spatial attention module and the channel attention module in sequence to obtain the feature map input F1″ of the second layer;

[0053] Perform a two-dimensional nearest neighbor pooling operation on the initial segmentation region mask seg_index obtained in step 2, that is, divide the input image initial segmentation region mask seg_index into several 3×3 regions, and set the sliding step size of the pooling operation to 2; for each pooling region, take the eigenvalue closest to the pooling center in the region as the pooling result, and obtain the image segmentation region mask seg_index2 of gcn2;

[0054] The second-layer feature map is input into the convolutional layer gcn2 of the image segmentation region mask seg_index2 of the second-layer feature map F1″ and gcn2 to obtain the second-scale map model. The second-layer pixel-level feature matrix F2 in the second-scale map model has a size of b×c×640×448; the second-layer pixel-level feature matrix F2 is upsampled until the second-layer pixel-level feature matrix F2 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the second-layer pixel-level feature matrix is ​​obtained.

[0055] The third layer gcn3:

[0056] Perform max pooling and attention model processing on the second-layer pixel-level feature matrix F2, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F2″ with a size of b×c×320×224; perform nearest neighbor pooling on the image segmentation region mask seg_index2 of gcn2 to obtain the image segmentation region mask seg_index3 of gcn3;

[0057] The feature map of the third layer is input into F2″ and the image segmentation region mask seg_index3 of gcn3 into the convolutional layer gcn3 to obtain the third scale graph model. The third layer pixel-level feature matrix F3 in the third scale graph model has a size of b×c×320×224; the third layer pixel-level feature matrix F3 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the third layer pixel-level feature matrix is ​​obtained.

[0058] The fourth layer gcn4:

[0059] Perform max pooling and attention model processing on the third-layer pixel-level feature matrix F3, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F3″ with a size of b×c×160×112; perform nearest neighbor pooling on the image segmentation region mask seg_index3 of gcn3 to obtain the image segmentation region mask seg_index4 of gcn4;

[0060] The feature map of the third layer is input into F3″ and the image segmentation region mask seg_index4 of gcn4 into the convolutional layer gcn4 to obtain the fourth scale graph model. The fourth layer pixel-level feature matrix F4 in the fourth scale graph model has a size of b×c×160×112; the fourth layer pixel-level feature matrix F4 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the fourth layer pixel-level feature matrix is ​​obtained.

[0061] Graph convolution output layer GCN all :

[0062] The first layer pixel-level feature matrix The third layer pixel-level feature matrix The fourth layer pixel-level feature matrix It is spliced ​​with the first layer pixel-level feature matrix F1 in the channel dimension direction to obtain the fused pixel-level total feature matrix F all , the size is b×4c×1280×896; the fused pixel-level total feature matrix Fall Input graph convolution output layer gcn all , the number of input channels of the fusion classification layer is 4c, and the number of output channels is the number of segmentation categories; the total pixel-level feature pixel level F after fusion all The length and width values ​​of the image are consistent with the RGB image preprocessed in step 1.3, and the classification feature matrix F is output. out .

[0063] The specific process of step 6 is:

[0064] Step 6.1: Substitute the classification feature matrix F in the graph convolution pre-training network constructed in step 5 out Input the Softmax classifier, which maps the input feature vector to the probability distribution of each category and outputs the predicted probability distribution matrix of RGB image classification in the RGB image dataset with a size of b×n×1280×896, where n is the number of categories, to obtain the classification result;

[0065] Step 6.2: Initialize the weight matrix W of step 3.2 using the Kaiming network parameter initialization method d and the weight matrix D from step 3; input the training set data from the RGB image dataset in step 1.4 into the graph convolution pre-training network constructed in step 5 for training. The number of training samples is 70% of the total dataset, the initial learning rate is 0.1, and the training generation number is 10;

[0066] Step 6.3: Perform label registration on the feature value of each pixel of the RGB image training set in step 1.4 with the label value of the pixel of the hand segmentation label map read in step 1.1 to obtain a hand segmentation label image dataset corresponding to the RGB image training set;

[0067] Step 6.4: Substitute the classification results in step 6.1 and the hand segmentation label image dataset corresponding to the RGB image training set in step 6.3 into the cross-entropy loss function. Optimize the network parameters of the graph convolution pre-training network constructed in step 5 by backpropagating the cross-entropy loss function value. The specific formula of cross-entropy loss is as follows:

[0068]

[0069] Among them, p(x) is the true probability distribution of the classification (true label value), and q(x) is the predicted probability distribution of the classification;

[0070] Repeatedly train and optimize the graph convolution pre-trained network, learning and updating the weight matrix W in step 3.2 multiple times during the training process dand the parameters of the weight matrix D in step 3.3; after 10 iterations, when the cross entropy loss function value no longer decreases, the graph convolution pre-training network reaches convergence and a classification model is obtained.

[0071] A gesture segmentation system based on graph convolution and attention mechanism, including:

[0072] Image preprocessing module: obtains multiple daily scene images containing hands and preprocesses the images;

[0073] Image segmentation module: performs super-pixel segmentation on the image in the image preprocessing module;

[0074] Graph convolution layer building module: used to build graph convolution layers;

[0075] Attention mechanism model building module: used to build the attention mechanism model;

[0076] Graph convolution pre-training network construction module: used to build graph convolution pre-training network;

[0077] Graph convolution pre-training network training module: The pre-processed image is input into the constructed graph convolution pre-training network for training until the network converges to obtain a trained classification model;

[0078] Result output module: The preprocessed image is input into the classification model of the graph convolution pre-training network training module to obtain the image segmentation result.

[0079] A gesture segmentation device based on graph convolution and attention mechanism, comprising:

[0080] Memory: used to store a computer program for implementing a gesture segmentation method based on graph convolution and attention mechanism;

[0081] Processor: used to implement a gesture segmentation method based on graph convolution and attention mechanism when executing the computer program.

[0082] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a gesture segmentation method based on graph convolution and attention mechanism.

[0083] Compared with the prior art, the present invention has the following advantages:

[0084] 1. When constructing a graph structure, the present invention first performs a superpixel segmentation operation on the input image and constructs a graph using regional representative features as graph nodes. Compared with the existing technology, this greatly reduces the computational cost of constructing the graph structure at the pixel level, reduces network parameters, makes the model more lightweight, and is convenient for porting to a variety of computing devices.

[0085] 2. Compared with traditional graph node update methods, this paper updates pixel features in the image using a pixel-level graph node update (PGNU) strategy and then re-averages the updated pixel features to update the graph nodes. The PGNU strategy allows multi-scale feature learning, and graph convolution operations can achieve multi-scale feature fusion by stacking node updates at different levels, thereby improving image segmentation performance, significantly saving computational costs, and reducing over-smoothing.

[0086] 3. This invention applies a spatial attention mask feature map to the feature map to express the relevance of that region to key information. The larger the weight value, the higher the relevance, and the more attention will be paid to the key region during network training. Furthermore, considering the data differences between various human-computer interaction image capture devices such as depth cameras and skeletal cameras, similar processing is performed on channels. Channels with high relevance to key information are given larger weights and receive more attention, which is beneficial for improving network training results. The adaptive selection of space and channels makes gesture edge segmentation clearer.

[0087] In summary, the present invention uses an RGB image containing a hand as network input and utilizes the characteristics of graph convolution to focus on the long-range contextual relationship in the input image, making the gesture segmentation effect more accurate and the edge segmentation clearer; by utilizing superpixel segmentation and graph convolution processing, the network scale and number of parameters for gesture segmentation are greatly reduced, which can be more conveniently deployed on hardware devices and has higher operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is an implementation flow chart of the present invention.

[0089] Figure 2 It is a schematic diagram of the graph convolution pre-training network structure of the present invention.

[0090] Figure 3 It is the segmentation result (part) of the present invention on the data set. DETAILED DESCRIPTION

[0091] The working principle of the present invention will be described in detail below with reference to the accompanying drawings.

[0092] A gesture segmentation method based on graph convolution and attention mechanism includes the following steps:

[0093] Step 1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain an RGB image dataset and a hand segmentation label image dataset; preprocess the RGB images in the RGB image dataset, and divide the preprocessed RGB image dataset into an RGB image training set and an RGB image test set;

[0094] Step 1.1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain RGB image datasets and hand segmentation label image datasets; set the RGB image to a three-dimensional matrix, the three-dimensional matrix F∈R c×m×n , where R represents the entire real number domain, m represents the length of the input image, n represents the width of the input image, and c represents the number of channels of the input image. The RGB image is read in, and then the values ​​of all pixels of the RGB image are extracted as eigenvalues ​​in the three-dimensional matrix F;

[0095] Step 1.2: Set the size of the RGB image in step 1.1 to c × 1280 × 896 to facilitate unified processing by the feature extraction network and achieve good performance;

[0096] Step 1.3: Normalize the RGB image processed in step 1.2, that is, normalize the eigenvalues ​​in step 1.1 to between [0, 1]. The normalization method is as follows:

[0097]

[0098] Where input refers to the input single RGB image data, min(input) is the minimum pixel value in the RGB image, and max(input) is the maximum pixel value in the RGB image;

[0099] Step 1.4: Divide the RGB image dataset processed in step 1.3 into 70% of the dataset as the training set and the remaining 30% as the test set.

[0100] Step 2: Perform superpixel segmentation on the RGB image in the RGB image dataset obtained in step 1 based on the SLIC algorithm to obtain the initial segmentation region mask seg_index of the RGB image;

[0101] Divide the single RGB image in step 1.3 into 20 uniform regions. This means that adjacent pixels with similar texture, color, and brightness are grouped into the same visually meaningful superpixel block. Set the number of superpixel segments, block_num, to 20. Since hands are usually irregularly shaped pixel blocks, set the compactness parameter, compactness, to 0.92. Obtain the initial segmentation region mask, seg_index, for the RGB image.

[0102] Step 3: Constructing the graph convolutional layer (GCN)

[0103] Step 3.1: Input the pixel-level feature matrix F of the RGB image into the initial segmentation region mask seg_index obtained in step 2, F={f 11 , f12 ,...,f hl}, where h is the number of rows of the pixel-level feature matrix F, l is the number of columns of the pixel-level feature matrix F, and f ij is the eigenvector contained in the pixel in the i-th row (i=1,2,...,h) and j-th column (j=1,2,...,l) of the pixel-level feature matrix F, n is the number of pixels in the pixel-level feature matrix F, that is, n=h×l; the pixel-level feature matrix F is divided into N regions {R1, R2,...,R N}, R k Refers to the kth region, k = 1, 2, ..., N; uses the center vector r k Represents each region within the pixel-level feature matrix F, the center vector r k For region R k The mean of all pixel features in is calculated as:

[0104]

[0105] Among them, num k Indicates region R k The number of pixels in the image, k = 1, 2, ..., N;

[0106] All regions in the pixel-level feature matrix F are regarded as nodes for constructing the graph structure. The nodes represent the value sets and are represented by F. g ={r1, r2, ..., r N} means, where r i is the value of the i-th node, and the representative values ​​of all nodes are obtained;

[0107] Step 3.2: Construct the adjacency matrix A through the connection relationship between the nodes in step 3.1. Set the constructed adjacency matrix A to the node fully connected state, that is, each position in this adjacency matrix is ​​1, indicating that all nodes are connected to each other, and the size is block_num×block_num; in order to deeply capture the relationship between different areas, set a weight matrix W of size 3×3 d In the graph convolution layer designed by the present invention, the Mahalanobis distance metric is used to represent the similarity between nodes. The specific calculation formula is as follows:

[0108]

[0109] Among them, (r i -r j ) T Yes (r i -r j ), M is a symmetric positive semidefinite matrix, and the calculation method of M is

[0110] The Mahalanobis distance D(r i , r j ) Construct the weight matrix W of the trainable dynamic adjacency matrix. The calculation formula of the weight matrix W is as follows:

[0111]

[0112] Among them, γ is a hyperparameter, r i and r j represents two nodes, D(·) represents the Mahalanobis distance metric;

[0113] Update the adjacency matrix A through the weight matrix W to reflect the graph structure at the current time step, that is, multiply the node's adjacency matrix A by the weight matrix W element by element to obtain the dynamic adjacency matrix A at time step t t ;

[0114] Step 3.3: The dynamic adjacency matrix A obtained in step 3.2 t Use the pixel-level graph node updating (PGNU) strategy to perform graph convolution operations and obtain the pixel-level feature matrix F new And the updated adjacency matrix A new Specifically, the PGNU strategy is used to propagate and update the features of the nodes obtained in step 3.1 to better represent the pixel-level features. The node update method is as follows:

[0115]

[0116] Among them, r i is the eigenvalue of the node in the i-th region, r i new is the i-th region node r i Updated eigenvalue, N i For node r i The neighbor node set of node r, u is the node r i Neighbor node's node, A t is the dynamic adjacency matrix at time step t, A t The parameters in are trainable parameters, and σ is a nonlinear activation function;

[0117] During the propagation process, the characteristic information between nodes is aggregated and transmitted on the graph, and the representative value of each node is updated. Get the updated node representative value set F in step 3.1 g ; According to the column direction of the matrix, the dynamic adjacency matrix A at time step t obtained in step 3.2 is tSum and divide by the number of non-zero elements in the adjacency matrix A to obtain the average adjacency value; add the adjacency matrix A to the average adjacency value to update the adjacency matrix A to obtain the updated adjacency matrix A new ; Through the updated node feature value set F in step 3.1 g Update each region {R1, R2, ..., R N}, update the pixel-level feature matrix F in step 3.1, and obtain the pixel-level feature matrix F new , the specific formula is as follows:

[0118]

[0119] in, and are the input and output features belonging to the jth region, A .j represents the jth row of the adjacency matrix A, r i represents the eigenvalue of the node in the i-th region, and D is the weight matrix of feature propagation learned by the back-propagation gradient algorithm.

[0120] Step 4: Build an attention mechanism model, including spatial attention module and channel attention module

[0121] (1) Constructing a spatial attention module: In the spatial attention module, the average eigenvalue and the maximum eigenvalue of the channel dimension of the initial input feature matrix X are calculated to obtain an average feature matrix X1 and a maximum feature matrix X2 of size 1×h×w, where h is the number of rows of the input feature matrix and w is the number of columns of the input feature matrix; the average feature matrix X1 and the maximum feature matrix X2 are concatenated by channels to obtain a feature matrix X with a channel number of 2. all , the feature matrix X all Input to the classic convolutional layer CNN, set the convolution kernel size of the classic convolutional layer to 5×5, and perform edge padding before convolution. The padding data is 0 and the padding size is 2, and the spatial attention mask feature map M with a size of 1×h×w is obtained. s ; Combine the initial input feature matrix X with the spatial attention mask feature map M s Multiply them to get the feature matrix X′ after spatial attention enhancement, and add the feature matrix X′ after spatial attention enhancement to the initial input feature matrix X to get the feature map of the final output of the spatial attention module;

[0122] (2) Construct a channel attention module: In the channel attention module, perform a global pooling operation on the initial input feature matrix Y, compress the eigenvalue of each channel of the initial input feature matrix Y to a value, and obtain a c×1×1 feature matrix, where c is the number of channels of the input feature matrix; reduce the dimension of the c×1×1 feature matrix through the fully connected layer linear1 to obtain a (c / r)×1×1 feature matrix, where r is the dimensionality reduction ratio parameter; increase the dimension of the (c / r)×1×1 feature matrix through the fully connected layer linear2 to obtain a c×1×1 feature matrix after the increase in dimension; finally, use the Sigmoid function to limit the value of each channel in the c×1×1 feature matrix after the increase in dimension to between [0, 1], and obtain the channel attention mask M c ; Combine the initial input feature matrix Y with the channel attention mask M c Multiply them to get the feature matrix Y' after channel attention enhancement, add the feature matrix Y' after channel attention enhancement to the initial input feature matrix Y to get the feature map of the final output of the channel attention module;

[0123] Step 5: Based on the graph convolution layer constructed in step 3 and the attention mechanism model constructed in step 4, build a graph convolution pre-training network;

[0124] Refer to the network structure diagram Figure 2 As shown, based on the graph convolution layer constructed in step 3, four graph convolution layers for feature extraction are established, namely gcn1, gcn2, gcn3, gcn4, and the graph convolution output layer gcn all , as follows:

[0125] The first layer gcn1:

[0126] Input the initial image segmentation region mask seg_index obtained in step 2 and the RGB image preprocessed in step 1.3 into the graph convolution layer gcn1 constructed in step 3 to obtain the first-scale graph model. The first-layer pixel-level feature matrix F1 in the first-scale graph model has a size of b×c×1280×896, where b refers to the number of samples processed simultaneously during each training session and c is the number of channels.

[0127] The second layer gcn2:

[0128] Perform maximum pooling on the first-layer pixel-level feature matrix F1, that is, divide the input first-layer pixel-level feature matrix F1 into several 3×3 areas, and set the sliding step size of the pooling operation to 2; for each pooling area, take the largest eigenvalue in the area as the pooling result, and obtain the pooled pixel-level feature matrix F1′. Input the pooled pixel-level feature matrix F1′ into the attention mechanism model constructed in step 4, that is, the pooled pixel-level feature matrix F1′ is processed by the spatial attention module and the channel attention module in sequence to obtain the feature map input F1″ of the second layer;

[0129] Perform a two-dimensional nearest neighbor pooling operation on the initial image segmentation region mask seg_index obtained in step 2, that is, divide the input image initial segmentation region mask seg_index into several 3×3 regions, and set the sliding step size of the pooling operation to 2; for each pooling region, take the eigenvalue closest to the pooling center in the region as the pooling result, and obtain the image segmentation region mask seg_index2 of gcn2; It is worth noting that although the sizes of seg_index and seg_index2 are different, the number of regions inside is the same, that is, the number of nodes in the graph structure constructed in the first and second layers is the same;

[0130] The second-layer feature map is input into the convolutional layer gcn2 of the image segmentation region mask seg_index2 of the second-layer feature map F1″ and gcn2 to obtain the second-scale map model. The second-layer pixel-level feature matrix F2 in the second-scale map model has a size of b×c×640×448; the second-layer pixel-level feature matrix F2 is upsampled until the second-layer pixel-level feature matrix F2 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the second-layer pixel-level feature matrix is ​​obtained.

[0131] The third layer gcn3:

[0132] Perform max pooling and attention model processing on the second-layer pixel-level feature matrix F2, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F2″ with a size of b×c×320×224; perform nearest neighbor pooling on the image segmentation region mask seg_index2 of gcn2 to obtain the image segmentation region mask seg_index3 of gcn3;

[0133] The feature map of the third layer is input into F2″ and the image segmentation region mask seg_index3 of gcn3 into the convolutional layer gcn3 to obtain the third scale graph model. The third layer pixel-level feature matrix F3 in the third scale graph model has a size of b×c×320×224; the third layer pixel-level feature matrix F3 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the third layer pixel-level feature matrix is ​​obtained.

[0134] The fourth layer gcn4:

[0135] Perform max pooling and attention model processing on the third-layer pixel-level feature matrix F3, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F3″ with a size of b×c×160×112; perform nearest neighbor pooling on the image segmentation region mask seg_index3 of gcn3 to obtain the image segmentation region mask seg_index4 of gcn4;

[0136] The feature map of the third layer is input into F3″ and the image segmentation region mask seg_index4 of gcn4 into the convolutional layer gcn4 to obtain the fourth scale graph model. The fourth layer pixel-level feature matrix F4 in the fourth scale graph model has a size of b×c×160×112; the fourth layer pixel-level feature matrix F4 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the fourth layer pixel-level feature matrix is ​​obtained.

[0137] Graph convolution output layer GCN all :

[0138] The first layer pixel-level feature matrix The third layer pixel-level feature matrix The fourth layer pixel-level feature matrix It is spliced ​​with the first layer pixel-level feature matrix F1 in the channel dimension direction to obtain the fused pixel-level total feature matrix F all , the size is b×4c×1280×896; the fused pixel-level total feature matrix F all Input graph convolution output layer gcn all , the number of input channels of the fusion classification layer is 4c, and the number of output channels is the number of segmentation categories; the total pixel-level feature pixel level F after fusion all The length and width of the image are consistent with the RGB image preprocessed in step 1.3, and the classification feature matrix F is output. out ;

[0139] Step 6: Build a classification model

[0140] Step 6.1: Substitute the classification feature matrix F in the graph convolution pre-training network constructed in step 5 out Input the Softmax classifier, which maps the input feature vector to the probability distribution of each category and outputs the predicted probability distribution matrix of RGB image classification in the RGB image dataset with a size of b×n×1280×896, where n is the number of categories, to obtain the classification result;

[0141] Step 6.2: Initialize the weight matrix W of step 3.2 using the Kaiming network parameter initialization method d and the weight matrix D from step 3; input the training set data from the RGB image dataset in step 1.4 into the graph convolution pre-training network constructed in step 5 for training. The number of training samples is 70% of the total dataset, the initial learning rate is 0.1, and the training generation number is 10;

[0142] Step 6.3: Perform label registration on the feature value of each pixel of the RGB image training set in step 1.4 with the label value of the pixel of the hand segmentation label map read in step 1.1 to obtain a hand segmentation label image dataset corresponding to the RGB image training set;

[0143] Step 6.4: Substitute the classification results in step 6.1 and the hand segmentation label image dataset corresponding to the RGB image training set in step 6.3 into the cross-entropy loss function. Optimize the network parameters of the graph convolution pre-training network constructed in step 5 by backpropagating the cross-entropy loss function value. The specific formula of cross-entropy loss is as follows:

[0144]

[0145] Among them, p(x) is the true probability distribution of the classification (true label value), and q(x) is the predicted probability distribution of the classification;

[0146] Repeatedly train and optimize the graph convolution pre-trained network, learning and updating the weight matrix W in step 3.2 multiple times during the training process d and the parameters of the weight matrix D in step 3.3; after 10 iterations, when the cross entropy loss function value no longer decreases, the graph convolution pre-training network reaches convergence and the classification model is obtained;

[0147] Step 7: Use the classification model in step 6 to perform gesture segmentation on the test set in the RGB image dataset in step 1.4 to obtain the segmentation results in the test set.

[0148] A gesture segmentation system based on graph convolution and attention mechanism, including:

[0149] Image preprocessing module: obtains multiple daily scene images containing hands and preprocesses the images;

[0150] Image segmentation module: performs super-pixel segmentation on the image in the image preprocessing module;

[0151] Graph convolution layer building module: used to build graph convolution layers;

[0152] Attention mechanism model building module: used to build the attention mechanism model;

[0153] Graph convolution pre-training network construction module: used to build graph convolution pre-training network;

[0154] Graph convolution pre-training network training module: The pre-processed image is input into the constructed graph convolution pre-training network for training until the network converges to obtain a trained classification model;

[0155] Result output module: The preprocessed image is input into the classification model of the graph convolution pre-training network training module to obtain the image segmentation result.

[0156] A gesture segmentation device based on graph convolution and attention mechanism, comprising:

[0157] Memory: used to store a computer program for implementing a gesture segmentation method based on graph convolution and attention mechanism;

[0158] Processor: used to implement a gesture segmentation method based on graph convolution and attention mechanism when executing the computer program.

[0159] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a gesture segmentation method based on graph convolution and attention mechanism.

[0160] The simulation experiment is as follows:

[0161] The hardware platform is: NVIDIA GeForce GTX 1650, 8GB RAM.

[0162] The software platform is: Python, PyTorch deep learning framework (python3.8+pytorch1.8.1+cu102).

[0163] Simulation content and results:

[0164] The dataset used in the simulation experiments is the publicly available dataset of everyday hand movement scenes, ego_hands. The dataset consists of 4800 RGB images with a size of 3×1280×720. The labels are the pixel values ​​of the hand segmentation label map, and the label information is divided into two categories: hand or background. These everyday scenes include playing chess and cards. Most training sets contain more than one hand target, and the background often contains complex lines and colors as distractions.

[0165] Table 1 shows the overall accuracy, average accuracy, kappa coefficient, and accuracy of hand and background categories when tested using the classic method, fully convolutional network (FCN), U-Net, and the three methods of the present invention, with the same training hyperparameters.

[0166] Table 1 Comparison of results between the present invention and the classical method

[0167] method FCN U-Net The present invention Overall accuracy 90.7 88.9 86.6 Average accuracy 88.4 89.7 89.2 Kappa coefficient 55.3 53.4 46.6 Background category accuracy 91.1 88.7 86.0 Hand category accuracy 85.9 90.7 92.4

[0168] As can be seen from Table 1, compared with the FCN and U-Net classic methods in the prior art, the accuracy of the hand category obtained by the segmentation method of the present invention is improved, indicating that the gesture segmentation effect is more accurate and the edge segmentation is clearer.

[0169] More importantly, a comparison of the file sizes of model parameters saved after training (Table 2) shows that the model parameter files of our present invention are significantly smaller than those of the existing FCN and U-Net approaches for the same gesture segmentation task. This means that our method is easier to deploy on hardware devices and offers higher operational efficiency.

[0170] Table 2 Comparison of model parameter file sizes between the present invention and the classical method

[0171] method FCN U-Net The present invention Model parameter file size 576KB 588KB 13KB

[0172] Figure 3 This is the segmentation result (part) of the present invention on the data set. It can be seen from the figure that the gesture segmentation method proposed in the present invention has clearer edge segmentation for the hand, especially the edge detection effect for multiple fingers. Even for some pictures where the background between the fingers is not marked in the label, they are also clearly segmented in the method of the present invention.

Claims

1. A gesture segmentation method based on graph convolution and attention mechanism, characterized in that: The following steps are involved: Step 1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain an RGB image dataset and a hand segmentation label image dataset; preprocess the RGB images in the RGB image dataset, and divide the preprocessed RGB image dataset into a training set and a test set; Step 2: Perform superpixel segmentation on the RGB image in the RGB image dataset obtained in step 1 based on the SLIC algorithm to obtain the initial segmentation region mask seg_index of the RGB image; Step 3: Construct the graph convolution layer, which includes the following steps: Step 3.1: Input the pixel-level feature matrix F of the RGB image into the initial segmentation region mask seg_index obtained in step 2, and regard all regions in the pixel-level feature matrix F as nodes of the constructed graph structure. The nodes represent the value set, and F is used to represent the pixel-level feature matrix F. g ={r1,r2,...,r N } means, where r i is the eigenvalue of the i-th region node; Step 3.2: Construct the adjacency matrix A through the connection relationship between nodes in step 3.1, and set the constructed adjacency matrix A to the node full connection state; use the Mahalanobis distance metric to represent the similarity between nodes, and the Mahalanobis distance D (r i ,r j ) Construct a trainable dynamic adjacency matrix weight matrix W, and update the adjacency matrix A through the weight matrix W to reflect the graph structure at the current time step, that is, multiply the node adjacency matrix A by the weight matrix W element by element to obtain the dynamic adjacency matrix A at time step t t ; Step 3.3: The dynamic adjacency matrix A obtained in step 3.2 t Use pixel-level graph nodes to update the PGNU strategy for graph convolution operations to obtain the pixel-level feature matrix F new And the updated adjacency matrix A new ; Step 4: Build an attention mechanism model, including spatial attention module and channel attention module; Step 5: Based on the graph convolution layer constructed in step 3 and the attention mechanism model constructed in step 4, build a graph convolution pre-training network; Step 6: Input the training set in step 1 into the graph convolution pre-training network constructed in step 5 for training, and then optimize the network parameters of the graph convolution pre-training network constructed in step 5 by backpropagating the cross entropy loss function value until the network converges and a trained classification model is obtained; Step 7: Use the classification model obtained in step 6 to perform gesture segmentation on the test set in step 1 to obtain the segmentation results of the test set.

2. A gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: The specific process of step 1 is: Step 1.1: Input multiple RGB images of daily scenes containing hands and hand segmentation label images into the feature extraction network to obtain RGB image datasets and hand segmentation label image datasets; set the RGB image to a three-dimensional matrix, the three-dimensional matrix F∈R c×m×n , where R represents the entire real number domain, m represents the length of the input image, n represents the width of the input image, and c represents the number of channels of the input image. The RGB image is read in, and then the values ​​of all pixels of the RGB image are extracted as eigenvalues ​​in the three-dimensional matrix F; Step 1.2: Resize the RGB image from step 1.1 to c × 1280 × 896. Step 1.3: Normalize the RGB image processed in step 1.2, that is, normalize the eigenvalues ​​in step 1.1 to between [0, 1]. The normalization method is as follows: Where input refers to the input single RGB image data, min(input) is the minimum pixel value in the RGB image, and max(input) is the maximum pixel value in the RGB image; Step 1.4: Divide the RGB image dataset processed in step 1.3, select 70% of the RGB image dataset as the training set, and the remaining 30% as the test set.

3. The gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: In step 2, the number of blocks block_num for superpixel segmentation is 20, and the compactness parameter compactness is 0.

92.

4. The gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 3.1: Input the pixel-level feature matrix F of the RGB image into the initial segmentation region mask seg_index obtained in step 2, F={f 11 ,f 12 ,…,f hl }, where h is the number of rows of the pixel-level feature matrix F, l is the number of columns of the pixel-level feature matrix F, and f ij is the eigenvector contained in the pixel in the i-th row (i=1,2,…,h) and j-th column (j=1,2,…,l) of the pixel-level feature matrix F, n is the number of pixels in the pixel-level feature matrix F, that is, n=h×l; the pixel-level feature matrix F is divided into N regions {R1,R2,…,R N }, R k Refers to the kth region, k = 1, 2, ..., N; uses the center vector r k Represents each region within the pixel-level feature matrix F, the center vector r k For region R k The mean of all pixel features in is calculated as: Among them, num k Indicates region R k The number of pixels in the image, k = 1, 2, ..., N; All regions in the pixel-level feature matrix F are regarded as nodes for constructing the graph structure. The nodes represent the value sets and are represented by F. g ={r1,r2,...,r N } means, where r i is the eigenvalue of the i-th region node; Step 3.2: Construct the adjacency matrix A through the connection relationship between the nodes in step 3.1, and set the constructed adjacency matrix A to the node fully connected state, that is, each position in this adjacency matrix is ​​1, and the size is block_num×block_num; in order to deeply capture the relationship between different areas, set a weight matrix W of size 3×3 d ; Use Mahalanobis distance metric to represent the similarity between nodes. The specific calculation formula is as follows: Among them, (r i -r j ) T Yes (r i -r j ), M is a symmetric positive semidefinite matrix, and the calculation method of M is The Mahalanobis distance D(r i ,r j ) Construct the weight matrix W of the trainable dynamic adjacency matrix. The calculation formula of the weight matrix W is as follows: Among them, γ is a hyperparameter, r i and r j represents two nodes, D(·) represents the Mahalanobis distance metric; Update the adjacency matrix A through the weight matrix W to reflect the graph structure at the current time step, that is, multiply the node's adjacency matrix A by the weight matrix W element by element to obtain the dynamic adjacency matrix A at time step t t ; Step 3.3: The dynamic adjacency matrix A obtained in step 3.2 t Use pixel-level graph nodes to update the PGNU strategy for graph convolution operations to obtain the pixel-level feature matrix F new And the updated adjacency matrix A new Specifically, the PGNU strategy is used to propagate and update the features of the nodes obtained in step 3.

1. The node update method is as follows: Among them, r i is the eigenvalue of the node in the i-th region, r i new is the i-th region node r i Updated eigenvalue, N i For node r i The neighbor node set of node r, u is the node r i Neighbor node's node, A t is the dynamic adjacency matrix at time step t, A t The parameters in are trainable parameters, and σ is a nonlinear activation function; During the propagation process, the characteristic information between nodes is aggregated and transmitted on the graph, and the representative value of each node is updated. Get the updated node representative value set F in step 3.1 g ; According to the column direction of the matrix, the dynamic adjacency matrix A at time step t obtained in step 3.2 is t Sum and divide by the number of non-zero elements in the adjacency matrix A to obtain the average adjacency value; add the adjacency matrix A to the average adjacency value to update the adjacency matrix A to obtain the updated adjacency matrix A new ; Through the updated node feature value set F in step 3.1 g Update each region {R1, R2, ..., R N }, update the pixel-level feature matrix F in step 3.1, and obtain the pixel-level feature matrix F new , the specific formula is as follows: in, and are the input and output features belonging to the jth region, A j represents the jth row of the adjacency matrix A, r i represents the eigenvalue of the node in the i-th region, and D is the weight matrix of feature propagation learned by the back-propagation gradient algorithm.

5. The gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: The specific process of step 4 is: (1) Constructing a spatial attention module: In the spatial attention module, the average eigenvalue and the maximum eigenvalue of the channel dimension of the initial input feature matrix X are calculated to obtain an average feature matrix X1 and a maximum feature matrix X2 of size 1×h×w, where h is the number of rows of the input feature matrix and w is the number of columns of the input feature matrix; the average feature matrix X1 and the maximum feature matrix X2 are concatenated by channels to obtain a feature matrix X with a channel number of 2. all , the feature matrix X all Input to the classic convolutional layer CNN, set the convolution kernel size of the classic convolutional layer to 5×5, and perform edge padding before convolution. The padding data is 0 and the padding size is 2, and the spatial attention mask feature map M with a size of 1×h×w is obtained. s ; Combine the initial input feature matrix X with the spatial attention mask feature map M s Multiply them to get the feature matrix X' after spatial attention enhancement, and add the feature matrix X' after spatial attention enhancement to the initial input feature matrix X to get the feature map of the final output of the spatial attention module; (2) Constructing a channel attention module: In the channel attention module, a global pooling operation is performed on the initial input feature matrix Y, compressing the eigenvalues ​​of each channel of the initial input feature matrix Y to a single value, obtaining a c×1×1 feature matrix, where c is the number of channels of the input feature matrix; The c×1×1 feature matrix is ​​reduced in dimension by the fully connected layer linear1 to obtain a (c / r)×1×1 feature matrix, where r is the dimensionality reduction ratio parameter; the (c / r)×1×1 feature matrix is ​​increased in dimension by the fully connected layer linear2 to obtain a c×1×1 feature matrix after the increase in dimension; finally, the Sigmoid function is used to limit the value of each channel in the increased c×1×1 feature matrix to between [0,1] to obtain the channel attention mask M c ; Combine the initial input feature matrix Y with the channel attention mask M c Multiply them to get the feature matrix Y' after channel attention enhancement, and add the feature matrix Y' after channel attention enhancement to the initial input feature matrix Y to get the feature map finally output by the channel attention module.

6. The gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: The specific process of step 5 is: Based on the graph convolution layer constructed in step 3, four graph convolution layers for feature extraction are established, namely gcn1, gcn2, gcn3, gcn4, and the graph convolution output layer gcn all , as follows: The first layer gcn1: Input the initial image segmentation region mask seg_index obtained in step 2 and the RGB image preprocessed in step 1.3 into the graph convolution layer gcn1 constructed in step 3 to obtain the first-scale graph model. The first-layer pixel-level feature matrix F1 in the first-scale graph model has a size of b×c×1280×896, where b refers to the number of samples processed simultaneously during each training session and c is the number of channels. The second layer gcn2: Perform maximum pooling on the first-layer pixel-level feature matrix F1, that is, divide the input first-layer pixel-level feature matrix F1 into several 3×3 areas, and set the sliding step size of the pooling operation to 2; for each pooled area, take the largest eigenvalue in the area as the pooling result, and obtain the pooled pixel-level feature matrix F1'. Input the pooled pixel-level feature matrix F1' into the attention mechanism model constructed in step 4, that is, the pooled pixel-level feature matrix F1' is processed by the spatial attention module and the channel attention module in sequence to obtain the feature map input F1" of the second layer; Perform a two-dimensional nearest neighbor pooling operation on the initial segmentation region mask seg_index obtained in step 2, that is, divide the input image initial segmentation region mask seg_index into several 3×3 regions, and set the sliding step size of the pooling operation to 2; for each pooling region, take the eigenvalue closest to the pooling center in the region as the pooling result, and obtain the image segmentation region mask seg_index2 of gcn2; Input the second-layer feature map F1' and the image segmentation region mask seg_index2 of gcn2 into the convolutional layer gcn2 to obtain the second-scale map model. The second-layer pixel-level feature matrix F2 in the second-scale map model has a size of b×c×640×448; upsample the second-layer pixel-level feature matrix F2 until it has the same resolution ratio as the RGB image preprocessed in step 1.3, and obtain the second-layer pixel-level feature matrix The third layer gcn3: Perform max pooling and attention model processing on the second-layer pixel-level feature matrix F2, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F2" with a size of b×c×320×224; perform nearest neighbor pooling on the image segmentation region mask seg_index2 of gcn2 to obtain the image segmentation region mask seg_index3 of gcn3; The feature map of the third layer is input into F2' and the image segmentation region mask seg_index3 of gcn3 into the convolutional layer gcn3 to obtain the third scale graph model. The third layer pixel-level feature matrix F3 in the third scale graph model has a size of b×c×320×224; the third layer pixel-level feature matrix F3 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the third layer pixel-level feature matrix is ​​obtained. The fourth layer gcn4: Perform max pooling and attention model processing on the third-layer pixel-level feature matrix F3, with the same parameter settings as the first-layer pixel-level feature matrix F1 in the first layer, to obtain the third-layer feature map input F3" with a size of b×c×160×112; perform nearest neighbor pooling on the image segmentation region mask seg_index3 of gcn3 to obtain the image segmentation region mask seg_index4 of gcn4; The feature map of the third layer is input into F3' and the image segmentation region mask seg_index4 of gcn4 into the convolutional layer gcn4 to obtain the fourth scale graph model. The fourth layer pixel-level feature matrix F4 in the fourth scale graph model has a size of b×c×160×112; the fourth layer pixel-level feature matrix F4 is upsampled until the third layer pixel-level feature matrix F3 has the same resolution ratio as the RGB image preprocessed in step 1.3, and the fourth layer pixel-level feature matrix is ​​obtained. Graph convolution output layer GCN all : The first layer pixel-level feature matrix The third layer pixel-level feature matrix The fourth layer pixel-level feature matrix It is spliced ​​with the first layer pixel-level feature matrix F1 in the channel dimension direction to obtain the fused pixel-level total feature matrix F all , the size is b×4c×1280×896; the fused pixel-level total feature matrix F all Input graph convolution output layer gcn all , the number of input channels of the fusion classification layer is 4c, and the number of output channels is the number of segmentation categories; the total pixel-level feature pixel level F after fusion all The length and width values ​​of the image are consistent with the RGB image preprocessed in step 1.3, and the classification feature matrix F is output. out .

7. The gesture segmentation method based on graph convolution and attention mechanism according to claim 1, characterized in that: The specific process of step 6 is: Step 6.1: Substitute the classification feature matrix F in the graph convolution pre-training network constructed in step 5 out Input the Softmax classifier, which maps the input feature vector to the probability distribution of each category and outputs the predicted probability distribution matrix of RGB image classification in the RGB image dataset with a size of b×n×1280×896, where n is the number of categories, to obtain the classification result; Step 6.2: Initialize the weight matrix W of step 3.2 using the Kaiming network parameter initialization method d and the weight matrix D from step 3; input the training set data from the RGB image dataset in step 1.4 into the graph convolution pre-training network constructed in step 5 for training. The number of training samples is 70% of the total dataset, the initial learning rate is 0.1, and the training generation number is 10; Step 6.3: Perform label registration on the feature value of each pixel of the RGB image training set in step 1.4 with the label value of the pixel of the hand segmentation label map read in step 1.1 to obtain a hand segmentation label image dataset corresponding to the RGB image training set; Step 6.4: Substitute the classification results in step 6.1 and the hand segmentation label image dataset corresponding to the RGB image training set in step 6.3 into the cross-entropy loss function. Optimize the network parameters of the graph convolution pre-training network constructed in step 5 by backpropagating the cross-entropy loss function value. The specific formula of cross-entropy loss is as follows: Among them, p(x) is the true probability distribution of the classification, that is, the true label value; q(x) is the predicted probability distribution of the classification; Repeatedly train and optimize the graph convolution pre-trained network, learning and updating the weight matrix W in step 3.2 multiple times during the training process d and the parameters of the weight matrix D in step 3.3; after 10 iterations, when the cross entropy loss function value no longer decreases, the graph convolution pre-training network reaches convergence and a classification model is obtained.

8. A gesture segmentation system based on graph convolution and attention mechanism, used to implement the gesture segmentation method based on graph convolution and attention mechanism according to any one of claims 1 to 7, characterized in that: include: Image preprocessing module: obtains multiple daily scene images containing hands and preprocesses the images; Image segmentation module: performs super-pixel segmentation on the image in the image preprocessing module; Graph convolution layer building module: used to build graph convolution layers; Attention mechanism model building module: used to build the attention mechanism model; Graph convolution pre-training network construction module: used to build graph convolution pre-training network; Graph convolution pre-training network training module: The pre-processed image is input into the constructed graph convolution pre-training network for training until the network converges to obtain a trained classification model; Result output module: The preprocessed image is input into the classification model of the graph convolution pre-training network training module to obtain the image segmentation result.

9. A gesture segmentation device based on graph convolution and attention mechanism, characterized in that: include: Memory: used to store a computer program for implementing the gesture segmentation method based on graph convolution and attention mechanism according to any one of claims 1 to 7; Processor: used to implement the gesture segmentation method based on graph convolution and attention mechanism as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement the gesture segmentation method based on graph convolution and attention mechanism described in any one of claims 1 to 7.

Images