Gesture segmentation network device and method based on multi-branch cascaded transformer

Abstract

This invention provides a gesture segmentation network device and method based on a multi-branch cascaded Transformer. The device includes a degree convolutional neural network (DCNN) to extract features from the original gesture image to obtain an intermediate feature map. The multi-branch cascaded Transformer module (MBCT) includes multiple cascaded Transformer branches, each of which includes a Patch Partition layer, a Linear Embedding layer, and a Multi-Window Self-Attention Block (MWSA) connected in series. A decoder is used to restore the gesture image to the same size as the original. The segmentation results of this invention have smoother hand edges, stronger ability to remove complex backgrounds, and greater robustness and effectiveness. This invention exhibits high accuracy and strong robustness even under conditions of uneven lighting, complex background noise, and varied gesture shapes.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, the field of computer vision, and specifically to a gesture segmentation method based on a multi-branch cascaded Transformer. Background Technology

[0002] With the rapid development of robotics, Human-Computer Interaction (HRI) has become increasingly important. Gestures can provide a simple, intuitive, and concise way to achieve HRI, thus vision-based gesture recognition has attracted much attention from scholars. Furthermore, vision-based gesture recognition mainly includes two steps: hand detection / segmentation and classification. Hand detection involves using bounding boxes in an image to detect and label the hand region when it exists. Hand segmentation involves segmenting the detected hand region from the image as the region of interest to reduce the computational burden of subsequent operations. Moreover, the hand detection / segmentation step is the initial and necessary step for classification; the accuracy of hand segmentation greatly affects the accuracy of gesture recognition.

[0003] Scholars have used traditional methods to detect / segment the hand region. Some researchers have used skin color[1], shape and texture (such as oriented gradient histogram[2], Haar-like[3], scale-invariant feature transform[4] features), combined with skin color, shape and texture[5], and fused motion and skin color[6] information to segment the hand region using a monocular camera.

[0004] With the development of deep learning methods, CNN-based methods have achieved good performance. Some researchers have applied convolutional neural network (CNN)-based methods to detect / segment hand regions. Tang et al. [7] used a single-shot multi-box detector to detect hand regions. Al-Hammadi et al. [8] used multiple deep learning architectures to segment hand regions. Dadashzadeh et al. [9] used convolutional residual networks and strous spatial pyramid sets to segment hand regions. Wei et al.

[10] incorporated the object detection model SSD (Single Shot multi-box Detector) into gesture segmentation, but this model caused the omission of hand detail information when performing thresholding. Cui et al.

[11] proposed the IASPP-ResNet model, which aims to enable the network to extract multi-scale information of gestures, enrich the feature representation of gestures, and obtain more accurate gesture segmentation results. Many gesture segmentation methods are based on fully convolutional neural networks. They apply a series of convolutional layers and downsampling layers to expand the receptive field of the network, obtain the features of gestures, and then use upsampling layers to recover the intermediate feature maps, thereby segmenting gestures. Chinese patent 201910130815.4 proposes an improved capsule network and algorithm method for gesture image segmentation and recognition. It employs an improved capsule network in deep learning to detect hands and generate binarized images for gesture recognition. The method includes the following steps: capturing and collecting gesture images against complex backgrounds; constructing and training a U-shaped residual capsule network to obtain binarized gesture images; locating the gesture bounding box; and constructing and training an improved matrix capsule network to achieve gesture image recognition. However, current vision-based gesture recognition methods suffer from slow network convergence and low recognition rates under complex backgrounds and non-uniform lighting conditions. Due to the varied shapes of gestures and complex background interference, gesture segmentation remains challenging, and the results directly affect the gesture recognition process, thus reducing human-computer interaction performance. Summary of the Invention

[0005] One of the objectives of this invention is to provide a gesture segmentation network device based on a multi-branch cascaded Transformer to solve the problem of low gesture recognition rate in existing network devices.

[0006] One of the objectives of this invention is achieved as follows: a gesture segmentation network device based on a multi-branch cascaded Transformer, comprising:

[0007] A deep convolutional neural network (DCNN) extracts features from the original gesture image to obtain an intermediate feature map.

[0008] The Multi-Branch Cascaded Transformer Module (MBCT) consists of multiple cascaded Transformer branches. Each Transformer branch includes a Patch Partition layer, a Linear Embedding layer, and a multi-window self-attention block (MWSA) connected in series.

[0009] A decoder used to restore the image to the same size as the original gesture image;

[0010] The deep convolutional neural network (DCNN) receives the original gesture image, performs calculations, and then inputs the calculation results into the multi-branch cascaded Transformer module (MBCT) in one direction. The result of the calculations in the other direction is superimposed with the result of the multi-branch cascaded Transformer module (MBCT) to obtain the superimposed gesture image, which is then input into the decoder.

[0011] The multi-branch cascaded Transformer module MBCT receives the gesture image from the deep convolutional neural network (DCNN) and performs calculations on it. The calculation result is then superimposed with the calculation result of the deep convolutional neural network (DCNN) to obtain the superimposed gesture image, which is then input into the decoder.

[0012] The decoder receives the original gesture image and the superimposed gesture image, decodes them, and obtains the corresponding gesture segmentation image.

[0013] Furthermore, the present invention can be implemented according to the following technical solution:

[0014] Each Transformer branch first feeds the intermediate feature map into the Patch Partition layer. The Patch Partition layer uses convolution operations to downsample the feature map, dividing the intermediate feature map into non-overlapping patches, and obtaining the feature map after patching.

[0015] The Linear Embedding layer transforms the dimension of the feature map output by the Patch Partition layer into a specific dimension, denoted as C;

[0016] The Multi-Window Self-Attention Block (MWSA) receives a specific graph output from the Linear Embedding layer and extracts multi-scale features of gestures from it.

[0017] The Transformer branch takes the output of the Multi-Window Self-Attention Block (MWSA) and passes it through the PatchMerging layer before passing it to the next Transformer branch. After fusing it with the output of the LinearEmbedding layer of the next Transformer branch, it enters the Multi-Window Self-Attention Block (MWSA) for feature extraction. The input of the last Transformer branch contains the gesture features extracted by the previous Transformer branches.

[0018] The output of each Transformer branch is upsampled layer by layer from bottom to top, and the results are concatenated level by level to obtain the overall feature map.

[0019] The Patch Merging layer selects pixel values ​​at intervals of 2 in both the row and column directions of the feature map, concatenates them together as a single tensor, and after stretching and unfolding, the channel dimension becomes 4 times the original. Then, a fully connected layer sets the channel dimension to a specific dimension.

[0020] The Multi-Window Self-Attention Block (MWSA) includes three cascaded Multi-Head Self-Attention Sub-Modules (MHSA). Each MHSA includes a cascaded LayerNorm (LN) layer, a Multi-Head Self-Attention Module (MSA), a Residual Connection (LN), and a 2-layer MLP with GELU nonlinearity.

[0021] The multi-window self-attention (MWSA) module can be described as follows:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] in Representing the lth th W l - Output of the MSA module, z l Representing the lth th The output of each MLP module;

[0029] The window sizes of the three Multi-Head Self-Attention Submodules (MHSA) are 3, 5, and 7, respectively.

[0030] The deep convolutional neural network (DCNN) is a feature extraction network consisting of a 7×7 convolution, a 3×3 convolution, and four residual groups. A 1×1 convolution is added to the fourth residual group. After feature extraction by the deep convolutional neural network (DCNN), the input image yields an intermediate feature map, which is 1 / 8 the size of the original image.

[0031] The decoder includes a serially connected upsampling module, a 3×3 convolution module, an upsampling module, a 3×3 convolution module, an upsampling module, and a 1×1 convolution module.

[0032] The second objective of this invention is to provide a gesture segmentation method based on a multi-branch cascaded Transformer to solve the problem of low human-computer interaction performance in the prior art.

[0033] The second objective of this invention is achieved as follows: a gesture segmentation method based on a multi-branch cascaded Transformer, comprising the following steps:

[0034] A. The gesture segmentation method based on multi-branch cascaded Transformer is applied to the gesture segmentation network device based on multi-branch cascaded Transformer as described in claim 1;

[0035] B. Preprocess the gesture image to fix its size;

[0036] C. Construct the gesture segmentation network as described in claim 1, input the preprocessed image into the gesture segmentation network, train the gesture segmentation network using a loss function and an optimizer, and obtain a gesture segmentation network with optimal parameters.

[0037] E. Inputting the gesture image into the trained gesture segmentation network will output the corresponding gesture segmentation image.

[0038] Furthermore, the present invention can be implemented according to the following technical solution:

[0039] The gesture segmentation network includes:

[0040] A deep convolutional neural network (DCNN) extracts features from a gesture image to obtain an intermediate feature map.

[0041] The Multi-Branch Cascaded Transformer Module (MBCT) consists of multiple cascaded Transformer branches. Each Transformer branch includes a Patch Partition layer, a Linear Embedding layer, and a multi-window self-attention block (MWSA) connected in series.

[0042] The decoder is used to restore the image to the same size as the original input image.

[0043] The deep convolutional neural network (DCNN) receives the original gesture image, performs calculations, and then inputs the calculation results into the multi-branch cascaded Transformer module (MBCT) in one direction. The result of the calculations in the other direction is superimposed with the result of the multi-branch cascaded Transformer module (MBCT) to obtain the superimposed gesture image, which is then input into the decoder.

[0044] The multi-branch cascaded Transformer module MBCT receives the gesture image from the deep convolutional neural network (DCNN) and performs calculations on it. The calculation result is then superimposed with the calculation result of the deep convolutional neural network (DCNN) to obtain the superimposed gesture image, which is then input into the decoder.

[0045] The decoder receives the original gesture image and the superimposed gesture image, decodes them, and obtains the corresponding gesture segmentation image.

[0046] Step B includes a gesture image acquisition operation and a preprocessing operation. The gesture image acquisition operation is used to acquire a color original gesture image, and the preprocessing operation is used to rotate, flip and crop the color original gesture image, and input the result into the gesture segmentation network.

[0047] The advantages of the gesture segmentation method based on multi-branch cascaded Transformer provided by this invention are as follows:

[0048] To address the issue of scale-variable gestures in complex backgrounds, this invention utilizes a multi-branch structure within the Multi-Branch Cascaded Transformer module (MBCT). This structure simultaneously extracts local detail features and models global semantic information, effectively capturing multi-scale gesture features. Furthermore, the cascading arrangement of branches increases the model's receptive field, enabling the modeling of global semantic information while acquiring local detail features. Multi-Window Self-Attention (MWSA) leverages the differences between windows to achieve both fine and coarse representations of gesture features, reducing computational cost while enhancing semantic information interaction between different windows. Finally, a gesture segmentation image is obtained through the overall encoding and decoding structure.

[0049] Experiments have proven that this invention outperforms other mainstream algorithms, producing smoother hand edges in the segmentation results, stronger ability to remove complex backgrounds, and greater robustness and effectiveness. This invention demonstrates high accuracy and robustness even under conditions of uneven lighting, complex background noise, and varied hand gestures. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of the present invention.

[0051] Figure 2 This is a framework diagram of the branch-cascaded Transformer module MBCT.

[0052] Figure 3 This is a framework diagram of the Multi-Window Self-Attention Block (MWSA).

[0053] Figure 4 This is a flowchart of the present invention.

[0054] Figure 5 This is a bar chart showing the test results of this invention and other methods.

[0055] Figure 6 This is a diagram illustrating the gesture segmentation effect of the present invention. Detailed Implementation

[0056] Example 1

[0057] like Figure 1 As shown, the gesture segmentation network device based on multi-branch cascaded Transformer of the present invention includes: a deep convolutional neural network (DCNN), a multi-branch cascaded Transformer module (MBCT), and a decoder. The encoding part consists of the deep convolutional neural network (DCNN) and the multi-branch cascaded Transformer (MBCT).

[0058] A Deep Convolutional Neural Network (DCNN) extracts features from the original gesture image to obtain an intermediate feature map. The DCNN is a feature extraction network consisting of one 7×7 convolution, one 3×3 convolution, and four residual groups. To reduce the number of feature channels and computational cost while incorporating more local detail information, a 1×1 convolution is added to the fourth residual group. After feature extraction by the DCNN, the input image yields an intermediate feature map, which is 1 / 8 the size of the original image.

[0059] like Figure 2As shown, the Multi-Branch Cascaded Transformer Module (MBCT) includes multiple cascaded Transformer branches. Each Transformer branch includes a Patch Partition layer, a Linear Embedding layer, and a Multi-Window Self-Attention Block (MWSA) connected in series. In gesture segmentation networks, the MBCT is designed to extract local details and global semantic information of gestures. This invention employs five cascaded Transformer branches.

[0060] Each Transformer branch first feeds the intermediate feature map into the Patch Partition layer. The Patch Partition layer uses convolution operations to downsample the feature map, dividing the intermediate feature map into non-overlapping patches, resulting in a patched feature map. The Patch Merging layer selects pixel values ​​at intervals of 2 in both the row and column directions of the feature map, concatenating them together as a single tensor. After stretching and unfolding, the channel dimension becomes four times the original. Then, a fully connected layer sets the channel dimension to a specific dimension. The patch size for each Transformer branch is different. In this invention, the patch sizes for each Transformer branch from top to bottom are 1, 2, 4, 8, and 16, respectively. The larger the patch, the smaller the feature map size obtained after passing through the Patch Partition layer, and the stronger the global modeling capability of the Multi-Window Self-Attention Block (MWSA) for that feature map. For example, when the patch size is 16, the resulting feature map size is 4×4, and the MWSA has a stronger global feature extraction capability for small-sized feature maps.

[0061] The Linear Embedding layer transforms the dimension of the feature map output by the Patch Partition layer into a specific dimension, represented as C, thereby facilitating the computation of Multi-Window Self-Attention Block (MWSA).

[0062] The Multi-Window Self-Attention Block (MWSA) receives a specific graph output from the Linear Embedding layer and extracts multi-scale features of gestures from it.

[0063] To expand the model's receptive field and capture more local gesture details, the Transformer branch passes the output of the Multi-Window Self-Attention Block (MWSA) through a Patch Merging layer before feeding it into the next Transformer branch. This output is then fused with the Linear Embedding layer output of the next Transformer branch before undergoing MWSA for feature extraction. The input to the final Transformer branch contains the gesture features extracted from the previous four branches. Specifically, each Transformer branch, except the last one, passes its output through a Patch Merging layer before feeding it into the next Transformer branch, where it is fused with the input of that branch for feature extraction.

[0064] The output of each Transformer branch is upsampled layer by layer from bottom to top, and the results are concatenated to obtain the overall feature map.

[0065] like Figure 3 As shown, the multi-window self-attention block MWSA includes three cascaded multi-head self-attention sub-modules MHSA. Each multi-head self-attention sub-module MHSA includes a cascaded LayerNorm (LN) layer, a multi-head self-attention module MSA, a residual connection (LN), and a 2-layer MLP with GELU nonlinearity.

[0066] The multi-window self-attention (MWSA) module can be described as follows:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] in Representing the lth th W l - Output of the MSA module, z l Representing the lth th The output of each MLP module.

[0074] To achieve both fine and coarse feature extraction at the same feature layer, this invention designs a multi-window self-attention block (MWSA). The window sizes of the three MHSA sub-modules are 3, 5, and 7, respectively. Utilizing a fixed-window self-attention mechanism reduces the computational cost of the multi-branch cascaded Transformer module MBCT, while the size differences between windows enhance semantic interaction between them, increasing the network's receptive field. MWSA is embedded within each Transformer branch of MBCT to extract gesture features. In MWSA, different window sizes are used to extract both fine and coarse gesture features at the same feature level, reducing model computation while enhancing semantic information interaction between different windows.

[0075] like Figure 1 As shown, after the deep convolutional neural network (DCNN) receives the original gesture image and performs calculations, one path of the calculation result is input into the multi-branch cascaded Transformer module (MBCT), and the other path is superimposed with the calculation result of the multi-branch cascaded Transformer module (MBCT) to obtain the superimposed gesture image, which is then input into the decoder.

[0076] The multi-branch cascaded Transformer module (MBCT) receives the gesture image from the deep convolutional neural network (DCNN), performs calculations, and superimposes the calculation results with the results of the deep convolutional neural network (DCNN) to obtain the superimposed gesture image, which is then input into the decoder.

[0077] The decoder receives the original gesture image and the superimposed gesture image, decodes them, and obtains the corresponding gesture segmentation image.

[0078] The decoder is used to restore the image to the same size as the original gesture image. The decoder includes a cascaded upsampling module, a 3×3 convolution module, an upsampling module, a 3×3 convolution module, an upsampling module, and a 1×1 convolution module.

[0079] During the decoding process, the outputs of the multi-branch cascaded Transformer module MBCT are superimposed to form the input feature map of the decoder. This input feature map undergoes a series of convolution and deconvolution operations to restore it to the same size as the original gesture image. To obtain more gesture details, the obtained feature map is concatenated with the original image and subjected to a 1×1 convolution to finally obtain the gesture segmentation result map.

[0080] The present invention also includes a gesture image acquisition module and a preprocessing module. The gesture image acquisition module is connected to the preprocessing module and is used to acquire a color gesture image. The preprocessing module is connected to both the gesture image acquisition module and the gesture segmentation network, and is used to perform rotation, flipping, and cropping operations on the color gesture image, and input the results into the gesture segmentation network.

[0081] Example 2

[0082] The gesture segmentation method based on multi-branch cascaded Transformer of the present invention includes the following steps:

[0083] A. The gesture segmentation method based on multi-branch cascaded Transformer is applied to the gesture segmentation network device based on multi-branch cascaded Transformer in Example 1.

[0084] B. Preprocess the gesture images to fix their size, so as to adapt to the input image size requirements of the gesture segmentation network.

[0085] Specifically, it includes gesture image acquisition operations and preprocessing operations. The gesture image acquisition operations are used to acquire original color gesture images, and the preprocessing operations are used to rotate, flip, and crop the original color gesture images, and input the results into the gesture segmentation network.

[0086] C. Construct the gesture segmentation network as in Example 1. Input the preprocessed image into the gesture segmentation network, and train the gesture segmentation network using a loss function and an optimizer to obtain a gesture segmentation network with optimal parameters. Figure 4 As shown.

[0087] The gesture segmentation network includes:

[0088] Deep convolutional neural networks (DCNNs) extract features from gesture images to obtain intermediate feature maps.

[0089] The Multi-Branch Cascaded Transformer Module (MBCT) consists of multiple cascaded Transformer branches. Each Transformer branch includes a Patch Partition layer, a Linear Embedding layer, and a multi-window self-attention block (MWSA) that are connected together.

[0090] A decoder used to restore the image to the same size as the original input image.

[0091] After receiving the original gesture image, the deep convolutional neural network (DCNN) performs calculations and inputs the results into the multi-branch cascaded Transformer module (MBCT) in one path. The results from the other path are superimposed with the results from the MBCT to obtain the superimposed gesture image, which is then input into the decoder.

[0092] The multi-branch cascaded Transformer module (MBCT) receives the gesture image from the deep convolutional neural network (DCNN), performs calculations, and superimposes the calculation results with the results of the deep convolutional neural network (DCNN) to obtain the superimposed gesture image, which is then input into the decoder.

[0093] The decoder receives the original gesture image and the superimposed gesture image, decodes them, and obtains the corresponding gesture segmentation image.

[0094] D. Inputting the gesture image into the trained gesture segmentation network will output the corresponding gesture segmentation image.

[0095] To further demonstrate the effectiveness of the proposed invention, we conducted gesture segmentation experiments on the OUTHANDS public dataset.

[0096] Table 1. Comparison of experimental results using different methods

[0097]

[0098] This method was compared with other methods, and the results are as follows: Figure 5 As shown in Table 1, the proposed gesture segmentation algorithm achieves 97.71%, 98.12%, 98.95%, and 98.85% accuracy on the four evaluation metrics of MIoU, PA, Re, and F-score, respectively. Furthermore, Figure 6 The results of this invention are shown in a comparison chart with other segmentation methods. The visualization results show that the proposed gesture segmentation network model performs better than other methods. This demonstrates that the gesture segmentation algorithm proposed in this paper exhibits strong robustness and effectiveness in the face of complex backgrounds, uneven lighting, and large gesture variations.

[0099] The references used in this invention specification are as follows:

[0100] [1]Zheng,Y.and P.Zheng.Hand segmentation based on improved gaussianmixture model.in 2015International Conference on Computer Science andApplications(CSA).2015.IEEE.

[0101] [2]Zhao,Y.,Z.Song,and X.Wu.Hand detection using multi-resolution HOGfeatures.in IEEE International Conference on Robotics and Biomimetics(ROBIO).2012.IEEE.

[0102] [3]Chen,Q.,N.D.Georganas,and E.M.Petriu,Hand gesture recognitionusing Haar-like features and a stochastic context-free grammar.IEEETransactions on Instrumentation and Measurement,2008.57(8):p.1562-1571.

[0103] [4]Dardas,N.H.and N.D.Georganas,Real-Time Hand Gesture Detection andRecognition Using Bag-of-Features and Support Vector Machine Techniques.IeeeTransactions on Instrumentation and Measurement,2011.60(11):p.3592-3607.

[0104] [5]Chuang,Y.L.,L.Chen,and G.C.Chen,Saliency-guided improvement forhand posture detection and recognition.Neurocomputing,2014.133:p.404-415.

[0105] [6]Mocanu,C.,G.Suciu,and Ieee.Automatic recognition of handgestures.in 11th International Conference on Electronics,Computers andArtificial Intelligence(ECAI).2019. Pitesti,ROMANIA.

[0106] [7]Tang,J.W.,et al.Position-Free Hand Gesture Recognition UsingSingle Shot MultiBoxDetector Based Neural Network.in 16th IEEE InternationalConference on Mechatronics and Automation(IEEE ICMA).2019.Tianjin,PEOPLES RCHINA.

[0107] [8]Al-Hammadi,M.,et al.,Deep Learning-Based Approach for SignLanguage GestureRecognition With Efficient Hand Gesture Representation.IeeeAccess,2020.8:p.192527-192542.

[0108] [9]Dadashzadeh,A.,et al.,HGR-Net:a fusion network for hand gesturesegmentation andrecognition.IET Comput.Vis.,2019.13(8):p.700-707.

[0109]

[10] Long,J.,E.Shelhamer,and T.Darrell.Fully convolutional networksfor semanticsegmentation.in Proceedings of the IEEE conference on computervision and pattern recognition. 2015.

[0110]

[11] Cui,Z.,et al.,Hand gesture segmentation against complexbackground based on improvedatrous spatial pyramid pooling.Journal of AmbientIntelligence and Humanized Computing,2022。

Claims

1. A gesture segmentation network device based on a multi-branch cascaded Transformer, characterized in that, Including: A deep convolutional neural network (DCNN) extracts features from the original gesture image to obtain an intermediate feature map. The Multi-Branch Cascaded Transformer module (MBCT) consists of multiple cascaded Transformer branches. Each Transformer branch includes a Patch Partition layer, a Linear Embedding layer, and a multi-window self-attention block (MWSA) connected in series. A decoder used to restore the image to the same size as the original gesture image; The deep convolutional neural network (DCNN) receives the original gesture image, performs calculations, and then inputs the calculation results into the multi-branch cascaded Transformer module (MBCT) in one direction. The result is then superimposed with the calculation result of the multi-branch cascaded Transformer module (MBCT) in another direction to obtain the superimposed gesture image, which is then input into the decoder. The multi-branch cascaded Transformer module MBCT receives the gesture image from the deep convolutional neural network (DCNN), performs calculations, and superimposes the calculation result with the calculation result of the deep convolutional neural network (DCNN) to obtain the superimposed gesture image, which is then input into the decoder. The decoder receives the original gesture image and the superimposed gesture image, decodes them, and obtains the corresponding gesture segmentation image. Each Transformer branch first feeds the intermediate feature map into the Patch Partition layer. The Patch Partition layer uses convolution operations to downsample the feature map, dividing the intermediate feature map into non-overlapping patches, and obtaining the feature map after patching. The Linear Embedding layer transforms the dimension of the feature map output by the Patch Partition layer into a specific dimension, denoted as C; The Multi-Window Self-Attention Block (MWSA) receives a specific graph output from the Linear Embedding layer and extracts multi-scale features of gestures from it. The Transformer branch takes the output of the Multi-Window Self-Attention Block (MWSA) and passes it through the Patch Merging layer before passing it to the next Transformer branch. After fusing it with the output of the Linear Embedding layer of the next Transformer branch, it enters the Multi-Window Self-Attention Block (MWSA) for feature extraction. The input of the last Transformer branch contains the gesture features extracted by the previous Transformer branches. The output of each Transformer branch is upsampled layer by layer from bottom to top, and the results are concatenated level by level to obtain the overall feature map.

2. The gesture segmentation network device based on multi-branch cascaded Transformer according to claim 1, characterized in that, The Patch Merging layer selects pixel values ​​at intervals of 2 in the row and column directions of the feature map, concatenates them together as a whole tensor, and after stretching and unfolding, the channel dimension becomes 4 times the original. Then, a fully connected layer sets the channel dimension to a specific dimension.

3. The gesture segmentation network device based on multi-branch cascaded Transformer according to claim 1, characterized in that, The Multi-Window Self-Attention Block (MWSA) includes three cascaded Multi-Head Self-Attention Sub-Modules (MHSA). Each MHSA includes a cascaded LayerNorm layer, a Multi-Head Self-Attention Module (MSA), a residual connection, and a 2-layer MLP with GELU nonlinearity. The multi-window self-attention (MWSA) module can be described as follows: , in represent The module's output, Representing the indivual The module's output.

4. The gesture segmentation network device based on multi-branch cascaded Transformer according to claim 3, characterized in that, The window sizes of the three Multi-Head Self-Attention Submodules (MHSA) are 3, 5, and 7, respectively.

5. The gesture segmentation network device based on multi-branch cascaded Transformer according to claim 1, characterized in that, The deep convolutional neural network (DCNN) is a feature extraction network consisting of a 7×7 convolution, a 3×3 convolution, and four residual groups. A 1×1 convolution is added to the fourth residual group. After feature extraction by the deep convolutional neural network (DCNN), the input image yields an intermediate feature map, which is 1 / 8 the size of the original image.

6. The gesture segmentation network device based on multi-branch cascaded Transformer according to claim 1, characterized in that, The decoder includes a serially connected upsampling module, a 3×3 convolution module, an upsampling module, a 3×3 convolution module, an upsampling module, and a 1×1 convolution module.

7. A gesture segmentation method based on multi-branch cascaded Transformer, characterized in that, Includes the following steps: A. The gesture segmentation method based on multi-branch cascaded Transformer is applied to the gesture segmentation network device based on multi-branch cascaded Transformer as described in claim 1; B. Preprocess the gesture image to fix its size; C. Input the preprocessed image into the gesture segmentation network device described in claim 1, and train the gesture segmentation network with the loss function and optimizer to obtain a gesture segmentation network with optimal parameters. By inputting the gesture image into the trained gesture segmentation network, the corresponding gesture segmentation image can be output.

8. The gesture segmentation method based on multi-branch cascaded Transformer according to claim 7, characterized in that, Step B includes a gesture image acquisition operation and a preprocessing operation. The gesture image acquisition operation is used to acquire a color original gesture image, and the preprocessing operation is used to rotate, flip and crop the color original gesture image, and input the result into the gesture segmentation network.

Images