Hand and object feature decoupling method based on semantic segmentation

Abstract

The invention relates to a hand and object feature decoupling method based on semantic segmentation, and the method comprises the steps: constructing a hand and object feature decoupling network which comprises a semantic segmentation network and a feature decoupling module; training the hand-object feature decoupling network based on the training data set to obtain a trained hand-object feature decoupling network; and inputting a to-be-processed hand-object interaction RGB into the trained hand-object feature decoupling network to obtain hand features and object features. Aiming at the problems of serious feature coupling, segmentation and decoupling target splitting and the like in object interaction analysis in the prior art, a hand-object feature decoupling network is designed, hand-object feature decoupling can be effectively realized, and hand features and object features are accurately obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a hand-object feature decoupling method based on semantic segmentation. BACKGROUND

[0002] In recent years, with the rapid development of computer vision and deep learning technology, image-based hand-object interaction analysis has been widely applied in human-computer interaction, augmented reality (AR) and robot operation fields. However, the existing technology still faces significant challenges in processing hand-object interaction scenes, especially in feature decoupling and independence guarantee.

[0003] Traditional methods mostly rely on a single segmentation model (such as FCN, Mask R-CNN) to distinguish hand and object regions. Generally, an encoder-decoder structure is used to achieve pixel-level segmentation, but it does not consider the complex coupling problem of hand and object contact areas. When the hand and object overlap closely in space (such as holding a cup, operating a tool), the segmentation boundary is blurred, causing the hand and object features to interfere with each other during feature extraction. Studies have shown that such coupled features can significantly reduce the accuracy of subsequent tasks.

[0004] Although the existing mainstream segmentation models (such as DeepLabv3+, PSPNet) perform well in multi-class target segmentation tasks, their original design is to improve global segmentation accuracy, not for the specific needs of hand-object interaction scenes. For example, DeepLabv3+ captures multi-scale context information through Atrous Spatial Pyramid Pooling (ASPP), but its feature fusion mechanism does not explicitly constrain the independence of hand and object features, which leads to a high degree of mixing of hand and object semantic information in the channel dimension during feature extraction, making it difficult to effectively separate them.

[0005] In terms of feature decoupling, some studies attempt to optimize feature independence through post-processing methods by cutting the feature maps of hand and object regions with hard masks. However, such methods only rely on the geometric information of the segmentation results, ignoring the deep statistical correlation between features. In addition, existing methods mostly use a phased optimization strategy, i.e., first optimizing the segmentation network, then fixing the segmentation results for feature decoupling, which leads to inconsistent training objectives between the two, resulting in suboptimal overall performance. SUMMARY

[0006] To overcome at least one deficiency in the prior art, the present application provides a hand-object feature decoupling method based on semantic segmentation.

[0007] In a first aspect, a hand-object feature decoupling method based on semantic segmentation is provided, comprising: The hand-object feature decoupling network is constructed, and the hand-object feature decoupling network comprises a semantic segmentation network and a feature decoupling module; the semantic segmentation network comprises an encoder and a decoder, the encoder comprises sequentially connected down-sampling modules and multi-scale feature fusion modules, and the decoder comprises a plurality of sequentially connected decoding units, each decoding unit comprising an up-sampling module and a double-flow decoupling convolution module; The hand-object feature decoupling network is trained based on a training data set to obtain a trained hand-object feature decoupling network; samples in the training data set are hand-object interaction RGB images; in the training process, the samples are input into the semantic segmentation network, the down-sampling modules perform multi-level down-sampling on the samples to obtain high-dimensional semantic features, the high-dimensional semantic features are input into the multi-scale feature fusion modules for hand feature enhancement, object feature enhancement and interaction feature mining, and after fusion processing, the fusion features are obtained; the fusion features are input into the decoder, the up-sampling modules in each decoding unit perform up-sampling operations on the input to obtain up-sampling results, the up-sampling results are input into the double-flow decoupling convolution modules for hand reconstruction and object reconstruction, and after fusion processing, double-flow decoupling results are obtained; the double-flow decoupling results output by the last decoding unit are subjected to convolution and Sigmoid activation to generate hand masks and object masks; the hand masks, the object masks and the samples are input into the feature decoupling module to obtain hand features and object features; The hand-object interaction RGB image to be processed is input into the trained hand-object feature decoupling network to obtain hand features and object features.

[0008] In one embodiment, the down-sampling module comprises a plurality of sequentially connected multi-target oriented convolution modules, each multi-target oriented convolution module comprising two parallel branches, namely a hand feature branch and an object feature branch; The hand feature branch adopts double continuous convolution layers, each convolution layer adopts 3×3 standard convolution for capturing fine motion features; The object feature branch adopts double continuous convolution layers, the first convolution layer adopts a dilated convolution with a dilation rate of 2, and the second convolution adopts 3×3 standard convolution for enhancing the perception of the macro structure of the object; The features output by the two branches are spliced and fused in the channel dimension, and then subjected to 2×2 maximum pooling for spatial down-sampling to obtain the output of the multi-target oriented convolution module.

[0009] In one embodiment, the multi-scale feature fusion module comprises three parallel branches, namely a hand feature enhancement branch, an object feature enhancement branch and an interaction feature mining branch; The hand feature enhancement branch adopts 3×3 standard convolution with a dilation rate of 1 to focus on the local details of the hand joints and posture; The object feature enhancement branch adopts 3×3 dilated convolution with a dilation rate of 3 to capture the overall structure and surface characteristics of the object; The interaction feature mining branch adopts a 3*3 cavity convolution with an expansion rate of 5 to model the context dependency relationship of the hand-object contact area. The outputs of the three branches are dynamically fused through an adaptive feature interaction gating mechanism to obtain fused features.

[0010] In one embodiment, the outputs of the three branches are dynamically fused through an adaptive feature interaction gating mechanism to obtain fused features, including: The output of the hand feature enhancement branch and the output of the object feature enhancement branch are subjected to convolution operation and Sigmoid function to obtain a mutual information weight matrix; According to the mutual information weight matrix, the outputs of the three branches are fused to obtain fused features.

[0011] In one embodiment, the feature decoupling module includes a hand feature decoupling branch and an object feature decoupling branch; The hand feature decoupling branch element-wise multiplies the hand mask with the sample and obtains the hand feature through channel attention; The object feature decoupling branch element-wise multiplies the object mask with the sample and obtains the object feature through channel attention.

[0012] In a second aspect, a hand-object feature decoupling device based on semantic segmentation is provided, including: A network construction module is configured to construct a hand-object feature decoupling network, which includes a semantic segmentation network and a feature decoupling module. The semantic segmentation network includes an encoder and a decoder. The encoder includes sequentially connected down-sampling modules and multi-scale feature fusion modules. The decoder includes a plurality of sequentially connected decoding units. Each decoding unit includes an up-sampling module and a dual-flow decoupling convolution module. A network training module is configured to train the hand-object feature decoupling network based on a training data set to obtain a trained hand-object feature decoupling network. The samples in the training data set are hand-object interactive RGB images. During the training process, the samples are input into the semantic segmentation network. The down-sampling modules perform multi-level down-sampling on the samples to obtain high-dimensional semantic features. The high-dimensional semantic features are input into the multi-scale feature fusion modules for hand feature enhancement, object feature enhancement, and interaction feature mining. After fusion processing, fused features are obtained. The fused features are input into the decoder. The up-sampling modules in each decoding unit perform up-sampling operation on the input to obtain up-sampling results, which are input into the dual-flow decoupling convolution modules for hand reconstruction and object reconstruction. After fusion processing, dual-flow decoupling results are obtained. The dual-flow decoupling results output by the last decoding unit are subjected to convolution and Sigmoid activation to generate hand masks and object masks. The hand masks, object masks, and samples are input into the feature decoupling module to obtain hand features and object features. The feature decoupling module is configured to input the hand-object interaction RGB image to be processed into the trained hand-object feature decoupling network to obtain hand features and object features.

[0013] Compared with the prior art, the present application has the following beneficial effects: The hand-object feature decoupling network is designed to solve the problems of serious feature coupling, segmentation and decoupling target fragmentation in the prior art object interaction analysis. In the network, a semantic segmentation network is used to strengthen hand features, strengthen object features, mine interaction features, and fuse the features to obtain fused features. Further, the fused features are reconstructed to generate hand masks and object masks. The feature decoupling module is used to perform spatial explicit cutting on the original feature map according to the hand masks and object masks, and force the non-target region features to be zero, thereby eliminating the coupling basis from the geometric level. The present application can effectively realize hand-object feature decoupling and accurately obtain hand features and object features. BRIEF DESCRIPTION OF DRAWINGS

[0014] The present application can be better understood by referring to the following description in conjunction with the accompanying drawings that form a part of this specification. In the drawings: Figure 1 A flowchart of a hand-object feature decoupling method based on semantic segmentation is shown; Figure 2 A schematic diagram of a semantic segmentation network is shown; Figure 3 A schematic diagram of a hand-object feature decoupling network training process is shown; Figure 4 A schematic diagram of a multi-target oriented convolution module is shown; Figure 5 A schematic diagram of a multi-scale feature fusion module is shown; Figure 6 A schematic diagram of a double-flow decoupling convolution module is shown; Figure 7 A schematic diagram of a feature decoupling module is shown. DETAILED DESCRIPTION

[0015] In the following, exemplary embodiments of the present application will be described with reference to the accompanying drawings. In the specification, not all the features of the actual embodiments are described for the sake of clarity and conciseness. However, it should be understood that many embodiment-specific decisions can be made in the process of developing any such actual embodiment to achieve the specific goals of the developer, and these decisions can vary from embodiment to embodiment.

[0016] It should be noted that, in order to avoid unnecessary details from obscuring the application, only the structures closely related to the application are shown in the drawings, and other details not closely related to the application are omitted.

[0017] It should be understood that the application is not limited to the described embodiments by virtue of the description below with reference to the drawings. In this context, the embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in an embodiment, if possible.

[0018] The embodiment of the application provides a hand-object feature decoupling method based on semantic segmentation, Figure 1 A flowchart of the hand-object feature decoupling method based on semantic segmentation is shown, referring to Figure 1 The method mainly comprises the following steps: Step S1, a hand-object feature decoupling network is constructed, the hand-object feature decoupling network comprises a semantic segmentation network and a feature decoupling module; the semantic segmentation network comprises an encoder and a decoder, the encoder comprises a down-sampling module and a multi-scale feature fusion module connected in sequence, and the decoder comprises a plurality of decoding units connected in sequence, each decoding unit comprising an up-sampling module and a double-flow decoupling convolution module. Figure 2 A schematic diagram of the semantic segmentation network is shown.

[0019] Step S2, the hand-object feature decoupling network is trained based on a training data set to obtain a trained hand-object feature decoupling network; the sample in the training data set is a hand-object interaction RGB image; in the training process, the sample is input to the semantic segmentation network, the down-sampling module performs multi-level down-sampling on the sample to obtain high-dimensional semantic features; the high-dimensional semantic features are input to the multi-scale feature fusion module for hand feature enhancement, object feature enhancement and interaction feature mining, and after fusion processing, the fusion features are obtained; the fusion features are input to the decoder, the up-sampling module in each decoding unit performs up-sampling operation on the input to obtain up-sampling results, the up-sampling results are input to the double-flow decoupling convolution module for hand reconstruction and object reconstruction, and after fusion processing, the double-flow decoupling results are obtained; the double-flow decoupling results output by the last decoding unit are subjected to convolution and Sigmoid activation to generate hand masks and object masks; the hand masks, object masks and samples are input to the feature decoupling module to obtain hand features and object features. Figure 3 A schematic diagram of the hand-object feature decoupling network training process is shown.

[0020] The training data set is obtained, and first, the obtained hand-object interaction RGB image is standardized and preprocessed, including: the pixel value is normalized by subtracting the mean vector and dividing by the variance vector through the channel by channel, to generate a standardized image that meets the network input requirements; then in the training stage, dynamically apply data enhancement techniques, including random affine transformation based on uniform sampling (rotation angle range ± 30°, translation offset controlled at ± 10% of the image size), color space disturbance, and occlusion simulation for the hand-object interaction area - generate a random rectangular occlusion area of 10%-20% of the image area with a probability of 30% and fill in the noise, and finally output the enhanced preprocessed image. Through the comprehensive enhancement strategy of geometric deformation, light disturbance and local information destruction, the adaptability of the model to complex scenes is significantly improved.

[0021] Step S3, input the hand-object interaction RGB image to be processed into the trained hand-object feature decoupling network to obtain hand features and object features.

[0022] In this embodiment, aiming at the problems of serious feature coupling, segmentation and decoupling target fragmentation in the prior art object interaction analysis, a hand-object feature decoupling network is designed. In the network, a semantic segmentation network is used to strengthen hand features, object features and interaction features of the extracted high-dimensional semantic features, and to obtain fused features by fusion processing. Further, hand reconstruction and object reconstruction are performed on the fused features to generate hand masks and object masks. A feature decoupling module is used to perform spatial explicit cutting on the original feature map according to the hand mask and the object mask, and to force the non-target region features to be zero, thereby eliminating the coupling basis from the geometric level to obtain hand features and object features. This embodiment can effectively realize hand-object feature decoupling and accurately obtain hand features and object features.

[0023] In one embodiment, the downsampling module includes a plurality of sequentially connected multi-target oriented convolution modules. Here, five multi-target oriented convolution modules can be provided, and each multi-target oriented convolution module includes two parallel branches, i.e., a hand feature branch and an object feature branch. Figure 4 A schematic diagram of a multi-target oriented convolution module is shown.

[0024] The hand feature branch adopts double continuous convolution layers, and each convolution layer adopts a 3x3 standard convolution for capturing fine motion features and extracting local texture features related to hand posture and joints (such as skin texture and knuckle curvature).

[0025] The object feature branch adopts double continuous convolution layers, and the first convolution layer adopts a dilated convolution with a dilation rate of 2, and the second convolution adopts a 3x3 standard convolution for enhancing the perception of object macrostructure and focusing on attribute features such as object contour and surface material (such as the geometric structure of the gripping area); The features of the two branch outputs are channel-dimensionally spliced and fused, and then spatially down-sampled through 2x2 max-pooling to obtain the output of the multi-target oriented convolution module.

[0026] In this embodiment, multiple multi-target oriented convolution modules are arranged, and the down-sampling process is sequentially performed from the first level to the fifth level, the output channel number is multiplied by one level (64→128→256→512→1024), and the spatial resolution is gradually compressed from 640x480 to 20x15, forming a hierarchical feature pyramid containing hand-object interaction semantics.

[0027] In one embodiment, Figure 5 A schematic diagram of the multi-scale feature fusion module is shown, see Figure 5 The multi-scale feature fusion module includes three parallel branches, namely a hand feature enhancement branch, an object feature enhancement branch, and an interaction feature mining branch. The hand feature enhancement branch adopts 3x3 standard convolution with a dilation rate of 1, focusing on local details of hand joints and postures. The object feature enhancement branch adopts 3x3 hollow convolution with a dilation rate of 3, capturing the overall structure and surface characteristics of the object. The interaction feature mining branch adopts 3x3 hollow convolution with a dilation rate of 5, modeling the context-dependent relationship of the hand-object contact area. The outputs of the three branches are dynamically fused through an adaptive feature interaction gating mechanism to obtain fused features, which specifically include: The output of the hand feature enhancement branch and the output of the object feature enhancement branch are convolved and passed through a Sigmoid function to obtain a mutual information weight matrix Who, Who = σ(Conv1x1([Fh, Fo])), Conv1x1 represents 1x1 convolution, Fh represents the output of the hand feature enhancement branch, Fo represents the output of the object feature enhancement branch, and σ represents the Sigmoid function.

[0028] According to the mutual information weight matrix Who, the outputs of the three branches are fused to obtain the fused features Ffusion, using the following formula: Ffusion = Who o (Fh + Fo) + (1 - Who) o Finter, where o represents element-wise multiplication, and Finter represents the output of the interaction feature mining branch.

[0029] The above process significantly improves the segmentation accuracy of the hand-object boundary area through differentiated convolution strategies and gating weight distribution, while maintaining the integrity of multi-scale semantics.

[0030] In one embodiment, Figure 6A schematic diagram of a dual-flow decoupled convolution module is shown. The decoder can include 5 decoding units connected in sequence, and the up-sampling module of each decoding unit is used to up-sample the input to restore spatial details. The up-sampling result is input to the dual-flow decoupled convolution module, which includes a hand reconstruction branch and an object reconstruction branch. The hand reconstruction branch extracts gesture joint motion trajectory features through 3x3 convolution, and the object reconstruction branch focuses on object contour geometric constraints. The two outputs are added after being compressed through 1x1 convolution, and finally the feature map output by the last decoding unit is compressed through 1x1 convolution and Sigmoid activation to generate a binary mask. At this time, the hand mask and the object mask are output through independent channels. = channel 0, = channel 1.

[0031] In one embodiment, Figure 7 A schematic diagram of a feature decoupling module is shown. The feature decoupling module includes a hand feature decoupling branch and an object feature decoupling branch. The hand feature decoupling branch element-wise multiplies the hand mask with the sample and obtains the hand feature through channel attention. The object feature decoupling branch element-wise multiplies the object mask with the sample and obtains the object feature through channel attention.

[0032] Here, the hand feature decoupling branch strengthens the hand joint area response, and the object feature decoupling branch focuses on the object structure feature. The hand feature is forced to zero in the non-hand area, and the object feature is the same. Explicit separation in feature space is achieved, and finally the hand feature and the object feature are output through channel attention.

[0033] Further, during network training, a joint loss function is designed by combining segmentation loss, mutual information minimization loss, adversarial loss and classification loss for back propagation to update network parameters at the same time. The joint loss function :

[0034] wherein, is the segmentation loss, which adopts a weighted cross-entropy loss, is the mutual information minimization loss, is the classification loss, is the adversarial loss. α, β, γ, δ are weight coefficients corresponding to each loss.

[0035] The training process includes: The network parameters are initialized, the encoder part loads the pre-trained weights on the COCO dataset to inherit the general visual representation ability, and the newly constructed decoder module and multi-scale fusion layer are randomly initialized using He normal distribution; the training process is divided into two key stages - the primary feature adaptation stage (rounds 1-50) freezes all weights of the encoder, and only optimizes the parameters of the decoder and the feature fusion module with a learning rate The global joint fine-tuning stage (rounds 51-150) unfreezes all network layers, uses the Adam optimizer with dynamic decay for end-to-end training, where the learning rate is exponentially decayed by 5% every 10 rounds, and the weight coefficient β (initial value 0.5) of the feature decoupling term in the joint loss function is dynamically adjusted according to the performance of the validation set. Finally, the model converges after 150 rounds of training.

[0036] To further verify the effectiveness of the method of the application, experimental analysis is performed. Table 1 shows the multi-target oriented convolution module ablation experiment using the H2O Interaction dataset. Table 2 shows the multi-scale feature fusion module ablation experiment using the H2O Interaction dataset.

[0037] Table 1

[0038] Table 2

[0039] According to Table 1, the network containing the complete multi-target oriented convolution module of the application has better mIoU and boundary F1-score. According to Table 2, the network containing the complete multi-scale feature fusion module of the application has better mIoU and boundary F1-score.

[0040] Using the same inventive concept as the hand-object feature decoupling method based on semantic segmentation, the embodiment also provides a hand-object feature decoupling device based on semantic segmentation corresponding thereto, comprising: A network construction module is configured to construct a hand-object feature decoupling network, the hand-object feature decoupling network comprising a semantic segmentation network and a feature decoupling module; the semantic segmentation network comprises an encoder and a decoder, the encoder comprising a down-sampling module and a multi-scale feature fusion module connected in sequence, and the decoder comprising a plurality of decoding units connected in sequence, each decoding unit comprising an up-sampling module and a dual-flow decoupling convolution module; The network training module is configured to train the hand-object feature decoupling network based on a training data set to obtain a trained hand-object feature decoupling network. The samples in the training data set are hand-object interaction RGB images. In the training process, the samples are input into the semantic segmentation network, and the multi-level down-sampling module performs multi-level down-sampling on the samples to obtain high-dimensional semantic features. The high-dimensional semantic features are input into the multi-scale feature fusion module for hand feature enhancement, object feature enhancement, and interaction feature mining, and after fusion processing, the fusion features are obtained. The fusion features are input into the decoder, the up-sampling module in each decoding unit performs up-sampling operation on the input, and the up-sampling result is input into the dual-flow decoupling convolution module for hand reconstruction and object reconstruction, and after fusion processing, the dual-flow decoupling result is obtained. The dual-flow decoupling result output by the last decoding unit is subjected to convolution and Sigmoid activation to generate a hand mask and an object mask. The hand mask, the object mask, and the sample are input into the feature decoupling module to obtain hand features and object features. The feature decoupling module is configured to input a hand-object interaction RGB image to be processed into the trained hand-object feature decoupling network to obtain hand features and object features.

[0041] The hand-object feature decoupling device based on semantic segmentation of the embodiment has the same inventive concept as the hand-object feature decoupling method based on semantic segmentation described above, and therefore the specific embodiments of the device can be seen in the embodiment part of the hand-object feature decoupling method based on semantic segmentation described above, and the technical effects thereof correspond to the technical effects of the method described above, which will not be described here.

[0042] The above is only various embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for hand-object feature decoupling based on semantic segmentation, characterized in that, The method comprises the following steps: constructing a hand-object feature decoupling network, the hand-object feature decoupling network comprising a semantic segmentation network and a feature decoupling module; the semantic segmentation network comprises an encoder and a decoder, the encoder comprising sequentially connected down-sampling modules and multi-scale feature fusion modules, and the decoder comprising a plurality of sequentially connected decoding units, each decoding unit comprising an up-sampling module and a double-flow decoupling convolution module; training the hand-object feature decoupling network based on a training data set to obtain a trained hand-object feature decoupling network; the samples in the training data set are hand-object interaction RGB images; in the training process, the samples are input into the semantic segmentation network, the down-sampling modules perform multi-level down-sampling on the samples to obtain high-dimensional semantic features, the high-dimensional semantic features are input into the multi-scale feature fusion modules for hand feature enhancement, object feature enhancement and interaction feature mining, and after fusion processing, fusion features are obtained, the fusion features are input into the decoder, the up-sampling modules in each decoding unit perform up-sampling operations on the inputs to obtain up-sampling results, the up-sampling results are input into the double-flow decoupling convolution modules for hand reconstruction and object reconstruction, and after fusion processing, double-flow decoupling results are obtained, and the double-flow decoupling results output by the last decoding unit are subjected to convolution and Sigmoid activation to generate hand masks and object masks, and the hand masks, the object masks and the samples are input into the feature decoupling module to obtain hand features and object features; inputting a hand-object interaction RGB image to be processed into the trained hand-object feature decoupling network to obtain hand features and object features.

2. The method of claim 1, wherein, The down-sampling module comprises a plurality of sequentially connected multi-target oriented convolution modules, each multi-target oriented convolution module comprising two parallel branches, namely a hand feature branch and an object feature branch; the hand feature branch adopts double continuous convolution layers, each convolution layer adopts 3×3 standard convolution for capturing fine motion features; the object feature branch adopts double continuous convolution layers, the first convolution layer adopts dilated convolution with a dilation rate of 2, and the second convolution adopts 3×3 standard convolution for enhancing the perception of the macro structure of the object; the features output by the two branches are spliced and fused in the channel dimension, and then subjected to 2×2 maximum pooling for spatial down-sampling to obtain the output of the multi-target oriented convolution module.

3. The method of claim 1, wherein, The multi-scale feature fusion module comprises three parallel branches, namely a hand feature enhancement branch, an object feature enhancement branch and an interaction feature mining branch; the hand feature enhancement branch adopts 3×3 standard convolution with a dilation rate of 1 to focus on the local details of hand joints and postures; the object feature enhancement branch adopts 3×3 dilated convolution with a dilation rate of 3 to capture the overall structure and surface characteristics of the object; the interaction feature mining branch adopts 3×3 dilated convolution with a dilation rate of 5 to model the context-dependent relationship of the hand-object contact area; the outputs of the three branches are dynamically fused through an adaptive feature interaction gating mechanism to obtain fusion features.

4. The method of claim 3, wherein, wherein the outputs of the three branches are dynamically fused through an adaptive feature interaction gating mechanism to obtain fusion features, comprising: The output of the hand feature enhancement branch and the output of the object feature enhancement branch are subjected to a convolution operation and a Sigmoid function to obtain a mutual information weight matrix; According to the mutual information weight matrix, the outputs of the three branches are fused to obtain fused features.

5. The method of claim 1, wherein, The feature decoupling module includes a hand feature decoupling branch and an object feature decoupling branch; The hand feature decoupling branch element-wise multiplies the hand mask and the sample, and obtains hand features through channel attention; The object feature decoupling branch element-wise multiplies the object mask and the sample, and obtains object features through channel attention.

6. A hand-object feature decoupling apparatus based on semantic segmentation, characterized in that, It comprises: A network construction module is configured to construct a hand-object feature decoupling network, which includes a semantic segmentation network and a feature decoupling module; The semantic segmentation network includes an encoder and a decoder, the encoder includes a down-sampling module and a multi-scale feature fusion module connected in sequence, and the decoder includes a plurality of decoding units connected in sequence, each decoding unit including an up-sampling module and a dual-flow decoupling convolution module; A network training module is configured to train the hand-object feature decoupling network based on a training data set to obtain a trained hand-object feature decoupling network; The samples in the training data set are hand-object interaction RGB images; During training, the sample is input to the semantic segmentation network, the down-sampling module performs multi-level down-sampling on the sample to obtain high-dimensional semantic features; the high-dimensional semantic features are input to the multi-scale feature fusion module for hand feature enhancement, object feature enhancement, and interaction feature mining, and after fusion processing, fused features are obtained; the fused features are input to the decoder, the up-sampling module in each decoding unit performs up-sampling operation on the input to obtain up-sampling results, which are input to the dual-flow decoupling convolution module for hand reconstruction and object reconstruction, and after fusion processing, dual-flow decoupling results are obtained; the dual-flow decoupling results output by the last decoding unit are subjected to convolution and Sigmoid activation to generate hand masks and object masks; the hand mask, the object mask, and the sample are input to the feature decoupling module to obtain hand features and object features; The feature decoupling module is configured to input a hand-object interaction RGB image to be processed into the trained hand-object feature decoupling network to obtain hand features and object features.

Images