A Human Body Analysis Method Based on Edge-Guided Diffusion

Abstract

This invention discloses a human body parsing method based on edge-guided diffusion. In human body parsing tasks, existing algorithms typically incorporate edge detection tasks based on multi-task learning to improve the model's segmentation at boundaries. However, due to semantic confusion between human body parts, these algorithms suffer from incomplete segmentation, where boundary segmentation is good but semantic inconsistencies within regions persist. This invention proposes a novel edge-guided diffusion mechanism that calculates a diffusion matrix through semantic edges, guiding semantic diffusion between pixels within a region, thereby enhancing semantic consistency within the region. Furthermore, this invention also enhances features through horizontal-vertical modules and channel aggregation modules. This invention improves the performance of human body parsing and promotes the development of related applications.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a human body parsing method based on edge-guided diffusion. Background Technology

[0002] Human body segmentation is a fine-grained semantic segmentation task that aims to break down human targets in an image into multiple intrinsic parts and predict corresponding semantic labels for each human body pixel, such as head, torso, and arms. Human body segmentation has significant research value and broad application prospects, and can be applied to many tasks such as virtual try-on, pedestrian re-identification, and clothing retrieval.

[0003] Edge information plays a crucial role in human parsing tasks. A common approach to utilizing edge information is to construct multi-task learning models, such as Context Embedding with EdgePerceiving (CE2P) and Self-Correction for Human Parsing (SCHP). These methods treat edge detection as an auxiliary task, generating edge-aware features and fusing them with segmentation features to enhance the boundaries between different body parts. While this method improves the model's segmentation performance at boundaries, it still results in semantic inconsistencies within regions and incomplete segmentation. This is because, as a fine-grained semantic segmentation task, human parsing involves many easily confused semantics, such as coat and top, skirt and hem. Edge detection only focuses on edge and non-edge regions, without specifically addressing these easily confused fine-grained semantics. This leads to accurate edge segmentation, but semantic inconsistencies within regions and incomplete segmentation.

[0004] To address the shortcomings of the aforementioned methods, some works have further explored the potential of leveraging predicted edges to enhance the human body parsing process. An intuitive approach is to propagate the semantic meaning of correctly classified pixels to neighboring pixels, guided by semantic edges. The key to this process is estimating the direction and distance of propagation for each pixel. Directly estimating the direction and distance of propagation for each pixel is quite difficult. To address this, SegFix proposed an alternative method that trains an additional network to predict edge maps and offset maps for each pixel, and optimizes the segmentation results for boundary regions through post-processing. However, this method still has limitations in handling semantic inconsistencies within regions, and accurately predicting offset maps remains a challenging task. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a human body parsing method based on edge-guided diffusion. In human body parsing tasks, existing algorithms typically incorporate edge detection tasks based on multi-task learning to improve the model's segmentation at boundaries. However, due to semantic confusion between human body parts, these algorithms suffer from incomplete segmentation, where boundary segmentation is good but semantic inconsistencies within regions persist. This invention proposes a novel edge-guided diffusion mechanism that calculates a diffusion matrix through semantic edges, guiding semantic diffusion between pixels within a region, thereby enhancing semantic consistency within the region. Furthermore, this invention also enhances features through horizontal-vertical modules and channel aggregation modules. This invention improves the performance of human body parsing and promotes the development of related applications.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step 1: Convert the input image I∈R 3×H×W The input is fed into the backbone network, which then extracts four features at different resolutions. Where m∈{4, 3, 2, 1}, C4=256, H4=1 / 4H, W4=1 / 4W, C m =1 / 2C m-1 H m =2H m-1 W m =2W m-1 ;

[0008] Step 2: Input the features {F4, F3, F2} into the edge module to predict and obtain the semantic edge map E∈R. 2×H×W The edge module uses CE2P, and the features {F4, F3, F2} are each reduced to 256 channels through a 1×1 convolution to obtain the edge features. After that, all E m Edge maps of different scales are generated through 3×3 convolution. During this process, the resolution of all features remains unchanged, and the number of channels of the features becomes 2 after convolution. E2 and E3 are upsampled to the same resolution as E1 and concatenated along the channels. Finally, a 1×1 convolution is used for the final prediction to generate the semantic edge map E.

[0009] Step 3: Feature F1 is input into the horizontal-vertical aggregation module for aggregation and enhancement to obtain feature F. hv Feature F hv After fusing with edge features e, segmentation prediction is performed using 1×1 convolution to generate the original coarse prediction P0∈R. N×H×W N represents the total number of categories; first, downsample the edge feature e to the level of feature F. hv Merge them only after they are the same size;

[0010] Step 4: Obtain the original coarse prediction P0∈R N×H×W Then, it is refined through an edge-guided diffusion mechanism; it includes 4 edge-guided diffusions. In the m-th diffusion, the prediction P from the previous stage is used. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m ;

[0011] Step 4-1: Before the first diffusion, P0 and the semantic edge map E are interpolated to ensure their spatial resolution matches that of F1. Then, the semantic edge map E is convolved with an L×L layer to generate edge-aware features E′, and P0 is projected to the same channel dimension as E′. During diffusion, F1 and E′ are concatenated along the channels and further fused using a 1×1 convolution. The fusion result is used to generate the diffusion weight matrix. Therefore, the diffusion operation in the first stage is as follows:

[0012]

[0013] Step 4-2: The semantic edge map E and the backbone feature F1 also contain detailed information missing from the previous prediction results; therefore, when performing the diffusion operation, E and F1 are also used to refine the diffusion results. To supplement and enhance; specifically, after E and F1 are merged, they are added element by element to... To supplement; the output of the first stage is as follows:

[0014]

[0015] Where CA(·) represents the channel aggregation module, which is implemented based on channel attention; F represents the input features, which are first generated through 1×1 convolution and 3×3 depthwise convolution respectively. Transform the dimensions to The attention level is then calculated as shown in the following formula:

[0016] Q = DWConv 3×3 (Conv 1×1 (F))

[0017] K = DWConv 3×3 (Conv 1×1 (F))

[0018] V = DWConv 3×3 (Conv 1×1 (F))

[0019]

[0020] Where α is a learnable scaling parameter; The input is fed into a feedforward network with residual connections to obtain the output of the channel aggregation module; finally, it is adjusted by convolution. The channel yields the output of the first stage.

[0021] And so on, using the prediction P from the previous stage. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m A total of four edge-guided diffusion processes were carried out.

[0022] Preferably, the calculation process of the horizontal-vertical aggregation module is as follows: Feature F1 is first linearly projected, and the number of channels becomes C. k It is then divided in two along the channel to obtain the characteristic and characteristics These are used for feature aggregation in the horizontal and vertical directions, respectively; in the horizontal direction, feature F... h Divide the data into H windows, each with a width of 1, meaning each row is considered a window, denoted as H. Self-attention is calculated within each window, so the calculation results of each window are concatenated along the channels to obtain the horizontally enhanced features. Furthermore, when calculating attention within each window, positional information is captured through a 3×3 depthwise convolution, as shown in the following equation:

[0023]

[0024]

[0025] F hv =FFN(Linear([F′)) h F′ v ]))

[0026] MHSA represents bullish self-attention, and DWConv 3×3 The depthwise convolution is 3×3; similarly, the vertically enhanced features are calculated. The two are concatenated along the channel and projected back to their original dimensions. Finally, they are input into a feedforward network with residual connections to obtain the output F of the horizontal-vertical enhancement module. hv .

[0027] The beneficial effects of this invention are as follows:

[0028] 1. A novel diffusion scheme is proposed, utilizing edge information to enhance the performance of human body parsing. This scheme not only strengthens the segmentation boundaries but also effectively alleviates the problem of semantic inconsistency within the segmented object, thereby obtaining more accurate segmentation results.

[0029] 2. A novel edge-guided diffusion network for human body segmentation is proposed, comprising a multi-stage edge-guided diffusion and horizontal-vertical aggregation module. This network effectively utilizes the spatial correlation features between different parts of the human body and refines the segmentation results from coarse to fine using semantic edge information.

[0030] 3. This invention achieves excellent performance in human body parsing tasks, and shows significant performance improvement on multiple public human body parsing datasets (LIP, Pasca-Person-Part, etc.). Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a human body analysis method based on edge-guided diffusion.

[0032] Figure 2 This is a schematic diagram illustrating the core idea of ​​the edge-guided diffusion mechanism.

[0033] Figure 3 This is a schematic diagram illustrating the specific process of the edge-guided diffusion mechanism. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] This invention proposes a human body analysis method based on edge-guided diffusion, such as... Figure 1 As shown. Its core is the edge-guided diffusion mechanism. Unlike predicting the diffusion direction and distance of each pixel individually, the edge-guided diffusion mechanism employs a predefined strategy. Specifically, each pixel has a tendency to diffuse towards the surrounding area, and the direction and distance of diffusion are fixed. The purpose of the edge-guided diffusion mechanism is to diffuse the semantics of surrounding pixels towards the center under the guidance of semantic edges, enhancing the semantic consistency of the internal regions of human body parts, thereby improving human body resolution performance. Thus, the problem of predicting the diffusion direction and distance can be simplified to calculating a diffusion weight towards the center pixel for all pixels within a local window. The diffusion weight of all pixels within a local window can be represented as the diffusion matrix of the center pixel, as shown... Figure 2 As shown.

[0036] Furthermore, this invention proposes a horizontal-vertical aggregation module to mine and utilize the feature distribution of human images in the horizontal and vertical directions, thereby enhancing the performance of human image parsing. This invention also proposes a channel aggregation module to enhance important semantic channels and suppress harmful noise.

[0037] 1. Edge-guided diffusion mechanism;

[0038] The specific calculation process for the edge-guided diffusion mechanism is shown in the attached instruction manual. Figure 3 As shown. Given an input image I∈R 3×H×W By using an existing multi-task model that combines edge detection and human body parsing, the original segmentation prediction P∈R is generated. N×H×W and semantic edge graph E∈R 2×H×W Where H×W represents the spatial dimension and N is the total number of semantic categories. The edge-guided diffusion mechanism takes the segmentation feature P and the semantic edge map E as input. Its purpose is to calculate the diffusion matrix for each pixel based on E and apply these diffusion matrices to P to generate a refined segmentation result. This method can effectively propagate the corrected semantic categories to misclassified regions, and the entire process is shown in the following formula:

[0039]

[0040] Specifically, for the pixel at position (i, j), assuming all pixels within the window centered at position (i, j), denoted by N(i, j), exhibit a tendency to diffuse towards (i, j). By weighting the diffusion with different weights, the degree of influence of different pixels on (i, j) can be controlled. The diffusion process of all pixels N(i, j) within the window towards pixel (i, j) is shown in the following equation:

[0041]

[0042] in, P represents the segmentation prediction result of pixel (i, j) after diffusion. (m，n) This represents the original segmentation prediction result for the pixel at position (m, n), where position (m, n) is located within a local window centered at (i, j). The diffusion weight represents the degree of influence of pixel (m, n) on pixel (i, j) in the final prediction.

[0043] The key to the above diffusion mechanism is how to determine the diffusion weight. The value of . In the edge map, if there is no edge between two pixels, it indicates that these two pixels are likely to belong to the same semantic region, that is, to the same semantic category. In this case, the semantic weight . The value of should be relatively large to facilitate the diffusion of information from pixel (m, n) to pixel (i, j). Conversely, if there is an edge between two pixels, these two pixels are likely to belong to different categories, and the semantic weight... The value of should be relatively small to reduce the influence of pixel (m, n) on pixel (i, j). Therefore, it is feasible to generate a diffusion matrix using a semantic edge map, and edge information can also improve the semantic consistency of the segmentation results and clarify the segmentation boundaries during human body parsing.

[0044] To obtain the diffusion weights from the semantic edge map E, an edge-aware feature E′ needs to be obtained by processing E through a projection layer. This projection layer is implemented using an L×L convolutional layer to ensure that all pixels N(i,j) within the window are associated with pixel (i,j), and L is set to 7. In the calculation... Previously, it was necessary to first process the feature E′ through two modulation layers. (i，j) and feature E′ (m，n) These modulation layers, implemented using linear projection layers, perform necessary transformations on the features and extract useful information. In summary, the process of calculating the diffusion weights is shown in the following equation:

[0045] E′=Conv L×L (E)

[0046]

[0047] in and This is a linear projection layer. Softmax represents the softmax normalization operation. Within a window of size L×L centered at (i, j), the diffusion weights of all pixels towards (i, j) are transformed into a diffusion matrix D. (i，j) ∈R L×L After calculating the diffusion matrix for each pixel, it is used to diffuse the original segmentation feature P. Inspired by multi-head self-attention, multiple sets of diffusion matrices can be calculated to make the diffusion process more stable and reliable.

[0048] 2. Horizontal-vertical aggregation module;

[0049] For human body analysis tasks, the input human images typically exhibit strong correlations in both the horizontal and vertical directions, and the pixels in each row (or column) can be aggregated together. In the horizontal-vertical aggregation module, feature F is first linearly projected, and the number of channels becomes C. k It is then divided in two along the channel to obtain the characteristic and characteristics These are used for feature aggregation in the horizontal and vertical directions, respectively. Taking the horizontal direction as an example, feature F... h Divide the data into H windows, each with a width of 1, meaning each row is considered a window, denoted as H. Self-attention is calculated within each window, so the calculation results of each window are concatenated along the channels to obtain the horizontally enhanced features. Furthermore, when calculating attention within each window, positional information is captured through a 3×3 depthwise convolution, as shown in the following equation:

[0050]

[0051]

[0052] F hv =FFN(Linear([F′)) h F′ v ]))

[0053] MHSA represents bullish self-attention, and DWConv 3×3 This is a 3×3 depthwise convolution. Similarly, the vertically enhanced features can be calculated. The two are concatenated along the channel and projected back to their original dimensions. Finally, they are input into a feedforward network with residual connections to obtain the output F of the horizontal-vertical enhancement module. hv .

[0054] 3. Channel aggregation module;

[0055] Considering the advantages of Transformer in global modeling, the channel aggregation module is implemented based on channel attention. Let F represent the input features, which are first generated through 1×1 convolutions and 3×3 depthwise convolutions. Transform the dimensions to The attention level is then calculated as shown in the following formula:

[0056] Q = DWConv 3×3 (Conv 1×1 (F))

[0057] K = DWConv 3×3 (Conv 1×1 (F))

[0058] V = DWConv 3×3 (Conv 1×1 (F))

[0059]

[0060] Where α is a learnable scaling parameter. The input is fed into a feedforward network with residual connections to obtain the output of the channel aggregation module. Finally, it is adjusted by convolution. The channel is used to obtain the output of the channel aggregation module.

[0061] Example:

[0062] The human body segmentation method proposed in this invention uses multi-stage edge-guided diffusion to predict segmentation results from coarse to fine. For ease of explanation, the specific implementation method will be described below using a single image and C semantic categories as an example.

[0063] Step 1: Convert the input image I∈R 3×H×W The input is fed into the backbone network, and four features at different resolutions can be extracted from it. Where m∈{4, 3, 2, 1}, C4=256, H4=1 / 4H, W4=1 / 4W, C m =1 / 2C m-1 H m =2H m-1 W m =2W m-1 .

[0064] Step 2: Input the features {F4, F3, F2} into the edge module to predict and obtain the semantic edge map E∈R. 2×H×W The edge module is consistent with CE2P. Features {F4, F3, F2} are each reduced to 256 channels through a 1×1 convolution to obtain edge features. After that, all E m Edge maps of different scales are generated through 3×3 convolutions. During this process, the resolution of all features remains unchanged, and the number of channels for each feature becomes 2 after convolution. E2 and E3 are upsampled to the same resolution as E1 and concatenated along the channels. Finally, a 1×1 convolution is used for the final prediction to generate the semantic edge map E.

[0065] Step 3: Feature F1 is input into the horizontal-vertical aggregation module for aggregation and enhancement to obtain feature F. hv Feature F hv After fusing with edge features e, segmentation prediction is performed using 1×1 convolution to generate the original coarse prediction P0∈R. N×H×W N represents the total number of categories. To reduce parameters and computational cost, the edge feature e is first downsampled to the level of feature F. hv They are then merged after they are of the same size.

[0066] The calculation process of the horizontal-vertical aggregation module is as follows: Feature F1 is first linearly projected, and the number of channels becomes C. k It is then divided in two along the channel to obtain the characteristic and characteristics These are used for feature aggregation in the horizontal and vertical directions, respectively. Taking the horizontal direction as an example, feature F... h Divide the data into H windows, each with a width of 1, meaning each row is considered a window, denoted as H. Self-attention is calculated within each window, so the calculation results of each window are concatenated along the channels to obtain the horizontally enhanced features. Furthermore, when calculating attention within each window, positional information is captured through a 3×3 depthwise convolution, as shown in the following equation.

[0067]

[0068]

[0069] F hv =FFN(Linear([F′)) h F′ v ]))

[0070] MHSA represents bullish self-attention, and DWConv 3×3 This is a 3×3 depthwise convolution. Similarly, the vertically enhanced features can be calculated. The two are concatenated along the channel and projected back to their original dimensions. Finally, they are input into a feedforward network with residual connections to obtain the output F of the horizontal-vertical enhancement module. hv .

[0071] Step 4: Obtain the original coarse prediction P0∈R N×H×W Subsequently, it is refined through an edge-guided diffusion mechanism. This invention includes four edge-guided diffusion processes. During the m-th diffusion process, the predicted P from the previous stage is used. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m .

[0072] Step 4-1: Before the first diffusion, P0 and E are interpolated to ensure their spatial resolution matches that of F1. Then, E is convolved with an L×L layer to generate an edge-aware feature E′, and P0 is projected onto the same channel dimension as E′. Since feature F1 contains rich spatial information, during diffusion, F1 and E′ are concatenated along the channels and further fused using a 1×1 convolution. The fusion result is used to generate the diffusion weight matrix. Therefore, the diffusion operation in the first stage is as follows:

[0073]

[0074] Step 4-2: The semantic edge E and backbone feature F1 also contain detailed information missing from the previous prediction results. Therefore, E and F1 are also used to refine the diffusion results during the diffusion operation. This will be supplemented and enhanced. Specifically, after E and F1 are merged, they will be supplemented element by element. Further details are needed. However, directly aggregating these features into high-level semantic features may introduce harmful noise. Therefore, a channel aggregation module is required to enhance the semantic distribution by calculating the covariance between different channels to strengthen important channels. Thus, the output of the first stage is as follows:

[0075]

[0076] Where CA(·) represents the channel aggregation module. The channel aggregation module is implemented based on channel attention. Let F represent the input features, which are first generated through 1×1 convolution and 3×3 depthwise convolution respectively. Transform the dimensions to The attention level is then calculated as shown in the following formula:

[0077] Q = DWConv 3×3 (Conv 1×1 (F))

[0078] K = DWConv 3×3 (Conv 1×1 (F))

[0079] V = DWConv 3×3 (Conv 1×1 (F))

[0080]

[0081] Where α is a learnable scaling parameter. The input is fed into a feedforward network with residual connections to obtain the output of the channel aggregation module. Finally, it is adjusted by convolution. The channel yields the output of the first stage.

[0082] And so on, using the prediction P from the previous stage. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m A total of four edge-guided diffusion processes were performed.

Claims

1. A human body analysis method based on edge-guided diffusion, characterized in that, Includes the following steps: Step 1: Convert the input image I∈R 3×H×W The input is fed into the backbone network, which then extracts four features at different resolutions. Where m∈{4,3,2,1}, C4=256, H4=1 / 4H, W4=1 / 4W, C m =1 / 2C m-1 H m =2H m-1 W m =2W m-1 ; Step 2: Input the features {F4, F3, F2} into the edge module to predict and obtain the semantic edge map E∈R. 2×H×W The edge module uses CE2P, and the features {F4, F3, F2} are each reduced to 256 channels through a 1×1 convolution to obtain the edge features. After that, all E m Edge maps of different scales are generated through 3×3 convolution. During this process, the resolution of all features remains unchanged, and the number of channels of the features becomes 2 after convolution. E2 and E3 are upsampled to the same resolution as E1 and concatenated along the channels. Finally, a 1×1 convolution is used for the final prediction to generate the semantic edge map E. Step 3: Feature F1 is input into the horizontal-vertical aggregation module for aggregation and enhancement to obtain feature F. hv Feature F hv After fusing with edge features e, segmentation prediction is performed using 1×1 convolution to generate the original coarse prediction P0∈R. N×H×W N represents the total number of categories; first, downsample the edge feature e to the level of feature F. hv Merge them only after they are the same size; Step 4: Obtain the original coarse prediction P0∈R N×H×W Then, it is refined through an edge-guided diffusion mechanism; it includes 4 edge-guided diffusions. In the m-th diffusion, the prediction P from the previous stage is used. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m ; Step 4-1: Before the first diffusion, P0 and the semantic edge map E are interpolated to ensure their spatial resolution matches that of F1. Then, the semantic edge map E is convolved with an L×L layer to generate edge-aware features E′, and P0 is projected to the same channel dimension as E′. During diffusion, F1 and E′ are concatenated along the channels and further fused using a 1×1 convolution. The fusion result is used to generate the diffusion weight matrix. Therefore, the diffusion operation in the first stage is as follows: Step 4-2: The semantic edge map E and the backbone feature F1 also contain detailed information missing from the previous prediction results; therefore, when performing the diffusion operation, E and F1 are also used to refine the diffusion results. To supplement and enhance; specifically, after E and F1 are merged, they are added element by element to... To supplement; the output of the first stage is as follows: Where CA(·) represents the channel aggregation module, which is implemented based on channel attention; F represents the input features, which are first generated through 1×1 convolution and 3×3 depthwise convolution respectively. Transform the dimensions to The attention level is then calculated as shown in the following formula: Q=DWConv 3×3 (Conv 1×1 (F)) K=DWConv 3×3 (Conv 1×1 (F)) V=DWConv 3×3 (Conv 1×1 (F)) Where α is a learnable scaling parameter; The input is fed into a feedforward network with residual connections to obtain the output of the channel aggregation module; finally, it is adjusted by convolution. The channel yields the output of the first stage. And so on, using the prediction P from the previous stage. m-1 Semantic edge graph E and backbone network features F m Generate refined prediction P m A total of four edge-guided diffusion processes were carried out.

2. The human body analysis method based on edge-guided diffusion according to claim 1, characterized in that, The calculation process of the horizontal-vertical aggregation module is as follows: Feature F1 is first linearly projected, and the number of channels becomes C. k It is then divided in two along the channel to obtain the characteristic and characteristics These are used for feature aggregation in the horizontal and vertical directions, respectively; in the horizontal direction, feature F... h Divide the data into H windows, each with a width of 1, meaning each row is considered a window, denoted as H. Self-attention is calculated within each window, so the calculation results of the windows are concatenated along the channels to obtain the horizontally enhanced features. Furthermore, when calculating attention within each window, positional information is captured through a 3×3 depthwise convolution, as shown in the following equation: F hv =FFN(Linear([F′ h ,F′ v ])) MHSA represents bullish self-attention, and DWConv 3×3 The depthwise convolution is 3×3; similarly, the vertically enhanced features are calculated. The two are concatenated along the channel and projected back to their original dimensions. Finally, they are input into a feedforward network with residual connections to obtain the output F of the horizontal-vertical enhancement module. hv .

Images