Weakly supervised human parsing method and system based on hierarchical semantic graph neural network

Abstract

The application discloses a weakly supervised human parsing method and system based on a hierarchical semantic graph neural network, relates to the field of image processing, and sets a human target frame and initial labeled semantic pixel points on an original image; the human target frame and the initial labeled semantic pixel points are taken as weak supervision information, and initial pixel-level pseudo labels are generated in combination with regional semantic unsupervised image segmentation; the initial pixel-level pseudo labels are screened based on a human parsing preliminary prediction result, high-confidence pseudo labels are generated, an initial dense graph is constructed, a dynamic hierarchical semantic graph is generated in combination with the initial dense graph and dynamic graph pooling, and a hierarchical human parsing result is generated by decoding. The application preliminarily acquires human parsing pseudo labels by using unsupervised image segmentation and a small amount of labeled semantic pixel points, greatly reduces the labeling cost, and the proposed hierarchical semantic graph neural network can extract accurate human semantic structure features under the supervision of pseudo labels containing noise, and generate a high-performance human parsing model.

Description

Technical Field

[0001] This application relates to the technical field of image processing, specifically to a weakly supervised human body parsing method and system based on hierarchical semantic graph neural networks. Background Technology

[0002] Currently, human body parsing, through refined pixel-level semantic segmentation of various body parts and clothing, enables a comprehensive understanding of human behavior in visual scenes, and has gained increasing attention from researchers in recent years. Utilizing prior knowledge of human body structure to encode inherent semantic structures effectively improves the accuracy of human body parsing models. Existing techniques include multi-level semantic segmentation of body parts, combining the powerful learning capabilities of neural networks and the robust representational capabilities of graph models to effectively mine the internal structure of the human body, acquire semantic features from all perspectives, and improve the accuracy of human body parsing. Furthermore, using pose estimation annotations as weakly supervised information to solve human body parsing tasks significantly reduces annotation costs. A part-adaptive module is designed to refine the initial limb segmentation results using pose estimation and target segmentation. Pose estimation performs location localization while enhancing affinity representation of specific parts, and target segmentation alleviates discrete noise pixels, generating clear target regions. In addition, there is instance-aware human body parsing, which utilizes multi-granular structural information of the human body to alleviate the difficulty of human instance segmentation. By explicitly associating dense human semantics with sparse keypoints, a dense-to-sparse mapping field is constructed, gradually enhancing the robustness of network features.

[0003] Graph neural networks have proven effective at modeling graph-structured data. Convolutional operations on graph-structured data have been shown to improve processing performance and are therefore widely used. However, there is still significant room for improvement in applying pooling operations to graph-structured data. Graph pooling is limited by irregular graph data structures and cannot be easily manipulated, resulting in relatively little research on graph pooling operations. Nevertheless, pooling mechanisms are crucial for improving the ability of graph neural network models to extract hierarchical feature representations and for achieving graph model compression, and have attracted increasing attention from researchers in recent years. A sampling-based graph pooling method designs a top-K node selection strategy. By selecting some important nodes to generate a coarser-grained graph to replace the original graph as the input graph for the next convolutional layer, the goal of graph pooling is achieved.

[0004] The problems in existing parsing methods include: (1) High-performance human parsing networks use a large number of pixel-level labeled images as supervision information to achieve accurate human parsing prediction, but pixel-level labeling requires a lot of human resources, which is too costly, and there will also be noise interference such as incorrect labeling.

[0005] (2) Existing sampling-based graph pooling methods usually generate coarser-grained graphs from some important nodes to replace the original graph as the input graph for the next convolutional layer. However, the coarsened graphs generated from important nodes in the sampling graph may not retain key local structural information and may destroy the integrity of the graph topology, thereby affecting the transmission of information in subsequent layers. Summary of the Invention

[0006] To address at least one of the problems mentioned in the background art, this application provides a weakly supervised human body parsing method and system based on hierarchical semantic graph neural networks. For weakly supervised human body parsing, a semantic segmentation strategy is proposed. This strategy utilizes unsupervised image segmentation and semantic pixels from human body bounding boxes and annotations to obtain initial fine-grained human body parsing pseudo-labels, significantly reducing annotation costs. A reliable pixel selection strategy is designed to reliably sample the generated initial pseudo-labels, selecting high-confidence pseudo-labels to provide accurate semantic structure features for subsequent model learning. A dynamic graph pooling module is designed for fine-grained human body parsing. This module generates a dynamic semantic graph by utilizing multi-label prediction probability values ​​and a predefined static semantic graph. Multiple dynamic graph pooling operations generate a dynamic hierarchical semantic graph from the fine-grained semantic graph to the coarse-grained semantic graph. Decoding this dynamic hierarchical semantic graph achieves accurate fine-grained human body parsing.

[0007] The specific technical solutions provided in this application are as follows:

[0008] Firstly, a weakly supervised human body parsing method based on a hierarchical semantic graph neural network is provided, the method comprising:

[0009] Receive the original image, and set the human target bounding box and initial semantic annotation pixels on the original image;

[0010] Human target bounding boxes and initial labeled semantic pixels are used as weakly supervised information, and combined with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0011] The initial pixel-level pseudo-labels are used to obtain preliminary prediction results of human body parsing, and the initial pixel-level pseudo-labels are filtered using the preliminary prediction results of human body parsing to generate high-confidence pseudo-labels.

[0012] An initial dense graph is constructed based on the high-confidence pseudo-labels. Dynamic graph pooling is then performed on the initial dense graph to generate a dynamic hierarchical semantic graph. Finally, the dynamic hierarchical semantic graph is decoded to generate a hierarchical human body parsing result.

[0013] In one specific embodiment, the human target bounding box and initially labeled semantic pixels are used as weakly supervised information, and initial pixel-level pseudo-labels are generated by combining region semantic unsupervised image segmentation. Specifically, this includes:

[0014] Based on the human target bounding box, the original image is segmented foreground to obtain a foreground segmentation image;

[0015] Unsupervised image segmentation is performed on the foreground segmentation image to obtain an unsupervised segmentation image;

[0016] The image segmentation blocks in the unsupervised segmentation image are semantically represented using initial labeled semantic pixels to generate initial pixel-level pseudo-labels.

[0017] In one specific embodiment, the initial pixel-level pseudo-labels are filtered using the preliminary prediction results of human body analysis to generate high-confidence pseudo-labels, specifically including:

[0018] A backbone feature extraction network is trained using the initial pixel-level pseudo-labels as network supervision information, and outputs high-level semantic features and shallow features.

[0019] The high-level semantic features and the shallow features are decoded by the decoder to generate preliminary prediction results for human body analysis;

[0020] Based on the preliminary prediction results of the human body analysis, reliable pixel sampling is performed on the initial pixel-level pseudo-labels to obtain the high-confidence pseudo-labels.

[0021] In one specific embodiment, the method further includes: segmenting the original image according to human body parts and / or clothing and connecting them to generate a first-level static semantic map;

[0022] The original image is coarsely segmented based on human body structure and then connected to generate a second-level static semantic map.

[0023] The entire human body is treated as a node to generate the third-level static semantic graph.

[0024] In one specific embodiment, the dynamic hierarchical semantic graph includes at least a first-level dynamic semantic graph, a second-level dynamic semantic graph, and a third-level dynamic semantic graph. The dynamic hierarchical semantic graph is generated by dynamic graph pooling based on an initial dense graph, specifically including:

[0025] Obtain the feature vector corresponding to the high-confidence pseudo-label as the initial dense graph node, and connect the initial dense graph node to construct the initial dense graph;

[0026] Obtain the multi-label prediction probability value of the original image, and dynamically adjust the adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value to obtain the first-level dynamic adjacency matrix.

[0027] The initial dense graph and the first-level dynamic adjacency matrix are combined to perform the first dynamic graph pooling to generate the first-level dynamic semantic graph;

[0028] The adjacency matrix of the second-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix.

[0029] A second dynamic graph pooling is performed based on the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate the second-level dynamic semantic graph.

[0030] The adjacency matrix of the third-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix.

[0031] A third dynamic graph pooling is performed based on the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate a third-level dynamic semantic graph.

[0032] In one specific embodiment, multi-label prediction is performed on the original image based on the high-level semantic features extracted by the backbone network to obtain the multi-label prediction probability value. The specific formula is as follows:

[0033] P m =S(f 2_MLP (GMP(F H )))

[0034] Among them, F H High-level semantic features extracted from the backbone network; GMP(·) is global max pooling; f 2_MLP (·) represents a two-layer fully connected neural network; S(·) is the activation function; P m This represents the probability value for multi-label prediction.

[0035] In a specific embodiment, the formula for calculating the first-level dynamic adjacency matrix is:

[0036]

[0037] Among them, P m A1 represents the multi-label prediction probability value; A2 represents the adjacency matrix of the first-level static semantic graph. This is the first-level dynamic adjacency matrix;

[0038] The first-level dynamic semantic graph consists of first-level dynamic semantic graph nodes and a first-level dynamic adjacency matrix. The formula for calculating the nodes of the first-level dynamic semantic graph is:

[0039]

[0040] in, These are nodes in the first-level dynamic semantic graph; Here, V0 is the first-level dynamic adjacency matrix; A0 is the initial dense graph, f is the initial dense adjacency matrix, and f is the initial dense graph. cluster (·) is the clustering function.

[0041] Secondly, a weakly supervised human body parsing system based on a hierarchical semantic graph neural network is provided, the system comprising:

[0042] The receiving module is used to receive the original image and set the human target bounding box and initial semantic annotation pixels on the original image;

[0043] The segmentation module is used to take the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0044] The filtering module is used to obtain preliminary prediction results of human body analysis using the initial pixel-level pseudo-labels, filter the initial pixel-level pseudo-labels using the preliminary prediction results of human body analysis, and generate high-confidence pseudo-labels.

[0045] The graph pooling module is used to construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate hierarchical human body parsing results.

[0046] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0047] Step A: Receive the original image, and set the human target bounding box and initial semantic annotation pixels on the original image;

[0048] Step B: Use the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0049] Step C: Use the initial pixel-level pseudo-labels to obtain preliminary prediction results of human body parsing, and use the preliminary prediction results of human body parsing to filter the initial pixel-level pseudo-labels to generate high-confidence pseudo-labels;

[0050] Step D: Construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate hierarchical human body parsing results.

[0051] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, performs the following steps:

[0052] Step A: Receive the original image, and set the human target bounding box and initial semantic annotation pixels on the original image;

[0053] Step B: Use the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0054] Step C: Use the initial pixel-level pseudo-labels to obtain preliminary prediction results of human body parsing, and use the preliminary prediction results of human body parsing to filter the initial pixel-level pseudo-labels to generate high-confidence pseudo-labels;

[0055] Step D: Construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate hierarchical human body parsing results.

[0056] The embodiments of this application have the following beneficial effects:

[0057] The embodiments of this application provide the following: receiving the original image, setting human target bounding boxes and initial labeled semantic pixels on the original image, using the human target bounding boxes and initial labeled semantic pixels as weakly supervised information, and combining them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels, greatly reducing the labeling cost; designing a reliable pixel selection strategy, performing reliable pixel sampling on the generated initial pseudo-labels, and selecting high-confidence pseudo-labels to provide accurate semantic structure features for subsequent model learning; designing a dynamic graph pooling module for fine-grained human parsing, generating a dynamic semantic graph by utilizing multi-label prediction probability values ​​and predefined static semantic graphs, and generating a dynamic hierarchical semantic graph from a fine-grained semantic graph to a coarse-grained semantic graph through multiple dynamic graph pooling, and decoding the dynamic hierarchical semantic graph to achieve accurate fine-grained human parsing. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A schematic diagram of a weakly supervised human body parsing method based on a hierarchical semantic graph neural network according to this application is shown.

[0060] Figure 2 A schematic diagram of a weakly supervised human parsing model based on a hierarchical semantic graph neural network according to this application is shown.

[0061] Figure 3This diagram illustrates the process of generating initial pixel-level pseudo-tags according to this application.

[0062] Figure 4 This diagram illustrates the structure of the dynamic hierarchical semantic graph according to this application.

[0063] Figure 5 A schematic diagram showing the parsing method according to Embodiment 1 of this application compared with other parsing methods;

[0064] Figure 6 A schematic diagram of a weakly supervised human body parsing system based on a hierarchical semantic graph neural network according to this application is shown.

[0065] Figure 7 A schematic diagram of an electronic device according to this application is shown. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] As described in the background section, high-performance human body parsing networks achieve accurate human body parsing prediction by using a large number of pixel-level labeled images as supervision information. However, pixel-level annotation requires a large amount of human resources, which is too costly, and is also subject to noise interference such as annotation errors. Graph pooling based on node sampling achieves graph pooling by sampling important nodes to generate a coarsened graph, but it may not retain key local structural information and may destroy the integrity of the graph topology, thus affecting the transmission of information in subsequent layers. Based on the above problems, this application proposes a weakly supervised human body parsing method and system based on hierarchical semantic graph neural networks.

[0068] Example 1

[0069] A weakly supervised human body parsing method based on hierarchical semantic graph neural networks, such as Figure 1 and 2 As shown, the method includes the following steps:

[0070] Step S1: Receive the original image and set the human target bounding box and initial semantic annotation pixels on the original image;

[0071] Since fine-grained human body segmentation tasks require pixel-level segmentation of clothing, including small accessories such as scarves and glasses, current high-performance human body segmentation models utilize a large number of pixel-level labeled images as supervision information for model training. However, pixel-level image annotation is resource-intensive, while weak supervision has attracted much attention due to its lower manual annotation cost. To address this, a two-stage weakly supervised human body segmentation model is proposed. Specifically, the original image is first received and annotated to obtain the human body bounding box and a small number of annotated semantic pixels. Then, unsupervised image segmentation with joint region semanticization is designed to generate initial pixel-level pseudo-labels for human body segmentation.

[0072] Step S2: Use the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels.

[0073] In one specific embodiment, the human target bounding box and initially labeled semantic pixels are used as weakly supervised information, and initial pixel-level pseudo-labels are generated by combining region semantic unsupervised image segmentation. Specifically, this includes:

[0074] Step S2.1: Perform foreground segmentation on the original image based on the human target bounding box to obtain a foreground segmentation image;

[0075] Step S2.2: Perform unsupervised image segmentation on the foreground segmentation image to obtain an unsupervised segmentation image;

[0076] Step S2.3: Use the initial labeled semantic pixels to semantically represent the segmented regions of the image segmentation blocks in the unsupervised segmentation image, and generate initial pixel-level pseudo-labels.

[0077] like Figure 3 As shown, the process of generating pixel-level pseudo-labels involves inputting the original image and a label image with a human body rectangular target box. First, the Grab-cut algorithm is used to segment the target human body from the background in the input image. The algorithm formula is shown in formula (1). After segmentation, the pixel values ​​of the foreground remain unchanged, and the pixel values ​​of the background obtained after segmentation are set to "0".

[0078]

[0079] Where M represents the original input image, and B represents the label image with the human body rectangular target box marked. This represents the obtained foreground segmentation image. This represents the label value of pixel i in the obtained foreground segmentation image. Then, an unsupervised image segmentation network is used to process the obtained foreground segmentation image. The division is performed using the following formula (2):

[0080]

[0081] Among them, Net UIS (·) represents an unsupervised image segmentation network, S k This represents the segmented region block, and K represents the number of region blocks obtained after unsupervised image segmentation.

[0082] Unsupervised image segmentation, by constraining the pixel features and spatial locations of an image, can obtain segmented regions with clear boundaries. Furthermore, a semantic region representation strategy is designed. Using the obtained segmented region blocks S = {S1, S2, ..., S...} K}, and the initial labeled semantic pixels T = {T1, T2, ..., T} N}, generating the final pixel-level pseudo-tags, i.e. Specifically, when the labeled semantic pixel T n Contained in segmented region block S k Then the semantic pixel T n The label is assigned to the segmented region block S k Specifically, N represents the number of semantic pixel categories labeled, and K >> N. The pixel value of a segmentation region block that does not contain semantic pixels is set to 0. After the segmentation region block undergoes the region semanticization strategy, the final pixel-level pseudo-label y is generated.

[0083] Step S3: Use the pixel-level pseudo-labels to obtain the preliminary prediction results of human body analysis, use the preliminary prediction results of human body analysis to filter the initial pixel-level pseudo-labels, and generate high-confidence pseudo-labels.

[0084] In one specific embodiment, the initial pixel-level pseudo-labels are filtered using the preliminary prediction results of human body analysis to generate high-confidence pseudo-labels, specifically including:

[0085] Step 3.1: Train the backbone feature extraction network using the initial pixel-level pseudo-labels as network supervision information, and output high-level semantic features and shallow features;

[0086] Step 3.2: Decode the high-level semantic features and the shallow features using a decoder to generate preliminary prediction results for human body analysis;

[0087] Step 3.3: Based on the preliminary prediction results of the human body analysis, reliable pixel sampling is performed on the initial pixel-level pseudo-labels to obtain the high-confidence pseudo-labels.

[0088] Specifically, ResNet-101 and ASPP are used as the backbone network for feature extraction, and the generated initial pixel-level pseudo-labels are used as the supervision information for training the human body parsing network. Specifically, the high-level semantic features output by the network and the shallow features output by the second layer of ResNet-101 are used to generate preliminary prediction results for human body parsing through a decoder, as shown in formula (3). The preliminary prediction results for human body parsing are obtained through encoding and decoding.

[0089]

[0090] X represents the high-level semantic features obtained by the backbone network; X2 represents the shallow features obtained by the backbone network. This represents the preliminary fine-grained prediction results obtained from human body analysis, F fine This indicates further acquired high-level semantic features.

[0091] By utilizing high-quality annotations as supervision, the manually parsed network achieved significant performance, stemming from its ability to capture rich and accurate features from the training process using precise labels. However, weakly supervised manual parsing cannot generate accurate labels. Noisy labels undoubtedly negatively impact the final performance. To handle noisy labels, an intuitive solution is to sample only reliable labels. Based on this, the initial prediction results are then combined... Based on the obtained pixel-level pseudo-label y, a reliable pixel sampling strategy is proposed to mitigate the negative impact of noisy labels. Specifically, when the initial prediction result is output... When the annotation result is consistent with the initial pseudo-label y, this pixel is determined to be a reliable pixel. Based on this, the set of reliable pixels I (high-confidence pseudo-labels) satisfies the following consistency constraint strategy:

[0092]

[0093] in, y(i) represents the preliminary prediction result of human body analysis for pixel i, and y(i) represents the pseudo-label value of pixel i.

[0094] By employing this strict constraint strategy, reliable pixels are sampled and noisy labels are removed, thereby obtaining high-confidence pseudo-labels.

[0095] Step S4: Construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate a hierarchical human body parsing result.

[0096] First, a corresponding static semantic map is obtained based on the original image segmentation. Specifically, the original image is segmented according to human body parts and / or clothing and connected to generate the first-level static semantic map; the original image is coarsely segmented according to human body structure and connected to generate the second-level static semantic map; the entire human body is treated as a node to generate the third-level static semantic map.

[0097] Specifically, the first-level static semantic graph includes detailed human body parts and clothing. For example, for the LIP dataset, this level includes: hat, hair, glasses, top, dress, pants, belt, left foot, right foot, face, left leg, right leg, left arm, right arm, bag, and scarf, etc. The second-level static semantic graph includes a coarse-grained division of the human body structure, including the upper and lower body. The third-level static semantic graph treats the entire human body as a node. The first, second, and third-level static semantic graphs are connected according to the topological structure of the human body, and a static adjacency matrix is ​​constructed. For example:

[0098] In one specific embodiment, the dynamic hierarchical semantic graph includes at least a first-level dynamic semantic graph, a second-level dynamic semantic graph, and a third-level dynamic semantic graph. The dynamic hierarchical semantic graph is generated by dynamic graph pooling based on an initial dense graph, specifically including:

[0099] Obtain the feature vector corresponding to the high-confidence pseudo-label as the initial dense graph node, and connect the initial dense graph node to construct the initial dense graph;

[0100] Obtain the multi-label prediction probability value of the original image, and dynamically adjust the adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value to obtain the first-level dynamic adjacency matrix.

[0101] The initial dense graph and the first-level dynamic adjacency matrix are combined to perform the first dynamic graph pooling to generate the first-level dynamic semantic graph;

[0102] The adjacency matrix of the second-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix.

[0103] A second dynamic graph pooling is performed based on the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate the second-level dynamic semantic graph.

[0104] The adjacency matrix of the third-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix.

[0105] A third dynamic graph pooling is performed based on the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate a third-level dynamic semantic graph.

[0106] In one specific embodiment, due to the inherent hierarchical structure of the human body and the close semantic correlation between different levels of human body structures, this application divides the human body structure into four levels from fine-grained to coarse-grained, G = {G0, G1, G2, G3 | G...} n =(V n E n A n )}.

[0107] Specifically, the feature vectors corresponding to the sampled reliable pixels are used as initial graph nodes to construct an initial dense graph G0(V0, E0, A0), where G0, V0, E0, and A0 represent the dense graph, the set of dense graph nodes, the set of dense edges, and the dense adjacency matrix, respectively. The initial dense graph is constructed from the feature map vectors corresponding to the sampled reliable pixels as graph nodes. The edges connecting the graph nodes are constructed based on the feature similarity of the graph nodes. Here, cosine similarity is used, as shown in formula (5) below:

[0108]

[0109] Among them, F fine(i) e represents the feature vector corresponding to pixel i (graph node i). ij This represents the weight of the edge connecting graph nodes i and j.

[0110] In a specific implementation, the extraction of hierarchical features is expressed as a graph pooling process, that is, gradually coarsening a dense graph into a sparse semantic graph until the entire human body is represented. First, multi-label prediction is performed on the input image based on the high-level semantic features extracted by the backbone network.

[0111] In a specific embodiment, the formula (6) for multi-label prediction of the original image based on the high-level semantic features extracted by the backbone network is as follows:

[0112]

[0113] Among them, F H High-level semantic features extracted from the backbone network; GMP(·) is global max pooling; f 2_MLP (·) represents a two-layer fully connected neural network; S(·) is the activation function; P represents a multi-label prediction function; m This represents the probability value for multi-label prediction.

[0114] Based on multi-label prediction probability value P mThe adjacency matrix A1 of the first-level semantic graph is dynamically adjusted to obtain the first dynamic adjacency matrix. In a specific embodiment, the formula (7) for calculating the first dynamic adjacency matrix is:

[0115]

[0116] Among them, P m A1 represents the multi-label prediction probability value; A2 represents the adjacency matrix of the first-level semantic graph. This is the first dynamic adjacency matrix;

[0117] Similarly, the adjacency matrix of the second-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix, and the adjacency matrix of the third-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix.

[0118] Based on the constructed initial dense graph and the first-level dynamic adjacency matrix The first graph pooling is performed to obtain the first-level dynamic semantic graph, which is composed of first-level dynamic semantic graph nodes and a first-level dynamic adjacency matrix. The formula (8) for calculating the first-level dynamic semantic graph nodes is as follows:

[0119]

[0120] in, These represent the nodes of the first-level dynamic semantic graph, where V0 is the initial dense graph and A0 is the initial dense adjacency matrix. This is a first-level dynamic adjacency matrix. and Together they form the first-level dynamic semantic graph f cluster (·) is the clustering function.

[0121] Similarly, the second-level and third-level dynamic semantic graphs are calculated based on the nodes of the first-level dynamic semantic graph. Second-level dynamic adjacency matrix A second dynamic graph pooling is performed to generate second-level dynamic semantic graph nodes, as shown in formula (9):

[0122]

[0123] in, These are nodes in the second-level dynamic semantic graph. and Together they form a second-level dynamic semantic graph

[0124] Finally, based on the nodes of the second-level dynamic semantic graph Second-level dynamic adjacency matrix The formula (10) for generating the third-level dynamic semantic graph nodes by performing the third dynamic graph pooling is as follows:

[0125]

[0126] in, For third-level dynamic semantic graph nodes, These are nodes in the second-level dynamic semantic graph. This is a second-level dynamic adjacency matrix. and Together they form a third-level dynamic semantic graph

[0127] Furthermore, using the obtained dynamic hierarchical semantic graph node features and the shallow features extracted by the backbone network, a decoder with the same structure is used to obtain the hierarchical human body parsing results, as shown in formulas (11) to (13):

[0128]

[0129] in, X2 represents the first-level dynamic semantic graph node features obtained through dynamic graph pooling, while X2 represents the shallow features extracted from the backbone network. For decoder, To obtain refined (first-level) human body analysis results.

[0130]

[0131] in, X2 represents the second-level dynamic semantic graph node features obtained through dynamic graph pooling, while X2 represents the shallow features extracted from the backbone network. For decoder, This is the result of coarse-grained (second-level) human body analysis.

[0132]

[0133] in, X2 represents the node features of the third-level dynamic semantic graph obtained through dynamic graph pooling, while X2 represents the shallow features extracted from the backbone network. For decoder, The results of the third-level human body analysis were obtained.

[0134] The formulas (14) to (16) for the cross-entropy loss function of the multi-level prediction results of human body analysis and pseudo-labels are shown below:

[0135]

[0136] in, and y fine These are the first-level pseudo-labels and the first-level human body parsing results predicted by the network, respectively. pix represents the number of pixels in the image, and N... n This indicates the number of categories with refined labels in the dataset. This represents the first-level loss function. Specifically, y fine These are the pseudo-tags obtained as mentioned above.

[0137]

[0138] in, and y coarse These represent the second-level pseudo-labels and the second-level human body parsing results predicted by the network, respectively. pix represents the number of pixels in the image, and N... n This indicates the number of categories with coarse-grained labels in the dataset. This represents the second-level loss function. Specifically, y coarse The pseudo-tags obtained above are used to obtain the correspondence between the defined static hierarchical semantic graph.

[0139]

[0140] in, and y human These represent the second-level pseudo-labels and the third-level human body parsing results predicted by the network, respectively. pix represents the number of pixels in the image. This represents the second-level loss function. Specifically, y human The pseudo-tags obtained above are used to obtain the correspondence between the defined static hierarchical semantic graph.

[0141] Furthermore, to verify the effectiveness of the method in this embodiment, the model trained with pixel-level fully supervised labels was compared with the weakly supervised human body parsing method based on hierarchical semantic graph neural networks in this embodiment. Specifically, LIP is currently the largest human body parsing dataset and is typically used as a standard benchmark for evaluating algorithm performance; therefore, the proposed weakly supervised algorithm was compared with other fully supervised methods on the LIP dataset, and the results are shown in Table 1.

[0142] Table 1 shows the test results of the weakly supervised algorithm and the comparison algorithm in this embodiment on the LIP dataset.

[0143]

[0144] The comparison results in Table 1 show that, compared with the fully supervised method, the weakly supervised algorithm in this embodiment still has the highest performance on sunglasses, left leg, right leg, left foot, and right foot. Specifically, the algorithm in this embodiment achieves final test results of 42.22% mIoU and 43.64% mIoU for the left and right legs, respectively. Compared with other fully supervised algorithms, the proposed weakly supervised algorithm has a significant advantage in parsing the left and right legs, exceeding the fully supervised algorithm Deeplab (ResNet-101), which is second only to the proposed weakly supervised algorithm, by 2.86% mIoU and 5.37% mIoU. It is worth noting that, supervised only by semantic pixels and a bounding box, the parsing method in this embodiment has the advantage of high accuracy, with an average accuracy of 57.16%, which is superior to all reported fully supervised methods.

[0145] like Figure 5 The figure shows a comparison between the analysis results obtained by other methods and the analysis results obtained by the method in this embodiment. In the figure, "groundtruth" represents the pixel-level fine-grained human body analysis results of the image using manual annotation; "baseline" represents the fine-grained human body analysis results obtained by training only using the pseudo-labels and backbone network generated by the proposed algorithm. As can be seen from the figure, the method in this embodiment, i.e., the part predicted by the white dashed box, provides more accurate predictions, further verifying the effectiveness of the proposed algorithm.

[0146] The scheme in this embodiment sets human target boxes and initial labeled semantic pixels on the original image, using the human target boxes and initial labeled pixels as weak supervision information, and combines them with region semantic image segmentation to generate initial pixel-level pseudo-labels, greatly reducing the labeling cost. A reliable pixel selection strategy is proposed, which combines the preliminary prediction results of human parsing to select initial pixel-level pseudo-labels and generate high-confidence pseudo-labels, effectively removing the interference of noisy pixels in the pseudo-labels and providing accurate structural features for the subsequent learning of the network model. An initial dense graph is constructed based on the high-confidence pseudo-labels, and a dynamic hierarchical semantic graph is generated based on dynamic graph pooling, which is then decoded to generate hierarchical human parsing results, achieving accurate fine-grained human parsing.

[0147] Example 2

[0148] Corresponding to the above embodiments, this application provides a weakly supervised human body parsing system based on hierarchical semantic graph neural networks, such as... Figure 6 As shown, the system includes:

[0149] The receiving module is used to receive the original image and set the human target bounding box and initial annotation pixels on the original image;

[0150] The segmentation module uses the human target bounding box and the initial labeled pixels as weakly supervised information, and combines them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0151] The filtering module is used to obtain preliminary prediction results of human body parsing using the initial pixel-level pseudo-labels, and to filter the initial pixel-level pseudo-labels using the preliminary prediction results of human body parsing to generate high-confidence pseudo-labels.

[0152] The dynamic graph pooling module is used to construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate a hierarchical human body parsing result.

[0153] In a specific embodiment, the segmentation module specifically includes a foreground segmentation module, an unsupervised segmentation module, and a semanticization module, specifically:

[0154] The foreground segmentation module is used to perform foreground segmentation on the original image based on the human target bounding box to obtain a foreground segmentation image;

[0155] The unsupervised segmentation module is used to perform unsupervised image segmentation on the foreground segmentation image to obtain an unsupervised segmentation image;

[0156] The semanticization module is used to semanticize the segmented regions of the image segmentation blocks in the unsupervised segmentation image using initially labeled semantic pixels, and generate initial pixel-level pseudo-labels.

[0157] In one specific embodiment, the filtering module includes an encoding module and a decoding module for feature extraction, specifically:

[0158] The feature extraction encoding module is used to train a backbone feature extraction network with the initial pixel-level pseudo-labels as network supervision information, and output high-level semantic features and shallow features.

[0159] The decoding module is used to decode the high-level semantic features and the shallow features through the decoder to generate a preliminary prediction result of human body analysis; and to combine the preliminary prediction result of human body analysis to perform reliable pixel sampling on the initial pixel-level pseudo-labels, and filter to obtain the high-confidence pseudo-labels.

[0160] In one specific embodiment, the system further includes a segmentation module, which is used to segment the original image according to human body parts and / or clothing and connect them to generate a first-level static semantic graph;

[0161] The original image is coarsely segmented based on human body structure and then connected to generate a second-level static semantic map.

[0162] The entire human body is treated as a node to generate the third-level static semantic graph.

[0163] In one specific embodiment, the dynamic graph pooling module includes a construction module, a first pooling module, a second pooling module, and a third pooling module. The dynamic hierarchical semantic graph includes at least a first-level dynamic semantic graph, a second-level dynamic semantic graph, and a third-level dynamic semantic graph. Specifically:

[0164] The construction module is used to obtain the feature vector corresponding to the high-confidence pseudo-label as the initial dense graph node, and connect the initial dense graph node to construct the initial dense graph;

[0165] The first pooling module is used to obtain the multi-label prediction probability value of the original image, dynamically adjust the adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value, and obtain the first-level dynamic adjacency matrix; and perform the first dynamic graph pooling by combining the initial dense graph and the first-level dynamic adjacency matrix to generate the first-level dynamic semantic graph.

[0166] The second pooling module is used to dynamically adjust the adjacency matrix of the second-level static semantic graph according to the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix; and to perform a second dynamic graph pooling based on the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate the second-level dynamic semantic graph.

[0167] The third pooling module is used to dynamically adjust the adjacency matrix of the third-level static semantic graph according to the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix; and to perform a third dynamic graph pooling based on the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate the third-level dynamic semantic graph.

[0168] In one specific embodiment, multi-label prediction is performed on the original image based on the high-level semantic features extracted by the backbone network to obtain the multi-label prediction probability value. The specific formula is as follows:

[0169] P m =S(f 2_MLP (GMP(F H )))

[0170] Among them, F H High-level semantic features extracted from the backbone network; GMP(·) is global max pooling; f 2_MLP (·) represents a two-layer fully connected neural network; S(·) is the activation function; P m This represents the probability value for multi-label prediction.

[0171] In a specific embodiment, the formula for calculating the first-level dynamic adjacency matrix is:

[0172]

[0173] Among them, P m A1 represents the multi-label prediction probability value; A2 represents the adjacency matrix of the first-level static semantic graph. This is the first-level dynamic adjacency matrix;

[0174] The first-level dynamic semantic graph consists of first-level dynamic semantic graph nodes and a first-level dynamic adjacency matrix. The formula for calculating the nodes of the first-level dynamic semantic graph is:

[0175]

[0176] in, These are nodes in the first-level dynamic semantic graph; Here, V0 is the first-level dynamic adjacency matrix; A0 is the initial dense graph, f is the initial dense adjacency matrix, and f is the initial dense graph. cluster (·) is the clustering function.

[0177] Example 3

[0178] An electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0179] Step 101: Receive the original image and set the human target bounding box and initial semantic annotation pixels on the original image;

[0180] Step 102: Use the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0181] Step 103: Use the initial pixel-level pseudo-labels to obtain the preliminary prediction results of human body parsing, and use the preliminary prediction results of human body parsing to filter the initial pixel-level pseudo-labels to generate high-confidence pseudo-labels;

[0182] Step 104: Construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate a hierarchical human body parsing result.

[0183] In one specific embodiment, step 102, which uses the human target bounding box and the initially labeled semantic pixels as weakly supervised information and combines them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels, specifically includes:

[0184] Based on the human target bounding box, the original image is segmented foreground to obtain a foreground segmentation image;

[0185] Unsupervised image segmentation is performed on the foreground segmentation image to obtain an unsupervised segmentation image;

[0186] The image segmentation blocks in the unsupervised segmentation image are semantically represented using initial labeled semantic pixels to generate initial pixel-level pseudo-labels.

[0187] In a specific embodiment, step 103, which uses the preliminary prediction results of human body analysis to filter the initial pixel-level pseudo-labels and generate high-confidence pseudo-labels, specifically includes:

[0188] A backbone feature extraction network is trained using the initial pixel-level pseudo-labels as network supervision information, and outputs high-level semantic features and shallow features.

[0189] The high-level semantic features and the shallow features are decoded by the decoder to generate preliminary prediction results for human body analysis;

[0190] Based on the preliminary prediction results of the human body analysis, reliable pixel sampling is performed on the initial pixel-level pseudo-labels to obtain the high-confidence pseudo-labels.

[0191] In one specific embodiment, it also includes:

[0192] The original image is segmented according to human body parts and / or clothing and connected to generate a first-level static semantic map;

[0193] The original image is coarsely segmented based on human body structure and then connected to generate a second-level static semantic map.

[0194] The entire human body is treated as a node to generate the third-level static semantic graph.

[0195] In a specific embodiment, the dynamic hierarchical semantic graph in step 104 includes at least a first-level dynamic semantic graph, a second-level dynamic semantic graph, and a third-level dynamic semantic graph. The generation of the dynamic hierarchical semantic graph based on the initial dense graph using dynamic graph pooling specifically includes:

[0196] Obtain the feature vector corresponding to the high-confidence pseudo-label as the initial dense graph node, and connect the initial dense graph node to construct the initial dense graph;

[0197] Obtain the multi-label prediction probability value of the original image, and dynamically adjust the adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value to obtain the first-level dynamic adjacency matrix.

[0198] The initial dense graph and the first-level dynamic adjacency matrix are combined to perform the first dynamic graph pooling to generate the first-level dynamic semantic graph;

[0199] The adjacency matrix of the second-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix.

[0200] A second dynamic graph pooling is performed based on the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate the second-level dynamic semantic graph.

[0201] The adjacency matrix of the third-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix.

[0202] A third dynamic graph pooling is performed based on the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate a third-level dynamic semantic graph.

[0203] In one specific embodiment, multi-label prediction is performed on the original image based on the high-level semantic features extracted by the backbone network to obtain the multi-label prediction probability value. The specific formula is as follows:

[0204] P m =S(f 2_MLP (GMP(F H )))

[0205] Among them, F H High-level semantic features extracted from the backbone network; GMP(·) is global max pooling; f 2_MLP (·) represents a two-layer fully connected neural network; S(·) is the activation function; P m This represents the probability value for multi-label prediction.

[0206] In a specific embodiment, the formula for calculating the first-level dynamic adjacency matrix is:

[0207]

[0208] Among them, P m A1 represents the multi-label prediction probability value; A2 represents the adjacency matrix of the first-level static semantic graph. This is the first-level dynamic adjacency matrix;

[0209] The first-level dynamic semantic graph consists of first-level dynamic semantic graph nodes and a first-level dynamic adjacency matrix. The formula for calculating the nodes of the first-level dynamic semantic graph is:

[0210]

[0211] in, These are nodes in the first-level dynamic semantic graph; Here, V0 is the first-level dynamic adjacency matrix; A0 is the initial dense graph, f is the initial dense adjacency matrix, and f is the initial dense graph. cluster (·) is the clustering function.

[0212] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores raw image data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a weakly supervised human body parsing method based on a hierarchical semantic graph neural network.

[0213] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0214] Example 4

[0215] In one embodiment of this invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0216] Step 201: Receive the original image and set the human target bounding box and initial semantic annotation pixels on the original image;

[0217] Step 202: Use the human target bounding box and the initial labeled semantic pixels as weakly supervised information, and combine them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels;

[0218] Step 203: Use the initial pixel-level pseudo-labels to obtain preliminary prediction results of human body parsing, and use the preliminary prediction results of human body parsing to filter the initial pixel-level pseudo-labels to generate high-confidence pseudo-labels;

[0219] Step 204: Construct an initial dense graph based on the high-confidence pseudo-labels, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate a hierarchical human body parsing result.

[0220] In a specific embodiment, step 202, which uses the human target bounding box and the initially labeled semantic pixels as weakly supervised information and combines them with region semantic unsupervised image segmentation to generate initial pixel-level pseudo-labels, specifically includes:

[0221] Based on the human target bounding box, the original image is segmented foreground to obtain a foreground segmentation image;

[0222] Unsupervised image segmentation is performed on the foreground segmentation image to obtain an unsupervised segmentation image;

[0223] The image segmentation blocks in the unsupervised segmentation image are semantically represented using initial labeled semantic pixels to generate initial pixel-level pseudo-labels.

[0224] In a specific embodiment, step 203, which uses the preliminary prediction results of human body analysis to filter the initial pixel-level pseudo-labels and generate high-confidence pseudo-labels, specifically includes:

[0225] A backbone feature extraction network is trained using the initial pixel-level pseudo-labels as network supervision information, and outputs high-level semantic features and shallow features.

[0226] The high-level semantic features and the shallow features are decoded by the decoder to generate preliminary prediction results for human body analysis;

[0227] Based on the preliminary prediction results of the human body analysis, reliable pixel sampling is performed on the initial pixel-level pseudo-labels to obtain the high-confidence pseudo-labels.

[0228] In one specific embodiment, it also includes:

[0229] The original image is segmented according to human body parts and / or clothing and connected to generate a first-level static semantic map;

[0230] The original image is coarsely segmented based on human body structure and then connected to generate a second-level static semantic map.

[0231] The entire human body is treated as a node to generate the third-level static semantic graph.

[0232] In a specific embodiment, the dynamic hierarchical semantic graph in step 204 includes at least a first-level dynamic semantic graph, a second-level dynamic semantic graph, and a third-level dynamic semantic graph. The generation of the dynamic hierarchical semantic graph based on the initial dense graph using dynamic graph pooling specifically includes:

[0233] Obtain the feature vector corresponding to the high-confidence pseudo-label as the initial dense graph node, and connect the initial dense graph node to construct the initial dense graph;

[0234] Obtain the multi-label prediction probability value of the original image, and dynamically adjust the adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value to obtain the first-level dynamic adjacency matrix.

[0235] The initial dense graph and the first-level dynamic adjacency matrix are combined to perform the first dynamic graph pooling to generate the first-level dynamic semantic graph;

[0236] The adjacency matrix of the second-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the second-level dynamic adjacency matrix.

[0237] A second dynamic graph pooling is performed based on the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate the second-level dynamic semantic graph.

[0238] The adjacency matrix of the third-level static semantic graph is dynamically adjusted based on the multi-label prediction probability value to obtain the third-level dynamic adjacency matrix.

[0239] A third dynamic graph pooling is performed based on the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate a third-level dynamic semantic graph.

[0240] In one specific embodiment, multi-label prediction is performed on the original image based on the high-level semantic features extracted by the backbone network to obtain the multi-label prediction probability value. The specific formula is as follows:

[0241] P m =S(f 2_MLP (GMP(F H )))

[0242] Among them, F H High-level semantic features extracted from the backbone network; GMP(·) is global max pooling; f 2_MLP (·) represents a two-layer fully connected neural network; S(·) is the activation function; P m This represents the probability value for multi-label prediction.

[0243] In a specific embodiment, the formula for calculating the first-level dynamic adjacency matrix is:

[0244]

[0245] Among them, P m A1 represents the multi-label prediction probability value; A2 represents the adjacency matrix of the first-level static semantic graph. This is the first-level dynamic adjacency matrix;

[0246] The first-level dynamic semantic graph consists of first-level dynamic semantic graph nodes and a first-level dynamic adjacency matrix. The formula for calculating the nodes of the first-level dynamic semantic graph is:

[0247]

[0248] in, These are nodes in the first-level dynamic semantic graph; Here, V0 is the first-level dynamic adjacency matrix; A0 is the initial dense graph, f is the initial dense adjacency matrix, and f is the initial dense graph. cluster (·) is the clustering function.

[0249] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0250] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0251] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A weakly supervised human parsing method based on hierarchical semantic graph neural network, characterized in that, The method comprises: receiving an original image, setting a human target frame and an initial labeled semantic pixel point on the original image; using the human target frame and the initial labeled semantic pixel point as weak supervision information, and generating an initial pixel-level pseudo label in combination with regional semantic unsupervised image segmentation; using the initial pixel-level pseudo label to obtain a human parsing preliminary prediction result, using the human parsing preliminary prediction result to screen the initial pixel-level pseudo label, and generating a high-confidence pseudo label; wherein the original image is divided and connected according to human body parts and / or clothing to generate a first-level static semantic graph, the original image is coarsely divided according to human body structure and connected to generate a second-level static semantic graph, and the whole human body is taken as a node to generate a third-level static semantic graph; constructing an initial dense graph according to the high-confidence pseudo label, performing dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decoding the dynamic hierarchical semantic graph to generate a hierarchical human parsing result; wherein the dynamic hierarchical semantic graph at least includes a first-level dynamic semantic graph, a second-level dynamic semantic graph and a third-level dynamic semantic graph, the dynamic hierarchical semantic graph is generated based on the initial dense graph, and specifically includes: obtaining a feature vector corresponding to the high-confidence pseudo label as an initial dense graph node, and connecting the initial dense graph node to construct an initial dense graph; obtaining a multi-label prediction probability value of the original image, dynamically adjusting an adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value to obtain a first-level dynamic adjacency matrix; combining the initial dense graph and the first-level dynamic adjacency matrix to perform first dynamic graph pooling to generate a first-level dynamic semantic graph; dynamically adjusting an adjacency matrix of the second-level static semantic graph according to the multi-label prediction probability value to obtain a second-level dynamic adjacency matrix; performing second dynamic graph pooling according to the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate a second-level dynamic semantic graph; dynamically adjusting an adjacency matrix of the third-level static semantic graph according to the multi-label prediction probability value to obtain a third-level dynamic adjacency matrix; performing third dynamic graph pooling according to the second-level dynamic semantic graph and the third-level dynamic adjacency matrix to generate a third-level dynamic semantic graph.

2. The weakly supervised human parsing method based on hierarchical semantic graph neural network according to claim 1, characterized in that, using the human target frame and the initial labeled semantic pixel point as weak supervision information, and generating an initial pixel-level pseudo label in combination with regional semantic unsupervised image segmentation, specifically including: performing foreground segmentation on the original image according to the human target frame to obtain a foreground segmentation image; performing unsupervised image segmentation on the foreground segmentation image to obtain an unsupervised segmentation image; performing segmentation region semanticization on an image segmentation block in the unsupervised segmentation image using the initial labeled semantic pixel point to generate an initial pixel-level pseudo label.

3. The weakly supervised human parsing method based on hierarchical semantic graph neural network according to claim 1, characterized in that, using the human parsing preliminary prediction result to screen the initial pixel-level pseudo label to generate a high-confidence pseudo label, specifically including: using a backbone feature extraction network to train the initial pixel-level pseudo label as network supervision information to output high-level semantic features and shallow features; The high-level semantic features and the shallow features are decoded by a decoder to generate a human parsing preliminary prediction result; The initial pixel-level pseudo label is reliably pixel sampled in combination with the human parsing preliminary prediction result, and the high-confidence pseudo label is screened.

4. The weakly supervised human parsing method based on hierarchical semantic graph neural network according to claim 1, characterized in that, The original image is multi-label predicted according to the high-level semantic features extracted by the backbone network to obtain a multi-label prediction probability value, and the specific formula is as follows: ； wherein, high-level semantic features extracted for the backbone network; is a global max pooling; is a two-layer fully connected neural network; is an activation function; is a multi-label prediction probability value.

5. The weakly supervised human parsing method based on hierarchical semantic graph neural network according to claim 1 or 4, characterized in that, The formula for calculating the first-level dynamic adjacency matrix is: ； wherein, is a multi-label prediction probability value; is an adjacency matrix of the first level static semantic graph; is a first level dynamic adjacency matrix; The first-level dynamic semantic graph is composed of a first-level dynamic semantic graph node and a first-level dynamic adjacency matrix, and the formula for calculating the first-level dynamic semantic graph node is: ； wherein, is a first level dynamic semantic graph node; is a first level dynamic adjacency matrix; is an initial dense graph, is an initial dense adjacency matrix, is a clustering function.

6. A weakly supervised human parsing system based on the hierarchical semantic graph neural network according to any one of claims 1-5, characterized in that, The system comprises: A receiving module is configured to receive an original image, set a human target frame and initial labeled semantic pixel points on the original image; A segmentation module is configured to take the human target frame and the initial labeled semantic pixel points as weak supervision information, and generate an initial pixel-level pseudo label by combining regional semantic unsupervised image segmentation; A division module is configured to divide the original image according to human body parts and / or clothing and connect to generate a first-level static semantic graph, divide the original image according to human body structure and connect to generate a second-level static semantic graph, and take the whole human body as a node to generate a third-level static semantic graph; A screening module is configured to obtain a human parsing preliminary prediction result using the initial pixel-level pseudo label, screen the initial pixel-level pseudo label using the human parsing preliminary prediction result, and generate a high-confidence pseudo label; A dynamic graph pooling module is configured to construct an initial dense graph according to the high-confidence pseudo label, perform dynamic graph pooling based on the initial dense graph to generate a dynamic hierarchical semantic graph, and decode the dynamic hierarchical semantic graph to generate a hierarchical human parsing result; the dynamic hierarchical semantic graph at least includes a first-level dynamic semantic graph, a second-level dynamic semantic graph and a third-level dynamic semantic graph; The dynamic graph pooling module comprises a construction module, a first pooling module, a second pooling module and a third pooling module, the construction module is configured to obtain a feature vector corresponding to the high-confidence pseudo label as an initial dense graph node, and connect the initial dense graph node to construct an initial dense graph; The first pooling module is configured to obtain a multi-label prediction probability value of the original image, dynamically adjust an adjacency matrix of the first-level static semantic graph according to the multi-label prediction probability value, obtain a first-level dynamic adjacency matrix, and perform first dynamic graph pooling in combination with the initial dense graph and the first-level dynamic adjacency matrix to generate a first-level dynamic semantic graph; The second pooling module is configured to dynamically adjust an adjacency matrix of the second-level static semantic graph according to the multi-label prediction probability value, obtain a second-level dynamic adjacency matrix, and perform second dynamic graph pooling according to the first-level dynamic semantic graph and the second-level dynamic adjacency matrix to generate a second-level dynamic semantic graph; The third pooling module is configured to dynamically adjust an adjacency matrix of the third hierarchical static semantic graph according to the multi-label prediction probability value, to obtain a third hierarchical dynamic adjacency matrix; and perform third dynamic graph pooling according to the second hierarchical dynamic semantic graph and the third hierarchical dynamic adjacency matrix, to generate a third hierarchical dynamic semantic graph.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method in any one of claims 1-5.

Images