Semantic segmentation network construction method based on depth level perception
By building a tag tree and training a semantic segmentation network and a semantic mapping network, and integrating the hierarchy of tags, the problem that network structures in the existing technology need to be designed for different scenarios is solved, and high-versatility and high-precision semantic segmentation are achieved.
Patent Information
- Application Number
- CN202411712836.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-25
AI Technical Summary
The existing semantic segmentation network needs to design different network structures for different scenarios, lacking universality and scalability, and not having high accuracy.
By obtaining the training sample image set and labels, building a tag tree, and simultaneously training the semantic segmentation network and semantic mapping network, the hierarchy of the tags is fused using the tree maximum and minimum loss function and the tree triple loss function.
It realizes the high universality and wide applicability of semantic segmentation networks, and can be applied in different scenarios without complex adjustments, improving segmentation accuracy.
Smart Images
Figure CN119693640A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a method for constructing a semantic segmentation network based on deep layer perception. Background Art
[0002] Semantic segmentation is a core task in computer vision, which aims to assign a specific category label to each pixel in an image. Unlike traditional image classification tasks, semantic segmentation not only identifies objects in an image, but also accurately determines the boundaries of these objects.
[0003] Existing semantic segmentation methods usually predict the category of each pixel independently, ignoring the potential hierarchical relationship between labels. In the few studies on the fusion of semantic hierarchical structures, the network architecture is usually changed to adapt to the hierarchy of labels. In order to integrate the results of semantic label tomography, the network architecture designed needs to add specific hierarchical modules to process label information at different levels. Some will change the connection mode of the network so that information can be effectively transmitted and integrated between different semantic levels. These operations are a relatively complex process and require special personnel to design, which requires a lot of time and effort. Secondly, different datasets have different semantic label structures. When facing new datasets, it may be necessary to redesign a new network structure to adapt to the new label hierarchy. This makes it difficult for these methods to be directly applied to different scenarios and tasks, reducing their versatility and scalability. Finally, the existing methods only integrate the hierarchical dependencies in the semantic labels and ignore the integration of hierarchical similarities, which limits the further improvement of segmentation accuracy. Summary of the invention
[0004] In view of the above analysis, an embodiment of the present invention aims to provide a method for constructing a semantic segmentation network based on deep hierarchical perception, so as to solve the problem that the existing semantic segmentation network needs to design different network structures for different scenarios, is not universal and scalable, and has low accuracy.
[0005] The purpose of the present invention is mainly achieved through the following technical solutions:
[0006] The present invention provides a method for constructing a semantic segmentation network based on deep layer perception, comprising:
[0007] Obtain the training sample image set and the labels corresponding to each image in the image set, and construct a label tree based on the hierarchical relationship of each label;
[0008] Based on the training sample image set, the labels corresponding to each image in the image set and the label tree, the semantic segmentation network and the semantic mapping network are trained simultaneously to obtain a trained semantic segmentation network; wherein,
[0009] The semantic segmentation network includes a feature extraction layer shared with the semantic mapping network, which is used to extract feature information of the input image;
[0010] The semantic segmentation network also includes a segmentation layer, which is used to perform semantic segmentation on the input image based on the number of nodes in the label tree; during training, the segmented category probability value is transformed based on the hierarchical relationship of each label in the label tree;
[0011] The semantic mapping network also includes a mapping layer for reorganizing the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree, the new feature vectors of similar categories are close to each other in the feature space, and the new feature vectors of dissimilar categories are far away from each other.
[0012] Furthermore, the semantic segmentation network and the semantic mapping network are trained simultaneously to obtain a trained semantic segmentation network, including:
[0013] The training sample image set, the labels corresponding to each image and the label tree are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.
[0014] Furthermore, the tree maximum and minimum loss function is expressed as:
[0015]
[0016] Wherein, B represents the number of images in the training sample image set; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.
[0017] Furthermore, the following formula is used to obtain the category probability value of each pixel after transformation:
[0018]
[0019] in, A represents the transformed category probability value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; k represents the parent node set of category k in the label tree; D kRepresents the set of child nodes of category k in the label tree; represents the (i,j)th pixel in the bth image. The category probability value corresponding to each class; Indicates category The label mask of .
[0020] Furthermore, the tree triplet loss function is expressed as:
[0021]
[0022] Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; m s Representing the optimal label interval of the sth valid label triple vector set; optionally selecting three new feature vectors output by the mapping layer and the corresponding labels to form a triple vector set; traversing the triple vector set, and obtaining a valid label triple vector set based on the label tree.
[0023] Furthermore, the traversal of the triple vector set to obtain a valid label triple vector set based on the label tree includes: the shortest path length between the label of the first feature vector and the label of the second feature vector in the triple vector set in the label tree is less than the shortest path length between the label of the first feature vector and the label of the third feature vector in the label tree.
[0024] Furthermore, the optimal label interval of the valid label triple vector set is calculated using the following formula:
[0025]
[0026] in, Represents the label of the first eigenvector in the sth valid label triplet vector set; Represents the label of the third eigenvector in the sth valid label triplet vector set; represents the label of the second eigenvector in the sth valid label triple vector set; Ψ() represents the shortest path length between two labels in the label tree.
[0027] Furthermore, the step of constructing a tag tree based on the hierarchical relationship of each tag includes:
[0028] For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag;
[0029] Based on the hierarchical relationship of each tag, a tag tree T = {V, E} is constructed; where V represents the node set of the tag tree, which is the tag class and its parent class in the WordNet dataset; E represents the node relationship of the tag tree.
[0030] Furthermore, in the semantic segmentation network,
[0031] The extracted feature information is passed through a segmentation layer, and each pixel in the image is classified based on the extracted feature information to obtain a semantic segmentation result; wherein,
[0032] The feature information is convolved on the feature map through a convolution layer of size 1×1×|V| to obtain the transformed feature information; wherein |V| represents the number of nodes in the label tree;
[0033] The transformed feature information is subjected to a nonlinear transformation through a ReLU activation function to obtain the nonlinearly transformed feature information;
[0034] The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×|V| again, and then passes through a sigmoid function to obtain the probability distribution of the semantics of each pixel.
[0035] Furthermore, in the semantic mapping network,
[0036] The extracted feature information is mapped to a new space through a mapping layer; wherein,
[0037] The feature information is convolved on the feature map through a convolution layer with a size of 1×1×32 to obtain transformed feature information;
[0038] The transformed feature information is subjected to a nonlinear transformation through a ReLU activation function to obtain the nonlinearly transformed feature information;
[0039] The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×32 again, and then passes through a sigmoid function to obtain a feature vector mapped to a new space.
[0040] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0041] 1. The semantic segmentation network constructed by the present invention does not need to change the existing network architecture of deep semantic segmentation, has high versatility and wide applicability, and can be applied to any common segmentation network without complex adjustment or redesign for a specific network architecture, thereby greatly reducing the difficulty of semantic segmentation.
[0042] 2. When constructing a semantic segmentation network, the present invention provides a hierarchical category representation method by constructing a label tree, so that the model can understand the inclusion relationship and hierarchical structure between categories. During the training process, the model can learn more discriminative feature representations.
[0043] 3. The semantic segmentation network constructed by the present invention can effectively integrate the hierarchical structure of labels, including hierarchical dependency and hierarchical similarity, based on the constructed label tree during training by introducing the tree maximum and minimum loss function and the tree triplet loss function. This fusion helps to further improve the accuracy of segmentation, so that the network can more accurately identify and segment different objects and scenes in the image.
[0044] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0046] Figure 1 Schematic diagram of the process of constructing a semantic segmentation network based on deep layer perception in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0048] A specific embodiment of the present invention discloses a method for constructing a semantic segmentation network based on deep layer perception, such as Figure 1 As shown, including:
[0049] Step S1: obtain a training sample image set and a label corresponding to each image in the image set, and construct a label tree based on the hierarchical relationship of each label.
[0050] Specifically, the training sample image set is selected to include diverse images to ensure that the model can learn the representation of objects in different environments and conditions; each image in the image set is equipped with corresponding annotation information. Exemplarily, the present invention selects the street scene semantic segmentation dataset Cityscapes. This dataset is specifically for urban street scenes, consisting of street scenes in 50 different cities, including 5,000 finely annotated images and 20,000 roughly annotated images, and uses 19 commonly used categories to evaluate segmentation accuracy, such as roads, sidewalks, buildings, vehicles, etc.
[0051] Furthermore, the step of constructing a tag tree based on the hierarchical relationship of each tag includes:
[0052] For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag;
[0053] Based on the hierarchical relationship of each tag, a tag tree T = {V, E} is constructed; where V represents the node set of the tag tree, which is the tag class and its parent class in the WordNet dataset; E represents the node relationship of the tag tree.
[0054] Specifically, the label tree is a tree-like hierarchical structure used to represent the hierarchical relationship between categories. Each node represents a category, and the edge represents the parent-child relationship between categories. The WordNet dataset is a large-scale English vocabulary database, in which words are organized in the form of synonym sets, which are connected by semantic relationships (such as parent class, child class, etc.).
[0055] In this embodiment, for several labels corresponding to each image in the image set, each label represents a category, and its parent category and the parent category of the parent category are searched in the WordNet dataset until the root node of the WordNet dataset is reached. The category of the label and all parent nodes up to the root node are used as nodes of the label tree. For example, in a city street view picture, "vehicle" can be a parent node, and "car", "truck" and "motorcycle" can be child nodes.
[0056] Step S2, based on the training sample image set, the labels corresponding to each image in the image set and the label tree, the semantic segmentation network and the semantic mapping network are trained simultaneously to obtain a trained semantic segmentation network; wherein,
[0057] The semantic segmentation network includes a feature extraction layer shared with the semantic mapping network, which is used to extract features of the input image;
[0058] The semantic segmentation network also includes a segmentation layer, which is used to perform semantic segmentation on the input image based on the number of nodes in the label tree; during training, the segmented category probability values are transformed based on the hierarchical relationship of each label in the label tree to obtain the maximum and minimum loss function of the tree during training;
[0059] The semantic mapping network also includes a mapping layer for reorganizing the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree, the new feature vectors of similar categories are close to each other in the feature space, and the new feature vectors of dissimilar categories are far away from each other, so that the feature extraction layer can extract more distinguishing features, providing a better basis for the segmentation layer to perform segmentation.
[0060] Furthermore, in the semantic segmentation network, the input image is passed through a feature extraction layer to extract feature information of the image.
[0061] Specifically, the feature extraction layer of the semantic segmentation network uses an existing segmentation network, removing its last classification layer, to extract features of the input image. Exemplarily, the existing segmentation network can use DeepLab, PSPNet, UNet, etc., which is not limited in this embodiment.
[0062] Furthermore, the feature information is passed through a segmentation layer to classify each pixel in the image to obtain a semantic segmentation result; wherein,
[0063] The feature information is convolved on the feature map through a convolution layer of size 1×1×|V| to obtain the transformed feature information; wherein |V| represents the number of nodes in the label tree.
[0064] Specifically, the 1×1 convolutional layer is used to fuse the feature information of different layers and compress the feature dimension at the same time. The high-dimensional feature map obtained from the feature extraction layer is compressed to the same dimension as the number of label tree nodes, and the multi-channel feature map can be converted into a feature map of |V| channels, each channel corresponding to a label tree node.
[0065] The transformed feature information is activated by a ReLU function to perform nonlinear transformation on each feature to obtain the feature information after nonlinear transformation.
[0066] Specifically, the ReLU activation function introduces nonlinear transformation by setting all negative values to 0 and keeping positive values unchanged, enabling the network to learn more complex feature representations and patterns.
[0067] The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×|V| again, and then passes through a sigmoid function to obtain the probability distribution of the semantics of each pixel.
[0068] Specifically, the convolutional layer that passes through again can adjust the feature weights, strengthen the features that are useful for the segmentation task, and suppress irrelevant features. The final sigmoid function is used to obtain the probability distribution of each pixel for |V| categories.
[0069] It should be noted that the segmentation layer can be expressed as f seg (I)∈R M×N×|V| , where M represents the height of the image; N represents the width of the image; and |V| represents the number of nodes in the label tree. The segmentation layer receives the output of the feature extraction layer, and classifies the output of the feature extraction layer at the pixel level according to the category nodes defined in the label tree to predict the category of each pixel in the image.
[0070] Furthermore, in the semantic mapping network, the input image is passed through a feature extraction layer to extract feature information of the image.
[0071] Specifically, through the feature extraction layer shared by the semantic mapping network and the semantic segmentation network, the semantic mapping network and the semantic segmentation network can share the features extracted from the input image to improve the generalization ability of the model, and at the same time help to improve the expressive ability of the features during the training stage.
[0072] Furthermore, the feature information is mapped to a new space through a mapping layer; wherein,
[0073] The feature information is convolved on the feature map through a convolution layer with a size of 1×1×32 to obtain the transformed feature information.
[0074] The transformed feature information is activated by a ReLU function to perform nonlinear transformation on each feature to obtain the feature information after nonlinear transformation.
[0075] The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×32 again, and then passes through a sigmoid function to obtain a feature vector mapped to a new space.
[0076] Specifically, the mapping layer is represented by f proj (I)∈R M×N×32, the features extracted by the feature extraction layer are reorganized into a new space of 32 channels to obtain a new feature vector. During the training process, a loss function is designed based on the distance of the new feature vector in the feature space and the distance of the annotated label in the label tree. The designed loss function fuses the similarity information of the labels by penalizing the categories with close but dissimilar distances of the new feature vectors in the feature space, as well as the categories with far but similar distances. By fusing the similarity information of the labels, the feature extraction layer can extract more discriminative features, providing a better basis for the segmentation layer to perform segmentation.
[0077] Furthermore, the semantic segmentation network and the semantic mapping network are trained simultaneously to obtain a trained semantic segmentation network, including:
[0078] The training sample image set, the labels corresponding to each image and the label tree are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.
[0079] Specifically, the semantic segmentation network and the semantic mapping network are based on the total loss function: L total =L TMM +L TT Perform simultaneous training; where L TMM is the maximum and minimum loss function of the tree; L TT is the tree triplet loss function. The two networks are optimized as a whole at the same time, which influences each other and learns collaboratively during the training process, while reducing the risk of overfitting and increasing the robustness of the model.
[0080] Furthermore, the tree maximum and minimum loss function is:
[0081]
[0082] Wherein, B represents the number of images in the training sample image set; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.
[0083] Specifically, the semantic segmentation network obtains the difference between the predicted category probability value and its true label through the tree maximum and minimum loss function. In the process of minimizing the loss function, the segmentation result of the semantic segmentation network is made more consistent with the hierarchical structure defined in the label tree.
[0084] Furthermore, the following formula is used to perform the tree maximum and minimum transformation to obtain the category probability value of each pixel after transformation:
[0085]
[0086] in, A represents the transformed category probability value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; k represents the parent node set of category k in the label tree; D k Represents the set of child nodes of category k in the label tree; represents the (i,j)th pixel in the bth image. The category probability value corresponding to each class; Indicates category The label mask of .
[0087] Specifically, when performing semantic segmentation, the segmentation layer will output a probability distribution for each pixel, indicating the possibility that the pixel belongs to each category. Through the hierarchical structure between the category labels on the label tree, when a pixel in the segmentation layer is predicted to be a subcategory, the prediction probability of its parent category should also be high. For example, if the model predicts that a pixel belongs to the subcategory of "car", then it should also predict that this pixel is likely to belong to its parent category "vehicle". The tree maximum and minimum transformation is used to ensure that the hierarchical structure of the output of the network segmentation layer is consistent with the hierarchical structure defined in the label tree, that is, the output value of the subcategory will not exceed the output value of its parent category. This transformation ensures that when performing pixel-level classification, the output value of the segmentation layer conforms to the hierarchical relationship between labels, which is conducive to improving the logical consistency and accuracy of segmentation.
[0088] Furthermore, the tree triplet loss function
[0089]
[0090] Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; m s Representing the optimal label interval of the sth valid label triple vector set; optionally selecting three new feature vectors output by the mapping layer and the corresponding labels to form a triple vector set;
[0091] The triple vector set is traversed, and a valid label triple vector set is obtained based on the label tree.
[0092] Specifically, the tree triplet loss function measures the hierarchical similarity between labels by calculating the distance between feature vectors. By minimizing the tree triplet loss function, the feature vectors of similar categories in the features extracted by the feature extraction layer are closer, and the feature vectors of dissimilar categories are farther apart, which is beneficial for the segmentation layer to perform semantic segmentation more accurately.
[0093] The triplet vector set is all feature vectors mapped to the new space for the output of the mapping layer Take out three different feature vectors and their corresponding labels at random to form a triple vector set, recorded as Among them, N t Denotes the triple vector set D T The number of triple vectors in ; represents the first eigenvector in the rth triple vector set; represents the second eigenvector in the rth triple vector set; represents the third eigenvector in the rth triplet vector set; Represents the label corresponding to the first eigenvector in the rth triple vector set; Represents the label corresponding to the second eigenvector in the rth triple vector set; Represents the label corresponding to the third eigenvector in the rth triple vector set.
[0094] Further, the triple vector set is traversed to obtain a valid label triple vector set based on the label tree, including: the shortest path length between the label of the first feature vector and the label of the second feature vector in the triple vector set in the label tree is less than the shortest path length between the label of the first feature vector and the label of the third feature vector in the label tree.
[0095] Specifically, the valid label triple vector is a label of three feature vectors in the triple vector based on their positions in the label tree, satisfying the following conditions:
[0096]
[0097] Wherein, Ψ() represents the shortest path length between two label nodes on the label tree.
[0098] In the label tree, the shortest path length between two nodes is considered as a measure of similarity in the hierarchy. The shortest path length indicates that the two categories are more similar. During the training process, constructing a valid label triple vector set can enable the entire model to more accurately distinguish between visually similar but semantically different categories, which can help the model capture the differences and similarities between categories and improve the segmentation accuracy of the semantic segmentation network.
[0099] The set of triple vectors that meet the above conditions constitutes a valid label triple vector set: Among them, N b Represents the number of triplets in the set of valid label triple vectors.
[0100] Specifically, for the first eigenvector, the second eigenvector, and the third eigenvector in the valid label triple vector set, the distances between the vectors are calculated using the following formulas:
[0101]
[0102]
[0103] The formula is a variant of the cosine similarity formula, and the result is that the distance value ranges from 0 to 1, where 0 means that the vectors are exactly the same and 1 means that the vectors are completely opposite.
[0104] Furthermore, the optimal label interval of the valid label triple vector set is calculated using the following formula:
[0105]
[0106] in, Represents the label of the first eigenvector in the sth valid label triplet vector set; Represents the label of the third eigenvector in the sth valid label triplet vector set; represents the label of the second eigenvector in the sth valid label triple vector set; Ψ() represents the shortest path length between two labels in the label tree.
[0107] Specifically, the optimal label interval is to quantify the hierarchical similarity between labels and use the quantified similarity to optimize the performance of the model when training the model. s When is a positive value, it means the label and tags The distance is greater than the label and tags The distance between With label On the contrary, when m s When it is a negative value, it means the label With label More similar.
[0108] During the joint training process of the semantic segmentation network and the semantic mapping network, the mapping layer maps the features extracted by the feature extraction layer to a new space. Through feature transformation, the feature extraction layer can extract feature information that is more suitable for segmentation from the original features, thereby enhancing the feature expression capability. At the same time, during the training process, the mapping layer integrates the hierarchical structure of labels into the information representation, so that the feature extraction layer can not only identify local features, but also combine the hierarchical relationship between categories. By improving the feature representation, the segmentation layer is provided with features that are more suitable for pixel-level classification, thereby directly improving the classification accuracy of the segmentation layer at each pixel in the image.
[0109] Another embodiment of the present invention is a method for performing semantic segmentation of an image based on the semantic segmentation model obtained by the above-mentioned construction method, comprising: obtaining an image to be segmented; inputting the image to be segmented into a trained semantic segmentation network to obtain a segmented semantic image, including: passing the image to be segmented through a feature extraction layer of the semantic segmentation network to extract features in the image; passing the features in the image through the segmentation layer of the semantic segmentation network to obtain a segmented image of the same size as the input image, in which each pixel is assigned a category label.
[0110] In summary, the method for constructing a semantic segmentation network based on deep layer perception according to an embodiment of the present invention has the following beneficial effects:
[0111] 1. The semantic segmentation network constructed by the present invention does not need to change the existing network architecture of deep semantic segmentation, has high versatility and wide applicability, and can be applied to any common segmentation network without complex adjustment or redesign for a specific network architecture, thereby greatly reducing the difficulty of semantic segmentation.
[0112] 2. When constructing a semantic segmentation network, the present invention provides a hierarchical category representation method by constructing a label tree, so that the model can understand the inclusion relationship and hierarchical structure between categories. During the training process, the model can learn more discriminative feature representations.
[0113] 3. The semantic segmentation network constructed by the present invention can effectively integrate the hierarchical structure of labels, including hierarchical dependency and hierarchical similarity, based on the constructed label tree during training by introducing the tree maximum and minimum loss function and the tree triplet loss function. This fusion helps to further improve the accuracy of segmentation, so that the network can more accurately identify and segment different objects and scenes in the image.
[0114] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A method for constructing a semantic segmentation network based on deep layer perception, characterized in that: include: Obtain the training sample image set and the labels corresponding to each image in the image set, and construct a label tree based on the hierarchical relationship of each label; Based on the training sample image set, the labels corresponding to each image in the image set and the label tree, the semantic segmentation network and the semantic mapping network are trained simultaneously to obtain a trained semantic segmentation network; wherein, The semantic segmentation network includes a feature extraction layer shared with the semantic mapping network, which is used to extract feature information of the input image; The semantic segmentation network also includes a segmentation layer, which is used to perform semantic segmentation on the input image based on the number of nodes in the label tree; during training, the segmented category probability value is transformed based on the hierarchical relationship of each label in the label tree; The semantic mapping network also includes a mapping layer for reorganizing the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree, the new feature vectors of similar categories are close to each other in the feature space, and the new feature vectors of dissimilar categories are far away from each other.
2. The method according to claim 1, characterized in that: The method of simultaneously training the semantic segmentation network and the semantic mapping network to obtain a trained semantic segmentation network includes: The training sample image set, the labels corresponding to each image and the label tree are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.
3. The method according to claim 2, characterized in that: The tree maximum and minimum loss function is expressed as: Wherein, B represents the number of images in the training sample image set; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.
4. The method according to claim 3, characterized in that: Use the following formula to get the category probability value of each pixel after transformation: in, A represents the transformed category probability value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; k represents the parent node set of category k in the label tree; D k Represents the set of child nodes of category k in the label tree; represents the (i,j)th pixel in the bth image. The category probability value corresponding to each class; Indicates category The label mask of .
5. The method according to claim 2, characterized in that: The tree triplet loss function is expressed as: Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; m s Representing the optimal label interval of the sth valid label triple vector set; optionally selecting three new feature vectors output by the mapping layer and the corresponding labels to form a triple vector set; traversing the triple vector set, and obtaining a valid label triple vector set based on the label tree.
6. The method according to claim 5, characterized in that: The traversing of the triple vector set to obtain a valid label triple vector set based on the label tree includes: the shortest path length between the label of the first feature vector and the label of the second feature vector in the triple vector set in the label tree is less than the shortest path length between the label of the first feature vector and the label of the third feature vector in the label tree.
7. The method according to claim 6, characterized in that: The optimal label interval of the valid label triple vector set is calculated using the following formula: in, Represents the label of the first eigenvector in the sth valid label triplet vector set; Represents the label of the third eigenvector in the sth valid label triplet vector set; represents the label of the second eigenvector in the sth valid label triple vector set; Ψ() represents the shortest path length between two labels in the label tree.
8. The method according to any one of claims 1 to 7, characterized in that: The step of constructing a tag tree based on the hierarchical relationship of each tag includes: For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag; Based on the hierarchical relationship of each tag, a tag tree T = {V, E} is constructed; where V represents the node set of the tag tree, which is the tag class and its parent class in the WordNet dataset; E represents the node relationship of the tag tree.
9. The method according to claim 8, characterized in that: In the semantic segmentation network, The extracted feature information is passed through a segmentation layer, and each pixel in the image is classified based on the extracted feature information to obtain a semantic segmentation result; wherein, The feature information is convolved on the feature map through a convolution layer of size 1×1×|V| to obtain the transformed feature information; wherein |V| represents the number of nodes in the label tree; The transformed feature information is subjected to a nonlinear transformation through a ReLU activation function to obtain the nonlinearly transformed feature information; The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×|V| again, and then passes through a sigmoid function to obtain the probability distribution of the semantics of each pixel.
10. The method according to claim 8, characterized in that: In the semantic mapping network, The extracted feature information is mapped to a new space through a mapping layer; wherein, The feature information is convolved on the feature map through a convolution layer with a size of 1×1×32 to obtain transformed feature information; The transformed feature information is subjected to a nonlinear transformation through a ReLU activation function to obtain the nonlinearly transformed feature information; The feature information after the nonlinear transformation passes through a convolution layer with a size of 1×1×32 again, and then passes through a sigmoid function to obtain a feature vector mapped to a new space.