A semantic segmentation method for traditional cultural images based on edge prediction

Through the combination of label relaxation method and edge loss, combined with edge prediction and point classifier, the problem of insufficient edge prediction in traditional cultural image semantic segmentation is solved, and the semantic segmentation of fine edges is realized, and labeling errors and edges are overcome. The labeling difficulties caused by fuzziness are relatively versatile.

CN114494698BActive Publication Date: 2024-11-05BEIJING UNIV OF POSTS & TELECOMM +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210102991.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-27
Publication Date
2024-11-05
Estimated Expiration
2042-01-27

AI Technical Summary

Technical Problem

The prior art is not fine enough in traditional cultural image semantic segmentation, and pixel labeling errors are prone to occur during labeling, which fails to effectively solve the prediction accuracy problem of edge parts.

Method used

The label relaxation method is used to perform label relaxation processing on image pixels, calculate the maximum likelihood estimation of labeled image pixels, define edge loss, and supervise and train the model using edge loss and initial cross entropy loss, and determine the blur through edge prediction method. The categories of edge pixel points are used to classify uncertain blurred edge pixel points using a shared point classifier, and finally a fine edge prediction map is obtained.

Benefits of technology

It realizes the acquisition of more refined segmentation edges in semantic segmentation, solves the problem of edge pixel labeling errors during labeling, improves the edge prediction fineness of semantic segmentation of traditional cultural images, and has good versatility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114494698B_ABST
    Figure CN114494698B_ABST
Patent Text Reader

Abstract

The present invention discloses a traditional cultural image semantic segmentation method based on edge prediction, which includes the following steps: using a label relaxation method to relax the labels of the pixels of the image and replace the pixel labels; defining edge loss, using edge loss and initial cross entropy loss to supervise the model training; using an edge prediction method to determine the category of fuzzy edge pixels, and selecting multiple fuzzy edge pixels with uncertain labels; using a shared point classifier to classify the fuzzy edge pixels with uncertain labels and determine the specific labels of the fuzzy edge pixels; mixing the predicted category of the fuzzy edge pixels and the label prediction map after bilinear interpolation to obtain the final fine edge prediction map. The present invention can eliminate the problems of artificial negligence when labeling data and the difficulty of labeling caused by edge blur, has good versatility, and can realize the semantic segmentation of fine edges through the algorithm of this article.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a traditional cultural image semantic segmentation method based on edge prediction. Background Art

[0002] The construction of the national cultural big data system is a new infrastructure for the propaganda and cultural sectors and is a key development direction for my country. As the supply end of the national cultural big data system, the Chinese Cultural Material Library has very valuable research on related technologies. The core research content of cultural computing includes deconstruction, annotation, reconstruction, presentation, knowledge graphs, etc. Among them, image segmentation is an important part of deconstruction, which provides a material basis for the subsequent vectorization and reconstruction work of the laboratory. Traditional cultural images are semantically rich, and the extraction of cultural image structures of different periods and different nationalities faces challenges. Therefore, this topic is of great practical application value.

[0003] Image segmentation is a classic problem in computer vision research and has become a hot topic in the field of image understanding. The classic method of interactive segmentation is relatively mature and can distinguish the foreground and background in the picture through a small amount of interaction. In recent years, deep learning semantic segmentation methods have made significant progress, and deep learning methods can directly predict multi-category targets, which overcomes the single-category limitation of non-deep learning methods. Semantic segmentation technology has developed rapidly and is widely used. It has now been successfully used in image segmentation, video segmentation, 3D point cloud segmentation and other fields. The use scenarios include autonomous driving, medical image analysis, quality inspection, etc., greatly improving the efficiency of data processing. Judging from the current development trend, deep learning is still the dominant method.

[0004] Most image segmentation methods do not focus on the prediction accuracy of edge parts because their usage scenarios do not have high requirements for edge fineness. Therefore, the segmentation results often have insufficient edges and cannot meet the requirements of traditional cultural image semantic segmentation.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In response to the problems in the related art, the present invention proposes a traditional cultural image semantic segmentation method based on edge prediction, which can obtain relatively fine segmentation edges while performing semantic segmentation and solve the inevitable problem of edge pixel labeling errors during labeling, so as to overcome the above-mentioned technical problems existing in the existing related art.

[0007] To this end, the specific technical solution adopted by the present invention is as follows:

[0008] A traditional cultural image semantic segmentation method based on edge prediction, the method comprising the following steps:

[0009] S1, using the label relaxation method to relax the pixels of the image and replace the pixel labels;

[0010] S2. Calculate the maximum likelihood estimate of the pixel labels of the annotated image, define the edge loss, and use the edge loss and the initial cross entropy loss to supervise the model training;

[0011] S3. In the process of upsampling prediction of the model, the category of the fuzzy edge pixel point is determined by using the edge prediction method, and a plurality of fuzzy edge pixel points with uncertain labels are selected;

[0012] S4, using a shared point classifier to classify the fuzzy edge pixel points with uncertain labels, and determine the specific labels of the fuzzy edge pixel points;

[0013] S5. Mix the predicted category of the blurred edge pixel points and the label prediction map after bilinear interpolation to obtain the final fine edge prediction map.

[0014] Furthermore, the method of performing label relaxation processing on pixels of the image by using the label relaxation method and replacing pixel labels includes the following steps:

[0015] S11. During the training process, a label relaxation method is used to relax and modify the label range of edge pixels of the image: pixels in eight areas around each pixel are selected as the label relaxation range of the pixel;

[0016] S12. Replace the labels of the initially marked pixels with the intersection of all labels within the relaxed range.

[0017] Furthermore, the edge pixel indicates that the pixels adjacent to a pixel have different labels, and the label relaxation method will not affect the pixels whose surrounding pixel labels are the same as the pixel label.

[0018] Furthermore, the labels of the initially annotated pixels are obtained by manual annotation, and for the training set to be manually annotated, the label range of edge pixels is modified by using a label relaxation method to solve the problem of difficulty in accurate pixel-level annotation during the annotation process caused by blurred object edges.

[0019] Furthermore, the step of calculating the maximum likelihood estimate of the pixel labels of the annotated image, defining the edge loss, and using the edge loss and the initial cross entropy loss to perform supervised training on the model includes the following steps:

[0020] S21, define the intersection of each pixel's own label and the labels of pixels in the eight surrounding areas as the relaxed label map;

[0021] S22, calculating the maximum likelihood estimate of the pixel labels of the annotated image and defining the edge loss;

[0022] S23. Combine the edge loss with the initial cross entropy loss to form the final cost loss function, and use the cost loss function for model training supervision.

[0023] Furthermore, the step of calculating the maximum likelihood estimate of the pixel labels of the annotated image and defining the edge loss includes the following steps:

[0024] Define the surrounding and self categories of an edge pixel to be classified as A and B respectively;

[0025] By calculating the A∪B category instead of calculating the maximum likelihood estimate of the annotated image pixel label, the calculation formula is as follows:

[0026] P(A∪B)=P(A)+P(B);

[0027] Among them, P(*) represents the softmax calculation of the probability of each category;

[0028] Define N as a 3*3 window around a pixel, and define the edge loss as:

[0029]

[0030] In the formula, L is the loss, boundary is the edge, N is a natural number set, and C is the union of the label of any point and the labels of the surrounding eight neighborhoods. When |C|=1, the edge loss L boundary Degenerates to the standard one-hot label cross entropy loss.

[0031] Furthermore, in the process of model upsampling prediction, the category of fuzzy edge pixels is determined by edge prediction method, and a plurality of fuzzy edge pixels with uncertain labels are selected, which includes the following steps:

[0032] S31, in the process of model upsampling prediction, using edge prediction method to determine the category of blurred edge pixel points;

[0033] S32. After the rough prediction image is subjected to bilinear interpolation, a predetermined point selection strategy is used to select a plurality of fuzzy edge pixel points with uncertain labels.

[0034] Furthermore, the predetermined point selection strategy is to obtain the probability of each fuzzy edge pixel point for each category in the prediction tensor, perform logarithmic operation, and select the two highest items to calculate the difference, wherein the higher the difference, the stronger the certainty that the label of the fuzzy edge pixel point belongs to a certain category. This method can realize the screening of the uncertainty of fuzzy edge pixels.

[0035] Furthermore, the shared point classifier is a three-layer perceptron, wherein the first layer is an input layer, the second layer is a hidden layer, and the third layer is an output layer, which is used to output the category of the predicted point, and different layers are associated in a fully connected manner.

[0036] Furthermore, the method of using a shared point classifier to classify fuzzy edge pixels with uncertain labels and determining specific labels of the fuzzy edge pixels also includes training the shared point classifier by random point sampling.

[0037] The beneficial effects of the present invention are as follows: by relaxing the labels of edge pixels, the loss is calculated by replacing the original pixel labels with the intersection of the labels of the surrounding pixels when calculating the pixel labels, and on this basis, the edge loss and cross entropy loss are proposed to supervise the training process of the model, and at the same time, uncertain edge pixels are selected in the sampling prediction stage of the model and the selected predicted edge points are classified by the shared point classifier to determine the edge pixel labels on the original label map after bilinear interpolation, so as to solve the problem that the edges of semantic segmentation predictions are not fine enough, complete the entire process of semantic segmentation of traditional cultural images, and obtain the prediction results of fine edges. For the predicted label maps of different other semantic segmentations, the relatively fine predicted object edges are basically retained, which can eliminate the problems of artificial negligence when annotating data and the difficulty of labeling caused by edge blur, and has good versatility. Therefore, for traditional cultural images with complex edges, the present invention can achieve semantic segmentation of fine edges through the algorithm of this article. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flowchart of a traditional cultural image semantic segmentation method based on edge prediction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0041] According to an embodiment of the present invention, a traditional cultural image semantic segmentation method based on edge prediction is provided. The principle of the method is: first, the label of the pixel point is modified by the label relaxation method, and on this basis, the edge loss function and the original cross entropy loss function are proposed to jointly supervise the training of the model. In the prediction process, the edge prediction module is used to select the fuzzy edge points of uncertain pixels, and the point predictor is used for classification to obtain a predicted image with fine edges.

[0042] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a method for semantic segmentation of traditional cultural images based on edge prediction comprises the following steps:

[0043] S1. Use the label relaxation method to relax the labels of the pixels of the image, and replace the label of the pixel with the intersection of the pixels around the pixel when calculating the label;

[0044] The step of performing label relaxation processing on pixels of an image by using a label relaxation method and replacing pixel labels comprises the following steps:

[0045] S11. During the training process, the label relaxation method is used to relax the labels of edge pixels of the image, where edge pixels indicate that the pixels adjacent to a pixel have different labels, and for each pixel, the pixels in the eight neighborhoods around it are selected as a label relaxation range;

[0046] S12. Replace the label of the initially labeled pixel with the intersection of all labels within the relaxed range, and this relaxation step will not affect the pixel points whose surrounding pixel labels are the same as the pixel label.

[0047] The labels of the initially labeled pixels are obtained by manual labeling, and for the training set to be manually labeled, the labels of edge pixels are modified by using a label relaxation method to solve the problem of difficulty in accurate pixel-level labeling during the labeling process caused by blurred object edges.

[0048] S2. Based on label relaxation, calculate the maximum likelihood estimate of the pixel labels of the annotated image, define the edge loss, and use the edge loss and initial cross entropy loss to supervise the model training;

[0049] The step of calculating the maximum likelihood estimate of the pixel labels of the annotated image, defining the edge loss, and using the edge loss and the initial cross entropy loss to perform supervised training on the model includes the following steps:

[0050] S21. The relaxed label map is the intersection of each pixel's own label and the labels of the pixels in the eight surrounding areas; specifically, the label map represents the image after the image is annotated at the pixel level.

[0051] S22. Calculate the maximum likelihood estimate of the pixel label of the labeled image for the label, and define the edge loss L on this basis boundary ;

[0052] Specifically, the step of calculating the maximum likelihood estimate of the pixel labels of the labeled image and defining the edge loss includes the following steps:

[0053] Define the surrounding and self categories of an edge pixel to be classified as A and B respectively;

[0054] Since it is different from the conventional calculation of the maximum likelihood estimate of the pixel labels of the labeled image, in this embodiment, the maximum likelihood estimate of the pixel labels of the labeled image is replaced by calculating the A∪B category. Since A and B are mutually exclusive, the calculation formula is as follows:

[0055] P(A∪B)=P(A)+P(B);

[0056] Among them, P(*) represents the softmax calculation of the probability of each category;

[0057] Define N as a 3*3 window around a pixel, and define the edge loss as:

[0058]

[0059] In the formula, L is the loss, boundary is the edge, N is a natural number set, and C is the union of the label of any point and the labels of the surrounding eight neighborhoods. When |C|=1, the edge loss L boundary Degenerates to the standard one-hot label cross entropy loss.

[0060] S23, the edge loss L boundary With the initial cross entropy loss L ce The two parts are combined to form the final cost loss function L, with the weight coefficients of the two parts being equal, and the cost loss function L is used in the training supervision process of the model.

[0061] By mixing cross entropy loss and margin loss, the model training process can be better supervised.

[0062] S3. In the process of upsampling prediction of the model, an edge prediction method is used to determine the category of the fuzzy edge pixel points, and a plurality of fuzzy edge pixel points with uncertain labels are selected after bilinear interpolation upsampling of the feature map after convolution compression;

[0063] The method of determining the category of fuzzy edge pixels using an edge prediction method and selecting a plurality of fuzzy edge pixels with uncertain labels in the model upsampling prediction process includes the following steps:

[0064] S31, in the process of model upsampling prediction, using edge prediction method to determine the category of blurred edge pixel points;

[0065] Specifically, the edge prediction method constructs features for uncertain points in the image, integrates their semantic features and low-level visual features, and inputs the constructed features into a point classifier for classification;

[0066] S32. After the rough prediction image is subjected to bilinear interpolation, a predetermined point selection strategy is used to select a plurality of fuzzy edge pixel points with uncertain labels.

[0067] Specifically, since upsampling is an iterative process, the prediction map at the beginning of each iteration is regarded as a coarse prediction map, and a label prediction map with the same size as the input image is generated at the end of the iteration;

[0068] The predetermined point selection strategy is to obtain the probability of each fuzzy edge pixel for each category in the prediction tensor, perform logarithmic operations, and select the highest two items to calculate the difference. Generally speaking, the higher the difference, the more certain the label of the fuzzy edge pixel belongs to a certain category. This method can be used to screen the uncertainty of the fuzzy edge pixel points, so as to quantify the degree of blurriness of the edge points and perform edge prediction. The number of selected points can be determined by setting a threshold or by limiting the ratio according to the number of pixels in the image. This method selects a certain number of fuzzy edge points with uncertain labels.

[0069] S4, using a shared point classifier to classify the fuzzy edge pixel points with uncertain labels, and determine the specific labels of the fuzzy edge pixel points;

[0070] Among them, after selecting uncertain pixels, a shared point classifier is used to determine the pixel category. The shared point classifier is a multi-layer perceptron (MLP) with three layers, wherein the first layer is the input layer, the second layer is the hidden layer, and the third layer is the output layer, which is used to output the category of the predicted point, and different layers are associated in a fully connected manner.

[0071] In the S4, random sampling is adopted to train the MLP during the training process, and the difficult problem of edge pixel prediction for semantic segmentation is simplified to a classification problem. A three-layer multi-layer perceptron is used to predict fuzzy edge pixels.

[0072] S5. Mix the predicted category of the blurred edge pixel points and the label prediction map after bilinear interpolation to obtain the final fine edge prediction map.

[0073] Among them, the backbone network obtains the heat map by replacing the last fully connected layer with a convolutional layer, and restores the resolution to the original input size through bilinear interpolation or deconvolution, which is the label prediction map.

[0074] In summary, with the help of the above technical solution of the present invention, by relaxing the labels of edge pixels, the loss is calculated by replacing the original pixel labels with the intersection of the labels of the surrounding pixels when calculating the pixel labels, and on this basis, it is proposed that the edge loss and cross entropy loss are combined to supervise the training process of the model, and at the same time, in the sampling prediction stage of the model, uncertain edge pixels are selected and the selected predicted edge points are classified by the shared point classifier, and the labels of the edge pixels on the original label map after bilinear interpolation are determined, so as to solve the problem that the edges of semantic segmentation prediction are not fine enough, complete the entire process of semantic segmentation of traditional cultural images, and obtain the prediction results of fine edges. For the predicted label maps of different other semantic segmentations, the relatively fine predicted object edges are basically retained, which can eliminate the problems of artificial negligence when annotating data and the difficulty of labeling caused by edge blur, and has good versatility. Therefore, for traditional cultural images with complex edges, the present invention can achieve semantic segmentation of fine edges through the algorithm of this article.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A traditional cultural image semantic segmentation method based on edge prediction, characterized in that: The method comprises the following steps: S1, using the label relaxation method to relax the pixels of the image and replace the pixel labels; S2. Calculate the maximum likelihood estimate of the pixel labels of the annotated image, define the edge loss, and use the edge loss and the initial cross entropy loss to supervise the model training; S3. In the process of upsampling prediction of the model, the category of the fuzzy edge pixel point is determined by using the edge prediction method, and a plurality of fuzzy edge pixel points with uncertain labels are selected; S4, using a shared point classifier to classify the fuzzy edge pixel points with uncertain labels, and determine the specific labels of the fuzzy edge pixel points; S5. Mix the predicted category of the blurred edge pixel points and the label prediction map after bilinear interpolation to obtain the final fine edge prediction map.

2. According to the method of traditional cultural image semantic segmentation based on edge prediction in claim 1, it is characterized in that: The method of using the label relaxation method to relax the pixels of the image and replace the pixel labels includes the following steps: S11. During the training process, a label relaxation method is used to relax and modify the label range of edge pixels of the image: pixels in eight neighborhoods around each pixel are selected as the label relaxation range of the pixel; S12. Replace the labels of the initially marked pixels with the intersection of all labels within the relaxed range.

3. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 2, characterized in that: The edge pixel indicates that the pixels adjacent to a pixel have different labels, and the label relaxation method will not affect the pixels whose surrounding pixel labels are the same as the pixel label.

4. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 2, characterized in that: The labels of the initially labeled pixels are obtained by manual labeling, and for the training set to be manually labeled, the labels of edge pixels are modified by using a label relaxation method to solve the problem of difficulty in accurate pixel-level labeling during the labeling process caused by blurred object edges.

5. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 1, characterized in that: The method of calculating the maximum likelihood estimate of the pixel labels of the labeled image, defining the edge loss, and using the edge loss and the initial cross entropy loss to supervise the model training includes the following steps: S21, define the intersection of each pixel's own label and the labels of its eight neighboring pixels as the relaxed label graph; S22, calculating the maximum likelihood estimate of the pixel labels of the annotated image and defining the edge loss; S23. Combine the edge loss with the initial cross entropy loss to form the final cost loss function, and use the cost loss function for model training supervision.

6. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 5, characterized in that: The method of calculating the maximum likelihood estimate of the pixel labels of the labeled image and defining the edge loss includes the following steps: Define the surrounding and self categories of an edge pixel to be classified as A and B respectively; By calculating the A∪B category instead of calculating the maximum likelihood estimate of the annotated image pixel label, the calculation formula is as follows: P(A∪B)=P(A)+P(B); Among them, P(*) represents the softmax calculation of the probability of each category; Define N as a 3*3 window around a pixel, and define the edge loss as: In the formula, L is the loss, boundary is the edge, N is a natural number set, and C is the union of the label of any point and the labels of the surrounding eight neighborhoods. When |C|=1, the edge loss L boundary Degenerates to the standard one-hot label cross entropy loss.

7. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 1, characterized in that: In the model upsampling prediction process, the edge prediction method is used to determine the category of the fuzzy edge pixel points, and multiple fuzzy edge pixel points with uncertain labels are selected, which includes the following steps: S31, in the process of model upsampling prediction, using edge prediction method to determine the category of blurred edge pixel points; S32. After the rough prediction image is subjected to bilinear interpolation, a predetermined point selection strategy is used to select a plurality of fuzzy edge pixel points with uncertain labels.

8. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 7, characterized in that: The predetermined point selection strategy is to obtain the probability of each fuzzy edge pixel point for each category in the prediction tensor, perform logarithmic operation, and select the two highest items to calculate the difference, wherein the higher the difference, the greater the certainty that the label of the fuzzy edge pixel point belongs to a certain category. This method can realize the screening of the uncertainty of the fuzzy edge pixels.

9. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 1, characterized in that: The shared point classifier is a three-layer perceptron, wherein the first layer is an input layer, the second layer is a hidden layer, and the third layer is an output layer, which is used to output the category of the predicted point, and different layers are associated in a fully connected manner.

10. The method for semantic segmentation of traditional cultural images based on edge prediction according to claim 1, characterized in that: The method of using a shared point classifier to classify fuzzy edge pixels with uncertain labels and determining specific labels of the fuzzy edge pixels also includes training the shared point classifier by random point sampling.