Posterior probability calibration method of semantic segmentation model based on multi-scale temperature scaling

Abstract

The invention provides a semantic segmentation model calibration method based on multi-scale temperature scaling, and particularly relates to fusion of semantic information content and realization of accurate calibration through a multi-scale temperature scaling technology. The method comprises the following steps: firstly, inputting batch samples in a training set into a preset semantic segmentation model to obtain an image segmentation result and an original posterior probability of each pixel point; according to the calibrated posterior probability of the sample, constructing a probability loss function to calculate a loss value; and according to the loss value, back propagation is carried out to adjust model parameters, and model training optimization is realized. Through the method, the model can output the posterior probability more consistent with the real situation, the phenomenon of blind confidence is effectively avoided, the probability accuracy of semantic segmentation is remarkably improved, and the method has wide application prospects in the fields of road scene understanding of intelligent driving, lesion region segmentation of medical images and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic segmentation, and particularly relates to a posterior probability calibration method of a semantic segmentation model based on multi-scale temperature scaling, aiming to improve the accuracy and reliability of the prediction results of the semantic segmentation model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, semantic segmentation has been widely applied in many fields such as intelligent driving, medical image analysis, image editing, etc. The goal of semantic segmentation model is to assign a class label to each pixel in an image, for example, in the automatic driving scenario, different regions such as roads, vehicles, pedestrians, etc. are accurately segmented. However, although the existing semantic segmentation model has achieved certain results in accuracy, the posterior probability output by the model often deviates from the actual situation.

[0003] In practical applications, the model may have a "overconfidence" situation, that is, the prediction result gives a too high probability estimate, but the actual prediction accuracy is not ideal. This mismatch between probability and actual accuracy can have a serious impact on subsequent decision-making. For example, in medical images, if the model incorrectly segments the lesion area and gives a high confidence result, it may lead to incorrect diagnosis by doctors; in automatic driving, the incorrect segmentation and high confidence judgment of the road scene may cause traffic accidents. Therefore, it is of great significance to accurately calibrate the posterior probability of the semantic segmentation model. SUMMARY

[0004] The purpose of the present application is to provide a posterior probability calibration method of a semantic segmentation model based on multi-scale temperature scaling, which fuses multi-scale temperature scaling technology of semantic information and a series of statistical analysis and optimization operations, so that the posterior probability output by the model is closer to the real situation, and the reliability of the model in practical applications is improved. The technical solution is as follows: A posterior probability calibration method of a semantic segmentation model based on multi-scale temperature scaling, comprising the following steps: A batch of samples in the training set is input into a preset semantic segmentation model. The semantic segmentation model processes the input samples and outputs the segmentation results of the image and the original posterior probability of each pixel. The original posterior probability here is the initial estimate of the possibility of each pixel belonging to each class based on the parameters and learned feature representation inside the model. For example, for an image containing multiple objects, the model will calculate the original posterior probability of each pixel in the image belonging to different object classes (such as people, buildings, background, etc.); According to the original posterior probability, the samples are divided into several probability intervals. The probability intervals are divided in an equidistant manner, for example, divided into [0, 0.1), [0.1, 0.2), …, [0.9, 1] intervals, and the boundary values of each interval can also be dynamically adjusted according to the distribution characteristics of the samples. For each probability interval, the confidence of each interval (i.e. the mean of the original posterior probability) is calculated, and the number of samples of different prediction categories in the interval and the number of correctly predicted samples are counted, which are used to calculate the accuracy of each interval (i.e. the percentage of the actual correct sample amount to the total sample amount); The confidence in each interval is calculated, and the calculation formula is: ; Where conf(bm) represents the confidence of the set, |bm| represents the number of elements in the set, represents the sum of the probability values corresponding to each element in the set. The confidence reflects the subjective judgment of the model's prediction at different probability levels; The accuracy in each interval is calculated, and the calculation formula is: ; Where acc(bm) represents the accuracy of the set, |bm| represents the number of elements in the set, represents the sum of the probability values corresponding to each element in the set, and l is an indicator function that takes the value 1 when the predicted value equals the true value, and 0 otherwise. The accuracy reflects the actual situation of the model's prediction at different probability levels. Combining the confidence and the accuracy, training data can be provided for the subsequent probability calibration model; A temperature scaling factor (temperature scaling factor) is introduced through the probability calibration model, denoted as T. The original posterior probability is adjusted using the temperature scaling factor, and the adjustment formula is: ; Where T is a vector of the same size as the input image, and each point is a learnable parameter greater than 0, and z represents the probability prediction vector. When T>1, the temperature scaling will smooth the output of the softmax, thereby reducing the confidence of the prediction and alleviating the problem of overconfidence of the model; on the contrary, when 0<T<1, the output of the softmax will become more and more sharp, and the confidence of the prediction will tend to 1, which will help to alleviate the problem of lack of confidence in prediction. In addition, since T is a parameter greater than zero, the order between the dimensions of the vector will not change after the temperature scaling transformation, so the model's prediction will not change, and therefore using temperature scaling technology to correct the confidence will not affect the classification performance of the model; Based on the calibrated posterior probabilities, the samples are divided into several probability intervals. The probability intervals are divided using equal intervals, such as [0, 0.1), [0.1, 0.2), …, [0.9, 1], etc. The boundary values ​​of each interval can also be dynamically adjusted according to the distribution characteristics of the samples. For each probability interval, the number of samples in different predicted categories and the number of correctly predicted samples within that interval are counted. A loss function is constructed based on the calibrated posterior probabilities of the samples. According to the predicted posterior probability corresponding to each sample, the probability loss value for each batch size is calculated using the loss function. The loss function can take the following form: ; Where L is the loss function for model training, N is the number of pixels in the training image, and C are the categories. SM represents the softmax operation on the features, z represents the probability prediction vector, and T is the temperature parameter. By minimizing this loss function, a suitable temperature parameter T can be found, making the calibrated probability distribution closer to the true probability distribution, thereby improving the calibration performance of the model. Based on the calculated loss value, the parameters are adjusted in the calibration model using the backpropagation algorithm. Gradient descent or its variants (such as stochastic gradient descent, Adagrad, Adadelta, etc.) can be used in the calibration model to update the calibration model parameters according to the probabilistic loss value. Through multiple iterations of training, the calibration model parameters are continuously optimized, ensuring that the adjustment amount output by the calibration model effectively corrects the posterior probability output by the semantic segmentation model. This results in the final posterior probability being closer to the real situation, improving the accuracy of semantic segmentation.

[0005] Therefore, the embodiments of the present invention enable the predicted posterior probability output by the semantic segmentation model to match the true confidence level. This allows the model to calibrate the posterior probability after outputting the predicted category of a sample, thereby matching the prediction accuracy of the model with the output confidence level. This avoids the situation of "blind confidence" in the model and improves the prediction accuracy of the model. Attached Figure Description

[0006] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0007] Figure 1 This is a schematic diagram of the overall training process of the semantic segmentation model and calibration model provided by the present invention; Figure 2This is a schematic diagram of the probabilistic calibration model training process provided in an embodiment of the present invention; Figure 3 This is the dilated convolutional pooling pyramid structure used for extracting multi-scale features in the probability calibration model provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the structure of a specific application of the probability calibration model provided by the present invention; Detailed Implementation

[0008] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0009] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0010] Reference Figure 1 , Figure 1 This invention illustrates a schematic diagram of the collaborative application between the semantic segmentation model and the calibration model, including: S100: Input the sample into the trained semantic segmentation model to obtain the predicted semantic segmentation image and the predicted posterior probability result. The training set and validation set include multiple image datasets and corresponding label maps. The predicted posterior probability represents the probability that the semantic segmentation model will correctly predict the sample at the corresponding pixel.

[0011] It should be noted that the samples in the embodiments of this invention can be understood as image samples; the architecture of the semantic segmentation model is not limited to the model architecture used in the embodiments of this invention, and other semantic segmentation models are also applicable to the method of this invention. Furthermore, the dataset of the semantic segmentation model should be diverse and representative, covering various situations such as different scenes, different lighting conditions, and different object shapes, to ensure the model's generalization ability. After image input, the semantic segmentation model, based on its complex internal neural network architecture, performs in-depth analysis and processing of multi-dimensional data such as pixel information, texture features, and color distribution of the image, thereby obtaining a predicted semantic segmentation image that clearly presents the division of different semantic regions. Simultaneously, based on its probability calculation submodule, the semantic segmentation model integrates various feature information of the image and the model's decision-making process information to calculate the predicted posterior probability result. This result, in numerical form, represents the confidence level of each semantic classification in the current image.

[0012] S101, the image and the predicted posterior probability results are used as samples to train the probability calibration model. The evaluation criterion for the model's performance is the difference between the predicted posterior probability and the actual posterior probability.

[0013] It should be noted that the difference between the predicted posterior probability and the actual posterior probability here is represented by the average value after dividing the interval, i.e., confidence and accuracy.

[0014] S102 utilizes the trained semantic segmentation model and probability calibration model to perform semantic segmentation and probability calibration.

[0015] Specifically, the computer uses a semantic segmentation model to segment the input image and obtains the predicted posterior probability while outputting the semantic segmentation result. By combining the predicted posterior probability with the original image, the posterior probability is calibrated to obtain a predicted posterior probability result that is closer to the actual result. This improves the mismatch between the model's prediction accuracy and confidence, thereby avoiding the current situation of "blind confidence".

[0016] In one specific embodiment, the process of performing step S101 may specifically include the following steps: S200: Construct an image semantic segmentation model. The model uses Deeplabv3+ and is trained using ResNet50 as the backbone network. Based on the model, the predicted output and predicted posterior probability are obtained.

[0017] S201, input the image and the predicted posterior probability results into the probability calibration model, and use the multi-scale features of the original image and the predicted posterior probability to obtain the calibrated predicted posterior probability.

[0018] It is important to note that in the initial stage of model training, the model parameters are initialized. During the training process, the parameters are iteratively updated through backpropagation to obtain the optimal model.

[0019] S202 utilizes a temperature factor T, calibrated predicted posterior probabilities, and empirical posterior probabilities to construct a loss function, training a probability calibration model. In semantic segmentation tasks, models typically use multi-class cross-entropy loss for training. Even when the segmentation error has been reduced to zero, the multi-class cross-entropy loss may still be positive. This is because the optimization process continues to push the predicted probabilities towards the extreme value of 1, attempting to further reduce the loss to zero. This leads to the model being overconfident in its own predictions, i.e., overfitting. To eliminate this incorrect calibration, the predicted distribution should be guided to have higher entropy to avoid overconfident predictions and improve calibration quality. In other words, the probability distribution given by the model should be smoother, reflecting more uncertainty. Therefore, using a temperature scaling method to avoid the tendency of entropy to minimize can effectively avoid the situation of "blind confidence."

[0020] Temperature scaling techniques aim to make the model's predicted probabilities closer to the true empirical posterior probabilities by avoiding the tendency of entropy to minimize. Based on the predicted posterior probability for each sample, a loss function is used to calculate the loss value for each batch size, and the model is trained and calibrated based on this loss value. This process is repeated until the difference between the model's confidence and accuracy continuously decreases. Finally, the probabilistic model with the best calibration effect is obtained.

[0021] The loss function is: ; By adjusting the temperature factor T, the probability distribution of the model output was modified, which improved the model's ability to express its own predictive uncertainty, thus achieving better probabilistic accuracy.

[0022] S203. Based on the predicted posterior probability of each sample, divide each pixel into the corresponding equidistant probability interval. For each probability interval, count the number of samples of different predicted categories and the number of correctly predicted samples within that interval.

[0023] As an example, the spacing between the equidistant intervals is set to 0.1, that is, 0-1 is divided into ten equal parts. Each probability interval has an equal length of 0.1. Therefore, the probability intervals in this embodiment include: a first probability interval corresponding to [0, 0.1]; a second probability interval corresponding to [0.1, 0.2); a third probability interval corresponding to [0.2, 0.3); a fourth probability interval corresponding to [0.3, 0.4); a fifth probability interval corresponding to [0.4, 0.5]; a sixth probability interval corresponding to [0.5, 0.6]; a seventh probability interval corresponding to [0.6, 0.7); an eighth probability interval corresponding to [0.7, 0.8); a ninth probability interval corresponding to [0.8, 0.9); and a tenth probability interval corresponding to [0.9, 1]. If the predicted posterior probability of sample A is 0.19 and the predicted posterior probability of sample B is 0.39, then sample A falls into the second probability interval, and sample B falls into the fourth probability interval.

[0024] It should be noted that the length of the probability interval can be adjusted as needed. For example, in one embodiment of the present invention, the length of each probability interval is equal, while in another embodiment of the present invention, the length of each probability interval is unequal. Furthermore, the samples A and B mentioned above specifically refer to a pixel in an image.

[0025] S204: Count the number of pixels in each interval and the number of correctly predicted pixels, calculate the accuracy and confidence, and repeat the training to obtain the model with the best calibration effect.

[0026] Let's continue with the example of a 0.1 interval. If 10 samples fall within the fourth probability interval, their predicted probabilities are 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, and 0.39, respectively, with 2 samples correctly predicted. Therefore, the confidence level for the fourth probability interval is 0.345, corresponding to the average of its predicted probabilities, and the accuracy is 0.2. In this case, the model exhibits "blind confidence." We calculate the difference between the accuracy and confidence level each time. When the difference is less than the difference of the previous optimal calibration model, we update the model parameters. We repeat steps S202-S204 until the optimal calibration model is obtained.

[0027] Specifically, the multi-scale feature extraction process described in S201 is as follows: Figure 3 As shown: S300 inputs the feature map into a dilated convolutional pooling pyramid structure and processes it in five different paths to extract multi-scale feature maps.

[0028] Specifically, the first path uses a dilated convolution with a dilation rate of 1. This convolution extracts features with a relatively small receptive field, focusing primarily on local details and preserving the original spatial resolution and number of channels of the input feature map, outputting a feature map with specific features. The second path uses a dilated convolution with a dilation rate of 6, which has a larger receptive field than the convolution with a dilation rate of 1, capturing a wider range of contextual information, and also outputting a corresponding feature map with the same number of channels as the input feature map. The third path uses a dilated convolution with a dilation rate of 8, further expanding the receptive field and acquiring a wider range of spatial information, while the generated feature map also maintains the same number of channels as the input. The fourth path uses a dilated convolution with a dilation rate of 12, which has a larger receptive field, helping to extract more macroscopic features, while the number of channels in the output feature map remains unchanged. The fifth path is a pooling layer, which downsamples the input feature map through pooling operations to extract global features, resulting in a smaller output feature map size, but the number of channels may vary depending on the pooling method.

[0029] It should be noted that the dilated convolutional combinations with dilation rates of 1, 6, 8, and 12 mentioned in this patent, along with the path settings including pooling layers, represent a specific multi-scale feature extraction method. If other dilation rate combinations are used in practical applications, but the functions and effects achieved by these combinations are essentially the same as the structure described in this patent—that is, they can extract multi-scale features in a similar manner and ultimately improve the performance and accuracy of the model at different scales—then they should be considered equivalent to the method protected by this patent and remain within the scope of protection of this patent.

[0030] S301, the feature maps output from the five paths of the above-mentioned dilated convolution pooling pyramid are spliced ​​together along the channel dimension.

[0031] Specifically, if each feature map has C channels, then the number of channels in the concatenated feature map will become 5C, thereby integrating feature information at different scales into one feature map to form a comprehensive feature representation containing multi-scale information.

[0032] For channel restoration, this patent requires that the number of channels after concatenation (e.g., 5C) be restored to a specific number of channels C through operations such as convolution to meet the requirements of subsequent processing or models, while maintaining the fusion effect of multi-scale information. In actual implementation, regardless of the specific parameters such as kernel size, stride, and padding used, as long as the number of channels can be restored from 5C to C without destroying the integrity and effectiveness of multi-scale features during the restoration process, that is, the restored feature map can achieve similar effects as the method described in this patent in improving the performance and accuracy of the model at different scales when used for subsequent probability calibration tasks, then such a channel restoration method should be considered to meet the requirements of this patent.

[0033] S302, perform channel restoration operation on the feature map with 5C channels after splicing.

[0034] By performing operations such as convolution, the number of channels is reduced from 5C to a specific number C to adapt to the requirements of subsequent processing or models, while maintaining the fusion effect of multi-scale information. The resulting processed feature map can be used for subsequent probability calibration tasks to improve the performance and accuracy of the model at different scales.

[0035] For channel restoration, this patent requires that the number of channels after concatenation (e.g., 5C) be restored to a specific number of channels C through operations such as convolution to meet the requirements of subsequent processing or models, while maintaining the fusion effect of multi-scale information. In actual implementation, regardless of the specific parameters such as kernel size, stride, and padding used, as long as the number of channels can be restored from 5C to C without destroying the integrity and effectiveness of multi-scale features during the restoration process, that is, the restored feature map can achieve similar effects as the method described in this patent in improving the performance and accuracy of the model at different scales when used for subsequent probability calibration tasks, then such a channel restoration method should be considered to meet the requirements of this patent.

[0036] Reference Figure 4 , Figure 4 A schematic diagram illustrating a specific application of the probability calibration model provided in this embodiment of the invention is shown below: S400: Input the image into the semantic segmentation model to obtain the predicted semantic segmentation image and the predicted posterior probability result.

[0037] It should be noted that after completing image segmentation, the semantic segmentation model, using its built-in probability generation mechanism, directly evaluates and calculates the likelihood of each semantic category in the current image, outputting the predicted posterior probability result. This probability generation mechanism is based on the overall network structure and parameter configuration of the model, comprehensively considering multi-dimensional data such as image pixel information, texture features, and color distribution, as well as intermediate result information from the model's classification decision process. This results in a numerical representation of the confidence level of each semantic category in the current image, providing a quantitative basis for subsequent analysis and judgment to determine the reliability of each semantic category.

[0038] S401, input the image and the predicted posterior probability as samples into the probability calibration model to obtain the calibrated posterior probability result.

[0039] When the probabilistic calibration model receives the original image and the predicted posterior probabilities from the semantic segmentation model, it performs remapping and optimization adjustments on the input data based on the calibration algorithm and parameter configuration determined during its pre-training, thereby generating the calibrated posterior probability result. This result can more accurately reflect the true confidence level of image semantic classification, providing more reliable data support for subsequent decision-making based on image semantic segmentation results.

[0040] The embodiments of the present invention enable the predicted posterior probability output by the classification model to gradually approach the true confidence level. Thus, after training, the predicted posterior probability output by the model matches the true confidence level. Consequently, when the model outputs the predicted category of a sample, the output confidence level matches the actual prediction of the model. That is, the prediction accuracy of the model matches the output confidence level, thereby avoiding the situation of "blind confidence" of the model.

[0041] Furthermore, if the model training method for posterior probability calibration proposed in this invention is applied to medical image segmentation and autonomous driving, once the confidence level output by the classification model matches the actual situation, when the computer device detects that the confidence level of a target to be detected is too low using the classification model, it will directly ignore the prediction result of the classification model and prompt the computer device to contact a human doctor to conduct a second detection, or to switch the autonomous driving system to manual driving mode, thereby preventing misjudgment.

[0042] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation model calibration method based on multi-scale temperature scaling, characterized in that, The method includes: A batch of samples from the training set are input into a preset semantic segmentation model to obtain the image segmentation results corresponding to the samples and the original posterior probability of each pixel. The original posterior probability represents the probability that the semantic segmentation model predicts that the pixel belongs to each category. The original posterior probability and the sample are input into the probability calibration model to obtain a temperature scaling factor of the same size. The original posterior probability is then adjusted using the temperature scaling factor to obtain the calibrated posterior probability. Based on the calibrated posterior probabilities of the samples, a probability loss function is constructed to calculate the probability loss value. Simultaneously, based on the calibrated posterior probabilities, the samples are divided into several probability intervals. The total number of samples and the number of correctly predicted samples in each probability interval are counted. The empirical posterior probability of each category in each interval is calculated, whereby the empirical posterior probability represents the proportion of correctly predicted samples of each category in each probability interval relative to the total number of samples in that interval. The parameters of the probability calibration model are adjusted by backpropagation based on the probability loss value to achieve model training optimization, obtain the final probability calibration model, and use the model to perform probability calibration on the semantic segmentation model. Semantic segmentation of images is performed using a semantic segmentation model combined with a probabilistic calibration model.

2. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, The temperature scaling factor is a real number greater than 0. Adjusting the value of the temperature scaling factor changes the distribution of the original posterior probability.

3. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, The process of dividing the sample into several probability intervals includes dividing the probability intervals in a manner that is equidistant or non-equidistant, and the boundary value of each probability interval is determined according to the distribution characteristics of the sample.

4. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, During the training of the probabilistic calibration model, image features from the original image were incorporated, and dilated convolution was used to generate contextual information at different scales.

5. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, The probabilistic loss function includes a loss term based on the difference between the calibrated predicted posterior probability and the label, and the model is optimized by minimizing the loss function.

6. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, It also includes normalizing the original posterior probability and the calibrated posterior probability before calculating the loss function, so that the probability values ​​are within a reasonable range.

7. The semantic segmentation model calibration method based on multi-scale temperature scaling according to claim 1, characterized in that, The model training and optimization process also includes using gradient descent algorithm or its variants to adjust model parameters based on probability loss values ​​in order to improve the accuracy of model predictions.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The computer program, when running on the processor, executes the semantic segmentation model calibration method based on multi-scale temperature scaling as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a processor, executes the semantic segmentation model calibration method based on multi-scale temperature scaling as described in any one of claims 1-7.

Images