Method and device for improving robustness of segmentation network and storage medium

Abstract

The application discloses a method and device for improving the robustness of a segmentation network and a storage medium. The method comprises the following steps: obtaining a segmentation network and a test data set of the segmentation network; inputting a test image in the test data set into the segmentation network to obtain a segmentation image; determining a fusion image according to the test image and the segmentation image; inputting the fusion image into a target synthesis network to obtain a synthesis image; iteratively updating network parameters of the segmentation network according to the test image and the synthesis image until the updated segmentation network reaches an optimization target to obtain a target segmentation network. The application generates a corresponding synthesis image through the test image and the segmentation image, enables the segmentation network to accurately recognize its segmentation errors based on the test image and the synthesis image, and adaptively updates the network parameters. The application solves the problem that the existing method for solving the gray distribution drift is difficult to enable the segmentation network to accurately recognize the segmentation errors and adaptively correct, thereby improving the robustness of the segmentation network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep segmentation model optimization, and particularly relates to a method and device for improving robustness of a segmentation network and a storage medium. BACKGROUND

[0002] In recent years, deep learning has achieved very good results in segmentation, classification, detection and other tasks. However, the current deep learning-based model has a very high degree of dependence on data. If data from different distributions is used in the training and testing stages of the deep model, it may cause the performance of the deep model to decrease significantly. This "distribution drift" problem is particularly typical in the clinical environment of medical images, because different hospitals, equipment, models, operating doctors, imaging protocols and other reasons will cause the distribution of the sampled data to be different.

[0003] For the research and analysis of medical images, automatic and accurate segmentation of key anatomical structures based on deep models has important clinical significance for subsequent diagnosis. In order to achieve accurate segmentation of key structures of images with different distributions, the above-mentioned challenge of "distribution drift" needs to be solved.

[0004] Currently, the solutions to solve the model performance decline caused by the gray scale distribution drift and improve the model robustness mainly include the following categories: 1) domain adaptation: it usually needs to use labeled data of the source domain (i.e. known data distribution) and unknown domain (i.e. unknown data distribution used in testing) at the same time. Most of the domain adaptation methods are aligning at the image and feature level, or finding feature invariance between different domains and extracting these cross-domain consistent features. However, since the domain adaptation method needs to rely on a large amount of unknown domain data obtained in advance, it is very difficult for medical clinics. 2) Domain generalization: it only needs to use the data of the source domain to train the model, which can be generalized to different unknown distribution of unknown domain data. However, most of the current methods need to use multiple source domain data for training, which is also very difficult to obtain multiple domain data of different distribution in the field of medical images. Some methods use adversarial learning to enhance and expand the data set, generate more data, and try to meet the needs of multiple source domain data as much as possible. However, this method needs to accurately control the way of data expansion, and if the generated data distribution is unreasonable, it will cause the model training difficult and overfitting problem. 3) Test self-adaptation: this method refers to the stage of model testing, which makes a pre-trained model adapt to the gray scale distribution characteristics of each test image. One of the most common methods is test enhancement, that is, different types of data enhancement are performed on the test image, such as rotating different angles (90 degrees, 180 degrees, 270 degrees), and then voting and weighting the output structure to get the final result, which effectively improves the robustness of the network. However, test data enhancement can only be enhanced in a limited number of ways, which makes it difficult for this method to stably cope with unknown distribution of test data. Recently, some self-supervised learning-based methods have appeared and tried to use self-supervised signals to provide important information for model adaptation during testing. However, this self-supervised signal is often not strong enough and not accurate enough to allow the deep model to accurately recognize which part of the segmentation is wrong, so it cannot accurately correct the segmentation result in the testing stage.

[0005] In short, the existing methods for solving gray scale distribution drift are difficult to make the segmentation network accurately recognize the segmentation error and adaptively correct it, resulting in low robustness of the segmentation network.

[0006] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a method and device for improving the robustness of a segmentation network and a storage medium, which aims to solve the problem that the existing methods for solving gray scale distribution drift are difficult to make the segmentation network accurately recognize the segmentation error and adaptively correct it, resulting in low robustness of the segmentation network.

[0008] The technical scheme adopted by the present application to solve the problems is as follows:

[0009] In a first aspect, the embodiments of the present application provide a method for improving robustness of a segmentation network, wherein the method comprises:

[0010] obtaining a segmentation network and a test data set corresponding to the segmentation network, inputting a test image in the test data set into the segmentation network to obtain a segmentation image corresponding to the test image, wherein the segmentation network is trained in advance by a training data set;

[0011] determining a fusion image according to the test image and the segmentation image, inputting the fusion image into a preset target synthesis network to obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is trained in advance by the training data set;

[0012] updating network parameters of the segmentation network according to the test image and the synthesis image, judging whether the updated segmentation network reaches an optimization target, if the updated segmentation network does not reach the optimization target, continuing to perform the step of inputting the test image in the test data set into the segmentation network until the updated segmentation network reaches the optimization target, and obtaining a target segmentation network.

[0013] In a second aspect, the embodiments of the present application further provide an apparatus for improving robustness of a segmentation network, wherein the apparatus comprises:

[0014] an image segmentation module, configured to obtain a segmentation network and a test data set corresponding to the segmentation network, input a test image in the test data set into the segmentation network to obtain a segmentation image corresponding to the test image, wherein the segmentation network is trained in advance by a training data set;

[0015] an image synthesis module, configured to determine a fusion image according to the test image and the segmentation image, input the fusion image into a preset target synthesis network to obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is trained in advance by the training data set;

[0016] an iterative updating module, configured to update network parameters of the segmentation network according to the test image and the synthesis image, judge whether the updated segmentation network reaches an optimization target, if the updated segmentation network does not reach the optimization target, continue to perform the step of inputting the test image in the test data set into the segmentation network until the updated segmentation network reaches the optimization target, and obtain a target segmentation network.

[0017] In a third aspect, the embodiments of the present application further provide a computer readable storage medium having stored thereon a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the steps of any of the above-mentioned methods for improving robustness of a segmentation network.

[0018] The present application has the following beneficial effects: The embodiments of the present application generate a corresponding synthetic image by testing an image and a segmentation image, so that the segmentation network accurately recognizes its segmentation errors based on the testing image and the synthetic image, and adaptively updates the network parameters. The embodiments of the present application solve the problem that the existing methods for solving the gray distribution drift are difficult to make the segmentation network accurately recognize the segmentation errors and adaptively correct, resulting in low robustness of the segmentation network. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 is a flowchart of the method for improving robustness of a segmentation network provided by the embodiments of the present application.

[0021] Figure 2 is a flowchart of the method for updating the network parameters of a segmentation network provided by the embodiments of the present application.

[0022] Figure 3 is a schematic diagram of the principle of generating a segmentation image provided by the embodiments of the present application.

[0023] Figure 4 is a flowchart of the method for generating a synthetic image provided by the embodiments of the present application.

[0024] Figure 5 is a flowchart of the method for determining the similarity data of two images provided by the embodiments of the present application.

[0025] Figure 6 is a schematic diagram of the internal modules of the device for improving robustness of a segmentation network provided by the embodiments of the present application.

[0026] Figure 7 is a schematic diagram of the principle of the terminal provided by the embodiments of the present application. DETAILED DESCRIPTION

[0027] The application discloses a method and device for improving robustness of a segmentation network and a storage medium. To make the purpose, technical solutions and effects of the application clearer and more explicit, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0028] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the use of the term "include" in the specification of the application means that the stated features, integers, steps, operations, elements, and / or components are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any one of the associated listed items and all combinations thereof.

[0029] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0030] In view of the above defects of the prior art, the present application provides a method for improving the robustness of a segmentation network, which comprises the following steps: obtaining a segmentation network and a test data set corresponding to the segmentation network; inputting a test image in the test data set into the segmentation network to obtain a segmentation image corresponding to the test image, wherein the segmentation network is trained in advance by a training data set; determining a fusion image according to the test image and the segmentation image, inputting the fusion image into a preset target synthesis network to obtain a synthesized image corresponding to the fusion image, wherein the target synthesis network is trained in advance by the training data set; updating network parameters of the segmentation network according to the test image and the synthesized image; and judging whether the updated segmentation network reaches an optimization target, if not, continuing to input the test image in the test data set into the segmentation network until the updated segmentation network reaches the optimization target to obtain a target segmentation network. The present application generates a corresponding synthesized image from a test image and a segmentation image, so that the segmentation network can accurately recognize its segmentation errors based on the test image and the synthesized image and adaptively update the network parameters. The present application solves the problem that the existing method for solving the gray distribution drift cannot accurately recognize the segmentation errors and adaptively correct them, resulting in low robustness of the segmentation network.

[0031] As shown in Figure 1 the method comprises the following steps:

[0032] In step S100, a segmentation network and a test data set corresponding to the segmentation network are obtained, and a test image in the test data set is inputted into the segmentation network to obtain a segmentation image corresponding to the test image, wherein the segmentation network is trained in advance by a training data set.

[0033] Specifically, the training data set in the present embodiment is a set corresponding to source domain data, and the data distribution of the source domain data is known; the test data set is a set corresponding to unknown domain data, and the data distribution of the unknown domain data is unknown. The present embodiment obtains a segmentation network by training in advance according to the training data set, and in order to further improve the robustness of the segmentation network, the segmentation network needs to be updated according to the test data set, so the test image needs to be inputted into the segmentation network first, and the segmentation network will determine the segmentation categories corresponding to each local region in the test image according to the inputted test image, thereby outputting a segmentation image corresponding to the test image.

[0034] In an implementation, the training data set and / or the test data set is pre-processed. The data pre-processing includes one or more of normalization processing and standardization processing. The normalization processing refers to dividing all pixels of an image by 255, so that the pixel distribution of the image changes from 0 to 255 to 0 to 1, thereby facilitating learning and convergence of the deep network. The standardization processing refers to first calculating the mean and standard deviation of all gray values of an image, and performing the operation of subtracting the mean and then dividing by the standard deviation on all pixels of the image. After the standardization processing, the image well conforms to the normal distribution, thereby facilitating learning of the network. Since the robustness of the segmentation network is closely related to the generalization ability of the segmentation network to different unknown distribution test data, the data pre-processing is particularly important. For example, when the processing object of the segmentation network is a medical image, due to the fact that in a clinical environment, images obtained by different hospitals, devices, models, operating doctors, imaging protocols, and the like often differ greatly, including appearance differences and internal gray distribution imbalance, etc., if no data pre-processing is performed, there is a risk of serious decline in model performance, poor generalization ability, slow convergence speed, and the like. A suitable data pre-processing method can well reduce the difference between data, and lay a good foundation for subsequent model training and optimization.

[0035] In an implementation, the step S100 specifically includes the following steps:

[0036] Step S101, inputting the test image into the segmentation network, and outputting a plurality of sub-segmentation images by the segmentation network, wherein the plurality of sub-segmentation images respectively correspond to different segmentation categories, and the sub-segmentation image corresponding to each segmentation category is determined based on a channel map and an index value of a channel corresponding to the segmentation category.

[0037] Step S102, determining the segmentation image according to the plurality of sub-segmentation images.

[0038] Specifically, the segmentation network in the embodiment performs multi-category segmentation. After inputting the test image into the segmentation network, the segmentation network outputs a plurality of sub-segmentation images, wherein each sub-segmentation image corresponds to a different channel, each channel corresponds to a different segmentation category / segmentation target, and each channel has a channel index value. In order to preserve gradient information in the image, for each segmentation category, the embodiment needs to obtain a channel map output by a channel corresponding to the segmentation category, and obtain a channel index value of the channel corresponding to the segmentation category, and determine a sub-segmentation image corresponding to the segmentation category according to the channel map and the channel index value. Finally, the segmentation image output by the segmentation network is determined according to the sub-segmentation images of each segmentation category. It can be understood that the segmentation image can reflect the unique segmentation category corresponding to each region in the test image.

[0039] For example, it is assumed that the segmentation network in this embodiment performs k-class segmentation, which includes k-1 foreground target structures to be segmented and 1 background. A k-channel map is obtained for the segmentation output, and the k-channel map corresponds to the k segmentation categories one by one. For each channel map, the following calculation is performed:

[0040]

[0041] where p(i) represents the probability map of the i-th channel, and g(i) represents the index value (i.e., the value of the true label) of the i-th channel. Then, according to all the calculated channel maps, a k-class segmentation heat map Hm with gradient information is obtained (as shown in Figure 3

[0042] It should be noted that in a common segmentation task, the k-channel probability map of the segmentation output is usually obtained by using a non-differentiable argmax operation to obtain the final segmentation result. However, using the non-differentiable argmax operation will cause the segmentation result to lose all gradient information, which is not conducive to adaptive operations and weight parameter updates of the deep network during subsequent method testing. Therefore, in this embodiment, the probability map of each channel is recalculated according to the index value corresponding to each channel, so that the gradient information is retained in the final segmentation result, and the gradient propagation of the entire process is maintained.

[0043] In one implementation manner, the network skeleton corresponding to the segmentation network is one of classical U-net, deeplabv3+, and the like.

[0044] As shown in Figure 1 The method comprises the following steps:

[0045] Step S200, determining a fusion image according to the test image and the segmentation image, inputting the fusion image into a preset target synthesis network to obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is pre-trained by using the training data set.

[0046] ​Specifically, since the data distribution of the training data set is known, and the data distribution of the test data set is unknown, the segmentation network trained on the training data set will usually encounter very great difficulties in processing the test data set, resulting in a serious decline in model performance. Since the current segmentation network can only perform image segmentation based on the input image, it cannot perform deeper reflection learning on the segmentation result, such as recognizing where the segmentation is wrong and where further improvement and optimization are needed. Therefore, the embodiment introduces a target synthesis network. Since the target synthesis network is pre-trained on a training data set with known data distribution, it will synthesize a new synthetic image based on the segmentation image output by the segmentation network, and the places where the segmentation network is wrong will be clearly reflected in the new synthetic image, thereby helping the segmentation network to clearly recognize the places where the segmentation is wrong. In order to provide more abundant information to the target synthesis network to synthesize a new image, the embodiment needs to fuse the test image and the segmentation image to obtain a fused image.

[0047] In an implementation manner, the step S200 specifically includes the following steps:

[0048] The step S201 acquires an edge image corresponding to the test image, wherein the edge image has a greater degree of gray scale change than other images in the test image except the edge image.

[0049] The step S202 fuses the edge image and the segmentation image to obtain the fused image.

[0050] Specifically, the embodiment fuses the edge image corresponding to the test image and the segmentation image to provide more abundant information to the subsequent target synthesis network. The edge image of the test image is an image composed of points with strong changes in brightness in the test image, so the degree of gray scale change of the edge image is greater than that of other images in the test image.

[0051] In an implementation manner, the training method corresponding to the target synthesis network is a synthesis network training method of a generative adversarial network (GAN). Specifically, in the training phase, the untrained target synthesis network will work synchronously through the generator and the discriminator to generate more realistic real images, and the goal is to make the discriminator unable to distinguish between the synthetic image and the real image. In the test phase, the target synthesis network will only use the trained generator to work, and the fused image will be input into the target synthesis network to generate a realistic synthetic image (such as Figure 4

[0052] ​It should be noted that the network structure of the segmentation network and the synthesis network in the embodiment is changeable. Taking the segmentation network as an example, if an updated network structure appears, the existing network structure can be replaced. Similarly, for the synthesis network, a method based on a generative adversarial network can be used to synthesize an image, but other methods such as a transformer can also be used.

[0053] As shown in Figure 1 The method further includes the following steps:

[0054] Step S300, updating the network parameters of the segmentation network according to the test image and the synthesized image, determining whether the updated segmentation network reaches an optimization target, if the updated segmentation network does not reach the optimization target, continuing to perform the step of inputting the test image in the test data set into the segmentation network until the updated segmentation network reaches the optimization target, and obtaining a target segmentation network.

[0055] Specifically, since the test image is from an unknown domain with unknown data distribution, the synthesized image is from a source domain with known data distribution (because the synthesis network is trained on the source domain), and the segmentation error of the segmentation network will be obviously reflected in the synthesized image, therefore, by comparing the test image and the synthesized image, the segmentation network can recognize its segmentation error / failure, thereby accurately guiding the segmentation network to update the network parameters, and iteratively optimizing until the optimization target is reached, i.e. the network parameter updating is stopped, thereby obtaining a target segmentation network with higher robustness. A more accurate segmentation result can be obtained through the target segmentation network.

[0056] In an implementation manner, the optimization target can be a preset number of iterations.

[0057] In an implementation manner, the updating of the network parameters of the segmentation network according to the test image and the synthesized image specifically includes the following steps:

[0058] Step S301, determining similarity data according to the test image and the synthesized image;

[0059] Step S302, determining a loss function value corresponding to the segmentation network according to the similarity data;

[0060] Step S303, updating the network parameters of the segmentation network according to the loss function value.

[0061] Specifically, this embodiment first needs to measure the similarity between the test image and the synthesized image to obtain similarity data. Since this similarity data can reflect the differences between the test image and the synthesized image, the loss function value corresponding to the segmentation network can be calculated using the similarity data. The loss function value is then used to drive the segmentation network to recognize its own segmentation errors / failures.

[0062] In another implementation, step S301 is: determining similarity data based on the weighted test image corresponding to the test image and the weighted composite image corresponding to the composite image, wherein the weighted test image is an image after weighting the test image based on an attention matrix, and the weighted composite image is an image after weighting the composite image based on the attention matrix.

[0063] Specifically, in order to better enable the loss function of the segmentation network to focus on the structural regions to be segmented, and thus better guide the segmentation network to update its parameters, this embodiment introduces an attention matrix. Before calculating similarity data, the test image and the synthesized image are first weighted according to the attention matrix to obtain a weighted test image and a weighted synthesized image. Then, similarity data is calculated based on the weighted test image and the weighted synthesized image, thereby better modeling the structural similarity between the two images and enabling the segmentation network to recognize the segmentation errors.

[0064] In one implementation, the attention matrix is ​​determined based on the segmented image output by the segmentation network. Specifically, the segmented image is normalized to obtain a normalized segmented image, and the attention matrix is ​​determined based on the normalized segmented image.

[0065] For example, such as Figure 5 As shown, the segmented image Hm is first normalized (divided by 255) to obtain Hm. att , Hm att As an attention matrix, weights are applied to the test image A and the synthesized image B: A' = (I + Hm) att )*A,B'=(I+Hm att )*B. Where I is a matrix with all values ​​of 1 and the same size as A and B.

[0066] In one implementation, the method for determining the similarity data corresponding to two images can be performed at the feature level: first, feature extraction is performed on the two images to obtain the image features corresponding to the two images respectively, and then the similarity data is determined based on the image features corresponding to the two images respectively.

[0067] In another implementation, the method for determining the similarity data corresponding to two images can also be performed at the image level, specifically including the following steps:

[0068] Step S3011, determining a cross-correlation coefficient and a cross-information coefficient corresponding to the first image and the second image according to the first image and the second image, wherein the first image is the test image and the second image is the synthetic image, or the first image is the weighted test image and the second image is the weighted synthetic image, the cross-correlation coefficient is used to reflect the gray difference between the first image and the second image, and the cross-information coefficient is used to reflect the information difference between the first image and the second image.

[0069] Step S3012, determining the similarity data according to the cross-correlation coefficient and the cross-information coefficient.

[0070] Specifically, the two images for which the similarity data needs to be calculated in the embodiment correspond to different domain distributions. The conventional pixel-level content similarity calculation method is to subtract the corresponding pixels of the two images directly to determine whether the two images are similar. Therefore, the conventional similarity calculation method is only applicable to processing two images from the same domain and is not applicable to the scenario in the embodiment. For the convenience of description, the two images for which the similarity data needs to be calculated in the embodiment are defined as the first image and the second image. In order to calculate the similarity data corresponding to the first image and the second image with different domain distributions, the embodiment defines the difference between the two images from two angles. The first angle is the gray difference between the two images, that is, the cross-correlation coefficient of the two images is obtained. The higher the cross-correlation coefficient, the smaller the difference in gray scale between the two images. The second angle is the information difference between the two images, that is, the cross-information coefficient corresponding to the two images is obtained. The higher the cross-information coefficient, the smaller the difference in information between the two images. The structural difference between the two images with different domain distributions can be accurately determined through the cross-correlation coefficient and the cross-information coefficient, so as to obtain the similarity data corresponding to the two images.

[0071] In an implementation manner, the method for determining the cross-correlation coefficient specifically includes the following steps.

[0072] Step S30111, determining a plurality of local cross-correlation coefficients according to the first image and the second image, wherein the plurality of local cross-correlation coefficients correspond to different local image pairs respectively, two local images in each local image pair are located in the first image and the second image respectively, and each local cross-correlation coefficient is used to reflect the gray difference between the two local images.

[0073] Step S30112, determining the cross-correlation coefficient according to the plurality of local cross-correlation coefficients.

[0074] Specifically, in order to better balance the gray difference between two images with different domain distributions, the embodiment first divides the first image and the second image into several local images respectively, and matches them into a plurality of local image pairs. By comparing the gray difference of each local image pair, the local cross-correlation coefficient of each local image pair is obtained. Finally, according to the local cross-correlation coefficient corresponding to each local image pair, the gray difference between the first image and the second image is determined, that is, the cross-correlation coefficient is obtained. The embodiment can better enable the network to learn the difference between the first image and the second image locally, and balance the structural difference between the two images by dividing the area to calculate the local cross-correlation coefficient.

[0075] In an implementation manner, each of the local cross-correlation coefficients and the cross-correlation coefficient is calculated as follows:

[0076] The first image A and the second image B are both divided into N pairs of local images with a size of n, and the local cross-correlation coefficient C of each local image pair is calculated n :

[0077]

[0078] Wherein, A(x, y) and B(x, y) represent one of the local images in a local image pair, μ A and μ B are the gray mean values of A and B, σ A and σ B are the gray standard deviations of A and B. Finally, the N pairs of local cross-correlation coefficients are averaged, and the cross-correlation coefficient C corresponding to A and B is obtained:

[0079]

[0080] Wherein, the range of the cross-correlation coefficient is -1 to 1, -1 represents that the two images are completely irrelevant, and 1 represents that the two images are completely consistent.

[0081] In an implementation manner, the determination method of the mutual information coefficient specifically includes the following steps:

[0082] Step S30113, determining a first information amount corresponding to the first image according to the first image;

[0083] Step S30114, obtaining the gray information corresponding to the second image, and determining a second information amount corresponding to the first image according to the gray information corresponding to the second image and the first image;

[0084] Step S30115, determining the mutual information coefficient corresponding to the first image and the second image according to the first information amount and the second information amount.

[0085] In brief, compared with the cross-correlation coefficient, the mutual information coefficient can better reflect the common features between two images. Specifically, first, according to the first image, the information amount of the first image itself (i.e., the information amount of the entropy / gray scale of the image) is determined to obtain a first information amount. At the same time, the information amount of the first image is calculated in the case of obtaining the gray scale information of the second image to obtain a second information amount. It should be noted that if the first image and the second image are completely independent / entirely different, the information amount of the first image will not be reduced after the gray scale information of the second image is known, and the mutual information coefficient calculated according to the first information amount and the second information is the minimum value, for example, 0. Conversely, if the first image and the second image are exactly the same, that is, the first image and the second image are completely overlapped, the mutual information coefficient calculated according to the first information amount and the second information amount is the maximum value. Therefore, the mutual information coefficient can better reflect the information difference between two images with different domain distributions, especially the structural difference degree.

[0086] In an implementation manner, for the first image A and the second image B, the calculation manner of the corresponding mutual information coefficient MI(A, B) is as follows:

[0087] MI(A, B) = H(A) - H(A|B);

[0088] Wherein, H(A) represents the entropy / information amount of the A image, that is, H(A) represents the information amount of the gray scale of the image A. H(A|B) represents the information amount of the image A calculated in the case of knowing the gray scale information of the image B. In other words, MI(A, B) can represent how much the original information amount of A is reduced after knowing the gray scale information of B.

[0089] In an implementation manner, the step S302 specifically includes the following steps:

[0090] Step S3021, determining a cross-correlation loss function value according to the cross-correlation coefficient, wherein the cross-correlation coefficient and the cross-correlation loss function value are in inverse proportion;

[0091] Step S3022, determining a mutual information loss function value according to the mutual information coefficient, wherein the cross-correlation coefficient and the cross-correlation loss function value are in inverse proportion;

[0092] Step S3023, determining the loss function value according to the cross-correlation coefficient and the mutual information coefficient.

[0093] In brief, the loss function value of the segmentation network in the embodiment needs to comprehensively consider the cross-correlation coefficient and the mutual information coefficient. Specifically, the cross-correlation loss function value is determined according to the cross-correlation coefficient, the greater the cross-correlation coefficient, the smaller the cross-correlation loss function value, and vice versa; and the mutual information loss function value is determined according to the mutual information coefficient, the greater the mutual information coefficient, the smaller the mutual information loss function value, and vice versa. The final loss function value of the segmentation network is determined by the cross-correlation loss function value and the mutual information loss function value, which can make the segmentation network model the structural similarity between two images with different domain distributions, so as to make the segmentation network recognize the segmentation error and continuously iterate and optimize itself, finally improve the segmentation result and improve the robustness of the segmentation network as a whole.

[0094] In an implementation manner, the cross-correlation loss function value L ncc is calculated as follows:

[0095] L ncc = 1-C;

[0096] wherein C is the cross-correlation coefficient.

[0097] In an implementation manner, the mutual information loss function value L mi is calculated as follows:

[0098] L mi = 1-MI(A,B);

[0099] wherein MI(A,B) is the mutual information coefficient.

[0100] In an implementation manner, when the similarity data is calculated by using the test image A and the synthetic image B, the loss function value L of the segmentation network is calculated as follows:

[0101] L = L ncc (A,B) + L mi (A,B)

[0102] In another implementation manner, when the similarity data is calculated by using the weighted test image A' and the synthetic image B', the loss function value L of the segmentation network is calculated as follows:

[0103] L = L ncc (A',B') + L mi (A',B')

[0104] Compared with the loss function value calculated by using the test image A and the synthetic image B, the loss function value calculated by using the weighted test image A' and the synthetic image B' can make the segmentation network model the structural similarity between the two images well, so that the segmentation network can recognize the segmentation error and iteratively optimize itself, finally improve the segmentation result and improve the overall robustness of the network.

[0105] In an implementation manner, the method further includes:

[0106] obtaining an initial segmentation image output by the target segmentation network;

[0107] performing a post-processing operation on the initial segmentation image to obtain a target segmentation image corresponding to the target segmentation network, wherein the post-processing operation includes but is not limited to an erosion operation, a dilation operation and a connected domain operation.

[0108] Specifically, the erosion operation refers to "shrinking" the segmentation result (binary). The dilation operation refers to "enlarging" the segmentation result. The connected domain operation generally includes: 1) calculating the number of connected domains, 2) removing connected domains with an area less than a given threshold / retaining the largest connected domain, etc. The post-processing operation on the initial segmentation image output by the target segmentation network in the embodiment can further improve the segmentation performance of the target segmentation network.

[0109] In an implementation manner, the post-processing operation on the initial segmentation image is sequentially performing the erosion operation, the connected domain operation and the dilation operation on the initial segmentation image.

[0110] Specifically, the embodiment first performs the erosion operation on the initial segmentation image, which is beneficial to remove some unreasonable "burr" and the like at the edges of the image, and then performs the connected domain operation and the dilation operation to restore the segmentation result. The target segmentation image obtained after the post-processing operation can be well improved in the whole and the edge.

[0111] The advantages of the present application are:

[0112] 1) The present application proposes a new, general and effective network robustness improvement scheme. Through the online cycle iteration process of segmentation-synthesis-segmentation, the deep model can recognize the segmentation error based on the synthesis result and optimize the segmentation result, effectively solving the problems of distribution drift and performance degradation of the deep model.

[0113] 2) The present application is completely online, only the full-automatic optimization and parameter updating of the model are needed in the test stage, which avoids the difficulty of retraining the model and deploying the model due to the addition of new data.

[0114] 3) The application only needs source domain data when optimizing and updating the segmentation network, does not need to introduce additional target domain data, can directly adapt to data of any unknown distribution on test data, and obtains an optimized segmentation result.

[0115] Based on the above embodiment, the application further provides a device for improving the robustness of a segmentation network, as shown in Figure 6 The device comprises:

[0116] An image segmentation module 01 is configured to obtain a segmentation network and a test data set corresponding to the segmentation network, input a test image in the test data set into the segmentation network, and obtain a segmentation image corresponding to the test image, wherein the segmentation network is pre-trained by a training data set.

[0117] An image synthesis module 02 is configured to determine a fusion image according to the test image and the segmentation image, input the fusion image into a preset target synthesis network, and obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is pre-trained by the training data set.

[0118] An iterative updating module 03 is configured to update network parameters of the segmentation network according to the test image and the synthesis image, determine whether the updated segmentation network reaches an optimization target, continue to input the test image in the test data set into the segmentation network if the updated segmentation network does not reach the optimization target, and obtain a target segmentation network until the updated segmentation network reaches the optimization target.

[0119] Based on the above embodiment, the application further provides a terminal, and a principle block diagram thereof can be as shown in Figure 7 The terminal comprises a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide calculation and control capabilities. The memory of the terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement the method for improving the robustness of a segmentation network. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0120] Those skilled in the art can understand that Figure 7 the principle block diagram shown in the above embodiment is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. The specific terminal can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0121] In one implementation, the memory of the terminal stores one or more programs configured to be executed by one or more processors to include instructions for performing the method for improving robustness of a segmentation network.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the computer program can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] In summary, this invention discloses a method, apparatus, and storage medium for improving the robustness of segmentation networks. The method includes: acquiring a segmentation network and a corresponding test dataset; inputting test images from the test dataset into the segmentation network to obtain segmented images corresponding to the test images, wherein the segmentation network is pre-trained on a training dataset; determining a fusion image based on the test images and segmented images; inputting the fusion image into a preset target synthesis network to obtain a synthesized image corresponding to the fusion image, wherein the target synthesis network is pre-trained on a training dataset; updating the network parameters of the segmentation network based on the test images and synthesized images; determining whether the updated segmentation network has reached the optimization target; if the updated segmentation network has not reached the optimization target, continuing the step of inputting test images from the test dataset into the segmentation network until the updated segmentation network reaches the optimization target, thus obtaining the target segmentation network. This invention generates corresponding synthesized images from test images and segmented images, enabling the segmentation network to accurately recognize its own segmentation errors based on the test images and synthesized images, and adaptively update its network parameters. This solves the problem that existing methods for addressing grayscale distribution drift are difficult to enable the segmentation network to accurately recognize and adaptively correct segmentation errors, resulting in low robustness of the segmentation network.

[0124] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for improving robustness of a segmentation network, characterized in that, The method comprises: obtaining a segmentation network and a test data set corresponding to the segmentation network, inputting a test image in the test data set into the segmentation network to obtain a segmentation image corresponding to the test image, wherein the segmentation network is pre-trained by a training data set; determining a fusion image according to the test image and the segmentation image, comprising: obtaining an edge image corresponding to the test image, wherein the gray level variation degree of the edge image is greater than that of the images other than the edge image in the test image; fusing the edge image and the segmentation image to obtain the fusion image; inputting the fusion image into a preset target synthesis network to obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is pre-trained by the training data set; updating network parameters of the segmentation network according to the test image and the synthesis image, comprising: determining similarity data according to the test image and the synthesis image; or determining similarity data according to a weighted test image corresponding to the test image and a weighted synthesis image corresponding to the synthesis image, wherein the weighted test image is an image obtained by weighting the test image based on an attention matrix, and the weighted synthesis image is an image obtained by weighting the synthesis image based on the attention matrix; determining a loss function value corresponding to the segmentation network according to the similarity data; updating the network parameters of the segmentation network according to the loss function value; determining whether the updated segmentation network reaches an optimization target, if the updated segmentation network does not reach the optimization target, continuing to input the test image in the test data set into the segmentation network until the updated segmentation network reaches the optimization target to obtain a target segmentation network.

2. The method for improving robustness of a segmentation network according to claim 1, wherein, The step of inputting the test image in the test data set into the segmentation network to obtain a segmentation image corresponding to the test image comprises: inputting the test image into the segmentation network to output a plurality of sub-segmentation images through the segmentation network, wherein each of the plurality of sub-segmentation images corresponds to a different segmentation category, and the sub-segmentation image corresponding to each segmentation category is determined based on a channel map and an index value of a channel corresponding to the segmentation category; determining the segmentation image according to the plurality of sub-segmentation images.

3. The method for improving robustness of a segmentation network according to claim 1, wherein, The method for determining the similarity data corresponding to two images comprises: determining a cross-correlation coefficient and a mutual information coefficient corresponding to a first image and a second image according to the first image and the second image, wherein the first image is the test image or the weighted test image, the second image is the synthesis image or the weighted synthesis image, the cross-correlation coefficient is used to reflect the gray difference between the first image and the second image, and the mutual information coefficient is used to reflect the information difference between the first image and the second image; determining the similarity data according to the cross-correlation coefficient and the mutual information coefficient.

4. The method for improving robustness of a segmentation network according to claim 3, characterized in that, The method for determining the cross-correlation coefficient comprises: According to the first image and the second image, a plurality of local cross-correlation coefficients are determined, wherein a plurality of the local cross-correlation coefficients respectively correspond to different local image pairs, two local images in each of the local image pairs are respectively located in the first image and the second image, and each of the local cross-correlation coefficients is used to reflect a gray difference between the two local images. The cross-correlation coefficient is determined according to the plurality of local cross-correlation coefficients.

5. The method for improving robustness of a segmentation network according to claim 4, characterized in that, The method for determining the mutual information coefficient comprises: According to the first image, a first information amount corresponding to the first image is determined. Gray information corresponding to the second image is obtained, and a second information amount corresponding to the first image is determined according to the gray information corresponding to the second image and the first image. According to the first information amount and the second information amount, the mutual information coefficient corresponding to the first image and the second image is determined.

6. The method for improving robustness of a segmentation network according to claim 3, wherein, The determination of the loss function value corresponding to the segmentation network according to the similarity data comprises: a cross-correlation loss function value is determined according to the cross-correlation coefficient, wherein the cross-correlation coefficient and the cross-correlation loss function value are in an inverse proportional relationship; a mutual information loss function value is determined according to the mutual information coefficient, wherein the cross-correlation coefficient and the cross-correlation loss function value are in an inverse proportional relationship; the loss function value is determined according to the cross-correlation coefficient and the mutual information coefficient.

7. An apparatus for improving robustness of a segmentation network, the apparatus comprising: The device comprises: An image segmentation module is configured to obtain a segmentation network and a test data set corresponding to the segmentation network, input a test image in the test data set into the segmentation network, and obtain a segmentation image corresponding to the test image, wherein the segmentation network is pre-trained by a training data set. An image synthesis module is configured to determine a fusion image according to the test image and the segmentation image, comprising: obtaining an edge image corresponding to the test image, wherein a gray variation degree of the edge image is greater than that of an image other than the edge image in the test image; and fusing the edge image and the segmentation image to obtain the fusion image. The fusion image is input into a preset target synthesis network to obtain a synthesis image corresponding to the fusion image, wherein the target synthesis network is pre-trained by the training data set. An iterative updating module is configured to update network parameters of the segmentation network according to the test image and the synthesis image, comprising: determining similarity data according to the test image and the synthesis image; or determining similarity data according to a weighted test image corresponding to the test image and a weighted synthesis image corresponding to the synthesis image, wherein the weighted test image is an image obtained by weighting the test image based on an attention matrix, and the weighted synthesis image is an image obtained by weighting the synthesis image based on the attention matrix; determining a loss function value corresponding to the segmentation network according to the similarity data; and updating the network parameters of the segmentation network according to the loss function value. determining whether the updated segmentation network reaches an optimization goal, and if the updated segmentation network does not reach the optimization goal, continuing to input the test images in the test data set into the segmentation network until the updated segmentation network reaches the optimization goal, and obtaining a target segmentation network.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded and executed by a processor to implement the steps of the method for improving robustness of a segmentation network according to any one of claims 1-6.

Images