Image segmentation sample generation method, device and equipment

Abstract

The disclosure provides an image segmentation sample generation method, device and equipment, relates to the technical field of artificial intelligence, specifically to the technical field of computer vision, image processing and deep learning, and can be applied to the medical image segmentation scene. A specific embodiment of the method comprises: obtaining an image classification sample set, wherein the sample images in the image classification sample set comprise multiple class labels; determining the weight of each pixel point for each class label when the sample images are classified by an image classification model; selecting positive label pixel points and negative label pixel points of each class label in the sample images based on the weight of each pixel point for each class label, and generating an image segmentation sample set corresponding to the image classification sample set. The embodiment can obtain a large number of image segmentation samples at a very small cost, and improves the generation efficiency of the image segmentation samples.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, image processing and deep learning, and can be applied to the field of medical image segmentation. BACKGROUND

[0002] Image semantic segmentation is a traditional task in the field of computer vision, aiming to identify and classify each pixel point in an image. The image segmentation technology can be applied to various scenarios, such as medical image segmentation. At present, the image semantic segmentation is mainly realized by training an image segmentation model.

[0003] In theory, an image segmentation model can be trained by a large number of image segmentation samples. However, considering that the annotation of image segmentation samples is difficult, costly and time-consuming, the existing technology mainly uses existing image classification samples to pre-train the model backbone parameters of a deep learning model, and then further adjusts the deep learning model using image segmentation samples to finally obtain the required image segmentation model. SUMMARY

[0004] Embodiments of the present disclosure provide an image segmentation sample generation method, device, equipment, storage medium and program product.

[0005] In a first aspect, embodiments of the present disclosure provide an image segmentation sample generation method, comprising: obtaining an image classification sample set, wherein a sample image in the image classification sample set comprises a plurality of class labels; determining the weight of each pixel point for each class label when the sample image is classified by an image classification model; selecting positive label pixel points and negative label pixel points of each class label in the sample image based on the weight of each pixel point for each class label, and generating an image segmentation sample set corresponding to the image classification sample set.

[0006] In a second aspect, embodiments of the present disclosure provide an image segmentation sample generation device, comprising: a first obtaining module configured to obtain an image classification sample set, wherein a sample image in the image classification sample set comprises a plurality of class labels; a determining module configured to determine the weight of each pixel point for each class label when the sample image is classified by an image classification model; and a selecting module configured to select positive label pixel points and negative label pixel points of each class label in the sample image based on the weight of each pixel point for each class label, and generate an image segmentation sample set corresponding to the image classification sample set.

[0007] In a third aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any implementation manner of the first aspect.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to perform the method according to any implementation manner of the first aspect.

[0009] In a fifth aspect, a computer program product is provided, including a computer program, and the computer program, when executed by a processor, implements the method according to any implementation manner of the first aspect.

[0010] The image segmentation sample generation method provided by the embodiments of the present disclosure generates image segmentation samples by selecting positive label pixel points and negative label pixel points of each class label in a sample image based on the weights of each pixel point on each class label when the sample image is classified by using an image classification model. A large number of image segmentation samples can be obtained at a very small cost by adding corresponding class labels to each pixel point that plays an important role in the sample image, thereby improving the generation efficiency of the image segmentation samples and further improving the pre-training performance of the image segmentation model to a certain extent. Moreover, the sample image includes multiple class labels, and the image segmentation samples including multiple class labels can be generated, and thus an image segmentation model capable of recognizing objects with different attributes can be trained under the setting of multiple class labels.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. The drawings are intended to better understand the present scheme and do not constitute a limitation of the present disclosure. Among them:

[0013] Figure 1 is a flowchart of one embodiment of the image segmentation sample generation method according to the present disclosure;

[0014] Figure 2 is a flowchart of another embodiment of the image segmentation sample generation method according to the present disclosure;

[0015] Figure 3is a flowchart of another embodiment of the image segmentation sample generation method according to the present disclosure;

[0016] Figure 4 is a flowchart of still another embodiment of the image segmentation sample generation method according to the present disclosure;

[0017] Figure 5 is a schematic diagram of an image segmentation sample;

[0018] Figure 6 is a structural schematic diagram of one embodiment of the image segmentation sample generation apparatus according to the present disclosure;

[0019] Figure 7 is a block diagram of an electronic device for implementing the image segmentation sample generation method according to the present disclosure. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of embodiments of the present disclosure by a person of ordinary skill in the art, and should be considered in connection with the following detailed description, but do not limit the scope of the present disclosure. Thus, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein, without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted herein.

[0021] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0022] Figure 1 A flowchart 100 of one embodiment of the image segmentation sample generation method according to the present disclosure is shown. The image segmentation sample generation method comprises the following steps:

[0023] In step 101, an image classification sample set is obtained.

[0024] In this embodiment, the execution subject of the image segmentation sample generation method can obtain an image classification sample set. The image classification sample set is multi-classification data, which can generally be used as a training sample set for training a multi-classification image classification model. The image classification sample set can include a large number of sample images. The sample images can be labeled with multiple class labels. The class label can be used to represent the class to which the object in the sample image belongs. For example, the class labels of a chest radiograph image include thoracic cage, thoracic cavity, lung tissue, mediastinum, heart, etc.

[0025] In step 102, the weight of each pixel point for each class label when the sample image is classified by the image classification model is determined.

[0026] In the embodiment, the execution subject can determine the weight of each pixel point for each category label when classifying the sample image by using the image classification model.

[0027] The image classification model can be any multi-classification model capable of identifying the category to which an image belongs. The type of image classification model can include, but is not limited to, implementation based on CNN (Convolutional Neural Networks) or RNN (Recurrent Neural Network). When classifying the sample image by using the image classification model, different pixel points have different importance for each category label. Here, the importance is quantified as the weight of each pixel point for each category label.

[0028] It can be understood that the higher the weight of a pixel point in the sample image for a category label, the greater the reference degree of the image classification model to the pixel point when determining the category (i.e., the category label) to which the sample image belongs, and thus the greater the probability that the pixel point belongs to the pixel point corresponding to the category label. For example, in a sample image containing a zebra, the higher the weight of a pixel point, the greater the probability that the pixel point is a pixel point used to form a zebra pattern in the sample image.

[0029] In step 103, based on the weight of each pixel point for each category label, positive label pixel points and negative label pixel points of each category label are selected in the sample image to generate an image segmentation sample set corresponding to the image classification sample set.

[0030] In the embodiment, the execution subject can select positive label pixel points and negative label pixel points of each category label in the sample image based on the weight of each pixel point for each category label to generate an image segmentation sample set corresponding to the image classification sample set.

[0031] For a category label, the positive label pixel points can be a part of the pixel points determined from all the pixel points of the sample image according to the weight. The remaining pixel points belong to the negative label pixel points. Generally, each pixel point in the sample image can be screened according to a preset pixel point screening condition and the weight of each pixel point to obtain the positive label pixel points, i.e., the pixel points that determine the category label of the sample image, so as to obtain the pixel points that have an important influence on image classification in the sample image. The pixel point screening condition can include, but is not limited to, the number of pixel points to be screened or the weight of the pixel point being greater than a set threshold.

[0032] Image segmentation samples can be understood as images annotated with segmentation labels, and segmentation labels can be understood as the category labels that mark the segmented objects in the image at the pixel level. Image segmentation samples are generally used as training samples for image segmentation models. A trained image segmentation model possesses image segmentation capabilities, meaning it can identify objects with different attributes (category labels) in an image.

[0033] The image segmentation sample generation method provided by the embodiment of the present disclosure is based on the weight of each pixel for each category label when classifying the sample image using an image classification model, and selects the positive label pixel points and negative label pixel points of each category label in the sample image to generate image segmentation samples. By adding the corresponding category label to each pixel point that plays an important role in the sample image to form an image segmentation sample, a large number of image segmentation samples can be obtained at a very low cost, thereby improving the generation efficiency of the image segmentation sample and thus improving the pre-training performance of the image segmentation model to a certain extent. In addition, the sample image here includes multiple category labels, and image segmentation samples including multiple category labels can be generated, thereby training an image segmentation model that can recognize objects with different attributes under the setting of multiple category labels.

[0034] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the image segmentation sample generation method according to the present disclosure. The image segmentation sample generation method includes the following steps:

[0035] Step 201: Acquire multiple image classification data sets.

[0036] In this embodiment, the execution subject of the image segmentation sample generation method may obtain multiple image classification datasets. The images in each image classification dataset may include at least one category label. In other words, at least some of the image classification datasets are multi-category data.

[0037] Step 202 : By splicing the category labels in the multiple image classification data sets, the multiple image classification data sets are fused into an image classification sample set.

[0038] In this embodiment, the execution entity may combine the category labels in the multiple image classification data sets to fuse the multiple image classification data sets into an image classification sample set.

[0039] In practical applications, due to the limited sample size of a single multi-class image classification dataset, in order to increase the sample size, multiple image classification datasets can be fused to form a larger image classification sample set. Since most image classification datasets are multi-class data, the sample labels from different image classification datasets can be spliced together. For example, a first image is from a first image classification dataset and has C1 binary labels, and a second image is from a second image classification dataset and has C2 binary labels; when the first image classification dataset and the second image classification dataset are fused together, the first image and the second image both have C=C1+C2 binary labels. It should be noted that, but the first C1 binary labels of the first image are the same as the labels in the first image classification dataset, and the last C2 binary labels are negative labels. Similarly, the first C1 binary labels of the second image are negative labels, and the last C2 binary labels are the same as the labels in the second image classification dataset. The number of labels on the fused image classification sample set is denoted as C.

[0040] Step 203, training an image classification model by taking the sample image as input and taking the multiple class labels of the sample image as output.

[0041] In this embodiment, the above execution subject can train an image classification model by taking the sample image as input and taking the multiple class labels of the sample image as output.

[0042] Generally, the parameters of the classification model are initialized, the sample image is input into the classification model to obtain multiple class prediction results of the sample image. A loss function is calculated based on the multiple class labels of the sample image and the multiple class prediction results. The parameters of the classification model are adjusted based on the loss function to obtain an image classification model. Here, the image classification model is trained using the image classification sample set obtained in step 202, which improves the accuracy of classifying the sample images in the image classification sample set obtained in step 202 using the image classification model.

[0043] Step 204, inputting the sample image into the image classification model for classification, and determining the weight of each pixel point for each class label through an explainability algorithm.

[0044] In this embodiment, the above execution subject can input the sample image into the image classification model for classification, and determine the weight of each pixel point for each class label through an explainability algorithm.

[0045] The explainable algorithm can be an algorithm for making the prediction result of the image classification model interpretable, and is used to establish the association between the model input data and the model prediction process. Here, one of the known explainable algorithms can be selected, such as a gradient-based explainable algorithm, an integral gradient-based explainable algorithm, or LIME (Local Interpretable Model-Agnostic Explanations), and the like. Then, after the sample image is input to the image classification model, the importance of each pixel point in the sample image in determining the category of the sample image by the image classification model can be determined according to the explainable algorithm, and the importance is quantified as the weight of each pixel point for each category label.

[0046] It can be understood that the higher the weight of a pixel point in the sample image for a category label, the greater the reference degree of the image classification model to the pixel point in determining the category (i.e., the category label) of the sample image, and thus the greater the probability that the pixel point belongs to the pixel point corresponding to the category label.

[0047] In step 205, the sample image is binarized, and the first preselected proportion of pixel points are selected as positive label pixel points in the order of the weights of the category labels from large to small, and the remaining pixel points are selected as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0048] In this embodiment, the execution subject can binarize the sample image, and select the first preselected proportion of pixel points as positive label pixel points in the order of the weights of the category labels from large to small, and select the remaining pixel points as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0049] Considering the use efficiency and reducing the storage space, the sample image can be binarized. For a category label, the positive label pixel points thereof can be determined from part of the pixel points of the sample image according to the weight size. The remaining pixel points are negative label pixel points. Generally, the first preselected proportion of pixel points are selected as positive label pixel points of the category label in the order of the weights of the category labels from large to small, and the remaining pixel points are selected as negative label pixel points of the category label.

[0050] The image segmentation sample can be understood as an image labeled with a segmentation label. The segmentation label can be understood as a class label of a segmentation object labeled in the image in units of pixels. The image segmentation sample can generally be used as a training sample for training an image segmentation model. The trained image segmentation model has image segmentation capability, i.e., can recognize objects with different attributes (class labels) in the image.

[0051] From Figure 2 It can be seen that, compared with the embodiment corresponding to Figure 1 The process 200 of the image segmentation sample generation method in the embodiment highlights the image classification sample set generation step, the image classification model training step, and the weight determination step. Therefore, the scheme described in the embodiment increases the sample quantity of the image classification sample set by fusing multiple image classification data sets to form a larger image classification sample set. The image classification model is trained using the image classification sample set, which improves the accuracy of classifying sample images in the image classification sample set using the image classification model. The pixels that play an important role in image classification in the sample image are obtained using the explainable algorithm. The image segmentation sample is formed by adding the corresponding class label to each pixel that plays an important role in the sample image in a simple, convenient, and high-precision manner. A large number of image segmentation samples can be obtained at a very small cost, which improves the generation efficiency of the image segmentation sample and further improves the pre-training performance of the image segmentation model to some extent. Moreover, the sample image includes multiple class labels, and the image segmentation sample including multiple class labels can be generated, and the image segmentation model capable of recognizing objects with different attributes can be trained under the setting of multiple class labels.

[0052] Further reference is made to Figure 3 which shows a flowchart of another embodiment of the image segmentation sample generation method according to the present disclosure. The image segmentation sample generation method includes the following steps:

[0053] Step 301, obtaining multiple image classification data sets.

[0054] In the embodiment, the execution subject of the image segmentation sample generation method can obtain multiple image classification data sets. The images in each image classification data set can include at least one class label. That is, at least part of the image classification data set is multi-classification data.

[0055] Step 302, fusing the multiple image classification data sets into an image classification sample set by splicing the class labels in the multiple image classification data sets.

[0056] In the embodiment, the execution subject described above can fuse the multiple image classification data sets into an image classification sample set by splicing the class labels in the multiple image classification data sets.

[0057] In practical applications, due to the limited sample size of a single multi-class image classification dataset, in order to increase the sample size, multiple image classification datasets can be fused to form a larger image classification sample set. Since most image classification datasets are multi-class data, the sample labels from different image classification datasets can be spliced together.

[0058] Step 303, training a plurality of image classification models by taking the sample images as input and taking the plurality of class labels of the sample images as output.

[0059] In this embodiment, the above execution subject can train a plurality of image classification models by taking the sample images as input and taking the plurality of class labels of the sample images as output. The structures and / or hyperparameters of different image classification models are different.

[0060] Generally, the parameters of the classification model are initialized, the sample images are input into the classification model to obtain a plurality of class prediction results of the sample images. A loss function is calculated based on the plurality of class labels of the sample images and the plurality of class prediction results. The parameters of the classification model are adjusted based on the loss function to obtain an image classification model. Here, the image classification model is trained using the image classification sample set obtained in step 302, which improves the accuracy of classifying the sample images in the image classification sample set obtained in step 302 using the image classification model.

[0061] Step 304, inputting the sample images into each image classification model to determine the weight of each pixel point for each class label when each image classification model classifies the sample images by using an explainability algorithm.

[0062] In this embodiment, for an image classification model, the above execution subject can input the sample images into the image classification model to determine the weight of each pixel point for each class label when the image classification model classifies the sample images by using an explainability algorithm.

[0063] The explainability algorithm can be an algorithm that makes the prediction result of the image classification model interpretable, which is used to establish the association between the model input data and the model prediction process. Here, one of the known explainability algorithms can be selected, such as a gradient-based explainability algorithm, an integral gradient-based explainability algorithm, or LIME, and then after the sample images are input into the image classification model, the importance of each pixel point in the sample images in determining the class of the sample images by the image classification model can be determined according to the explainability algorithm, and the importance is quantified as the weight of each pixel point for each class label.

[0064] Step 305, the weights of the same pixel position and the same category label in the sample image are weighted and averaged to obtain the weights of each pixel point of the sample image for each category label.

[0065] In the embodiment, the execution subject can weight and average the weights of the same pixel position and the same category label in the sample image to obtain the weights of each pixel point of the sample image for each category label.

[0066] Since the single image classification model may have the problem of poor model accuracy, to reduce noise, multiple image classification models can be trained using the image classification sample set, the weights of each pixel point for each category label when each image classification model classifies the sample image are determined through an explainable algorithm, and then weighted and averaged, which can avoid the problem of large weight calculation error caused by the accuracy problem of the single image classification model itself, thereby obtaining more robust and less noisy segmentation labels.

[0067] Step 306, the sample image is binarized, the first preselected proportion of pixel points are selected as positive label pixel points in the order of the weights of each category label from large to small, and the remaining pixel points are selected as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0068] In the embodiment, the execution subject can binarize the sample image, select the first preselected proportion of pixel points as positive label pixel points in the order of the weights of each category label from large to small, and select the remaining pixel points as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0069] Considering the use efficiency and reducing the storage space, the sample image can be binarized. For a category label, the positive label pixel points thereof can be part of the pixel points determined from all the pixel points of the sample image according to the weight size. The remaining pixel points belong to the negative label pixel points. Usually, the first preselected proportion of pixel points are selected as the positive label pixel points of the category label in the order of the weights of the category label from large to small, and the remaining pixel points are selected as the negative label pixel points of the category label.

[0070] It can be seen from Figure 3 that the image segmentation sample set is more accurate than Figure 2Compared with the corresponding embodiments, the process 300 of the image segmentation sample generation method in this embodiment highlights the weight determination step. Thus, the scheme described in this embodiment trains multiple image classification models using an image classification sample set, determines the weight of each pixel point for each class label when each image classification model classifies a sample image through an explainable algorithm, and then performs weighted averaging, which can avoid the problem of large weight calculation error caused by the accuracy problem of a single image classification model itself, thereby obtaining more robust and less noisy segmentation labels.

[0071] Further reference is made to Figure 4 which shows a flowchart of still another embodiment of an image segmentation sample generation method according to the present disclosure. The image segmentation sample generation method comprises the following steps:

[0072] Step 401, obtaining multiple image classification data sets.

[0073] In this embodiment, the execution subject of the image segmentation sample generation method can obtain multiple image classification data sets. The images in each image classification data set can include at least one class label. That is, at least part of the image classification data sets are multi-classification data.

[0074] Step 402, fusing the multiple image classification data sets into an image classification sample set by splicing the class labels in the multiple image classification data sets.

[0075] In this embodiment, the execution subject described above can fuse the multiple image classification data sets into an image classification sample set by splicing the class labels in the multiple image classification data sets.

[0076] In actual applications, due to the limited sample size of a single multi-classification image classification data set, in order to increase the sample size, multiple image classification data sets can be fused to form a larger image classification sample set. Since most image classification data sets are multi-classification data, the sample labels from different image classification data sets can be spliced together.

[0077] Step 403, training multiple image classification models by taking a sample image as input and taking multiple class labels of the sample image as output.

[0078] In this embodiment, the execution subject described above can train multiple image classification models by taking a sample image as input and taking multiple class labels of the sample image as output. The structures and / or hyperparameters of different image classification models are different.

[0079] Generally, the parameters of the classification model are initialized, the sample image is input into the classification model to obtain a plurality of category prediction results of the sample image. A loss function is calculated based on the plurality of category labels and the plurality of category prediction results of the sample image. The parameters of the classification model are adjusted based on the loss function to obtain the image classification model. Here, the image classification model is trained by using the image classification sample set obtained in step 402, which improves the accuracy of classifying the sample images in the image classification sample set obtained in step 402 by using the image classification model.

[0080] In step 404, the sample image is input into each image classification model, and the weight of each pixel point with respect to each category label when each image classification model classifies the sample image is determined by using the explainability algorithm.

[0081] In this embodiment, for one image classification model, the execution subject can input the sample image into the image classification model, and determine the weight of each pixel point with respect to each category label when the image classification model classifies the sample image by using the explainability algorithm.

[0082] The explainability algorithm can be an algorithm that makes the prediction result of the image classification model explainable, and is used to establish the association between the model input data and the model prediction process. Here, one of the known explainability algorithms can be selected, such as a gradient-based explainability algorithm, an integral gradient-based explainability algorithm, or LIME, and then after the sample image is input into the image classification model, the importance of each pixel point in the sample image when the image classification model determines the category to which the sample image belongs can be determined according to the explainability algorithm, and the importance is quantified as the weight of each pixel point with respect to each category label.

[0083] In step 405, the weights of the same pixel position and the same category label in the sample image are weighted and averaged to obtain the weight of each pixel point of the sample image with respect to each category label.

[0084] In this embodiment, the execution subject can weight and average the weights of the same pixel position and the same category label in the sample image to obtain the weight of each pixel point of the sample image with respect to each category label.

[0085] Since a single image classification model may have a problem of poor model accuracy, to reduce noise, a plurality of image classification models can be trained by using the image classification sample set, the weight of each pixel point with respect to each category label when each image classification model classifies the sample image is determined by using the explainability algorithm, and then weighted and averaged, which can avoid the problem of large weight calculation error caused by the accuracy problem of a single image classification model itself, thereby obtaining a more robust and less noisy segmentation label.

[0086] In step 406, the sample image is binarized, and the first preset selection proportion of pixel points are selected as positive label pixel points in descending order of the weight of each category label, and the remaining pixel points are selected as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0087] In the embodiment, the execution subject can binarize the sample image, and select the first preset selection proportion of pixel points as positive label pixel points in descending order of the weight of each category label, and select the remaining pixel points as negative label pixel points, to generate an image segmentation sample set corresponding to the image classification sample set.

[0088] Considering the use efficiency and the reduction of storage space, the sample image can be binarized. For a category label, the positive label pixel points can be part of the pixel points determined from all the pixel points of the sample image according to the weight size. The remaining pixel points belong to the negative label pixel points. Generally, the first preset selection proportion of pixel points are selected as positive label pixel points of the category label in descending order of the weight of the category label, and the remaining pixel points are selected as negative label pixel points of the category label.

[0089] In step 407, the image segmentation sample set is used to pre-train the segmentation model to obtain a pre-trained image segmentation model.

[0090] In the embodiment, the execution subject can use the image segmentation sample set to pre-train the segmentation model to obtain a pre-trained image segmentation model.

[0091] Generally, the image segmentation sample set and the image classification sample set are used for pre-training to obtain a pre-trained image segmentation model. The supervision information of the pre-training is derived from the category labels of the image classification sample set, i.e. the negative label and the positive label, and the pixel point labels of each pixel point in the image segmentation sample set, i.e. the negative label and the positive label. Only when the category label is positive and the pixel point label is also positive, the pixel point is a positive label, otherwise the pixel point is a negative label. It should be noted that the output of the image segmentation model trained by the image segmentation sample set generated by the image classification sample set with C category labels is a pixel-level classification of C categories.

[0092] In step 408, a standard image segmentation sample set matching the target image segmentation task is obtained.

[0093] In the embodiment, the execution subject can obtain a standard image segmentation sample set matching the target image segmentation task.

[0094] The target image segmentation task can be any image segmentation task. The standard image segmentation sample set can be an image segmentation sample set matched with the target image segmentation task and having image features specific to the target image segmentation task. For example, the target image segmentation task is a medical image segmentation task, and the standard image segmentation sample set can be a medical image segmentation sample set. Here, the target image segmentation task can be determined first, and then an image classification sample set matched with the target image segmentation task is obtained, and a standard image segmentation sample set matched with the target image segmentation task is generated according to the image segmentation sample generation method in any of the above embodiments.

[0095] At step 409, the pre-trained image segmentation model is fine-tuned using the standard image segmentation sample set to obtain an image segmentation model matched with the target image segmentation task.

[0096] In this embodiment, the execution subject can fine-tune the pre-trained image segmentation model using the standard image segmentation sample set to obtain an image segmentation model matched with the target image segmentation task.

[0097] Generally, the sample image in the standard image segmentation sample set is input into the classification model to obtain a plurality of class prediction results of the sample image. A loss function is calculated based on the plurality of class labels and the plurality of class prediction results of the sample image. The parameters of the pre-trained image segmentation model are fine-tuned based on the loss function to obtain the image segmentation model.

[0098] The image segmentation model can be a model obtained by fine-tuning the pre-trained image segmentation model using the standard image segmentation sample set, so that the image segmentation model can perform high-precision image segmentation on images matched with the target image segmentation task. By training the pre-trained image segmentation model using the standard image segmentation sample set, the pre-trained image segmentation model can have stronger image segmentation capability for images under the target image segmentation task only by fine-tuning the model parameters.

[0099] From Figure 4 It can be seen that, compared with the embodiment corresponding to Figure 3 The image segmentation sample generation method in this embodiment highlights the image segmentation model training step in the flow 400. Therefore, the scheme described in this embodiment uses the image segmentation sample set for pre-training, uses the standard image segmentation sample set for fine-tuning, and can comprehensively improve the performance of the image segmentation model without changing the existing model architecture, so that the image segmentation model has stronger image segmentation capability for images under the target image segmentation task.

[0100] For ease of understanding, the following provides a scenario of an image segmentation sample generation method that can implement embodiments of the present disclosure. Since the sample quantity of a single chest radiograph dataset is limited, in order to increase the sample quantity and improve the performance of a pre-trained model, multiple multi-classification datasets such as NIHCC, Pneumonia, MURA, and CheXpert are selected, sample labels from different classification datasets are spliced together, and a larger image classification sample set is formed by fusion. Then, multiple image classification models are trained using this image classification sample set to ensure that their accuracy is high. When classifying the sample images in this image classification sample set using each image classification model, the weight of each pixel point for each class label is determined by an explainable algorithm. In order to reduce noise, the weights of the same pixel position and the same class label in the sample image are weighted and averaged to obtain more robust and less noisy segmentation labels. Finally, considering the use efficiency and reducing the storage space, the sample images are binarized, the top 30% of the pixel points are selected as positive label pixel points, and the remaining pixel points are selected as negative label pixel points to generate an image segmentation sample set. As shown in Figure 5 , which shows a schematic diagram of an image segmentation sample. Here, the image classification sample includes C class labels, and C image segmentation samples are generated. Since some class labels of the image classification sample are negative, the generated image segmentation sample is also negative.

[0101] Further referring to Figure 6 , as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of an image segmentation sample generation device, which corresponds to the method embodiment shown in Figure 1 . The device can be specifically applied to various electronic devices.

[0102] As shown in Figure 6 , the image segmentation sample generation device 600 of the present embodiment can include a first acquisition module 601, a determination module 602, and a selection module 603. The first acquisition module 601 is configured to acquire an image classification sample set, wherein the sample images in the image classification sample set include multiple class labels; the determination module 602 is configured to determine the weight of each pixel point for each class label when the sample images are classified using an image classification model; and the selection module 603 is configured to select positive label pixel points and negative label pixel points of each class label in the sample images based on the weight of each pixel point for each class label, and generate an image segmentation sample set corresponding to the image classification sample set.

[0103] In the present embodiment, in the image segmentation sample generation device 600: the specific processing of the first acquisition module 601, the determination module 602, and the selection module 603 and the technical effects brought by them can be respectively referred to Figure 1The related description of steps 101-103 in the corresponding embodiment will not be repeated here.

[0104] In some optional implementation of the embodiment, the acquisition module 601 is further configured to: acquire a plurality of image classification data sets, wherein the images in each image classification data set comprise at least one class label; and fuse the plurality of image classification data sets into an image classification sample set by splicing the class labels in the plurality of image classification data sets.

[0105] In some optional implementation of the embodiment, the image segmentation sample generation apparatus 600 further comprises a training module configured to take the sample image as input and take a plurality of class labels of the sample image as output, and train an image classification model.

[0106] In some optional implementation of the embodiment, the determination module 602 comprises a determination sub-module configured to input the sample image into the image classification model for classification, and determine the weight of each pixel point with respect to each class label by using an explainability algorithm.

[0107] In some optional implementation of the embodiment, the number of image classification models is a plurality, and the structures and hyperparameters of different image classification models are different.

[0108] In some optional implementation of the embodiment, the determination sub-module is further configured to: input the sample image into each image classification model, determine the weight of each pixel point with respect to each class label when each image classification model classifies the sample image by using an explainability algorithm; and perform weighted average on the weights of the same pixel position and the same class label in the sample image to obtain the weight of each pixel point of the sample image with respect to each class label.

[0109] In some optional implementation of the embodiment, the selection module 603 is further configured to: perform binaryzation processing on the sample image, and select the pixel points in a preset selection proportion as positive label pixel points and the remaining pixel points as negative label pixel points in the order of the weights of each class label from large to small.

[0110] In some optional implementation of the embodiment, the image segmentation sample generation apparatus 600 further comprises a pre-training module configured to pre-train a segmentation model by using the image segmentation sample set to obtain a pre-trained image segmentation model; a second acquisition module configured to acquire a standard image segmentation sample set matched with a target image segmentation task; and a fine-tuning module configured to fine-tune the pre-trained image segmentation model by using the standard image segmentation sample set to obtain an image segmentation model matched with the target image segmentation task.

[0111] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0112] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0113] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0114] As shown in Figure 7 The device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0115] Various components in the device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc., an output unit 707, such as various types of displays, speakers, etc., a storage unit 708, such as a magnetic disk, an optical disk, etc., and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0116] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the image segmentation sample generation method. For example, in some embodiments, the image segmentation sample generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the image segmentation sample generation method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the image segmentation sample generation method by any other appropriate means, such as by means of firmware.

[0117] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0118] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0121] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions provided by the present disclosure can be achieved, which is not limited herein.

[0124] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating image segmentation samples, comprising: Acquire an image classification sample set, wherein the sample images in the image classification sample set include multiple category labels; Determining the weight of each pixel for each category label when classifying the sample image using an image classification model; Based on the weight of each pixel point for each category label, select the positive label pixel points and the negative label pixel points of each category label in the sample image to generate an image segmentation sample set corresponding to the image classification sample set; The step of selecting positive label pixels and negative label pixels of each category label in the sample image based on the weight of each pixel for each category label includes: Binarization is performed on the sample image, and according to the order of the weight of each category label from large to small, a preset selection ratio of pixels are selected as positive label pixels, and the remaining pixels are selected as negative label pixels; The number of the image classification models is multiple; and The determining of the weight of each pixel for each category label when classifying the sample image using the image classification model includes: Inputting the sample image into each image classification model, determining the weight of each pixel for each category label when each image classification model classifies the sample image using an interpretable algorithm, and after inputting the sample image into each image classification model, determining the importance of each pixel in the sample image when each image classification model determines the category to which the sample image belongs according to the interpretable algorithm, and quantifying the importance as the weight of each pixel for each category label; The weights of the same category label at the same pixel position in the sample image are weighted averaged to obtain the weight of each pixel point in the sample image for each category label.

2. The method according to claim 1, wherein The obtaining of the image classification sample set includes: Acquire multiple image classification datasets, wherein images in each image classification dataset include at least one category label; The multiple image classification data sets are fused into the image classification sample set by splicing the category labels in the multiple image classification data sets.

3. The method according to claim 1, wherein The method further comprises: The sample image is used as input, and multiple category labels of the sample image are used as output to train the image classification model.

4. The method according to claim 3, wherein: Different image classification models have different structures and / or hyperparameters.

5. The method according to claim 1, wherein The method further comprises: Pre-training a segmentation model using the image segmentation sample set to obtain a pre-trained image segmentation model; Obtain a standard image segmentation sample set that matches the target image segmentation task; The pre-trained image segmentation model is fine-tuned using the standard image segmentation sample set to obtain an image segmentation model that matches the target image segmentation task.

6. An image segmentation sample generation device, comprising: A first acquisition module is configured to acquire an image classification sample set, wherein the sample images in the image classification sample set include multiple category labels; a determination module configured to determine the weight of each pixel for each category label when classifying the sample image using an image classification model; a selection module configured to select positive label pixels and negative label pixels of each category label in the sample image based on the weight of each pixel for each category label, and generate an image segmentation sample set corresponding to the image classification sample set; The selection module is further configured to: Binarization is performed on the sample image, and according to the order of the weight of each category label from large to small, a preset selection ratio of pixels are selected as positive label pixels, and the remaining pixels are selected as negative label pixels; The number of the image classification models is multiple; and The determining module is further configured to: Inputting the sample image into each image classification model, determining the weight of each pixel for each category label when each image classification model classifies the sample image using an interpretable algorithm, and after inputting the sample image into each image classification model, determining the importance of each pixel in the sample image when each image classification model determines the category to which the sample image belongs according to the interpretable algorithm, and quantifying the importance as the weight of each pixel for each category label; The weights of the same category label at the same pixel position in the sample image are weighted averaged to obtain the weight of each pixel point in the sample image for each category label.

7. The device according to claim 6, wherein The acquisition module is further configured to: Acquire multiple image classification datasets, wherein images in each image classification dataset include at least one category label; The multiple image classification data sets are fused into the image classification sample set by splicing the category labels in the multiple image classification data sets.

8. The device according to claim 6, wherein The device further comprises: The training module is configured to take the sample image as input and multiple category labels of the sample image as output, and train to obtain the image classification model.

9. The device according to claim 8, wherein Different image classification models have different structures and / or hyperparameters.

10. The device according to any one of claims 6 to 9, wherein: The device further comprises: A pre-training module is configured to pre-train the segmentation model using the image segmentation sample set to obtain a pre-trained image segmentation model; A second acquisition module is configured to acquire a standard image segmentation sample set that matches the target image segmentation task; The fine-tuning module is configured to use the standard image segmentation sample set to fine-tune the pre-trained image segmentation model to obtain an image segmentation model that matches the target image segmentation task.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images