Image segmentation method, device, equipment and storage medium

Abstract

The present application discloses an image segmentation method, apparatus, device and storage medium, and relates to the field of artificial intelligence technology. The method comprises: obtaining a first image sample and a second image sample, wherein the quality of the annotation data of the first image sample is higher than the quality of the annotation data of the second image sample; performing segmentation processing on the first image sample and the second image sample respectively through a student network and a teacher network, and obtaining a student segmentation result of the first image sample, a student segmentation result of the second image sample, a teacher segmentation result of the first image sample and a teacher segmentation result of the second image sample; determining a training loss based on the annotation data of the first image sample and the above segmentation results; and training the student network based on the training loss. The present application extracts image information from samples with low-quality annotations through a teacher network and trains the student network, thereby avoiding the problem of model overfitting caused by a lack of samples with high-quality annotations.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image segmentation method, apparatus, device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, developers are trying to use deep learning technology to achieve the task of segmenting blood vessels within organs.

[0003] Taking the task of liver blood vessel segmentation as an example, the relevant technology trains the image segmentation model based on liver images with high-quality blood vessel annotations to obtain an image segmentation model that can be used to segment the blood vessels in the liver.

[0004] However, due to the general lack of liver images with high-quality vascular annotations, related technologies can only rely on a small number of liver images with high-quality vascular annotations to train image segmentation models, resulting in overfitting of the image segmentation model and low segmentation accuracy. Summary of the Invention

[0005] The present invention provides an image segmentation method, apparatus, device, and storage medium that can assist a student network in training based on high-quality labeled image samples and low-quality labeled image samples through a teacher network. This ensures the segmentation performance of the student network while avoiding the overfitting problem of the student network caused by a lack of image samples. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a method for image segmentation is provided, the method comprising:

[0007] Acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotated data of the first image sample, and the second sample data includes at least one second image sample and annotated data of the second image sample, and the quality of the annotated data of the first image sample is higher than the quality of the annotated data of the second image sample;

[0008] Performing segmentation processing on the first image sample and the second image sample respectively through the student network of the image segmentation model to obtain a student segmentation result of the first image sample and a student segmentation result of the second image sample, and performing segmentation processing on the first image sample and the second image sample respectively through the teacher network of the image segmentation model to obtain a teacher segmentation result of the first image sample and a teacher segmentation result of the second image sample;

[0009] determining a supervision loss based on a student segmentation result of the first image sample and the labeled data of the first image sample;

[0010] determining a consistency loss based on difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample, and difference information between a student segmentation result of the second image sample and a teacher segmentation result of the second image sample;

[0011] Based on the supervision loss and the consistency loss, the student network is trained to perform segmentation processing on the target image through the trained student network.

[0012] According to one aspect of an embodiment of the present application, there is provided an image segmentation apparatus, the apparatus comprising:

[0013] a sample data acquisition module, configured to acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotated data of the first image sample, and the second sample data includes at least one second image sample and annotated data of the second image sample, and the quality of the annotated data of the first image sample is higher than the quality of the annotated data of the second image sample;

[0014] a segmentation result acquisition module, configured to perform segmentation processing on the first image sample and the second image sample respectively through a student network of an image segmentation model to obtain a student segmentation result of the first image sample and a student segmentation result of the second image sample, and to perform segmentation processing on the first image sample and the second image sample respectively through a teacher network of the image segmentation model to obtain a teacher segmentation result of the first image sample and a teacher segmentation result of the second image sample;

[0015] a supervision loss determination module, configured to determine a supervision loss based on a student segmentation result of the first image sample and the labeled data of the first image sample;

[0016] a consistency loss determining module, configured to determine a consistency loss based on difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample, and difference information between a student segmentation result of the second image sample and a teacher segmentation result of the second image sample;

[0017] The student network training module is used to train the student network based on the supervision loss and the consistency loss, so as to segment the target image through the trained student network.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned image segmentation method.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned image segmentation method.

[0020] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image segmentation method.

[0021] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0022] By performing consistency comparison between the prediction results of the teacher network for image samples with high-quality annotations and the results of the student network for image samples with high-quality annotations, and by performing consistency comparison between the prediction results of the teacher network for image samples with low-quality annotations and the prediction results of the student network for image samples with low-quality annotations, the teacher network can help the student network learn useful image information from both image samples with high-quality annotations and image samples with low-quality annotations, thereby avoiding the network overfitting problem caused by lack of image samples and ensuring the segmentation accuracy of the student network.

[0023] In addition, since the teacher network helps the student network learn useful image information from image samples with low-quality annotations, rather than the student network directly learning useful image information from image samples with low-quality annotations, the impact of image samples with low-quality annotations on the student network can be reduced, thereby ensuring the segmentation performance of the student network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a flowchart of an image segmentation method provided by an embodiment of the present application;

[0026] Figure 2 This is a flowchart of a method for training an image segmentation model provided by one embodiment of the present application;

[0027] Figure 3 is a schematic diagram of an image segmentation method provided by another embodiment of the present application;

[0028] Figure 4 is a schematic diagram of an original data set provided by an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of the final 3D output blood vessel segmentation map provided by one embodiment of the present application;

[0030] Figure 6 is a schematic diagram of a quantitative comparison result provided by an embodiment of the present application;

[0031] Figure 7 is a block diagram of an image segmentation device provided by one embodiment of the present application;

[0032] Figure 8 is a block diagram of an image segmentation device provided by another embodiment of the present application;

[0033] Figure 9 This is a block diagram of a computer device provided by one embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0036] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0037] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0038] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0039] The technical solutions provided in the embodiments of this application involve artificial intelligence machine learning technology, which uses machine learning technology to train an image segmentation model (such as the student network in the image segmentation model), and then use the trained student network to perform image segmentation tasks. Optionally, the technical solutions provided in the embodiments of this application are applicable to any segmentation task where it is difficult to obtain a large number of high-quality annotated image samples, such as medical image segmentation tasks and intra-organ vascular segmentation tasks.

[0040] Illustratively, the technical solutions provided in the embodiments of the present application are applicable to obtaining the image segmentation model required for the task of segmenting blood vessels in any organ (such as the liver, kidneys, heart, etc.). For example, by adopting the technical solutions provided in the embodiments of the present application, an image segmentation model for segmenting blood vessels in the liver can be obtained based on liver image samples with high-quality blood vessel annotations and liver image samples with low-quality blood vessel annotations. The technical solutions provided in the embodiments of the present application can train the image segmentation model based on image samples with high-quality annotations and image samples with low-quality annotations, thereby avoiding overfitting of the image segmentation model due to lack of image samples while ensuring the segmentation accuracy of the image segmentation model.

[0041] In the methods provided in the embodiments of the present application, the execution entity of each step may be a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. The computer device may be a terminal such as a PC (Personal Computer), a tablet computer, a smartphone, a wearable device, an intelligent robot, or a server. The server may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0042] Please refer to Figure 1 , which shows a flowchart of an image segmentation method provided by an embodiment of the present application. The execution body of each step of the method can be the computer device introduced above, and the method can include the following steps (101~105).

[0043] Step 101: Acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotation data of the first image sample, and the second sample data includes at least one second image sample and annotation data of the second image sample, and the quality of the annotation data of the first image sample is higher than the quality of the annotation data of the second image sample.

[0044] In an embodiment of the present application, an image segmentation model is used to perform image segmentation processing. For example, the target object in the image can be marked by the image segmentation model, or the target object can be directly segmented to form an image that only includes the target object. Optionally, the target object can be set according to the segmentation task. For example, in the medical field, the target object can be a target organ in an image of a human cavity, a blood vessel in an image of a target organ, etc.; in the field of autonomous driving, the target object can be an animal in an image, a person in an image, etc., and the embodiment of the present application is not limited here.

[0045] In an embodiment of the present application, both the first image sample and the second image sample are used to train the image segmentation model. The first image sample and the second image sample can be CT (Computed Tomography, electronic computed tomography) images, images, pictures, photos, etc. with the above-mentioned target object required for the segmentation task. Among them, the first image sample and the second image sample both carry annotation data of the target object, and the annotation data is used to identify the target object. For example, each pixel corresponding to the target object in the first image sample carries annotation data for indicating that it belongs to the target object. Optionally, the quality of the annotation data of the target object corresponding to the first image sample is higher than the quality of the annotation data of the target object corresponding to the second image sample. Exemplarily, the annotation data of the first image sample is high-quality annotation data, that is, the annotation error rate corresponding to the annotation data of the first image sample is extremely low, and the annotation data of the second image sample is low-quality annotation data, that is, there are certain erroneous annotations in the annotation data of the second image sample. Taking the annotated data of liver blood vessels as an example, the quality of the annotated data of liver blood vessels largely depends on the experience and skills of medical practitioners, and medical practitioners must manually annotate the blood vessels in liver images. In addition, due to the inherent noise, pathological changes of blood vessels, poor contrast and complex vascular geometry in CT images, high-quality annotated data of liver blood vessels is very limited, that is, the annotated data of the first image sample mentioned above is limited, while low-quality annotated data of liver blood vessels is very abundant, that is, the annotated data of the second image sample mentioned above is relatively abundant.

[0046] In one example, the process of acquiring the first sample data and the second sample data can be as follows: acquiring a first original data set and a second original data set; wherein the first original data set includes at least one first original image and annotation data of the first original image, and the second original data set includes at least one second original image and annotation data of the second original image; preprocessing the first original image and the second original image respectively to obtain a processed first original image and a processed second original image; wherein the preprocessing includes at least one of the following: cropping processing, clarity adjustment processing, and normalization processing; acquiring target object probability maps corresponding to the first original image and the second original image respectively, the target object probability maps being used to represent the distribution of target objects; performing splicing processing on the processed first original image and the target object probability maps corresponding to the first original image to obtain a first image sample, and performing splicing processing on the processed second original image and the target object probability maps corresponding to the second original image to obtain a second image sample.

[0047] The first original dataset is used to obtain the first image sample and the annotated data of the first image sample, and the second original dataset is used to obtain the annotated data of the second image sample and the second image sample. Both the first original dataset and the second original dataset can provide annotated data of the target object, and the quality of the annotated data of the target object provided by the first original dataset is higher than the quality of the annotated data of the target object provided by the second original dataset, that is, the quality of the annotated data of the first original image is higher than the quality of the annotated data of the second original image.

[0048] A target object probability map is a distribution probability map of target objects. For example, a blood vessel distribution probability map within the liver can be used to represent the distribution of blood vessels within the liver. The target object probability map can be used to provide auxiliary information about target objects, such as their location, number, and edges, to the image segmentation model. The target object probability map can be obtained using a multi-scale target object (e.g., blood vessel) enhancement filter network.

[0049] For example, the preprocessing process of the original image can be as follows:

[0050] 1. First, mask the original image and crop the target area. For example, in the task of segmenting blood vessels in the liver, the target area can be the liver area. Since the first original data set can provide a fine segmentation mask of the target area, the target area can be extracted directly from the first original image. Since the second original data set cannot provide a segmentation mask of the target area or cannot provide a fine segmentation mask of the target area, the segmentation mask of the target area corresponding to the second original data set can be obtained through the trained H-DenseUNet (hybrid densely connected U-shaped structure network), and then the target area is extracted from the second original image based on the obtained segmentation mask of the target area. Among them, the target area of ​​the first original image is the same as the target area of ​​the second original image, and the target area includes the target object. The original image is cropped based on the target area. During the cropping process, ensure that the scale of the cropped original image is larger than the maximum edge scale of the target area. Optionally, the original image can be cropped to a set pixel size according to actual needs (such as the scale requirements of the image segmentation model for image samples). In an embodiment of the present application, when the original image is an axial 2D slice of a 3D CT image, since the thickness of the axial 2D slice varies greatly (for example, ranging from 1 mm to 4 mm), no resampling operation is performed to avoid an increase in potential artificial noise caused by interpolation.

[0051] 2. Adjust the clarity of the cropped original image to more clearly visualize the structure of the target region and the target object therein. For example, if the original image is an axial 2D slice of a 3D CT image, after acquiring the first and second cropped original images, the Hounsfield Unit (HU) values ​​of the first and second cropped original images can be truncated to a range of [-100, 250] based on the experience of radiologists.

[0052] 3. The first and second original images processed in the above two steps may also be normalized. For example, taking the original images corresponding to the liver as an example, the original images corresponding to the liver may be normalized to have a mean of 0 and a variance of 1, so that the original images corresponding to the liver have the same voxel distribution, thereby eliminating differences in voxel distribution when each device acquires the original images corresponding to the liver.

[0053] In one example, an image segmentation model may be overly sensitive to regions with high HU values, but these regions are not necessarily the regions corresponding to the target object, thus affecting the image segmentation model's recognition of the target object. Alternatively, a target object probability map can be combined to provide the image segmentation model with auxiliary information about the target object. The specific method can be as follows: by calculating the eigenvectors of the Hessian matrix, the similarity between the processed original image and the target object is obtained, the edge information of the target object is detected, and the planar and spherical structures in the 3D CT image are eliminated. Thus, regions corresponding to potential target objects are assigned a higher probability, making the regions corresponding to the target object in the processed original image appear as brighter regions, thereby obtaining a target object probability map corresponding to the processed original image. Finally, the processed original image and its corresponding target object probability map are directly merged in the input space of the image segmentation model to obtain an image sample for the image segmentation model. By simultaneously considering the information in the original image and its corresponding target object probability map, this embodiment enables the image segmentation model to more robustly perceive target object signals, thereby improving the segmentation performance of the image segmentation model.

[0054] For example, taking the task of segmenting blood vessels in the liver as an example, refer to Figure 2 The original dataset includes limited high-quality annotated data (corresponding to the first sample data) and abundant low-quality noisy annotated data (corresponding to the second sample data). The original image 201 (a CT image) is preprocessed to obtain a processed original image 202. The processed original image 202 and its corresponding vascular probability map 203 are spliced ​​to obtain an image sample.

[0055] Step 102: The first image sample and the second image sample are segmented by the student network of the image segmentation model to obtain the student segmentation result of the first image sample and the student segmentation result of the second image sample, and the first image sample and the second image sample are segmented by the teacher network of the image segmentation model to obtain the teacher segmentation result of the first image sample and the teacher segmentation result of the second image sample.

[0056] Optionally, the image segmentation model includes a student network and a teacher network. The student network is the target training network, and the teacher network is used to assist in the training of the student network. Optionally, the teacher network can be pre-trained and has superior segmentation performance compared to the student network. During the training process of the image segmentation model, the student network can learn the superior segmentation performance of the teacher network, and through the auxiliary training of the teacher network, the student network can simultaneously learn useful image information from the first image sample and the second image sample, as well as learn useful annotation information from the annotated data of the first image sample and the annotated data of the second image sample.

[0057] Optionally, both the student network and the teacher network can employ a U-Net (U-shaped network) architecture. The U-Net can be used to classify each pixel in an image sample, with pixels of different categories being labeled using different channels. The network parameters of the student network differ from those of the teacher network. Optionally, the relationship between the network parameters of the student network and the teacher network satisfies a relationship corresponding to an EMA (Exponential Moving Average) strategy.

[0058] The student segmentation result of the first image sample refers to the segmentation result obtained by the student network performing segmentation processing on the first image sample. The segmentation result is used to represent the student network's prediction result for the category of each pixel point in the first image sample. The prediction result of the category of each pixel point can be represented by prediction annotation data (such as prediction probability value), and the prediction annotation data can be used to represent the possibility that the pixel point belongs to a certain category. The student segmentation result of the second image sample is used to represent the student network's prediction result for the category of each pixel point in the second image sample. The teacher segmentation result of the first image sample is used to represent the teacher network's prediction result for the category of each pixel point in the first image sample. The teacher segmentation result of the second image sample is used to represent the teacher network's prediction result for the category of each pixel point in the second image sample.

[0059] For example, the student network and the teacher network have the same segmentation process for the image sample. The following describes the process of obtaining the student segmentation result of the first image sample. The specific content can be as follows:

[0060] After obtaining the first image sample, the student network performs downsampling processing on the first image sample, that is, using continuous convolution pooling layers to extract feature information in the first image sample, and gradually maps the feature information to high dimensions to obtain rich feature information corresponding to the first image sample. The high-dimensional feature information is then deconvolved to map the high-dimensional feature information into low-dimensional feature information. In this process, the feature information in the deconvolution process is fused with the feature information with the same dimension in the downsampling process to enhance the segmentation accuracy of the student network until an output image with the same dimension as the first image sample is obtained. The output image can be an image that only includes the target object. Optionally, the student segmentation result of the first image sample can be a feature map before the output image, which can be used to represent the student network's prediction result for the category of each pixel in the first image sample.

[0061] Optionally, the student network and the teacher network may also adopt DenseNet, ResNet, etc., which is not limited in the embodiment of the present application.

[0062] For example, refer to Figure 2 The image segmentation model includes a student network 204 and a teacher network 205. The first image sample is segmented by the student network 204 to obtain a student segmentation result 206 of the first image sample. The second image sample is segmented by the student network 204 to obtain a student segmentation result 207 of the second image sample. The first image sample is segmented by the teacher network 205 to obtain a teacher segmentation result 208 of the first image sample. The second image sample is segmented by the teacher network 205 to obtain a teacher segmentation result 209 of the second image sample.

[0063] Step 103: Determine the supervision loss based on the student segmentation result of the first image sample and the labeled data of the first image sample.

[0064] In an embodiment of the present application, the supervised loss is used for the student network to learn useful information (i.e., the labeled data for the target object) from the labeled data of the first image sample. Optionally, the supervised loss may include at least one of the following: cross entropy loss, Dice loss, boundary loss, and focus loss, which can be selected according to the content of the supervised loss as needed. Exemplarily, the weighted sum of the cross entropy loss, Dice loss, boundary loss, and focus loss is set as the supervised loss. Among them, the weight parameters of each loss can be set to 0.5, 0.5, 1, and 0.5, respectively.

[0065] In one example, the process of obtaining the supervised loss can be as follows: the student segmentation result of the first image sample and the labeled data of the first image sample are calculated and processed by the cross entropy loss function to determine the first sub-supervised loss; the student segmentation result of the first image sample and the labeled data of the first image sample are calculated and processed by the Dice loss function to determine the second sub-supervised loss; the student segmentation result of the first image sample and the labeled data of the first image sample are calculated and processed by the boundary loss function to determine the third sub-supervised loss; the student segmentation result of the first image sample and the labeled data of the first image sample are calculated and processed by the focal loss function to determine the fourth sub-supervised loss; the first sub-supervised loss, the second sub-supervised loss, the third sub-supervised loss and the fourth sub-supervised loss are weightedly summed to determine the supervised loss.

[0066] Among them, the first sub-supervised loss (i.e., cross entropy loss) is used to represent the distance between the student segmentation result of the first image sample and the labeled data of the first image sample. By minimizing the first sub-supervised loss, the distance between the student segmentation result of the first image sample and the labeled data of the first image sample can be shortened, thereby enabling the student network to learn useful information in the labeled data of the first image sample. The second sub-supervised loss (i.e., Dice loss) is used to represent the overlap between the student segmentation result of the first image sample and the labeled data of the first image sample. By minimizing the second sub-supervised loss, the overlap between the student segmentation result of the first image sample and the labeled data of the first image sample can be improved. The third sub-supervised loss (i.e., focal loss) can be used to reduce the weight parameters of easily classified pixels and increase the weight parameters of difficult-to-classify pixels in the student segmentation result of the first image sample and the labeled data of the first image sample, and at the same time shorten the distance between the student segmentation result of the first image sample and the labeled data of the first image sample. The fourth sub-supervised loss (i.e., boundary loss) can be used to represent the distance between the segmentation boundary of the target object in the student segmentation result of the first image sample and the segmentation boundary of the target object in the labeled data of the first image sample. By minimizing the fourth sub-supervised loss, the distance between the segmentation boundary of the target object in the student segmentation result of the first image sample and the segmentation boundary of the target object in the labeled data of the first image sample can be shortened.

[0067] Step 104 , determining consistency loss based on difference information between the student segmentation result of the first image sample and the teacher segmentation result of the first image sample, and difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample.

[0068] The consistency loss can represent the degree of difference between the student segmentation result of the first image sample and the teacher segmentation result of the first image sample, as well as the degree of difference between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample. By minimizing the consistency loss, the degree of difference between the student network and the teacher network can be reduced, so that the student network can learn useful information from the teacher network. For example, the teacher network learns the annotation information from the annotation data of the first image sample and the annotation data of the second image sample, and then by minimizing the consistency loss, the student network can indirectly learn the annotation information from the annotation data of the first image sample and the annotation data of the second image sample based on the teacher network. In this way, the student network can learn additional image information and annotation information without being affected by the annotation data of the second image sample.

[0069] In one example, the process of obtaining the consistency loss can be as follows: the difference information between the student segmentation result of the first image sample and the teacher segmentation result of the first image sample is calculated and processed through the mean square error function to determine the first sub-consistency loss; the difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample is calculated and processed through the mean square error function to determine the second sub-consistency loss; the first sub-consistency loss and the second sub-consistency loss are weightedly summed to determine the consistency loss.

[0070] Step 105 : Based on the supervision loss and the consistency loss, the student network is trained to segment the target image using the trained student network.

[0071] Optionally, the parameters of the image segmentation model are adjusted by minimizing the supervision loss and the consistency loss to complete the training of the image segmentation model, that is, the network parameters of the student network and the network parameters of the teacher network of the image segmentation model are adjusted to complete the training of the image segmentation model.

[0072] In one example, the training process of the student network and the teacher network can be as follows: adjust the current network parameters of the student network based on the supervision loss and the consistency loss to obtain the adjusted student network; obtain the parameter difference information between the current network parameters of the teacher network and the network parameters of the adjusted student network; perform attenuation calculation on the parameter difference information to obtain the attenuated parameter difference information; replace the current network parameters of the teacher network with the sum of the attenuated parameter difference information and the network parameters of the adjusted student network to obtain the adjusted teacher network.

[0073] For example, the EMA strategy can be used to update the network parameters of the teacher network based on the network parameters of the student network. Assume that during the tth iteration, the network parameters of the adjusted student network are expressed as θ t, the network parameters θ′ of the adjusted teacher network can be obtained using the EMA strategy t :

[0074] θ′ t =αθ′ t-1 +(1-α)θ t =θ t +α(θ′ t-1 -θ t );

[0075] Among them, α is the decay rate corresponding to the EMA strategy, θ′ t-1 Refers to the network parameters of the teacher network before adjustment. Optionally, α can be set to 0.99.

[0076] Optionally, for the same image sample, perturbations can be added to the image sample before the teacher network obtains the image sample. For example, Figure 2 , random noise ξ (such as incorrectly labeled data) is added to the image sample corresponding to the teacher network 205, and the student network 204 directly segments the image sample, and the segmentation results corresponding to the teacher network 205 and the student network 204 are kept consistent through consistency loss, so as to enhance the robustness of the student network 204 and enable the student network 204 to capture the target object more accurately.

[0077] In one example, in order to further mine potential useful information from the labeled data of the second image sample and mitigate the negative impact of the labeled data of the second image sample on the segmentation performance of the student network, the training loss of the image segmentation network may further include a learning loss. The learning loss is used for the student network to learn useful information from the corrected labeled data of the second image sample. The specific process of obtaining the learning loss may be as follows: using the teacher segmentation result of the second image sample to correct the labeled data of the second image sample to obtain the corrected labeled data of the second image sample; and determining the learning loss based on the student segmentation result of the second image sample and the corrected labeled data of the second image sample. The corrected labeled data of the second image sample refers to the labeled data of the second image sample after the included erroneous labels have been corrected.

[0078] Exemplarily, the learning loss may include a cross-entropy loss and a focal loss. A first sub-learning loss is determined by calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a cross-entropy loss function; a second sub-learning loss is determined by calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a focal loss function; and a weighted summation of the first and second sub-learning losses is performed to determine the learning loss.

[0079] Optionally, the process of acquiring the corrected annotation data of the second image sample can be as follows: based on the teacher segmentation result of the second image sample, the annotation data of the second image sample is identified and processed to obtain a noise landmark map corresponding to the second image sample, and the noise landmark map is used to identify incorrect annotations in the second image sample; based on the noise landmark map, the annotation data of the second image sample is corrected and processed to obtain the corrected annotation data of the second image sample. The noise landmark map can be in binary form. For example, "1" in the noise landmark map indicates that the pixel point has incorrect annotation data, and "0" in the noise landmark map indicates that the annotation data corresponding to the pixel point is correct annotation data.

[0080] In one example, confidence learning technology can be used to identify incorrect labels from the labeled data of the second image sample at the pixel level. The specific content can be as follows: based on the labeled data of each pixel in the second image sample, the out-of-sample prediction probability of the first category and the out-of-sample prediction probability of the second category of each pixel, a confusion matrix of the second image sample is constructed, the confusion matrix including the number of first-category pixels labeled as the first category and whose out-of-sample prediction probability of the first category is greater than a first threshold, the number of second-category pixels labeled as the first category but whose out-of-sample prediction probability of the second category is greater than a second threshold, the number of third-category pixels labeled as the second category and whose out-of-sample prediction probability of the second category is greater than the second threshold, and the number of fourth-category pixels labeled as the second category but whose out-of-sample prediction probability of the first category is greater than the first threshold; based on the confusion matrix, a joint distribution matrix corresponding to the first category and the second category is constructed, the joint distribution matrix including the joint probability of the first-category pixels, the joint probability of the second-category pixels, the joint probability of the third-category pixels and the joint probability of the fourth-category pixels; based on the joint probability of the second-category pixels, a noise landmark map corresponding to the second image sample is obtained. Optionally, the categories corresponding to the pixel points may also include a third category, a fourth category, etc., which can be set according to actual conditions and are not limited in the embodiments of the present application.

[0081] For example, let’s take the task of segmenting blood vessels in the liver as an example. In this segmentation task, the annotation data of the liver image sample can include two categories: blood vessels and non-blood vessels. The blood vessels can be set as the first category (hereinafter referred to as i), and the non-blood vessels can be set as the second category (hereinafter referred to as j). Assume that the annotation data of the pixel point is The potential labeled data corresponding to the pixel is y* (i.e., the possible real labeled data corresponding to the pixel). The liver image sample includes n pixels, so the dataset corresponding to the liver image sample can be expressed as:

[0082] Among them, x is the pixel point.

[0083] The out-of-sample prediction probability that the pixel point is of the first category, the out-of-sample prediction probability that the pixel point is of the second category, and the joint out-of-sample prediction probability of the first category and the second category corresponding to the pixel point can be determined based on the teacher segmentation result of the second image sample (for example, the out-of-sample prediction probability that the pixel point is of the first category can be the probability that the pixel point is of the first category in the teacher segmentation result of the second image sample).

[0084] If the labeled data The labeled data y corresponding to the pixel point ＊ If the out-of-sample prediction probability is greater than the set threshold, it can be considered that the true annotation data of the pixel point is y ＊ instead of That is, the pixel contains incorrectly labeled data. For example, if pixel A is labeled as a blood vessel, but the out-of-sample prediction probability of the potential labeled data of pixel A is greater than the set threshold, then the pixel A can be considered as a non-blood vessel. The set threshold is calculated by calculating the labeled data in the teacher segmentation result of the second image sample. The average predicted probability is obtained by taking the second threshold (i.e. the set threshold corresponding to j) as an example, which can be expressed as follows:

[0085] Among them, t j is the set threshold corresponding to j. Optionally, the set threshold corresponding to i can be expressed as t i .

[0086] Furthermore, the number of pixels marked as i but possibly i, the number of pixels marked as i but possibly j, the number of pixels marked as j but possibly i, and the number of pixels marked as j but possibly j in the labeled data of the first image sample are counted to construct a confusion matrix corresponding to the labeled data of the first image sample.

[0087] in, Refers to the one marked as i (i.e. ) but may be j (ie y ＊ =j), which can be expressed as:

[0088]

[0089] in, is the out-of-sample prediction probability of j, M is the category set, and l is the intermediate parameter.

[0090] Using the constructed confusion matrix We can further construct a joint distribution matrix of size m*m (number of categories) for i and j:

[0091]

[0092] Alternatively, a PBC (Prune by Class) strategy can be used to identify incorrect annotations. The process can be as follows:

[0093] For pixels marked as i, sort them in ascending order according to the confidence value (i.e., the probability value of the marked data corresponding to i), and multiply n and the joint probability of the second type of pixels (i.e., pixels marked as i but possibly j) to obtain the expected value N. The expected value N can be expressed as follows:

[0094]

[0095] The labeled data corresponding to the first N pixels in the sorting are identified as incorrectly labeled, and a noise landmark map corresponding to the second image sample is constructed. Because the confident learning technique is computationally efficient and does not require any additional hyperparameters, it does not affect the training efficiency of the image segmentation model.

[0096] For example, refer to Figure 2 A confidence learning network 210 may be set in the image segmentation model, and the confidence learning network 210 may be used to identify and process the labeled data 211 of the second image sample based on the teacher segmentation result 209 of the second image sample to obtain a noise landmark map 212 corresponding to the second image sample.

[0097] In one example, due to the uncertainty in the identification process of incorrect annotations, the present application adopts a smooth soft correction strategy, and the correction process of the annotation data of the second image sample can be as follows:

[0098] The corrected annotation data of the second image sample is calculated according to the following formula:

[0099]

[0100] Where II is the indicator function, τ∈[0,1] is the smoothing factor, x is the pixel point in the second image sample, is the labeled data, X n is the second type of pixel set corresponding to the noise sign map, and n is the number of pixels in the second type of pixel set corresponding to the noise identification map.

[0101] For example, refer to Figure 2 A soft correction network 213 may be provided in the image segmentation model. The soft correction network 213 performs smooth soft correction on the labeled data 211 of the second image sample based on the noise landmark map 212 to obtain the corrected labeled data 214 of the second image sample.

[0102] Optionally, the student network can be trained based on the supervision loss, consistency loss and learning loss, and the network parameters of the teacher network can be adjusted based on the network parameters of the adjusted student network using the EMA strategy to obtain the adjusted teacher network.

[0103] For example, the supervision loss, consistency loss, and learning loss can be weighted and summed to obtain the total training loss of the student network. The student network is trained based on the total training loss, which can be expressed as follows:

[0104] L=L s +λ c L c +λ cl L cl ;

[0105] Among them, L s To monitor the loss, L c is the consistency loss, L cl is the learning loss, λ cl is the weight parameter of learning loss, λ c is the weight parameter of the consistency loss. Optionally, the present application automatically sets the weight parameter of the consistency loss by adopting a time-dependent Gaussian function, which can be expressed as follows:

[0106]

[0107] Among them, w max represents the final weight parameter, which can be set to an empirical value of 0.1, t represents the current training step, t max is the maximum training step.

[0108] Optionally, the teacher network needs to be "warmed up" to provide relatively reliable out-of-sample prediction probabilities, so the weight parameter of the learning loss can be set to 0 in the first iteration under the preset value (for example 4000), and modified to 0.5 during the remaining iterations.

[0109] Optionally, the student network can be trained for the next round based on at least one first image sample, the labeled data of the first image sample, at least one second image sample and the corrected labeled data of the second image sample, so as to further reduce the influence of the labeled data of the second image sample on the student network, thereby ensuring the segmentation performance of the student network.

[0110] Optionally, the target image is finally segmented by the trained student network to obtain a segmented image of the target object in the target image. The target image and the specific segmentation process of the target image will be described in detail below.

[0111] To summarize, the technical solution provided by the embodiments of the present application, by performing a consistency comparison between the prediction results of the teacher network for image samples with high-quality annotations and the results of the student network for image samples with high-quality annotations, and performing a consistency comparison between the prediction results of the teacher network for image samples with low-quality annotations and the prediction results of the student network for image samples with low-quality annotations, enables the teacher network to help the student network learn useful image information from both image samples with high-quality annotations and image samples with low-quality annotations, thereby avoiding the problem of network overfitting caused by lack of image samples, and thus ensuring the segmentation accuracy of the student network.

[0112] In addition, since the teacher network helps the student network learn useful image information from image samples with low-quality annotations, rather than the student network directly learning useful image information from image samples with low-quality annotations, the impact of image samples with low-quality annotations on the student network can be reduced, thereby ensuring the segmentation performance of the student network.

[0113] In addition, by determining the learning loss of the student network based on the corrected labeled data of the second image version, the student network can learn useful labeling information from image samples with low-quality labels without being affected by the low-quality labels, thereby improving the segmentation performance of the student network.

[0114] In addition, the image segmentation model is trained in the next round based on the corrected labeled data of the second image sample. Due to the improvement in the quality of the labeled data, the segmentation performance of the image segmentation model is further improved.

[0115] Please refer to Figure 3 , which shows a flowchart of an image segmentation method provided by another embodiment of the present application. The execution body of each step of the method can be the computer device introduced above, and the method can include the following steps (301~303).

[0116] Step 301: Obtain a target image and a target object probability map corresponding to the target image. The target object probability map is used to represent the distribution of target objects.

[0117] In the embodiments of the present application, the target image can be a CT image, an image, a picture, a photograph, etc. The target image includes a target object and a target region corresponding to the target object. The target object can be set according to the segmentation task. For example, in the task of intra-organ blood vessel segmentation, the target region is the organ, and the target object is the blood vessel within the organ; in the task of license plate recognition, the target region is the vehicle, and the target object is the license plate on the vehicle. The embodiments of the present application are not limited to this.

[0118] A target object probability map is a distribution probability map of target objects. For example, a blood vessel distribution probability map within the liver can be used to represent the distribution of blood vessels within the liver. The target object probability map can be used to provide auxiliary information about target objects, such as their location, number, and edges, to the image segmentation model. The target object probability map can be obtained using a multi-scale target object (e.g., blood vessel) enhancement filter network.

[0119] Step 302 : splicing the preprocessed target image with the target object probability map corresponding to the target image to obtain an input image; wherein the preprocessing includes at least one of the following: cropping, clarity adjustment, and normalization.

[0120] Optionally, the process of obtaining the preprocessed target image can be as follows: masking the target image and cropping the target area to obtain a cropped target image, then performing clarity adjustment processing on the cropped target image to obtain a clarity-adjusted cropped target image, and finally normalizing the clarity-adjusted cropped target image to obtain a preprocessed target image. Optionally, the input image is used as input to the image segmentation model.

[0121] Step 303 , segmenting the input image using the student network of the image segmentation model to obtain an output segmented image, where the output segmented image includes the target object segmented from the input image.

[0122] The image segmentation model's student network extracts feature information from both the preprocessed target image and its corresponding target object probability map. It then segments the preprocessed target image based on the feature information corresponding to the target object probability map, producing an output segmented image. The image segmentation model's student network is a pre-trained network, and the specific training method is not detailed here.

[0123] For example, taking the task of segmenting blood vessels in the liver as an example, if the input image is composed of a processed liver image and a corresponding blood vessel probability map, then after segmentation processing by the student network of the image segmentation model, an output segmented image containing only blood vessels can be obtained.

[0124] In one example, the target image is a 2D axial slice of a 3D image. After obtaining the output segmentation image of the target image, the multiple output segmentation images corresponding to the 3D image can be superimposed to obtain a 3D output segmentation image. To filter out unreasonable predictions, the 3D output segmentation image needs to be filtered. In other words, connected regions in the 3D output segmentation image with a volume less than a threshold are deleted to obtain the final 3D output segmentation map.

[0125] Then, based on the final 3D output segmentation map, guidance information for the target object is obtained.

[0126] In an exemplary embodiment, the task of liver blood vessel segmentation is taken as an example. We used two public datasets to train and test the image segmentation model. One of the datasets is the 3DIRCADb dataset and the other is the MSD8 dataset. Both the 3DIRCADb dataset and the MSD8 dataset provide annotation data of liver blood vessels. The 3DIRCADb dataset includes 20 abdominal CT images with high-quality liver annotation data and liver blood vessel annotation data. The axial 2D slice size of each abdominal CT image is the same, both are 512*512 pixels, and the number of axial 2D slices of each abdominal CT image is between 74 and 260. The MSD8 dataset includes 443 abdominal CT images with low-quality liver blood vessel annotation data, and the properties of the abdominal CT images are similar to those of the abdominal CT images in the 3DIRCADb dataset. Among them, approximately 65.5% of the voxels are not labeled, and approximately 8.5% of the voxels are incorrectly labeled as blood vessels. Reference Figure 4 It can be seen that the quality of the liver blood vessel annotation data in the 3DIRCADb dataset 401 is better than that in the MSD8 dataset 402.

[0127] Optionally, the axial 2D slices and the corresponding liver vascular annotation data corresponding to 10 CT images in the 3DIRCADb dataset were used as the first original dataset, and the axial 2D slices and the corresponding liver vascular annotation data corresponding to another 10 CT images were used to test the effectiveness of the image segmentation model. The axial 2D slices and the corresponding liver vascular annotation data corresponding to all abdominal CT images in the MSD8 dataset were used as the second original dataset.

[0128] Each axial 2D slice in the first original data set and each axial 2D slice in the second original data set are preprocessed to obtain a processed first axial 2D slice (hereinafter referred to as the processed first original liver image) and a processed second axial 2D slice (hereinafter referred to as the processed second original liver image).

[0129] Optionally, the scale of the processed first original liver image and the processed second original liver image is 320*320 pixels, the HU values ​​of the processed first original liver image and the processed second original liver image are within the range of [-100, 250], and the processed first original liver image and the processed second original liver image are normalized to a mean of 0 and a variance of 1. In addition, a vascular probability map corresponding to each of the processed first original liver image and the processed second original liver image is obtained using a multi-scale vascular enhancement filter network.

[0130] Finally, the processed first original liver image and its corresponding vascular probability map are spliced ​​into the first liver image sample (i.e., the liver image sample corresponding to the 3DIRCADb dataset), and the processed second original liver image and its corresponding vascular probability map are spliced ​​into the second liver image sample (i.e., the liver image sample corresponding to the MSD8 dataset).

[0131] Using the image segmentation method provided in an embodiment of the present application, an image segmentation model is trained based on the first liver image sample, the liver blood vessel annotation data of the first liver image sample, the second liver image sample, and the liver blood vessel annotation data of the second liver image sample to obtain a trained student network.

[0132] The image segmentation model was then tested using 2D axial slices corresponding to the remaining 10 CT images in the 3DIRCADb dataset, along with the corresponding liver vascular annotation data. The input liver image was acquired based on the target liver image using the same acquisition method as the liver image samples. Finally, the trained Student network was used to segment the input liver image to produce an output vascular segmentation map. This output vascular segmentation map was then superimposed to form a 3D output vascular segmentation map. Finally, the 3D output vascular segmentation map was filtered (for example, by filtering out connected regions less than 0.1% of the entire image volume) to obtain the final 3D output vascular segmentation map.

[0133] Optionally, the segmentation results of the image segmentation model can be comprehensively evaluated by using evaluation indicators such as the average Dice score, average surface distance, accuracy, and Hausdorff distance between the predicted segmentation mask obtained by the image segmentation model and the true segmentation mask.

[0134] In an exemplary embodiment, the image segmentation method provided in the present application is compared with other methods. For the convenience of description, "i" represents the processed original liver image and the corresponding high-quality liver vascular annotation data, "p" represents the vascular probability map and the corresponding high-quality liver vascular annotation data, "c" represents the input liver image spliced ​​with the processed original liver image and the vascular probability map and the corresponding high-quality liver vascular annotation data, U-Net(i) represents the U-shaped structure network trained only with "i", U-Net(p) represents the U-shaped structure network trained only with "p", U-Net(c) represents the U-shaped structure network trained with "c", U-Net(c, Mix) is the U-shaped structure network trained with "c" and additional sample data with low-quality liver vascular annotation data, and MTCL(c) represents the student network trained using the technical solution provided in the embodiment of the present application.

[0135] refer to Figure 5 and Figure 6 , Figure 5 The final 3D output vascular segmentation map 501 of U-Net(i), the final 3D output vascular segmentation map 502 of U-Net(p), the final 3D output vascular segmentation map 503 of U-Net(c), the final 3D output vascular segmentation map 504 of U-Net(c, Mix), and the final 3D output vascular segmentation map 505 of MTCL(c) are shown. Figure 6 The quantitative comparison results provided by the embodiments of the present application are shown. The scores of U-Net(c) under various evaluation indicators are better than those of U-Net(i) and U-Net(p), which indicates that considering "i" and "p" at the same time can make the image segmentation model perceive vascular signals more robustly. The scores of U-Net(c, Mix) under various evaluation indicators have degraded compared with U-Net(c). Among them, the average Dice score of U-Net(c, Mix) dropped from 0.6685 to 0.6338, and its accuracy dropped from 0.6699 to 0.6322. It can be seen that directly referencing additional sample data with low-quality liver vascular annotation data will cause the performance of the image segmentation model to degrade seriously. However, the scores of MTCL(c) under various evaluation indicators are better than those of U-Net(i), U-Net(p), U-Net(c) and U-Net(c, Mix), indicating that the image segmentation method provided in the embodiment of the present application is the most effective. It can not only eliminate the impact of low-quality liver blood vessel annotation data on the image segmentation model, but also extract useful image information and annotation information from it.

[0136] To summarize, the technical solution provided by the embodiments of the present application, by performing a consistency comparison between the prediction results of the teacher network for image samples with high-quality annotations and the results of the student network for image samples with high-quality annotations, and performing a consistency comparison between the prediction results of the teacher network for image samples with low-quality annotations and the prediction results of the student network for image samples with low-quality annotations, enables the teacher network to help the student network learn useful image information from both image samples with high-quality annotations and image samples with low-quality annotations, thereby avoiding the problem of network overfitting caused by lack of image samples, and thus ensuring the segmentation accuracy of the student network.

[0137] In addition, since the teacher network helps the student network learn useful image information from image samples with low-quality annotations, rather than the student network directly learning useful image information from image samples with low-quality annotations, the impact of image samples with low-quality annotations on the student network can be reduced, thereby ensuring the segmentation performance of the student network.

[0138] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0139] refer to Figure 7 , which shows a block diagram of a training device for an image segmentation model provided by an embodiment of the present application. The device has the function of implementing the above method example, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the computer device described above, or it can be set in a computer device. Figure 7 As shown, the device 700 includes: a sample data acquisition module 701, a segmentation result acquisition module 702, a supervision loss determination module 703, a consistency loss determination module 704 and a student network training module 705.

[0140] The sample data acquisition module 701 is used to acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotation data of the first image sample, and the second sample data includes at least one second image sample and annotation data of the second image sample, and the quality of the annotation data of the first image sample is higher than the quality of the annotation data of the second image sample.

[0141] The segmentation result acquisition module 702 is used to perform segmentation processing on the first image sample and the second image sample respectively through the student network of the image segmentation model to obtain the student segmentation result of the first image sample and the student segmentation result of the second image sample, and to perform segmentation processing on the first image sample and the second image sample respectively through the teacher network of the image segmentation model to obtain the teacher segmentation result of the first image sample and the teacher segmentation result of the second image sample.

[0142] The supervision loss determination module 703 is used to determine the supervision loss according to the student segmentation result of the first image sample and the labeled data of the first image sample.

[0143] The consistency loss determination module 704 is used to determine the consistency loss based on the difference information between the student segmentation result of the first image sample and the teacher segmentation result of the first image sample, and the difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample.

[0144] The student network training module 705 is used to train the student network based on the supervision loss and the consistency loss, so as to perform segmentation processing on the target image through the trained student network.

[0145] In an exemplary embodiment, Figure 8As shown, the device 700 further includes: a correction annotation acquisition module 706 and a learning loss determination module 707.

[0146] The correction annotation acquisition module 706 is used to use the teacher segmentation result of the second image sample to perform correction processing on the annotation data of the second image sample to obtain the corrected annotation data of the second image sample.

[0147] The learning loss determination module 707 is configured to determine the learning loss according to the student segmentation result of the second image sample and the corrected annotation data of the second image sample.

[0148] The student network training module 705 is further configured to train the student network based on the supervision loss, the consistency loss, and the learning loss.

[0149] In an exemplary embodiment, the correction annotation acquisition module 706 is further configured to:

[0150] performing identification processing on the labeled data of the second image sample based on the teacher segmentation result of the second image sample to obtain a noise signature map corresponding to the second image sample, wherein the noise signature map is used to identify incorrect labels in the second image sample;

[0151] Correction processing is performed on the labeled data of the second image sample based on the noise signature map to obtain corrected labeled data of the second image sample.

[0152] In an exemplary embodiment, the correction annotation acquisition module 706 is further configured to:

[0153] constructing a confusion matrix of the second image sample based on the labeling data of each pixel in the second image sample, the out-of-sample prediction probability of the first category and the out-of-sample prediction probability of the second category of each pixel, the confusion matrix including the number of first-category pixels labeled as the first category and whose out-of-sample prediction probability of the first category is greater than a first threshold, the number of second-category pixels labeled as the first category but whose out-of-sample prediction probability of the second category is greater than a second threshold, the number of third-category pixels labeled as the second category and whose out-of-sample prediction probability of the second category is greater than the second threshold, and the number of fourth-category pixels labeled as the second category but whose out-of-sample prediction probability of the first category is greater than the first threshold;

[0154] Based on the confusion matrix, construct a joint distribution matrix corresponding to the first category and the second category, the joint distribution matrix including the joint probability of the first category pixel points, the joint probability of the second category pixel points, the joint probability of the third category pixel points, and the joint probability of the fourth category pixel points;

[0155] Based on the joint probability of the second type of pixels, a noise signature map corresponding to the second image sample is obtained.

[0156] In an exemplary embodiment, the correction annotation acquisition module 706 is further configured to:

[0157] The corrected annotation data of the second image sample is calculated according to the following formula:

[0158]

[0159] Wherein, II is the indicator function, τ∈[0,1] is the smoothing factor, x is the pixel point in the second image sample, is the labeled data, X n is the second type of pixel point set corresponding to the noise sign map, and n is the number of pixels in the second type of pixel point set corresponding to the noise identification map.

[0160] In an exemplary embodiment, the learning loss determination module 707 is configured to:

[0161] calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a cross entropy loss function to determine a first sub-learning loss;

[0162] calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a focal loss function to determine a second sub-learning loss;

[0163] A weighted sum is performed on the first sub-learning loss and the second sub-learning loss to determine the learning loss.

[0164] In an exemplary embodiment, Figure 8 As shown, the device 700 also includes a teacher network training module 708.

[0165] The student network training module 705 is further configured to adjust the current network parameters of the student network based on the supervision loss, the consistency loss, and the learning loss to obtain an adjusted student network.

[0166] The teacher network training module 708 is used to obtain parameter difference information between the current network parameters of the teacher network and the adjusted network parameters of the student network.

[0167] The teacher network training module 708 is further configured to perform attenuation calculation processing on the parameter difference information to obtain attenuated parameter difference information.

[0168] The teacher network training module 708 is further configured to replace the current network parameters of the teacher network with the sum of the attenuated parameter difference information and the adjusted network parameters of the student network to obtain the adjusted teacher network.

[0169] In an exemplary embodiment, the student network training module 705 is further used to perform a next round of training on the student network based on the at least one first image sample, the labeled data of the first image sample, the at least one second image sample and the corrected labeled data of the second image sample.

[0170] In an exemplary embodiment, the supervision loss determination module 703 is configured to:

[0171] calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a cross entropy loss function to determine a first sub-supervisory loss;

[0172] Calculating the student segmentation result of the first image sample and the labeled data of the first image sample using a Dice loss function to determine a second sub-supervision loss;

[0173] calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a boundary loss function to determine a third sub-supervision loss;

[0174] calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a focal loss function to determine a fourth sub-supervision loss;

[0175] A weighted sum is performed on the first sub-supervisory loss, the second sub-supervisory loss, the third sub-supervisory loss, and the fourth sub-supervisory loss to determine the supervisory loss.

[0176] In an exemplary embodiment, the consistency loss determination module 704 is configured to:

[0177] Calculating difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample using a mean square error function to determine a first sub-consistency loss;

[0178] calculating difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample using the mean square error function to determine a second sub-consistency loss;

[0179] A weighted sum is performed on the first sub-consistency loss and the second sub-consistency loss to determine the consistency loss.

[0180] In an exemplary embodiment, the sample data acquisition module 701 is configured to:

[0181] Acquire a first original data set and a second original data set; wherein the first original data set includes at least one first original image and annotated data of the first original image, and the second original data set includes at least one second original image and annotated data of the second original image;

[0182] Preprocessing the first original image and the second original image respectively to obtain a processed first original image and a processed second original image; wherein the preprocessing includes at least one of the following: cropping, sharpness adjustment, and normalization;

[0183] Obtaining target object probability maps corresponding to the first original image and the second original image respectively, wherein the target object probability maps are used to represent the distribution of target objects;

[0184] The processed first original image and the target object probability map corresponding to the first original image are spliced ​​to obtain the first image sample, and the processed second original image and the target object probability map corresponding to the second original image are spliced ​​to obtain the second image sample.

[0185] To summarize, the technical solution provided by the embodiments of the present application, by performing a consistency comparison between the prediction results of the teacher network for image samples with high-quality annotations and the results of the student network for image samples with high-quality annotations, and performing a consistency comparison between the prediction results of the teacher network for image samples with low-quality annotations and the prediction results of the student network for image samples with low-quality annotations, enables the teacher network to help the student network learn useful image information from both image samples with high-quality annotations and image samples with low-quality annotations, thereby avoiding the problem of network overfitting caused by lack of image samples, and thus ensuring the segmentation accuracy of the student network.

[0186] In addition, since the teacher network helps the student network learn useful image information from image samples with low-quality annotations, rather than the student network directly learning useful image information from image samples with low-quality annotations, the impact of image samples with low-quality annotations on the student network can be reduced, thereby ensuring the segmentation performance of the student network.

[0187] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0188] Please refer to Figure 9 , which shows a block diagram of the structure of a computer device provided in one embodiment of the present application. The computer device can be used to implement the image segmentation method provided in the above embodiment. Specifically, it may include the following contents.

[0189] The computer device 900 includes a central processing unit (CPU, central processing unit), GPU (graphics processing unit), and FPGA (field programmable gate array) 901, a system memory 904 including RAM (random-access memory) 902 and ROM (read-only memory) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 also includes a basic input / output system (I / O system) 906 for facilitating information transmission between various components within the server, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0190] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 such as a mouse and a keyboard for user input. The display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing input from a variety of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, a printer, or other types of output devices.

[0191] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. In other words, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0192] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above-mentioned ones. The above-mentioned system memory 904 and mass storage device 907 can be collectively referred to as memory.

[0193] According to an embodiment of the present application, the computer device 900 can also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905. Alternatively, the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0194] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned image segmentation method.

[0195] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and when the at least one instruction, the at least one program, the code set, or the instruction set is executed by a processor, the above-mentioned image segmentation method is implemented.

[0196] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0197] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image segmentation method.

[0198] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0199] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image segmentation method, characterized in that: The method comprises: Acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotated data of the first image sample, and the second sample data includes at least one second image sample and annotated data of the second image sample, and the quality of the annotated data of the first image sample is higher than the quality of the annotated data of the second image sample; Performing segmentation processing on the first image sample and the second image sample respectively through the student network of the image segmentation model to obtain a student segmentation result of the first image sample and a student segmentation result of the second image sample, and performing segmentation processing on the first image sample and the second image sample respectively through the teacher network of the image segmentation model to obtain a teacher segmentation result of the first image sample and a teacher segmentation result of the second image sample; determining a supervision loss based on a student segmentation result of the first image sample and the labeled data of the first image sample; determining a consistency loss based on difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample, and difference information between a student segmentation result of the second image sample and a teacher segmentation result of the second image sample; Correcting the labeled data of the second image sample using the teacher segmentation result of the second image sample to obtain corrected labeled data of the second image sample; determining a learning loss based on the student segmentation result of the second image sample and the corrected labeled data of the second image sample; Based on the supervision loss, the consistency loss and the learning loss, the student network is trained to perform segmentation processing on the target image through the trained student network.

2. The method according to claim 1, characterized in that The correcting the labeled data of the second image sample using the teacher segmentation result of the second image sample to obtain the corrected labeled data of the second image sample includes: performing identification processing on the labeled data of the second image sample based on the teacher segmentation result of the second image sample to obtain a noise signature map corresponding to the second image sample, wherein the noise signature map is used to identify incorrect labels in the second image sample; Correction processing is performed on the labeled data of the second image sample based on the noise signature map to obtain corrected labeled data of the second image sample.

3. The method according to claim 2, characterized in that The step of performing identification processing on the labeled data of the second image sample based on the teacher segmentation result of the second image sample to obtain a noise signature map corresponding to the second image sample includes: constructing a confusion matrix of the second image sample based on the labeling data of each pixel in the second image sample, the out-of-sample prediction probability of the first category and the out-of-sample prediction probability of the second category of each pixel, the confusion matrix including the number of first-category pixels labeled as the first category and whose out-of-sample prediction probability of the first category is greater than a first threshold, the number of second-category pixels labeled as the first category but whose out-of-sample prediction probability of the second category is greater than a second threshold, the number of third-category pixels labeled as the second category and whose out-of-sample prediction probability of the second category is greater than the second threshold, and the number of fourth-category pixels labeled as the second category but whose out-of-sample prediction probability of the first category is greater than the first threshold; Based on the confusion matrix, construct a joint distribution matrix corresponding to the first category and the second category, the joint distribution matrix including the joint probability of the first category pixel points, the joint probability of the second category pixel points, the joint probability of the third category pixel points, and the joint probability of the fourth category pixel points; Based on the joint probability of the second type of pixels, a noise signature map corresponding to the second image sample is obtained.

4. The method according to claim 2, characterized in that The correcting the labeled data of the second image sample based on the noise signature map to obtain the corrected labeled data of the second image sample includes: The corrected annotation data of the second image sample is calculated according to the following formula: Wherein, II is the indicator function, τ∈[0,1] is the smoothing factor, x is the pixel point in the second image sample, is the labeled data, X n is the second type of pixel point set corresponding to the noise sign map, and n is the number of pixels in the second type of pixel point set corresponding to the noise sign map.

5. The method according to claim 1, wherein The determining of the learning loss according to the student segmentation result of the second image sample and the corrected labeled data of the second image sample includes: calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a cross entropy loss function to determine a first sub-learning loss; calculating and processing the student segmentation result of the second image sample and the corrected labeled data of the second image sample using a focal loss function to determine a second sub-learning loss; A weighted sum is performed on the first sub-learning loss and the second sub-learning loss to determine the learning loss.

6. The method according to claim 1, characterized in that The method further comprises: Adjusting the current network parameters of the student network based on the supervision loss, the consistency loss, and the learning loss to obtain an adjusted student network; Obtaining parameter difference information between current network parameters of the teacher network and the adjusted network parameters of the student network; performing attenuation calculation processing on the parameter difference information to obtain attenuated parameter difference information; The current network parameters of the teacher network are replaced by the sum of the attenuated parameter difference information and the adjusted network parameters of the student network to obtain the adjusted teacher network.

7. The method according to claim 1, characterized in that The method further comprises: The student network is trained in the next round based on the at least one first image sample, the labeled data of the first image sample, the at least one second image sample, and the corrected labeled data of the second image sample.

8. The method according to claim 1, characterized in that The determining of the supervision loss according to the student segmentation result of the first image sample and the labeled data of the first image sample includes: calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a cross entropy loss function to determine a first sub-supervisory loss; Calculating the student segmentation result of the first image sample and the labeled data of the first image sample using a Dice loss function to determine a second sub-supervision loss; calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a boundary loss function to determine a third sub-supervision loss; calculating and processing the student segmentation result of the first image sample and the labeled data of the first image sample using a focal loss function to determine a fourth sub-supervision loss; A weighted sum is performed on the first sub-supervisory loss, the second sub-supervisory loss, the third sub-supervisory loss, and the fourth sub-supervisory loss to determine the supervisory loss.

9. The method according to claim 1, characterized in that The determining the consistency loss according to difference information between the student segmentation result of the first image sample and the teacher segmentation result of the first image sample, and difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample, includes: Calculating difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample using a mean square error function to determine a first sub-consistency loss; calculating difference information between the student segmentation result of the second image sample and the teacher segmentation result of the second image sample using the mean square error function to determine a second sub-consistency loss; A weighted sum is performed on the first sub-consistency loss and the second sub-consistency loss to determine the consistency loss.

10. The method according to claim 1, characterized in that The obtaining of the first sample data and the second sample data includes: Acquire a first original data set and a second original data set; wherein the first original data set includes at least one first original image and annotated data of the first original image, and the second original data set includes at least one second original image and annotated data of the second original image; Preprocessing the first original image and the second original image respectively to obtain a processed first original image and a processed second original image; wherein the preprocessing includes at least one of the following: cropping, clarity adjustment, and normalization; Obtaining target object probability maps corresponding to the first original image and the second original image respectively, wherein the target object probability maps are used to represent the distribution of target objects; The processed first original image and the target object probability map corresponding to the first original image are spliced ​​to obtain the first image sample, and the processed second original image and the target object probability map corresponding to the second original image are spliced ​​to obtain the second image sample.

11. An image segmentation device, characterized in that: The device comprises: a sample data acquisition module, configured to acquire first sample data and second sample data; wherein the first sample data includes at least one first image sample and annotated data of the first image sample, and the second sample data includes at least one second image sample and annotated data of the second image sample, and the quality of the annotated data of the first image sample is higher than the quality of the annotated data of the second image sample; a segmentation result acquisition module, configured to perform segmentation processing on the first image sample and the second image sample respectively through a student network of an image segmentation model to obtain a student segmentation result of the first image sample and a student segmentation result of the second image sample, and to perform segmentation processing on the first image sample and the second image sample respectively through a teacher network of the image segmentation model to obtain a teacher segmentation result of the first image sample and a teacher segmentation result of the second image sample; a supervision loss determination module, configured to determine a supervision loss based on a student segmentation result of the first image sample and the labeled data of the first image sample; a consistency loss determining module, configured to determine a consistency loss based on difference information between a student segmentation result of the first image sample and a teacher segmentation result of the first image sample, and difference information between a student segmentation result of the second image sample and a teacher segmentation result of the second image sample; a correction annotation acquisition module, configured to perform correction processing on the annotation data of the second image sample using the teacher segmentation result of the second image sample to obtain the corrected annotation data of the second image sample; a learning loss determination module, configured to determine a learning loss based on the student segmentation result of the second image sample and the corrected annotation data of the second image sample; The student network training module is used to train the student network based on the supervision loss, the consistency loss and the learning loss, so as to perform segmentation processing on the target image through the trained student network.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image segmentation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the image segmentation method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises computer instructions, and the computer instructions are executed by a processor to implement the image segmentation method according to any one of claims 1 to 10.

Images