Image segmentation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image segmentation method and device, electronic equipment and storage medium. The method comprises: performing feature extraction on source domain images and target domain images based on a semantic segmentation model to obtain source domain features of the source domain images and target domain features of the target domain images; processing the source domain features and the target domain features based on a domain discriminator, and adjusting the semantic segmentation model using the processing result, so that the semantic segmentation model outputs an image segmentation result in which the target domain images are aligned to the source domain images in an output feature space, entropy map information and key point information; wherein the domain discriminator comprises an output feature space discriminator, an entropy discriminator and a key point discriminator. Embodiments of the present disclosure can improve the segmentation performance of the semantic segmentation model for images of different domains.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image segmentation is a technique for classifying and recognizing individual objects within an image. For example, medical image segmentation classifies different organs in an image based on their category labels. This image segmentation technique has made significant progress with the development of convolutional neural networks.

[0003] However, due to differences in imaging machine type, imaging mode, imaging sequence, and acquisition parameters, the visual appearance (contrast, brightness, structural differences, etc.) of medical images generated for the same object under different conditions varies significantly, i.e., domain shift. Thus, in real-world scenarios, deep learning segmentation models trained on the source domain experience a severe drop in segmentation performance on the target domain, which has a different data distribution. For example, a semantic segmentation model trained on MR (Magnetic Resonance) images will have significantly reduced segmentation accuracy when directly segmenting CT (Computed Tomography) images. Summary of the Invention

[0004] This disclosure provides an image segmentation method, apparatus, electronic device, and storage medium to solve or alleviate one or more technical problems in the prior art.

[0005] As a first aspect of the present disclosure, the present disclosure provides an image segmentation method, including:

[0006] The source domain image and the target domain image are processed based on the semantic segmentation model to obtain the source domain features of the source domain image and the target domain features of the target domain image.

[0007] The source domain features and the target domain features are processed by a domain discriminator to obtain an image segmentation result in which the target domain image is aligned with the source domain image in terms of output feature space, entropy map information, and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator.

[0008] As a second aspect of the present disclosure, an image segmentation apparatus is provided, comprising:

[0009] The semantic segmentation module is used to extract features from the source domain image and the target domain image based on the semantic segmentation model, so as to obtain the source domain features of the source domain image and the target domain features of the target domain image.

[0010] The domain discriminator module is used to process the source domain features and the target domain features based on the domain discriminator, and to adjust the semantic segmentation model using the processing results, so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information, and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator.

[0011] As a third aspect of this disclosure, this disclosure provides an electronic device, including:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image segmentation method provided in the embodiments of this disclosure.

[0015] As a fourth aspect of the present disclosure, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the image segmentation method provided in the present disclosure.

[0016] The technical solution provided in this disclosure, when segmenting a target domain image, utilizes an output space discriminator, an entropy discriminator, and a keypoint discriminator to process the source domain features and target domain features provided by the semantic segmentation model. This adjusts the semantic segmentation model, enabling it to obtain image segmentation results aligned with the source domain image in terms of output feature space, entropy map information, and keypoint information. Consequently, it improves the segmentation performance of the semantic segmentation model for images in different domains and demonstrates good generalization ability for segmenting images using different imaging methods.

[0017] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this disclosure will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0019] Figure 1 This is a flowchart of an image segmentation method according to an embodiment of the present disclosure;

[0020] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram of the model training process according to an embodiment of the present disclosure;

[0022] Figure 4 This is a structural block diagram of a key point prediction branch according to an embodiment of the present disclosure;

[0023] Figure 5 This is a structural block diagram of a key point discriminator according to an embodiment of the present disclosure;

[0024] Figure 6 This is a structural block diagram of an image segmentation apparatus according to an embodiment of the present disclosure;

[0025] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. Therefore, the drawings and description are to be considered exemplary in nature and not restrictive.

[0027] Figure 1 This is a flowchart of an image segmentation method provided in an embodiment of this disclosure. Figure 1 As shown, an image segmentation method may include the following steps:

[0028] S110, Based on the semantic segmentation model, feature extraction is performed on the source domain image and the target domain image to obtain the source domain features of the source domain image and the target domain features of the target domain image;

[0029] S120, based on the domain discriminator, processes the source domain features and target domain features, and uses the processing results to adjust the semantic segmentation model so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator and a key point discriminator.

[0030] In this example, when segmenting the target domain image, the output space discriminator, entropy discriminator, and keypoint discriminator are used to process the source and target domain features provided by the semantic segmentation model. This adjusts the semantic segmentation model so that when segmenting the target domain image, it can obtain image segmentation results aligned with the source domain image in terms of output feature space, entropy map information, and keypoint information. Therefore, the segmentation performance of the semantic segmentation model for images in different domains is improved, and the generalization effect for segmenting images with different imaging methods is good.

[0031] For example, semantic segmentation models are used to classify individual target objects in an image. This includes classifying organs in medical images, vehicles in traffic images, and objects in object images.

[0032] For example, the source domain image and the target domain image are imaged in different ways. For instance, the source domain image is an MR image, and the target domain image is a CT image.

[0033] For example, source domain features may include keypoint information, semantic segmentation results, and entropy map information. Target domain features may include keypoint information, semantic segmentation results, and entropy map information.

[0034] For example, a domain discriminator may include one or more. The domain discriminator is used to determine the source of the input information (source domain features or target domain features), i.e., whether it originates from a source domain image or a target domain image.

[0035] A domain discriminator may include at least one of an output feature space discriminator, an entropy discriminator, and a keypoint discriminator. The output feature space discriminator determines the source of the semantic segmentation result, i.e., whether it originates from a source domain image or a target domain image. The entropy discriminator determines the source of the entropy map information, i.e., whether it originates from a source domain image or a target domain image. The keypoint discriminator determines the source of the entropy map information, i.e., whether it originates from a source domain image or a target domain image.

[0036] Shannon entropy (also known as information entropy or simply entropy) describes the degree of uncertainty in an uncertain event and is a measure of the uncertainty of a random variable. The greater the uncertainty of a random variable, the greater the entropy; conversely, the less uncertain the variable, the smaller the entropy.

[0037] For example, feature distribution alignment refers to the degree of similarity between the feature distribution obtained by the semantic segmentation model for segmenting and recognizing the source domain image and the feature distribution obtained for segmenting and recognizing the target domain image. The higher the similarity, the higher the feature distribution alignment.

[0038] For example, both the semantic segmentation model and the domain discriminator can be constructed using neural networks.

[0039] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that incorporates convolutional computations and has a deep structure. They are one of the representative algorithms in deep learning. CNNs possess representation learning capabilities, enabling them to classify input information invariantly according to their hierarchical structure. Unlike conventional neural networks, neurons in each layer of a CNN are connected to only a small region in the previous layer, rather than using a fully connected approach.

[0040] For example, semantic segmentation models can be designed based on the classic UNet segmentation network architecture. For instance, DR-UNet, based on the classic UNet segmentation network architecture, combines the ideas of ResNet and DenseNet to establish dense residual modules, which are then integrated into the UNet encoder and decoder.

[0041] For example, the domain discriminator can be constructed using a generative adversarial network (GAN). GANs are an important generative model in deep learning, where two networks (a generator and a discriminator) are trained alternately at the same time and engage in a minimax game. They occupy a dominant position in unsupervised deep learning and can be used to capture higher-order correlations in observed or visible data when target class label information is unavailable.

[0042] PatchGAN is a discriminator structure of generative adversarial networks. Unlike directly evaluating the entire image generated by the generator, PatchGAN evaluates the entire image in blocks and then integrates the evaluation results of each block as the evaluation of the entire image.

[0043] In practical applications, training image segmentation models using domain adaptation is quite common. Unsupervised Domain Adaptation (UDA), a type of transfer learning, is a machine learning algorithm designed to address the domain shift between the source and target domains. It aims to learn domain-invariant features of both the source and target domains, allowing the classifier learned from the source domain to be applied to the target domain even when the target domain lacks labels. This reduces the gap between the source and target domains by generating labels for the target domain using labeled data from the source domain. Domain adaptation typically makes three assumptions: similar probability distributions between the source and target domains, similar features between the source and target domains, and similarity of certain subspaces between the source and target domains. Specifically, for the source and target domains, the source domain is the initial training set with labeled information, representing existing knowledge; the target domain is new knowledge to be learned, different from the source domain and following an independent and identically distributed distribution, without labeled information.

[0044] For example, the above-mentioned updating of the semantic segmentation model and domain discriminator mainly refers to updating or training the model parameters.

[0045] For example, the model parameters of the semantic segmentation model and the domain discriminator are updated alternately. For instance, the model parameters of the semantic segmentation model are first fixed, and then the model parameters of the domain discriminator are updated based on the discriminant loss function until the discriminant loss function of the domain discriminator meets a set condition or reaches a set number of iterations. Then, the model parameters of the domain discriminator are fixed, and the semantic segmentation model is trained based on the adversarial loss function until both the loss function of the semantic segmentation model and the adversarial loss function of the domain discriminator meet the set conditions or reach the set number of iterations, at which point the training of the semantic segmentation model can be stopped.

[0046] The loss function of the domain discriminator includes a discriminative loss function and an adversarial loss function. The discriminative loss function evaluates the accuracy of the domain discriminator's discrimination, while the adversarial loss function evaluates the domain discriminator's inability to discriminate the source of the input image. For example, a smaller adversarial loss function means that the domain discriminator has a harder time determining the source of the input image.

[0047] Training deep learning models requires a large amount of labeled data. However, creating annotations for each new data source (target domain) is a tedious, time-consuming task that is susceptible to subjective factors, making it impossible to quickly and robustly deploy the model on new data sources. Therefore, the solution provided in this disclosure can train a semantic segmentation network using both labeled source domain data and unlabeled target domain data simultaneously, even when the target domain is unlabeled, thereby improving the accuracy of target domain segmentation.

[0048] In some embodiments, the labeled source domain image is first input into the semantic segmentation model to obtain the predicted semantic segmentation result of the source domain image. The loss function of the semantic segmentation model is calculated using the predicted semantic segmentation result and the labeled semantic segmentation result (i.e., the true semantic segmentation result) to optimize the semantic segmentation model in a supervised manner. Then, a domain discriminator is used to optimize the semantic segmentation model so that the feature distribution of the unlabeled target domain image is aligned with that of the source domain image, thereby obtaining the target semantic segmentation model. Since the target domain image does not carry label information, the amount of label information can be reduced during model training, thus improving the model training efficiency.

[0049] For example, the above method may further include:

[0050] The source domain image is input into the first model to obtain the predicted semantic segmentation result of the source domain image output by the first model, wherein the source domain image includes the labeled semantic segmentation result;

[0051] Based on the predicted semantic segmentation results and the labeled semantic segmentation results, the loss function of the first model is determined;

[0052] Based on the gradient information of the loss function, the first model is updated, and the predicted semantic segmentation result of the source domain image is returned by inputting the source domain image into the first model and obtaining the output of the first model. Based on the predicted semantic segmentation result and the labeled semantic segmentation result, the loss function of the first model is determined. The first model is stopped updating when the loss function of the first model meets the set conditions, and the first model at the time of stopping the update is used as the semantic segmentation model.

[0053] In this example, the amount of labeled data is reduced during model training to improve training efficiency.

[0054] For example, the structure of the first model, namely the semantic segmentation model, can adopt the DR-UNet framework. The input is a source domain or target domain image. The encoder consists of multiple layers of convolution + BN + ReLU and pooling layers. The bottleneck layer after the encoder consists of dilated convolution + BN + ReLU to further process high-dimensional features. The decoder consists of convolution + BN + ReLU and bilinear interpolation upsampling to output the predicted segmentation probability map.

[0055] For example, the loss function of the first model can be the sum of the binary cross-entropy loss (BCE) and the Jaccard loss.

[0056] In some embodiments, since the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator, the semantic segmentation model can be adjusted based on these three domain discriminators to obtain a target semantic segmentation model, and the target semantic segmentation model can be used to segment the target domain image.

[0057] For example, in step S120 above, the source domain features and target domain features are processed based on the domain discriminator, and the semantic segmentation model is adjusted using the processing results, so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned with the source domain image in the output feature space, entropy map information, and key point information, including:

[0058] The semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model are processed by the output feature space discriminator, and the first semantic segmentation model is adjusted using the processing results to obtain the second semantic segmentation model.

[0059] The entropy discriminator processes the entropy map information of the source domain image and the target domain image extracted by the second semantic segmentation model, and adjusts the second semantic segmentation model using the processing results to obtain the third semantic segmentation model.

[0060] The key point information of the source domain image and the key point information of the target domain image extracted by the key point discriminator are processed, and the processing results are used to process the third semantic segmentation model and adjust the third semantic segmentation model to obtain the target semantic segmentation model.

[0061] Based on the target semantic segmentation model, semantic segmentation is performed on the target domain image to obtain the image segmentation result that aligns the target domain image with the source domain image in terms of output feature space, entropy map information, and key point information.

[0062] In this example, the source and target domain features extracted by the semantic segmentation model are discriminated using the output feature space discriminator, entropy discriminator, and key point discriminator. The semantic segmentation model is then adjusted to obtain a final semantic segmentation model that enables the target domain image to correspond to the source domain image in terms of output feature space, entropy map information, and key point information.

[0063] Among them, the first semantic segmentation model, the second semantic segmentation model, and the third semantic segmentation model are all models at a certain stage in the training process of semantic segmentation models. The structure of the models is the same, only the parameters are different.

[0064] In some embodiments, the order in which the model is adjusted using the discriminator can be adjusted. For example, the model can be adjusted first using the keypoint discriminator, and then the model can be adjusted using the output feature space discriminator and the entropy discriminator.

[0065] In some embodiments, since the domain discriminator may include an output feature space discriminator, an entropy discriminator, and a keypoint discriminator, these discriminators can be used to train the semantic segmentation model. Training of the semantic segmentation model can be stopped when the sum of the loss functions of these discriminators and the loss functions of each discriminator all meet predetermined conditions. The process of training the semantic segmentation model differs for each domain discriminator, as detailed below:

[0066] The process of training the semantic segmentation model for the output feature space discriminator is described in detail below:

[0067] For example, the above-mentioned semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model are processed by the output feature space discriminator, and the first semantic segmentation model is adjusted using the processing results to obtain the second semantic segmentation model, including:

[0068] The output feature space discriminator is used to determine the degree of alignment between the first feature distribution of the semantic segmentation result of the source domain image and the semantic segmentation result of the target domain image output by the first semantic segmentation model;

[0069] Based on the gradient information of the output feature space discriminator and the first semantic segmentation model, the first semantic segmentation model and the output feature space discriminator are updated, and the above discrimination steps are returned until the first feature distribution alignment degree and the loss function of the first semantic segmentation model meet the set conditions. Then the updating of the first semantic segmentation model and the output feature space discriminator is stopped, and the first semantic segmentation model at the time of stopping the update is used as the second semantic segmentation model.

[0070] The step of returning to perform the above discrimination refers to returning to the aforementioned step of "using the output feature space discriminator to determine the degree of alignment between the first feature distribution of the source domain image output by the first semantic segmentation model and the semantic segmentation result of the target domain image".

[0071] For example, the output feature space discriminator is used to determine the degree of alignment between the first feature distribution of the semantic segmentation result of the source domain image and the semantic segmentation result of the target domain image, including:

[0072] Input the source domain image and the target domain image into the first semantic segmentation model to obtain the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model.

[0073] Based on the image to which the semantic segmentation result belongs, determine the binary classification label of the semantic segmentation result; wherein the binary classification label includes the source domain or the target domain;

[0074] The semantic segmentation result is input into the output feature space discriminator to obtain the predicted classification result of the semantic segmentation result output by the output feature space discriminator; wherein, the predicted classification result includes the source domain or the target domain;

[0075] Based on the binary classification labels and predicted classification results of the semantic segmentation results, the loss function of the output feature space discriminator is determined, where the loss function is used to characterize the alignment degree of the first feature distribution of the semantic segmentation results of the source domain image and the semantic segmentation results of the target domain image.

[0076] In this example, the output feature space discriminator is used to align the output features of the semantic segmentation model in the source domain with the output features in the target domain.

[0077] For example, the output feature space discriminator D1 can be constructed using an adversarial network.

[0078] For example, the label of the source domain is defined as 1, and the label of the target domain is 0. The discriminator outputs the prediction of the domain label. Based on the generative adversarial network, D1 sets its loss function to include the discriminator loss L. D1 (S s ,S t ) and adversarial loss L adv1 (S t The gradient is propagated back from D1 to the semantic segmentation model G to promote a segmentation distribution in the target domain that is similar to that in the source domain, achieving alignment between the two domains in the output feature space. The discriminator loss L... D1 (S s ,S t ) and adversarial loss L adv1 (S t The definition is as follows:

[0079]

[0080]

[0081] in, Expressing expectations, D1(S) t )∈[0,1].

[0082] Among them, the discriminator loss is used to determine the discriminator's discrimination accuracy, and the adversarial loss is used to determine the degree to which the discriminator cannot discriminate the input information.

[0083] The process of training the semantic segmentation model for the entropy discriminator is described in detail below:

[0084] For example, the entropy map information of the source domain image and the target domain image extracted by the second semantic segmentation model are processed based on the entropy discriminator, and the processing result is used to adjust the second semantic segmentation model to obtain a third semantic segmentation model, including:

[0085] The second feature distribution alignment between the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model is determined based on the entropy discriminator.

[0086] Based on the gradient information of the entropy discriminator and the second semantic segmentation model, the second semantic segmentation model and the entropy discriminator are updated and the discrimination step is returned until the second feature distribution alignment degree and the loss function of the second semantic segmentation model meet the set conditions. Then, the second semantic segmentation model and the entropy discriminator are stopped from being updated, and the second semantic segmentation model at the time of stopping the update is used as the third semantic segmentation model.

[0087] The discrimination step is the aforementioned step of "discriminating the alignment degree of the second feature distribution between the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model based on the entropy discriminator".

[0088] For example, the alignment degree of the second feature distribution between the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model is determined based on the entropy discriminator, including:

[0089] Input the source domain image and the target domain image into the second semantic segmentation model to obtain the semantic segmentation results of the source domain image and the target domain image output by the second semantic segmentation model.

[0090] Based on the semantic segmentation results of the source domain image, the entropy map information of the source domain image is determined;

[0091] Based on the semantic segmentation results of the target domain image, the entropy map information of the target domain image is determined;

[0092] Based on the image attribution of entropy map information, determine the binary classification label of the entropy map information; wherein the binary classification label includes the source domain or the target domain;

[0093] The entropy map information is input into the entropy discriminator to obtain the predicted classification result of the entropy map information output by the entropy discriminator; wherein, the predicted classification result includes the source domain or the target domain;

[0094] Based on the binary classification labels and predicted classification results of the entropy map information, the loss function of the entropy discriminator is determined, whereby the loss function is used to characterize the degree of alignment between the second feature distribution of the entropy map information of the source domain image and the entropy map information of the target domain image.

[0095] In this example, an entropy discriminator is used to align the entropy graph of the semantic segmentation model in the source domain with the entropy graph in the target domain, and to minimize the entropy in the target domain.

[0096] For example, entropy, a measure of the degree of disorder in distribution, can be used to explore the structural consistency between two domains. If organs lack clear boundaries and are difficult to segment, the segmentation network will be forced to generate high entropy at organ edges, i.e., high uncertainty. Entropy minimization can reduce this gap at organ edges, thereby achieving structural consistency between the two domains and preventing overfitting, thus improving generalization performance. The semantic segmentation prediction results S for the source and target domains are shown below. s and S t Calculate Shannon entropy E s and E t Shannon entropy is calculated using a weighted self-information graph of the predicted segmentation results, and its calculation formula is as follows:

[0097]

[0098] Where n represents the number of images and c represents the number of channels.

[0099] For example, for a target domain image with a higher similarity to the source domain image distribution, the entropy is smaller after the model transfers to the target domain. Therefore, this disclosure reduces the performance gap of the segmentation model in the source and target domains by making the entropy maps of the two domains similar. The entropies of the two domains are fed into the entropy minimization discriminator D2. D2 is used to discriminate between the source and target domain entropy maps, distinguishing whether the entropy map comes from the source or target domain. At the same time, by making the target domain entropy map similar to the source domain entropy map, the entropy map is indirectly minimized.

[0100] For example, D2 has a discriminator loss L D2 (E s E t ) and adversarial loss L adv2 (E t The gradient is propagated back from D2 to the semantic segmentation model G to promote an entropy distribution in the target domain that is similar to that in the source domain, indirectly minimizing the entropy in the target domain. The discriminator loss L... D2 (E s E t ) and adversarial loss L adv2 (E t The definition is as follows:

[0101]

[0102]

[0103] The training process for the semantic segmentation model of the keypoint discriminator is described in detail below:

[0104] For example, the key point information of the source domain image and the target domain image extracted by the third semantic segmentation model are processed based on the key point discriminator, and the processing results are used to process the third semantic segmentation model and adjust it to obtain the target semantic segmentation model, including:

[0105] The keypoint discriminator is used to determine the alignment degree of the third feature distribution between the keypoint information of the source domain image and the keypoint information of the target domain image extracted by the third semantic segmentation model.

[0106] Based on the gradient information of the keypoint discriminator and the third semantic segmentation model, the third semantic segmentation model and the keypoint discriminator are updated and the discrimination step is returned. The updating of the third semantic segmentation model and the keypoint discriminator is stopped when the alignment degree of the third feature distribution and the loss function of the third semantic segmentation model meet the set conditions. The third semantic segmentation model at the time of stopping the update is used as the target semantic segmentation model.

[0107] The discrimination step is the aforementioned step of "discriminating the alignment degree of the third feature distribution between the key point information of the source domain image and the key point information of the target domain image extracted by the third semantic segmentation model based on the key point discriminator".

[0108] For example, the alignment degree of the third feature distribution between the keypoint information of the source domain image and the keypoint information of the target domain image extracted by the third semantic segmentation model is determined based on the keypoint discriminator, including:

[0109] Input the source domain image and the target domain image into the third semantic segmentation model to obtain the encoded features of the source domain image and the encoded features of the target domain image output by the encoder in the third semantic segmentation model;

[0110] Based on the coding features of the source domain image, the key point information of the source domain image is determined;

[0111] Based on the coding features of the target domain image, the key point information of the target domain image is determined;

[0112] Based on the image attribution of keypoint information, determine the binary classification labels of keypoint coordinates; wherein the binary classification labels include source domain or target domain;

[0113] The key point information is input into the key point discriminator to obtain the predicted classification result of the key point information output by the key point discriminator; wherein, the predicted classification result includes the source domain or the target domain;

[0114] Based on the binary classification labels and predicted classification results of key point information, the loss function of the key point discriminator is determined. The loss function is used to characterize the degree of alignment between the third feature distribution of key point information in the source domain image and the key point information in the target domain image.

[0115] In this example, a keypoint discriminator is used to align keypoint information in the source domain with keypoint information in the target domain in a semantic segmentation model.

[0116] For example, the key point discriminator D3 is provided with a discriminator loss L D3 (K s ,K t ) and adversarial loss L adv3 (K t The gradient is propagated back from D3 to the semantic segmentation model G to promote the generation of a key feature distribution in the target domain that is similar to that in the source domain. The discriminator loss L... D3 (K s ,K t ) and adversarial loss L adv3 (K t The definition is as follows:

[0117]

[0118]

[0119] For example, key point information of the source domain image is determined based on the coding features of the source domain image, and key point information of the target domain image is determined based on the coding features of the target domain image, including:

[0120] The encoded features of the source domain image are input into the second model to obtain the key point information of the source domain image output by the second model. The encoded features of the target domain image are input into the second model to obtain the key point information of the target domain image output by the second model.

[0121] For example, the above method further includes:

[0122] Based on the key point information output by the second model and the corresponding real key point information, the loss function of the second model is determined.

[0123] If the loss function of the second model does not meet the set conditions, the second model is updated based on the loss function of the second model, and the steps of determining the loss function of the second model based on the key point information output by the second model and the real key point information corresponding to the key point information are returned, until the loss function of the second model meets the set conditions and the second model is stopped from being updated.

[0124] In this example, supervised learning can be performed on the keypoint coordinates output by the second model, thereby enabling the second model to predict keypoint information.

[0125] Key points describe the shape and key information of a target, and combining this information can improve the performance of a medical image semantic segmentation model. The semantic segmentation model G consists of an encoder and a decoder. The encoder extracts high-dimensional features from the input image, continuously shrinking the feature map. In the decoder, the feature map is enlarged back to the original image size for classification of each pixel. This disclosure inserts a new prediction branch after the encoder in the semantic segmentation network G, using the high-dimensional features extracted by the encoder to predict key point coordinates. The source and target domains will each predict k key points K. s and K t and the source domain key point K s With ground real key point K gt Compare and set the keypoint prediction loss function L. kp Supervised learning of key point coordinates is performed.

[0126] Optionally, SmoothL1 can be used as the loss function for keypoint prediction.

[0127] To adapt to key point features (such as shape, edge, and corner information) between the two domains, the performance gap between the source and target domains is further reduced by forcing the key point information predicted by the target domain to be similar to that of the source domain. The key point coordinates K predicted by the two domains are then... s and K t The data are fed into the keypoint discriminator D3, which aligns the keypoints in the source and target domains to distinguish whether the keypoint coordinates come from the source or target domain.

[0128] For example, the domain discriminator includes multiple domain discriminators, and the third semantic segmentation model at the time of stopping updates is used as the target semantic segmentation model, including:

[0129] Under the condition that the sum of the loss functions of the first semantic segmentation model, the second semantic segmentation model, and the third semantic segmentation model satisfies the set condition, the third semantic segmentation model at the time of stopping updates is used as the target semantic segmentation model.

[0130] Secondly, if the sum of the loss functions of the first semantic segmentation model, the second semantic segmentation model, and the third semantic segmentation model does not meet the set condition, return to step S120 to continue processing the semantic segmentation model.

[0131] This disclosure employs an end-to-end approach for model training, with the overall loss function derived by weighting the semantic segmentation loss, keypoint prediction loss, three discriminator losses, and adversarial loss described above.

[0132] L total =L seg +L kp +μ adv1 μ adv1 (S t )-μ D1 L D1 (S s ,S t )+μ adv2 L adv2 (E t )-μ D2 L D2 (E s E t )+μ adv3 L adv3 (K t )-μ D3 L D3 (K s ,K t )

[0133] Where, μ adv1 μ adv2 μ adv3 and μ D1 μ D2 μ D3 These are the weighting coefficients used to balance the loss. In this example, the semantic segmentation model and each domain discriminator can be trained alternately until the loss function of each domain discriminator, the loss function of the semantic segmentation model, and the sum of the aforementioned loss functions reach a set condition, such as reaching the minimum value, at which point the training of the semantic segmentation model and each domain discriminator can be stopped.

[0134] like Figure 2 The diagram shown is a flowchart of an unsupervised domain-adaptive medical image semantic segmentation method using entropy and key points disclosed in this invention. The technical solution adopted includes the following steps:

[0135] Step 1: Acquire source and target domain data (e.g., medical images generated under different imaging conditions). Label each part of each source domain image, using keypoint detection methods to detect and label the coordinates of key points (e.g., organs) in each source domain image. Perform preprocessing such as cropping and image enhancement on the labeled source domain image set and the unlabeled target domain image set to obtain the final dataset.

[0136] Step 2: Construct an unsupervised domain-adaptive semantic segmentation model for medical images. The overall model structure is as follows: Figure 3 As shown, it includes a semantic segmentation model and three domain discriminators.

[0137] Both the source and target domain images are input into the same semantic segmentation model. After feature extraction, the semantic segmentation results for the source and target domain images are predicted. The segmentation result predicted for the source domain image is supervised by the ground truth labels of the source domain, thus enabling the semantic segmentation model to segment the source domain image. Next, the semantic segmentation results of both the source and target domain images are fed into a first discriminator. The discriminator adjusts the semantic segmentation model to achieve feature distribution alignment between the two domain images in the output feature space. Shannon entropy is calculated for both the source and target domain semantic segmentation results, and the results are fed into a second discriminator for adversarial learning through entropy minimization. Simultaneously, a keypoint prediction decoder is inserted into the encoder of the semantic segmentation model to predict the keypoint coordinates of the source and target domain images. These keypoint coordinates are then fed into a third discriminator, which adjusts the semantic segmentation model to achieve feature distribution alignment between the two domains in terms of key information such as shape. By updating the model parameters of the semantic segmentation model and the three discriminators using the discriminator loss and adversarial loss of the three discriminators, the performance gap between the semantic segmentation model in segmenting source and target domain images during testing or application is bridged. This allows the semantic segmentation model trained under supervision on source domain images to be generalized to target domain images, thus achieving semantic segmentation of target domain images.

[0138] Step 3: Using all source domain images and their labeled images and key point coordinate labels, as well as some target domain images as the training set, train the unsupervised domain adaptation model for medical images, and store the trained semantic segmentation model and the parameters of the three domain discriminators respectively.

[0139] Step 4: Using the remaining target domain images as the test set, predict the semantic segmentation results of the images in the test set using only the trained semantic segmentation model.

[0140] Specifically:

[0141] Define the source domain image set as in, Represents the source domain image. This represents the corresponding source domain mask label, m s This indicates the number of source domain images. The target domain image set is... in, This represents the target domain image, which is unlabeled. (m) tThis indicates the number of images in the target domain. The source and target domain images come from two different data sources. Images from different domains processed using the same semantic segmentation model often exhibit different feature distributions. This example uses an existing semantic segmentation model (which may include an encoder and decoder) as a foundation, inserting an output feature space discriminator, an entropy minimization discriminator, and a keypoint discriminator to make the distributions of features extracted by the semantic segmentation model from the two domains more similar or closer.

[0142] For example, data augmentation can be performed on the images in the training set by means of translation, rotation, scaling, etc., to obtain a data-augmented training set.

[0143] First, the source domain image The input is fed into a semantic segmentation model G for prediction, and the source domain segmentation result S obtained from the prediction is used. s Segmentation result with ground truth gt Calculate the semantic segmentation loss function L seg The loss function is then used to optimize the parameters of the semantic segmentation model, thereby optimizing the segmentation model in a supervised manner.

[0144] The sum of binary cross-entropy loss (BCE) and Jaccard loss can be used as the semantic segmentation loss function.

[0145] Next, the output feature spaces of the semantic segmentation model are aligned between the source and target domains. The semantic segmentation model G is used to receive the target domain image. The target domain segmentation result S is then predicted. t At this point, the target domain image lacks semantic labels. The segmentation result is the output of the semantic segmentation model G, while in order to make the source domain segmentation result S... s Segmentation result S of the target domain t As they get closer to each other, they are fed into the output feature space discriminator D1. D1 learns how to distinguish whether these predicted segmentation results come from the source domain or the target domain, thereby aligning the two domains in the output feature space.

[0146] For example, the label of the source domain is defined as 1, and the label of the target domain is 0. The discriminator outputs the prediction of the domain label. Based on the generative adversarial network, D1 has a discriminator loss L. D1 (S s ,S t ) and adversarial loss L adv1 (S t The gradient is propagated back from D1 to G to promote a segmentation distribution in the target domain that is similar to that in the source domain, achieving alignment of the two domains in the output feature space. The discriminator loss L... D1 (S s ,S t) and adversarial loss L adv1 (S t The definition is as follows:

[0147]

[0148]

[0149] in, Expressing expectations, D1(S) t )∈[0,1].

[0150] Next, entropy alignment is performed between the source and target domains of the semantic segmentation model. Entropy is a measure of the degree of disorder in distribution and can explore the structural consistency between two domains. If organs in an image lack clear boundaries and are difficult to segment, the segmentation network will be forced to generate high entropy at organ edges, i.e., high uncertainty. Entropy minimization can reduce this gap at organ edges, thereby ensuring consistency in the structure of the same organs in both domains, preventing overfitting, and improving generalization performance. The semantic segmentation prediction results S for the source and target domains are then compared. s and S t Calculate Shannon entropy E s and E t Shannon entropy is calculated using a weighted self-information graph of the predicted segmentation results, and its calculation formula is as follows:

[0151]

[0152] Where n represents the number of images and c represents the number of channels.

[0153] For a target domain image with a higher similarity to the source domain image distribution, its entropy will be smaller after the model is transferred to the target domain. Therefore, this disclosure reduces the performance gap between the segmentation model in the source and target domains by making the entropy maps of the two domains similar. The entropies of the two domains are fed into the entropy minimization discriminator D2. D2 discriminates the entropy maps of the source and target domains, distinguishing whether the entropy map comes from the source or target domain. At the same time, by making the entropy map of the target domain similar to the entropy map of the source domain, the entropy map is indirectly minimized.

[0154] For example, D2 has a discriminator loss L D2 (E s E t ) and adversarial loss L adv2 (E t The gradient is propagated back from D2 to the segmentation model G to promote an entropy distribution in the target domain that is similar to that in the source domain, indirectly minimizing the entropy in the target domain. The discriminator loss L... D2 (E s E t ) and adversarial loss L adv2 (Et The definition is as follows:

[0155]

[0156]

[0157] Then, keypoint alignment is performed between the source and target domains of the semantic segmentation model. Keypoints describe the shape and key information of the target, and combining this information can improve the performance of the medical image semantic segmentation model. The semantic segmentation model G is generally divided into an encoder and a decoder. The encoder extracts high-dimensional features from the input image, continuously shrinking the feature map. In the decoder, the feature map is enlarged back to the original image size for classification of each pixel. This disclosure inserts a new prediction branch after the encoder in the semantic segmentation network G, using the high-dimensional features extracted by the encoder to predict keypoint coordinates. K keypoints K will be predicted from both the source and target domains respectively. s and K t and the source domain key point K s With ground real key point K gt Compare and set the keypoint prediction loss function L. kp Supervised learning of key point coordinates is performed.

[0158] For example, SmoothL1 is used as the loss function for keypoint prediction.

[0159] To adapt to key point features (such as shape, edge, and corner information) between the two domains, the performance gap between the source and target domains is further reduced by forcing the key point information predicted by the target domain to be similar to that of the source domain. The key point coordinates K predicted by the two domains are then... s and K t The data are fed into the keypoint discriminator D3, which aligns the keypoints in the source and target domains to distinguish whether the keypoint coordinates come from the source or target domain.

[0160] For example, D3 sets the discriminator loss L D3 (K s ,K t ) and adversarial loss L adv3 (K t The gradient is propagated back from D3 to the segmentation model G to promote the generation of a key feature distribution in the target domain that is similar to that in the source domain. The discriminator loss L... D3 (K s ,K t ) and adversarial loss L adv3 (K t The definition is as follows:

[0161]

[0162]

[0163] This disclosure employs an end-to-end approach for model training, with the overall loss function derived by weighting the semantic segmentation loss, keypoint prediction loss, three discriminator losses, and adversarial loss described above.

[0164] L total =L seg +L kp +μ adv1 L adv1 (S t )-μ D1 L D1 (S s ,S t )+μ adv2 L adv2 (E t )-μ D2 L D2 (E s E t )+μ adv3 L adv3 (K t )-μ D3 L D3 (K s ,K t )

[0165] Where, μ adv1 μ adv2 μ adv3 and μ D1 μ D2 μ D3 These are weighting coefficients used to balance losses.

[0166] The overall minimum-maximum optimization problem is set as follows:

[0167]

[0168] Where D1, D2, and D3 refer to the discriminator loss. It is a network model where the remaining weights are shared.

[0169] The specific structure of each module in the model is as follows:

[0170] 1. Semantic segmentation model G:

[0171] For example, using the DR-UNet framework, the input source or target domain image is composed of an encoder consisting of multiple convolutional + BN + ReLU and pooling layers. The bottleneck layer after the encoder is composed of dilated convolutional + BN + ReLU to further process high-dimensional features. The decoder is composed of convolutional + BN + ReLU and bilinear interpolation upsampling to output the predicted segmentation probability map.

[0172] 2. Output feature space discriminator D1 and entropy minimization discriminator D2:

[0173] For example, using the structure proposed in PatchGAN, the semantic segmentation probability maps or entropy maps of the source and target domains are input. The input is mapped to an N×N matrix using a fully convolutional form, where each point represents a small patch of the original image. The N×N matrix result is used to evaluate the entire image. This disclosure divides the image into 8×8 patches and uses five 4×4 convolutional kernels to perform convolutions with a stride of 2. Each convolutional layer is followed by a Leaky ReLU activation function.

[0174] 3. Keypoint Prediction Branch:

[0175] like Figure 4 As shown, this is the keypoint prediction branch structure. The high-dimensional features input from the encoder are processed through three convolutional layers (Conv) to obtain corresponding feature maps. Then, upsampling and skip connections are used for multi-scale feature fusion. Finally, a convolutional layer, a tensor flattening layer, and two fully connected layers output the keypoint coordinates. This multi-scale feature fusion method can combine high-level semantic information with low-level keypoint geometric information, which is helpful for keypoint prediction.

[0176] 4. Key Point Discriminator D3:

[0177] like Figure 5 As shown, this is the structure of the keypoint discriminator D3. Since the order of the k keypoints in each part does not affect its shape and key information, a PointNet-like structure is used to process the two-dimensional keypoints. The input consists of predicted (c×k)×2-dimensional keypoint coordinates (c types of parts, k keypoints per type, each keypoint containing x and y coordinates). A combination of Multilayer Perceptron (MLP) and Max Pooling is used to obtain global features. These global features are then concatenated to each high-dimensional feature using the concat method (used to connect two or more arrays), performing simple local and global feature fusion. Finally, the MLP outputs a discrimination result of 0 or 1.

[0178] 5. Obtaining the coordinates of real ground key points:

[0179] For example, a method combining Scale-invariant Feature Transform (SIFT) and Farthest Point Sampling (FPS) can be used to find a fixed set of k ground truth keypoints. First, SIFT is used to detect highly representative points (such as corner points and edge points) that are independent of scale, rotation, and brightness changes. Then, the farthest point sampling method is used to obtain the k keypoints. The steps of the farthest point sampling method are: first, the center point coordinates are calculated directly based on the pixel coordinates of each part, and this is stored as the first keypoint in the keypoint set. Then, the keypoint set is updated by iteratively adding a new point that is farthest from all selected keypoints until the k keypoints are obtained.

[0180] Figure 6 This is a schematic diagram of an image segmentation apparatus according to an embodiment of the present disclosure. Figure 6 As shown, the image segmentation device is characterized by comprising:

[0181] The semantic segmentation module 610 is used to extract features from the source domain image and the target domain image based on the semantic segmentation model, so as to obtain the source domain features of the source domain image and the target domain features of the target domain image.

[0182] The domain discriminator module 620 is used to process the source domain features and the target domain features based on the domain discriminator, and to adjust the semantic segmentation model using the processing results, so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information, and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator.

[0183] For example, the domain discriminator module 620 further includes:

[0184] The first adjustment unit is used to process the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model based on the output feature space discriminator, and to adjust the first semantic segmentation model using the processing results to obtain the second semantic segmentation model.

[0185] The second adjustment unit is used to process the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model based on the entropy discriminator, and to adjust the second semantic segmentation model using the processing results to obtain the third semantic segmentation model.

[0186] The third adjustment unit is used to process the key point information of the source domain image and the key point information of the target domain image extracted by the third semantic segmentation model based on the key point discriminator, and to process the third semantic segmentation model using the processing result, and to adjust the third semantic segmentation model using the processing result to obtain the target semantic segmentation model.

[0187] The semantic segmentation unit is used to perform semantic segmentation on the target domain image based on the target semantic segmentation model, and obtain the image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy and key points.

[0188] The apparatus provided in this disclosure also includes other units capable of implementing the above method, which will not be described in detail here.

[0189] The functions of each unit, module, or sub-module in the various devices of this disclosure embodiment can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0190] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0191] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0192] like Figure 7 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0193] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0194] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as image segmentation methods. For example, in some embodiments, the image segmentation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 102 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image segmentation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the image segmentation method by any other suitable means (e.g., by means of firmware).

[0195] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0196] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable ambient lighting control device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0197] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0198] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0199] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0200] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0201] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0202] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image segmentation method, characterized in that, include: The source domain image and the target domain image are processed based on the semantic segmentation model to obtain the source domain features of the source domain image and the target domain features of the target domain image. The source domain features and the target domain features are processed based on a domain discriminator, and the semantic segmentation model is adjusted using the processing results so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information, and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator; The step of processing the source domain features and the target domain features based on the domain discriminator, and adjusting the semantic segmentation model using the processing results, so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned to the source domain image in terms of output feature space, entropy map information, and key point information, includes: Based on the output feature space discriminator, the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model are processed, and the first semantic segmentation model is adjusted using the processing results to obtain the second semantic segmentation model. The entropy discriminator processes the entropy map information of the source domain image and the target domain image extracted by the second semantic segmentation model, and adjusts the second semantic segmentation model using the processing results to obtain the third semantic segmentation model. The key point information of the source domain image and the key point information of the target domain image extracted by the key point discriminator are processed, and the processing results are used to process the third semantic segmentation model and adjust the third semantic segmentation model to obtain the target semantic segmentation model. Based on the target semantic segmentation model, semantic segmentation is performed on the target domain image to obtain the image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information, and key point information.

2. The method according to claim 1, characterized in that, The step of processing the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model based on the output feature space discriminator, and adjusting the first semantic segmentation model using the processing results to obtain the second semantic segmentation model, includes: The output feature space discriminator is used to determine the degree of alignment between the first feature distribution of the semantic segmentation result of the source domain image output by the first semantic segmentation model and the semantic segmentation result of the target domain image; Based on the gradient information of the output feature space discriminator and the first semantic segmentation model, the first semantic segmentation model and the output feature space discriminator are updated and the step of performing the discrimination is returned until the first feature distribution alignment degree and the loss function of the first semantic segmentation model meet the set conditions. Then the updating of the first semantic segmentation model and the output feature space discriminator is stopped, and the first semantic segmentation model at the time of stopping the update is used as the second semantic segmentation model.

3. The method according to claim 2, characterized in that, The step of using the output feature space discriminator to determine the alignment degree of the first feature distribution between the semantic segmentation result of the source domain image output by the first semantic segmentation model and the semantic segmentation result of the target domain image includes: The source domain image and the target domain image are input into the first semantic segmentation model to obtain the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model. Based on the image to which the semantic segmentation result belongs, a binary classification label for the semantic segmentation result is determined; wherein, the binary classification label includes the source domain or the target domain; The semantic segmentation result is input into the output feature space discriminator to obtain the predicted classification result of the semantic segmentation result output by the output feature space discriminator; wherein, the predicted classification result includes the source domain or the target domain; Based on the binary classification labels and predicted classification results of the semantic segmentation results, the loss function of the output feature space discriminator is determined, wherein the loss function is used to characterize the degree of alignment of the first feature distribution of the semantic segmentation results of the source domain image and the semantic segmentation results of the target domain image.

4. The method according to claim 1, characterized in that, The third semantic segmentation model is obtained by processing the entropy map information of the source domain image and the target domain image extracted by the second semantic segmentation model based on the entropy discriminator, and adjusting the second semantic segmentation model using the processing result. This process includes: The entropy discriminator determines the degree of alignment between the second feature distribution of the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model. Based on the gradient information of the entropy discriminator and the second semantic segmentation model, the second semantic segmentation model and the entropy discriminator are updated and the discrimination step is returned until the second feature distribution alignment degree and the loss function of the second semantic segmentation model meet the set conditions. Then, the updating of the second semantic segmentation model and the entropy discriminator is stopped, and the second semantic segmentation model at the time of stopping the update is used as the third semantic segmentation model.

5. The method according to claim 4, characterized in that, The step of determining the alignment degree of the second feature distribution between the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model based on the entropy discriminator includes: The source domain image and the target domain image are input into the second semantic segmentation model to obtain the semantic segmentation results of the source domain image and the target domain image output by the second semantic segmentation model. Based on the semantic segmentation results of the source domain image, the entropy map information of the source domain image is determined; Based on the semantic segmentation results of the target domain image, the entropy map information of the target domain image is determined; Based on the image to which the entropy map information belongs, a binary classification label for the entropy map information is determined; wherein, the binary classification label includes a source domain or a target domain; The entropy map information is input into the entropy discriminator to obtain the predicted classification result of the entropy map information output by the entropy discriminator; wherein, the predicted classification result includes the source domain or the target domain; Based on the binary classification labels and predicted classification results of the entropy map information, the loss function of the entropy discriminator is determined, wherein the loss function is used to characterize the degree of alignment between the second feature distribution of the entropy map information of the source domain image and the entropy map information of the target domain image.

6. The method according to claim 1, characterized in that, The keypoint information of the source domain image and the target domain image extracted by the third semantic segmentation model based on the keypoint discriminator is processed, and the processing result is used to process and adjust the third semantic segmentation model to obtain the target semantic segmentation model, including: Based on the key point discriminator, the alignment degree of the third feature distribution between the key point information of the source domain image and the key point information of the target domain image extracted by the third semantic segmentation model is determined; Based on the gradient information of the keypoint discriminator and the third semantic segmentation model, the third semantic segmentation model and the keypoint discriminator are updated and the discrimination step is returned until the alignment degree of the third feature distribution and the loss function of the third semantic segmentation model meet the set conditions. Then the updating of the third semantic segmentation model and the keypoint discriminator is stopped, and the third semantic segmentation model at the time of stopping the update is used as the target semantic segmentation model.

7. The method according to claim 6, characterized in that, The step of determining the alignment degree of the third feature distribution between the key point information of the source domain image and the key point information of the target domain image extracted by the third semantic segmentation model based on the key point discriminator includes: The source domain image and the target domain image are input into the third semantic segmentation model to obtain the encoded features of the source domain image and the encoded features of the target domain image output by the encoder in the third semantic segmentation model. Based on the coding features of the source domain image, the key point information of the source domain image is determined; Based on the coding features of the target domain image, the key point information of the target domain image is determined; Based on the image to which the key point information belongs, a binary classification label for the key point information is determined; wherein, the binary classification label includes a source domain or a target domain; The key point information is input into the key point discriminator to obtain the predicted classification result of the key point information output by the key point discriminator; wherein, the predicted classification result includes the source domain or the target domain; Based on the binary classification labels and predicted classification results of the key point information, the loss function of the key point discriminator is determined, wherein the loss function is used to characterize the degree of alignment between the third feature distribution of the key point information of the source domain image and the key point information of the target domain image.

8. The method according to claim 1, characterized in that, The method further includes: The source domain image is input into the first model to obtain the predicted semantic segmentation result of the source domain image output by the first model, wherein the source domain image includes the labeled semantic segmentation result; Based on the predicted semantic segmentation results and the labeled semantic segmentation results, the loss function of the first model is determined; Based on the gradient information of the loss function, the first model is updated, and the process of inputting the source domain image into the first model to obtain the predicted semantic segmentation result of the source domain image output by the first model is repeated. Based on the predicted semantic segmentation result and the labeled semantic segmentation result, the loss function of the first model is determined. The process continues until the loss function of the first model meets the set conditions, at which point the first model is stopped from being updated, and the first model at the time of stopping the update is used as the semantic segmentation model.

9. The method according to claim 7, characterized in that, The process of determining key point information of the source domain image based on the coding features of the source domain image, and determining key point information of the target domain image based on the coding features of the target domain image, includes: The encoded features of the source domain image are input into the second model to obtain the key point information of the source domain image output by the second model. The encoded features of the target domain image are input into the second model to obtain the key point information of the target domain image output by the second model.

10. The method according to claim 9, characterized in that, The method further includes: Based on the key point information output by the second model and the real key point information corresponding to the key point information, the loss function of the second model is determined; If the loss function of the second model does not meet the set conditions, the second model is updated based on the loss function of the second model, and the step of determining the loss function of the second model based on the key point information output by the second model and the real key point information corresponding to the key point information is returned, until the loss function of the second model meets the set conditions and the second model is stopped from being updated.

11. The method according to claim 7, characterized in that, The step of using the third semantic segmentation model at the time of stopping updates as the target semantic segmentation model includes: If the sum of the loss functions of the first semantic segmentation model, the second semantic segmentation model, and the third semantic segmentation model satisfies a set condition, the third semantic segmentation model at the time of stopping updates is used as the target semantic segmentation model.

12. An image segmentation apparatus, characterized in that, include: The semantic segmentation module is used to extract features from the source domain image and the target domain image based on the semantic segmentation model, so as to obtain the source domain features of the source domain image and the target domain features of the target domain image. The domain discriminator module is used to process the source domain features and the target domain features based on the domain discriminator, and to adjust the semantic segmentation model using the processing results, so that the semantic segmentation model outputs an image segmentation result of the target domain image aligned to the source domain image in terms of output feature space, entropy map information, and key point information; wherein, the domain discriminator includes an output feature space discriminator, an entropy discriminator, and a key point discriminator; The domain discriminator module also includes: The first adjustment unit is used to process the semantic segmentation results of the source domain image and the target domain image output by the first semantic segmentation model based on the output feature space discriminator, and to adjust the first semantic segmentation model using the processing results to obtain the second semantic segmentation model. The second adjustment unit is used to process the entropy map information of the source domain image and the entropy map information of the target domain image extracted by the second semantic segmentation model based on the entropy discriminator, and to adjust the second semantic segmentation model using the processing results to obtain the third semantic segmentation model. The third adjustment unit is used to process the key point information of the source domain image and the key point information of the target domain image extracted by the third semantic segmentation model based on the key point discriminator, and to process the third semantic segmentation model using the processing result, and to adjust the third semantic segmentation model using the processing result to obtain the target semantic segmentation model. The semantic segmentation unit is used to perform semantic segmentation on the target domain image based on the target semantic segmentation model, and obtain the image segmentation result of the target domain image aligned with the source domain image in terms of output feature space, entropy map information and key point information.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-11.

Images