Model training method and device, equipment and storage medium

Abstract

This application discloses a model training method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring classification labels for a first / second modality image and a second modality image; inputting the first / second modality image into an image translation model to obtain a first translated image; inputting the first translated image into a classification network to obtain a first predicted probability, and inputting the second modality image into the classification network to obtain a second predicted probability; calculating the structural consistency error based on the first / second predicted probabilities and classification labels, and performing backpropagation training on the image translation model to obtain a trained image translation model. This application fully mines the semantic information in the image through the first translated image, realizing translation between different modalities within the image; using the structural consistency error to train the image translation model can improve the accuracy of translated images, achieving unsupervised adaptation in medical images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method, apparatus, device, and storage medium. Background Technology

[0002] Image translation is used to translate and convert images of different modalities, which are due to different technologies used to acquire the images or different styles of the images.

[0003] In related technologies, when translating natural images, image style transfer and unsupervised training are usually used to translate information between different modalities, thereby achieving unsupervised domain adaptation (UDA).

[0004] However, because the distribution of different modal information in medical images differs greatly from that in natural images, the above methods cannot fully extract information from medical images and cannot translate between different modal information in medical images. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, and storage medium, the technical solution of which is as follows:

[0006] According to one aspect of this application, a method for training an image translation model is provided, the method comprising:

[0007] Obtain the classification labels of the first modality image, the second modality image, and the second modality image;

[0008] The first modal image and the second modal image are input into the image translation model to obtain a first translated image, wherein the first translated image is an image generated by the image translation model when predicting the first modal image based on the second modal image;

[0009] The first translated image is input into the classification network to obtain a first predicted probability, and the second modal image is input into the classification network to obtain a second predicted probability. The first predicted probability is the probability that the image content of the first translated image belongs to the target organization, and the second predicted probability is the probability that the image content of the second modal image belongs to the target organization.

[0010] Based on the first predicted probability, the second predicted probability, and the classification label, a structural consistency error is calculated, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively.

[0011] Based on the structural consistency error, the image translation model is trained by backpropagation to obtain the trained image translation model.

[0012] In an optional design of this application, the first modality discrimination error is an adversarial loss function of the first modality discrimination result.

[0013] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0014] The step of calling the image translation model to process the first modal image and the second modal image to obtain the second translated image includes:

[0015] The second modal image is input into the second modal encoder to obtain a third feature representation, which is a feature representation obtained by modal encoding of the second modal image;

[0016] The first modality image is input into the anatomical encoder to obtain a fourth feature representation, which is a feature representation obtained by anatomical encoding of the first modality image;

[0017] The third feature representation and the fourth feature representation are input into the generator to obtain the second translated image, which is an image generated by the image translation model when predicting the second modality image based on the first modality image.

[0018] In an optional design of this application, the second modality discrimination error is an adversarial loss function of the second modality discrimination result.

[0019] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, and a generator, wherein the anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in a first modal domain of the image;

[0020] The step of calling the image translation model to process the first modal image and the second modal image to obtain the third translated image includes:

[0021] The first modal image is input into the first modal encoder to obtain a second feature representation, which is a feature representation obtained by modal encoding of the first modal image;

[0022] The first modality image is input into the anatomical encoder to obtain a fourth feature representation, which is a feature representation obtained by anatomical encoding of the first modality image;

[0023] The second feature representation and the fourth feature representation are input into the generator to obtain the third translated image, which is an image generated by the image translation model when predicting the first modality image based on the first modality image.

[0024] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0025] The step of calling the image translation model to process the first modality image and the second modality image to obtain the fourth translated image includes:

[0026] The second modality image is input into the anatomical encoder to obtain a first feature representation, which is a feature representation obtained by anatomical encoding of the second modality image;

[0027] The second modal image is input into the second modal encoder to obtain a third feature representation, which is a feature representation obtained by modal encoding of the second modal image;

[0028] The first feature representation and the third feature representation are input into the generator to obtain the fourth translated image, which is an image generated by the image translation model when predicting the second modality image based on the second modality image.

[0029] In an optional design of this application, the reconstruction loss error is an L1 norm loss function.

[0030] According to another aspect of this application, a method for training an image semantic segmentation model is provided, the method comprising:

[0031] Obtain the classification labels of the first modality image, the second modality image, and the second modality image;

[0032] The first modality image and the second modality image are input into the trained image translation model to obtain a fifth translated image. The fifth translated image is an image generated by the trained image translation model when predicting the first modality image based on the second modality image. The trained image translation model is obtained according to the training method of any of the above image translation models.

[0033] The fifth translated image is input into the image semantic segmentation model to obtain a third prediction probability, and the second modality image is input into the image semantic segmentation model to obtain a fourth prediction probability. The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modality image belongs to the target organization.

[0034] Based on the third predicted probability, the fourth predicted probability, and the classification label, a classification error is calculated, wherein the classification error is used to represent the difference between the third predicted probability and the fourth predicted probability and the classification label, respectively;

[0035] Based on the classification error, the image semantic segmentation model is trained by backpropagation to obtain the trained image semantic segmentation model.

[0036] In an optional design of this application, the adversarial consistency error is an adversarial loss function between the first discrimination result and the second discrimination result.

[0037] In an optional design of this application, the rotational consistency error includes the cross-entropy loss function between the fifth predicted probability and the predicted classification label and / or the Dessian loss function between the fifth predicted probability and the predicted classification label.

[0038] According to another aspect of this application, a training apparatus for an image translation model is provided, the apparatus comprising:

[0039] The acquisition module is used to acquire the first modality image, the second modality image, and the classification label of the second modality image;

[0040] An input module is used to input the first modal image and the second modal image into the image translation model to obtain a first translated image, wherein the first translated image is an image generated by the image translation model when predicting the first modal image based on the second modal image;

[0041] The input module is further configured to input the first translated image into the classification network to obtain a first prediction probability, and input the second modal image into the classification network to obtain a second prediction probability, wherein the first prediction probability is the probability that the image content of the first translated image belongs to the target organization, and the second prediction probability is the probability that the image content of the second modal image belongs to the target organization;

[0042] The calculation module is used to calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively.

[0043] The training module is used to perform backpropagation training on the image translation model based on the structural consistency error, so as to obtain the trained image translation model.

[0044] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, and a generator, wherein the anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in a first modal domain of the image;

[0045] The input module is further configured to: input the second modality image into the anatomical encoder to obtain a first feature representation, wherein the first feature representation is a feature representation obtained by anatomical encoding of the second modality image; input the first modality image into the first modality encoder to obtain a second feature representation, wherein the second feature representation is a feature representation obtained by modality encoding of the first modality image; and input the first feature representation and the second feature representation into the generator to obtain the first translated image.

[0046] In an alternative design of this application, the classification network includes an anatomical encoder and a classifier;

[0047] The input module is further configured to: input the first translated image and the second modality image into the anatomical encoder respectively to obtain the fifth feature representation of the first translated image and the sixth feature representation of the second modality image respectively; input the fifth feature representation and the sixth feature representation into the classifier respectively to obtain the first prediction probability and the second prediction probability respectively.

[0048] In an optional design of this application, the structural consistency error includes a first cross-entropy loss function and / or a first Dessian loss function; wherein, the first cross-entropy loss function includes a cross-entropy loss function between the first predicted probability and the classification label, and a cross-entropy loss function between the second predicted probability and the classification label; the first Dessian loss function includes a Dessian loss function between the first predicted probability and the classification label, and a Dessian loss function between the second predicted probability and the classification label. In an optional design of this application, the input module is further configured to: input the first translated image and the first modality image respectively into a first modality discriminator to obtain a first discrimination result, wherein the first modality discriminator is used to discriminate whether the input image belongs to the first modality image, and the first discrimination result includes the first modality discrimination result of the first translated image and the first modality discrimination result of the first modality image;

[0049] The calculation module is further configured to: calculate the first mode discrimination error based on the first discrimination result;

[0050] The training module is further configured to: perform backpropagation training on the image translation model based on the structural consistency error and the first modality discrimination error, so as to obtain the trained image translation model.

[0051] In an optional design of this application, the first modality discrimination error is an adversarial loss function of the first modality discrimination result.

[0052] In an optional design of this application, the device further includes:

[0053] The calling module is used to call the image translation model to process the first modal image and the second modal image to obtain the second translated image;

[0054] The input module is further configured to: input the second translated image and the second modal image into the second modality discriminator respectively to obtain a second discrimination result, wherein the second modality discriminator is used to discriminate whether the input image belongs to the second modality image, and the second discrimination result includes the second modality discrimination result of the second translated image and the second modality discrimination result of the second modality image;

[0055] The calculation module is further configured to: calculate the second mode discrimination error based on the second discrimination result;

[0056] The training module is also used to: perform backpropagation training on the image translation model based on the structural consistency error and the second modality discrimination error, so as to obtain the trained image translation model.

[0057] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0058] The calling module is further configured to: input the second modal image into the second modal encoder to obtain a third feature representation, wherein the third feature representation is a feature representation obtained by modal encoding of the second modal image; input the first modal image into the anatomical encoder to obtain a fourth feature representation, wherein the fourth feature representation is a feature representation obtained by anatomical encoding of the first modal image; input the third feature representation and the fourth feature representation into the generator to obtain the second translated image, wherein the second translated image is an image generated by the image translation model when predicting the second modal image based on the first modal image.

[0059] In an optional design of this application, the second modality discrimination error is an adversarial loss function of the second modality discrimination result.

[0060] In an optional design of this application, the calling module is further configured to: call the image translation model to process the first modal image and the second modal image to obtain a third translated image;

[0061] The calculation module is also used to: calculate the reconstruction loss error based on the third translated image and the first modality image;

[0062] The training module is also used to: perform backpropagation training on the image translation model based on the structural consistency error and the reconstruction loss error, so as to obtain the trained image translation model.

[0063] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, and a generator, wherein the anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in a first modal domain of the image;

[0064] The calling module is further configured to: input the first modal image into the first modal encoder to obtain a second feature representation, wherein the second feature representation is a feature representation obtained by modal encoding of the first modal image; input the first modal image into the anatomical encoder to obtain a fourth feature representation, wherein the fourth feature representation is a feature representation obtained by anatomical encoding of the first modal image; input the second feature representation and the fourth feature representation into the generator to obtain the third translated image, wherein the third translated image is an image generated by the image translation model when predicting the first modal image based on the first modal image.

[0065] In an optional design of this application, the calling module is further configured to: call the image translation model to process the first modal image and the second modal image to obtain a fourth translated image;

[0066] The calculation module is also used to: calculate the reconstruction loss error based on the fourth translated image and the second modality image;

[0067] The training module is also used to: perform backpropagation training on the image translation model based on the structural consistency error and the reconstruction loss error, so as to obtain the trained image translation model.

[0068] In an optional design of this application, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0069] The calling module is further configured to: input the second modality image into the anatomical encoder to obtain a first feature representation, wherein the first feature representation is a feature representation obtained by anatomical encoding of the second modality image;

[0070] The second modal image is input into the second modal encoder to obtain a third feature representation, which is a feature representation obtained by modal encoding of the second modal image;

[0071] The first feature representation and the third feature representation are input into the generator to obtain the fourth translated image, which is an image generated by the image translation model when predicting the second modality image based on the second modality image.

[0072] In an optional design of this application, the reconstruction loss error is an L1 norm loss function.

[0073] According to another aspect of this application, a training apparatus for an image semantic segmentation model is provided, the apparatus comprising:

[0074] The acquisition module is used to acquire the first modality image, the second modality image, and the classification label of the second modality image;

[0075] The input module is used to input the first modal image and the second modal image into the trained image translation model to obtain a fifth translated image. The fifth translated image is an image generated by the trained image translation model when predicting the first modal image based on the second modal image. The trained image translation model is obtained according to the training method of any of the above image translation models.

[0076] The input module is further configured to input the fifth translated image into the image semantic segmentation model to obtain a third prediction probability, and to input the second modal image into the image semantic segmentation model to obtain a fourth prediction probability. The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modal image belongs to the target organization.

[0077] The calculation module is used to calculate the classification error based on the third prediction probability, the fourth prediction probability, and the classification label, wherein the classification error is used to represent the difference between the third prediction probability and the fourth prediction probability and the classification label, respectively;

[0078] The training module is used to perform backpropagation training on the image semantic segmentation model based on the classification error, so as to obtain the trained image semantic segmentation model.

[0079] In an optional design of this application, the image semantic segmentation model includes an anatomical encoder and a classifier, wherein the anatomical encoder is used to encode the anatomical structure of the image;

[0080] The input module is also used for:

[0081] The fifth translated image and the second modal image are respectively input into the anatomical encoder to obtain the seventh feature representation of the fifth translated image and the eighth feature representation of the second modal image, respectively;

[0082] The seventh feature representation and the eighth feature representation are respectively input into the classifier to obtain the third prediction probability and the fourth prediction probability.

[0083] In an optional design of this application, the classification error includes a second cross-entropy loss function and / or a second Descein loss function; wherein the second cross-entropy loss function includes a cross-entropy loss function between the third predicted probability and the classification label, and a cross-entropy loss function between the fourth predicted probability and the classification label; the second Descein loss function includes a Descein loss function between the third predicted probability and the classification label, and a Descein loss function between the fourth predicted probability and the classification label. In an optional design of this application, the input module is further configured to: rotate the first modality image, and input the rotated first modality image into the image semantic segmentation model to obtain a fifth predicted probability, wherein the fifth predicted probability is the probability that the image content of the rotated first modality image belongs to the target organization;

[0084] The calculation module is also used to: calculate the weighted entropy error based on the fifth prediction probability;

[0085] The training module is also used to: perform backpropagation training on the image semantic segmentation model based on the classification error and the weighted entropy error, so as to obtain the trained image semantic segmentation model.

[0086] In an optional design of this application, the input module is further configured to: rotate the first modal image and input the rotated first modal image into the image semantic segmentation model to obtain a fifth prediction probability, wherein the fifth prediction probability is the probability that the image content of the rotated first modal image belongs to the target organization;

[0087] The fourth prediction probability and the fifth prediction probability are respectively input into the discriminator to obtain the first discrimination result of the fourth prediction probability and the second discrimination result of the fifth prediction probability. The discriminator is used to distinguish whether the image corresponding to the prediction probability comes from the first mode or the second mode.

[0088] The calculation module is further configured to: calculate the adversarial consistency error based on the first discrimination result and the second discrimination result;

[0089] The training module is further configured to: perform backpropagation training on the image semantic segmentation model based on the classification error and the adversarial consistency error, so as to obtain the trained image semantic segmentation model.

[0090] In an optional design of this application, the adversarial consistency error is an adversarial loss function between the first discrimination result and the second discrimination result.

[0091] In an optional design of this application, the input module is further configured to: rotate the first modal image and input the rotated first modal image into the image semantic segmentation model to obtain a fifth prediction probability, wherein the fifth prediction probability is the probability that the image content of the rotated first modal image belongs to the target organization;

[0092] The first modality image is input into the teacher image semantic segmentation model, and the predicted probability of the first modality image output by the teacher image semantic segmentation model is rotated to obtain the sixth predicted probability. The parameters of the teacher image semantic segmentation model are determined based on the parameters of the image semantic segmentation model.

[0093] The determination module is used to: determine the predicted classification label based on the sixth prediction probability and through a confidence threshold;

[0094] The calculation module is also used to: calculate the rotation consistency error based on the predicted classification label and the fifth predicted probability;

[0095] The training module is further configured to: perform backpropagation training on the image semantic segmentation model based on the classification error and the rotation consistency error, so as to obtain the trained image semantic segmentation model.

[0096] In an optional design of this application, the rotational consistency error includes the cross-entropy loss function between the fifth predicted probability and the predicted classification label and / or the Dessian loss function between the fifth predicted probability and the predicted classification label.

[0097] In an optional design of this application, the acquisition module is further configured to: acquire the image to be segmented;

[0098] The input module is further configured to: input the image to be segmented into the trained image semantic segmentation model to obtain the predicted probability of the image to be segmented;

[0099] The determining module is further configured to: determine the predicted classification label of the image to be segmented based on the predicted probability of the image to be segmented by using a prediction confidence threshold.

[0100] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the model training method as described above.

[0101] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the model training method described above.

[0102] According to another aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the model training method described above.

[0103] The beneficial effects of the technical solution provided in this application include at least the following:

[0104] The first translated image includes information from both the first and second modal images, thus containing sufficient semantic information for predicting semantic labels. By using the first translated image, the semantic information in the image can be fully extracted, enabling the use of an image translation model to translate between different modal information within the image. Training the image translation model using structural consistency error can improve the accuracy of obtaining translated images between different modal information, overcoming the difficulty of significant differences in the distribution of different modal information in the image, and achieving unsupervised adaptation in medical images. Attached Figure Description

[0105] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0106] Figure 1 This is a block diagram of a computer system used for model training according to one embodiment of this application;

[0107] Figure 2 This is a schematic diagram of an image translation model provided in an exemplary embodiment of this application;

[0108] Figure 3 This is a schematic diagram of an image translation model provided in an exemplary embodiment of this application;

[0109] Figure 4 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0110] Figure 5This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0111] Figure 6 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0112] Figure 7 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0113] Figure 8 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0114] Figure 9 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0115] Figure 10 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0116] Figure 11 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0117] Figure 12 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0118] Figure 13 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0119] Figure 14 This is a flowchart of a training method for an image translation model provided in an exemplary embodiment of this application;

[0120] Figure 15 This is a schematic diagram of a trained image translation model provided in an exemplary embodiment of this application;

[0121] Figure 16 This is a schematic diagram of an image semantic segmentation model provided in an exemplary embodiment of this application;

[0122] Figure 17 This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0123] Figure 18 This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0124] Figure 19This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0125] Figure 20 This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0126] Figure 21 This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0127] Figure 22 This is a flowchart of a training method for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0128] Figure 23 This is a schematic diagram of a trained image semantic segmentation model provided in an exemplary embodiment of this application;

[0129] Figure 24 This is a flowchart illustrating a method for training and using an image semantic segmentation model provided in an exemplary embodiment of this application;

[0130] Figure 25 This is a schematic diagram of image semantic segmentation provided in an exemplary embodiment of this application;

[0131] Figure 26 This is a structural block diagram of a training apparatus for an image translation model provided in an exemplary embodiment of this application;

[0132] Figure 27 This is a structural block diagram of a training apparatus for an image semantic segmentation model provided in an exemplary embodiment of this application;

[0133] Figure 28 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0134] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0135] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Exemplary embodiments will be described in detail here, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The singular forms “a,” “the,” and “the” used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0136] Figure 1 A schematic diagram of a computer system provided in one embodiment of this application is shown. This computer system can implement a system architecture for training methods of image translation models and / or image semantic segmentation models. The computer system may include a terminal 100 and a server 200. The terminal 100 may be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), access control device, or unmanned vending terminal. A client application for a target application may be installed and run on the terminal 100. This target application may be a game application or other applications that provide training functions for image translation models and / or image semantic segmentation models; this application does not limit the specific application. Furthermore, this application does not limit the form of the target application, including but not limited to Apps (Applications), mini-programs, etc., installed on the terminal 100, and may also be in the form of a webpage. The server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 200 may be a backend server for the aforementioned target application, used to provide backend services to the client of the target application.

[0137] The training method for image translation models and / or image semantic segmentation models provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the training method of the image translation model and / or image semantic segmentation model can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executing the training method of the image translation model and / or image semantic segmentation model), or the training method of the image translation model and / or image semantic segmentation model can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it. This application does not limit this.

[0138] Furthermore, the technical solution of this application can be combined with blockchain technology. For example, some data involved in the training method of the image translation model and / or image semantic segmentation model disclosed in this application (such as 3D medical images, predicted instance segmentation results, etc.) can be stored on the blockchain. The terminal 100 and the server 200 can communicate through a network, such as a wired or wireless network.

[0139] Next, the image translation model in this application will be introduced:

[0140] Figure 2 A schematic diagram of an image translation model 210 provided in an exemplary embodiment of this application is shown.

[0141] The image translation model 210 includes: anatomical encoder E a First mode encoder E m t And generator G. Image translation model 210 is used to translate a second modal image into an image including a first modal image, that is, to predict the first modal image based on the second modal image; for example, taking the translation of medical images as an example, the reason why the modal information included in medical images is different is because the technologies for acquiring medical images are different. For example, the technologies for acquiring medical images include, but are not limited to, at least one of the following: computed tomography (CT) technology, magnetic resonance (MR) technology, digital radiography (DX) technology, and ultrasound scanning (US) technology. The first modal image x t Second mode image x s Enter E respectively a and E m t The first feature representation z is obtained. a sThe second feature represents z m t The first feature representation is the feature representation obtained by anatomical encoding of the second modality image; the second feature representation is the feature representation obtained by modality encoding of the first modality image.

[0142] z a s and z m t Input G to get the first translated image. This is the image generated by image translation model 210 when predicting the first modality image based on the second modality image. (The result is...) The formula is:

[0143]

[0144] Image translation model 210 implements x-based s Predict x t , received

[0145] exist Figure 2 Based on the image translation model 210 shown, the image translation model 210 may further include: a second modal encoder E m s . Figure 3 A schematic diagram of an image translation model 210 provided in an exemplary embodiment of this application is shown.

[0146] The image translation model 210 includes: anatomical encoder E a First mode encoder E m t Second mode encoder E m s And generator G. Image translation model 210 is used to translate a second modality image into a first modality image, i.e., to predict a first modality image based on a second modality image; and / or, to translate a first modality image into a second modality image, i.e., to predict a second modality image based on a first modality image.

[0147] The first modality image x t Second mode image x s Enter E respectively a E m t and E m s .

[0148] The first feature representation z is obtained a s The second feature represents z m t The third feature represents zm s and the fourth feature representation z a t The first feature representation is the feature representation obtained by anatomical encoding of the second modality image; the second feature representation is the feature representation obtained by modal encoding of the first modality image; the third feature representation is the feature representation obtained by modal encoding of the second modality image; and the fourth feature representation is the feature representation obtained by anatomical encoding of the first modality image.

[0149] z a s and z m t Input G to get the first translated image. This is the image generated by image translation model 210 when predicting the first modality image based on the second modality image; z m s and z a t Input G to get the second translation image. This is the image generated by image translation model 210 when predicting the second modality image based on the first modality image; z m t and z a t Input G to get the third translation image. It is the image generated by image translation model 210 when predicting the first modality image based on the first modality image; z a s and z m s Input G to get the fourth translation image. This is the image generated by image translation model 210 when predicting a second modality image based on a second modality image. (The result is...) and The formula is:

[0150]

[0151]

[0152] To improve the translation accuracy of image translation models, it is necessary to train the image translation models. The following examples will introduce the training methods for image translation models.

[0153] Figure 4 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0154] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0155] In the various embodiments of this application, the detailed information regarding modal information in the image has been described above. Please refer to the above description. Figure 2 The contents shown in the diagram.

[0156] Category labels are used to mark the target organization to which the image content belongs. For example, the target organization includes, but is not limited to, at least one of the following: • Biological organs; such as: heart, brain, lungs, stomach, spleen, kidneys, gills, bones; • Biological tissues; such as: lymphatic tissue, connective tissue; • Biological cells: such as: muscle cells, nerve cells; • Biological organelles or biological macromolecules: proteins, ribosomes, mitochondria, chloroplasts, viruses.

[0157] Taking the heart as an example, the classification labels for the heart include, but are not limited to, at least one of the following: ascending aorta (AA), left ventricular myocardium (MYO), left ventricular blood cavity (LVC), and left atrium blood cavity (LAC). In various embodiments of this application, the classification labels of the first modality image, the second modality image, and the second modality image are usually acquired simultaneously, but the possibility of acquiring the above three types of information separately is not excluded; when the classification labels of the second modality image are acquired separately, the acquisition time of the classification labels of the second modality image is earlier than the execution time of step 340, that is, it can be executed before, after, or simultaneously with step 320 or step 330.

[0158] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0159] The first translated image is the image generated by the image translation model when predicting the first modality image based on the second modality image. The first translated image has the same modal information as the first modality image.

[0160] For example, taking medical images as an example, the first modal image is a CT image obtained based on CT technology, the second modal image is an MR image obtained based on MR technology, and the first translated image is an image generated when predicting the CT image based on the MR image. The first translated image has modal information of the CT image.

[0161] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0162] The first prediction probability is the probability that the image content of the first translated image belongs to the target organization, and the second prediction probability is the probability that the image content of the second modality image belongs to the target organization;

[0163] For example, the target tissue is typically at least one of the classification labels; for the heart region, the predicted probability is the probability that the image content belongs to at least one of AA, MYO, LVC, and LAC.

[0164] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0165] Structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively; for example, the structural consistency error of an image translation model includes, but is not limited to, at least one of the following: Cross-Entropy Loss, Zero-One Loss, and Dice Loss.

[0166] Step 350: Based on the structural consistency error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0167] The purpose of backpropagation training of the image translation model is to minimize the error between the first translated image and the first modality image.

[0168] In summary, the method provided in this embodiment includes information from both a first modality image and a second modality image in the first translated image, thus containing sufficient semantic information for predicting semantic labels. Through the first translated image, the semantic information in the image can be fully extracted, enabling the use of an image translation model to translate between different modalities in the image. By using structural consistency error to train the image translation model, the accuracy of obtaining translated images between different modalities can be improved, overcoming the difficulty of significant differences in the distribution of different modalities in the image, and achieving unsupervised adaptation in medical images.

[0169] Figure 5 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0170] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0171] The details of this step have been described above; please refer to the above text. Figure 4 Step 310 in the illustrated embodiment.

[0172] Step 322: Input the second modality image into the anatomical encoder to obtain the first feature representation;

[0173] For example, the image translation model includes an anatomical encoder, a first modality encoder, and a generator. The anatomical encoder encodes the anatomical structures of the image, and the first modality encoder encodes modal information in a first modal domain within the image. For example, in medical images, the anatomical structures are used to represent the shape and / or arrangement of image content. The specific structure of the image translation model can be found in [reference needed]. Figure 2 The diagram shown illustrates the image translation model; optionally, the image translation model also includes a second modal encoder. The specific structure of the image translation model can be found in [reference needed]. Figure 3 The diagram shows an image translation model.

[0174] The first feature representation is the feature representation obtained by anatomical encoding of the second modality image.

[0175] For example, in various embodiments of this application, the anatomical encoder is structured with Joint Pyramid Upsampling (JPU) connected to the last residual layer (layer 4) of the backbone network, which is a Deep Residual Network 101 (ResNet101). The generator structure is at least one of the following: Adaptive Instance Normalization (AdaIN) structure, Generative Adversarial Networks (GAN) structure, Network in Network (NIN) structure, and Recurrent Neural Network (RNN) structure.

[0176] Step 324: Input the first modal image into the first modal encoder to obtain the second feature representation;

[0177] The first modal encoder is used to encode modal information in the first modal domain of the image; the second feature representation is the feature representation obtained by modal encoding the first modal image.

[0178] For example, the first modal encoder includes six convolutional units, the first of which includes one convolutional layer; the other convolutional units include one convolutional layer, one regularization layer, and one activation function; the activation function is a Rectified Linear Unit (ReLU), with a fixed negative slope of 0.2. In various embodiments of this application, the structure of the second modal encoder used to encode modal information in the second modal domain of an image is generally the same as that of the first modal encoder, but there are also cases where the structures of the two encoders differ. For example, the second feature representation is an 8-dimensional vector.

[0179] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between steps 322 and 324. Step 322 can be executed before, after, or simultaneously with step 324.

[0180] Step 326: Input the first feature representation and the second feature representation into the generator to obtain the first translated image;

[0181] The first translated image is an image generated by the image translation model when predicting the first modality image based on the second modality image. Exemplarily, in various embodiments of this application, the generator structure is at least one of the following: AdaIN structure, GAN structure, NIN structure, or RNN structure.

[0182] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0183] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0184] Step 350: Based on the structural consistency error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0185] The details of steps 330, 340, and 350 have been described above. Please refer to the above text. Figure 4 Steps 330, 340, and 350 are shown in the illustrated embodiment.

[0186] In summary, the method provided in this embodiment constructs an image translation model through an anatomical encoder, a first modality encoder, and a generator; it fully mines the semantic information in the image through the first translated image; it calculates the error between the predicted probability obtained by the classification network and the classification label, and uses the structural consistency error to train the image translation model. This can improve the accuracy of obtaining translated images between different modal information, overcome the difficulty of large differences in the distribution of different modal information in the image, and achieve unsupervised adaptation in medical images.

[0187] Figure 6 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0188] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0189] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0190] The details of steps 310 and 320 have been described above; please refer to the above text. Figure 4 Steps 310 and 320 in the illustrated embodiment.

[0191] Step 332: Input the first translated image and the second modality image into the anatomical encoder respectively to obtain the fifth feature representation of the first translated image and the sixth feature representation of the second modality image respectively;

[0192] The classification network includes an anatomical encoder and a classifier. It should be noted that inputting only the first translated image into the anatomical encoder yields the fifth feature representation of the first translated image; this fifth feature representation is the feature representation obtained by anatomically encoding the first translated image. Similarly, inputting only the second modality image into the anatomical encoder yields the sixth feature representation of the second modality image; this sixth feature representation is the feature representation obtained by anatomically encoding the second modality image. The anatomical encoder in the classification network typically has the same structure and / or parameters as the anatomical encoder in the image translation model described in the above embodiments, but differences in structure and / or parameters are not excluded.

[0193] Step 334: Input the fifth feature representation and the sixth feature representation into the classifier respectively to obtain the first prediction probability and the second prediction probability;

[0194] The first prediction probability is the probability that the image content of the first translated image belongs to the target organization, and the second prediction probability is the probability that the image content of the second modality image belongs to the target organization.

[0195] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0196] Step 350: Based on the structural consistency error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0197] The details of steps 340 and 350 have been described above; please refer to the above text. Figure 4 Steps 340 and 350 in the illustrated embodiment.

[0198] In summary, the method provided in this embodiment fully mines the semantic information in the image through the first translated image; a classification network is constructed by an anatomical encoder and a classifier, and the error between the predicted probability obtained by the classification network and the classification label is calculated. The image translation model is trained using the structural consistency error, which can improve the accuracy of the translated images obtained between different modal information, overcome the difficulty of the large differences in the distribution of different modal information in the image, and achieve unsupervised adaptation in medical images.

[0199] Next, for Figure 4 , Figure 5 and Figure 6 The structural consistency errors in the illustrated embodiments are described below:

[0200] For example, structural consistency error includes a first cross-entropy loss function and / or a first Dessell loss function;

[0201] The first cross-entropy loss function includes the cross-entropy loss function between the first predicted probability and the classification label, and the cross-entropy loss function between the second predicted probability and the classification label;

[0202] For example, the cross-entropy loss function between the first predicted probability and the classification label is:

[0203]

[0204] Among them, L ce (p s2t ,y s p represents the cross-entropy loss function between the first predicted probability and the classification label. s2t Indicates the first predicted probability. C represents the classifier, E a Indicates anatomical encoder, Represents the first translated image, y s H represents the category label, W represents the image height, and T represents the image width.

[0205] For example, the cross-entropy loss function between the second predicted probability and the classification label is:

[0206]

[0207] Among them, L ce (p s ,y s ) represents the cross-entropy loss function between the second predicted probability and the classification label, p s p represents the second prediction probability. s =C(E) a (x s C represents the classifier, E a Indicates the anatomical encoder, x s Represents the second modality image, y s H represents the category label, W represents the image height, and T represents the image width.

[0208] The first Dessian loss function includes the Dessian loss function between the first predicted probability and the classification label, and the Dessian loss function between the second predicted probability and the classification label.

[0209] For example, the Dess loss function between the first predicted probability and the classification label is:

[0210]

[0211] Among them, L dice (p s2t ,y s p represents the Desais loss function between the first predicted probability and the classification label. s2t Indicates the first predicted probability. C represents the classifier, E a Indicates anatomical encoder, Represents the first translated image, y s The symbol represents a category label, and |||| represents the norm operation.

[0212] For example, the Dess loss function between the second predicted probability and the classification label is:

[0213]

[0214] Among them, L dice (p s ,y s ) represents the Desais loss function between the second predicted probability and the classification label, p s p represents the second prediction probability. s =C(E) a (x s C represents the classifier, Ea Indicates the anatomical encoder, x s Represents the second modality image, y s The symbol represents a category label, and |||| represents the norm operation.

[0215] For example, when the structural consistency error includes the first cross-entropy loss function and the first Dessell loss function, the structural consistency error is:

[0216] L SC =L ce (p s ,y s )+L dice (p s ,y s )+L ce (p s2t ,y s )+L dice (p s2t ,y s );

[0217] To further improve the translation accuracy of image translation models, more errors can be added to the structural consistency error to train the image translation model. The following examples will introduce the training method of the image translation model.

[0218] Figure 7 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0219] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0220] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0221] The details of steps 310 and 320 have been described above; please refer to the above text. Figure 4 Steps 310 and 320 in the illustrated embodiment.

[0222] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0223] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0224] The details of steps 330 and 340 have been described above; please refer to the above text. Figure 4Steps 330 and 340 in the illustrated embodiment.

[0225] Step 330a: Input the first translated image and the first modality image into the first modality discriminator respectively to obtain the first discrimination result;

[0226] The first modality discriminator is used to determine whether the input image belongs to the first modality. The first discrimination result includes the first modality discrimination result of the first translated image and the first modality discrimination result of the first modality image. For example, the first modality discriminator includes four convolutional kernels, each with a size of 4. The stride of the first, second, and third convolutional kernels is 2, and the stride of the last convolutional kernel is 1. The structure of the first modality discriminator is at least one of the following: Patch Generative Adversarial Networks (PatchGAN), NIN structure, or RNN structure. In various embodiments of this application, the structure of the second modality discriminator used to determine whether the input image belongs to the second modality image is generally the same as the structure of the first modality discriminator, but there are also cases where the structures of the two discriminators differ.

[0227] Step 340a: Based on the first discrimination result, calculate the first mode discrimination error;

[0228] For example, the first mode discrimination error is the adversarial loss function of the first mode discrimination result;

[0229] For example, the first mode discrimination error is:

[0230]

[0231] in, Let x represent the first mode discrimination error, E represent the mathematical expectation operation, and x represent the first mode discrimination error. t Represents the first modality image, x s D represents the second modality image. t Let G represent the first mode discriminator, and E represent the generator. a Indicates anatomical encoder, This indicates the first mode encoder.

[0232] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and the second branch, that is, the first branch can be executed before, after or simultaneously with the second branch; wherein, the first branch includes steps 330 and 340, and the second branch includes steps 330a and 340a.

[0233] Step 350a: Based on the structural consistency error and the first mode discrimination error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0234] The purpose of backpropagation training of the image translation model is to minimize the error between the first translated image and the first modality image.

[0235] In summary, the method provided in this embodiment fully mines the semantic information in the image through the first translated image; it introduces a first modality discriminator to calculate the first modality discrimination error, and uses the structural consistency error and the first modality discrimination error to train the image translation model, which can improve the accuracy of the translated images obtained between different modal information, overcome the difficulty of the large differences in the distribution of different modal information in the image, and achieve unsupervised adaptation in medical images.

[0236] Figure 8 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0237] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0238] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0239] The details of steps 310 and 320 have been described above; please refer to the above text. Figure 4 Steps 310 and 320 in the illustrated embodiment.

[0240] Step 320b: Call the image translation model to process the first modality image and the second modality image to obtain the second translated image;

[0241] The second translated image is the image generated by the image translation model when predicting the second modality image based on the first modality image.

[0242] Optional, such as Figure 9 As shown, step 320b may include three sub-steps: step 322b, step 324b, and step 326b.

[0243] Step 322b: Input the second modality image into the second modality encoder to obtain the third feature representation;

[0244] The image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode the modal information in the first modal domain of the image, and the second modal encoder is used to encode the modal information in the second modal domain of the image.

[0245] The third feature representation is the feature representation obtained by modal encoding of the second modality image.

[0246] Step 324b: Input the first modality image into the anatomical encoder to obtain the fourth feature representation;

[0247] The fourth feature representation is the feature representation obtained by anatomical encoding of the first modality image.

[0248] It should be noted that, in this embodiment, no restrictive provisions are made on the timing relationship between steps 322b and 324b. Step 322b can be executed before, after, or simultaneously with step 324b.

[0249] Step 326b: Input the third feature representation and the fourth feature representation into the generator to obtain the second translated image;

[0250] The second translated image is the image generated by the image translation model when predicting the second modality image based on the first modality image. For example, the generator in this embodiment is... Figure 5 The generator in the illustrated embodiments is typically the same generator.

[0251] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0252] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0253] The details of steps 330 and 340 have been described above; please refer to the above text. Figure 4 Steps 330 and 340 in the illustrated embodiment.

[0254] Step 330b: Input the second translated image and the second modality image into the second modality discriminator respectively to obtain the second discrimination result;

[0255] The second modality discriminator is used to identify whether the input image belongs to the second modality. The second discrimination result includes the second modality discrimination result of the second translated image and the second modality discrimination result of the second modality image.

[0256] Step 340b: Based on the second discrimination result, calculate the second mode discrimination error;

[0257] For example, the second mode discrimination error is the adversarial loss function of the second mode discrimination result;

[0258] For example, the second mode discrimination error is:

[0259]

[0260] in, Let x represent the second mode discrimination error, E represent the mathematical expectation operation, and x represent the second mode discrimination error. t Represents the first modality image, x s D represents the second modality image. s G represents the second-mode discriminator, and E represents the generator. a Indicates anatomical encoder, This indicates the second mode encoder.

[0261] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and the second branch, that is, the first branch can be executed before, after or simultaneously with the second branch; wherein, the first branch includes steps 330 and 340, and the second branch includes steps 330b and 340b.

[0262] Step 350b: Based on the structural consistency error and the second modality discrimination error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0263] The purpose of backpropagation training of the image translation model is to minimize the error between the first translated image and the first modality image.

[0264] In summary, the method provided in this embodiment fully mines the semantic information in the image through the first translated image; introduces a second modality discriminator to calculate the second modality discrimination error, and uses the structural consistency error and the second modality discrimination error to train the image translation model, which can improve the accuracy of the translated images obtained between different modal information, overcome the difficulty of the large differences in the distribution of different modal information in the image, and realize unsupervised adaptation in medical images.

[0265] Figure 10 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0266] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0267] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0268] The details of steps 310 and 320 have been described above; please refer to the above text. Figure 4 Steps 310 and 320 in the illustrated embodiment.

[0269] Step 320c: Call the image translation model to process the first modality image and the second modality image to obtain the third translated image;

[0270] The third translated image is the image generated by the image translation model when predicting the first modality image based on the first modality image.

[0271] Optional, such as Figure 11 As shown, step 320c may include three sub-steps: step 322c, step 324c, and step 326c.

[0272] Step 322c: Input the first modal image into the first modal encoder to obtain the second feature representation;

[0273] The image translation model includes an anatomical encoder, a first modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode the modal information in the first modal domain of the image.

[0274] The second feature representation is the feature representation obtained by modal encoding of the first modality image.

[0275] Step 324c: Input the first modality image into the anatomical encoder to obtain the fourth feature representation;

[0276] The fourth feature representation is the feature representation obtained by anatomical encoding of the first modality image.

[0277] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between steps 322c and 324c. Step 322c can be executed before, after, or simultaneously with step 324c.

[0278] Step 326c: Input the second feature representation and the fourth feature representation into the generator to obtain the third translated image;

[0279] The third translated image is the image generated by the image translation model when predicting the first modality image based on the first modality image. For example, the generator in this embodiment is... Figure 5 The generator in the illustrated embodiments is typically the same generator.

[0280] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0281] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0282] The details of steps 330 and 340 have been described above; please refer to the above text. Figure 4 Steps 330 and 340 in the illustrated embodiment.

[0283] Step 340c: Calculate the reconstruction loss error based on the third translated image and the first modality image;

[0284] For example, the reconstruction loss error is an L1 norm loss function;

[0285] For example, the reconstruction loss error is:

[0286]

[0287] Among them, L rec Indicates reconstruction loss error, Indicates the third translation image, x t Let |||| denote the first modality image, and |||| denote the norm operation.

[0288] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and step 340c, that is, the first branch can be executed before, after or simultaneously with step 340c; wherein, the first branch includes steps 330 and 340.

[0289] Step 350c: Based on the structural consistency error and reconstruction loss error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0290] The purpose of backpropagation training of the image translation model is to minimize the error between the first translated image and the first modality image.

[0291] In summary, the method provided in this embodiment fully mines the semantic information in the image through the first translated image; calculates the reconstruction loss error by comparing the difference between the third translated image and the first modal image; and trains the image translation model using the structural consistency error and the reconstruction loss error. This can improve the accuracy of obtaining translated images between different modal information, overcome the difficulty of large differences in the distribution of different modal information in the image, and achieve unsupervised adaptation in medical images.

[0292] Figure 12 A flowchart illustrating a training method for an image translation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0293] Step 310: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0294] Step 320: Input the first modality image and the second modality image into the image translation model to obtain the first translated image;

[0295] The details of steps 310 and 320 have been described above; please refer to the above text. Figure 4Steps 310 and 320 in the illustrated embodiment.

[0296] Step 320d: Call the image translation model to process the first modality image and the second modality image to obtain the fourth translated image;

[0297] The fourth translated image is the image generated by the image translation model when predicting the second modality image based on the second modality image.

[0298] Optional, such as Figure 11 As shown, step 320d may include three sub-steps: step 322d, step 324d, and step 326d.

[0299] Step 322d: Input the second modality image into the anatomical encoder to obtain the first feature representation;

[0300] The image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode the modal information in the first modal domain of the image, and the second modal encoder is used to encode the modal information in the second modal domain of the image.

[0301] The first feature representation is the feature representation obtained by anatomical encoding of the second modality image.

[0302] Step 324d: Input the second modality image into the second modality encoder to obtain the third feature representation;

[0303] The third feature representation is the feature representation obtained by modal encoding of the second modality image.

[0304] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between steps 322d and 324d. Step 322d can be executed before, after, or simultaneously with step 324d.

[0305] Step 326d: Input the first feature representation and the third feature representation into the generator to obtain the fourth translation image;

[0306] The fourth translated image is the image generated by the image translation model when predicting the second modality image based on the second modality image. For example, the generator in this embodiment is... Figure 5 The generator in the illustrated embodiments is typically the same generator.

[0307] Step 330: Input the first translated image into the classification network to obtain the first predicted probability, and input the second modality image into the classification network to obtain the second predicted probability;

[0308] Step 340: Calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label;

[0309] The details of steps 330 and 340 have been described above; please refer to the above text. Figure 4 Steps 330 and 340 in the illustrated embodiment.

[0310] Step 340d: Calculate the reconstruction loss error based on the fourth translated image and the second modality image;

[0311] For example, the reconstruction loss error is an L1 norm loss function;

[0312] For example, the reconstruction loss error is:

[0313]

[0314] Among them, L rec Indicates reconstruction loss error, The fourth translation image, x s represents the second modality image, and |||| represents the norm operation.

[0315] Optionally, the reconstruction loss error is:

[0316]

[0317] Among them, L rec Indicates reconstruction loss error, This represents the third translated image. The fourth translation image, x t Represents the first modality image, x s represents the second modality image, and |||| represents the norm operation.

[0318] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and step 340d, that is, the first branch can be executed before, after or simultaneously with step 340d; wherein, the first branch includes steps 330 and 340.

[0319] Step 350d: Based on the structural consistency error and reconstruction loss error, perform backpropagation training on the image translation model to obtain the trained image translation model;

[0320] The purpose of backpropagation training of the image translation model is to minimize the error between the first translated image and the first modality image.

[0321] In summary, the method provided in this embodiment fully mines the semantic information in the image through the first translated image; calculates the reconstruction loss error by comparing the difference between the fourth translated image and the second modal image; and trains the image translation model using the structural consistency error and the reconstruction loss error. This can improve the accuracy of obtaining translated images between different modal information, overcome the difficulty of large differences in the distribution of different modal information in the image, and achieve unsupervised adaptation in medical images.

[0322] Figure 14 A flowchart illustrating a training method for an image translation model 210 according to this application is shown. This method can be executed by a computer device.

[0323] The first modality image x t Second mode image x s The first translated image is obtained by inputting the image translation model 210. Second translation image Third translation image and the fourth translation image

[0324] in, It is an image generated by image translation model 210 when predicting the first modality image based on the second modality image; It is the image generated by image translation model 210 when predicting the second modality image based on the first modality image; It is the image generated by image translation model 210 when predicting the first modality image based on the first modality image; It is an image generated by the image translation model 210 when predicting a second modality image based on a second modality image.

[0325] The classification network 220 includes an anatomical encoder E a And classifier C.

[0326] x s and Enter E respectively a The fifth feature representation z of the first translated image is obtained respectively. a s The sixth feature representation z of the second modality image a s2t ;

[0327] z a s and z a s2t Input C respectively, and obtain the first predicted probability p respectively. s Second prediction probability p s2t ;p sp is the probability that the image content of the first translated image belongs to the target organization. s2t It is the probability that the image content of the second modality belongs to the target organization.

[0328] Based on p s p s2t and category tags y s The structural consistency error of 302 is calculated; based on the third translated image, the fourth translated image, the first modal image, and the second modal image, the reconstruction loss error of 304 is calculated.

[0329] Will and x t Input the first mode discriminator D respectively t The first identification result is obtained; based on the first identification result, the first mode identification error of 306 is calculated. and x s Input the second modality discriminator D respectively s The second identification result is obtained; based on the second identification result, the second mode identification error 308 is calculated.

[0330] Based on the structural consistency error 302, reconstruction loss error 304, first mode discrimination error 306, and second mode discrimination error 308, the image translation model 210 is trained by backpropagation to obtain the trained image translation model.

[0331] For example, the total loss function of the image translation model 210 is:

[0332]

[0333] Among them, L SC This indicates a structural consistency error of 302. This indicates that the discrimination error for the first mode is 306. This indicates that the second mode discrimination error is 308, L rec The reconstruction loss error is 304, λ. adv This represents the balance factor for modal discrimination error. For example, λ adv The value range of λ is greater than or equal to 0 and less than or equal to 1; optionally, λ adv =0.01.

[0334] Figure 15 A schematic diagram of a trained image translation model 230 provided in an exemplary embodiment of this application is shown.

[0335] The trained image translation model 230 includes: anatomical encoder E a First mode encoder E m t And generator G.

[0336] The first modality image x t Second mode image x s Enter E respectively a and E m t We get x. s The corresponding feature representation z a s x t The corresponding feature representation z m t , z a s It is x s Feature representation obtained through anatomical encoding; z m t It is x t The feature representation obtained through modality encoding.

[0337] z a s and z m t Input G to obtain the fifth translation image x s2t x s2t It is the trained image translation model 230 based on x s Predict x t The image generated at that time.

[0338] Next, the image semantic segmentation model in this application will be introduced:

[0339] Figure 16 A schematic diagram of an image semantic segmentation model 240 provided in an exemplary embodiment of this application is shown.

[0340] The image semantic segmentation model 240 includes: anatomical encoder E a And classifier C. Image semantic segmentation model 240 segments the image according to the target organization to which the image content belongs. The fifth translated image x s2t Second mode image x s Enter E respectively a The seventh feature representation z of the fifth translated image is obtained respectively. a s2t The eighth feature representation z of the second modality image a s ;

[0341] z a s2t and z a s Input C respectively, and obtain the third prediction probability p respectively. s2t and the fourth prediction probability p s .

[0342] To improve the semantic segmentation accuracy of image semantic segmentation models, it is necessary to train the image semantic segmentation models. The following examples will introduce the training methods for image semantic segmentation models.

[0343] Figure 17 A flowchart illustrating a training method for an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0344] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0345] In the various embodiments of this application, the detailed information regarding modal information in the image has been described above. Please refer to the above description. Figure 2 The contents shown in the diagram.

[0346] Classification labels are used to indicate the target tissue to which the image content belongs. For example, taking the heart as an example, the classification labels for the heart include, but are not limited to, at least one of the following: AA, MYO, LVC, LAC. In various embodiments of this application, the classification labels of the first modality image, the second modality image, and the second modality image are usually acquired simultaneously, but the possibility of acquiring the above three types of information separately is not excluded. When the classification labels of the second modality image are acquired separately, the acquisition time of the classification labels of the second modality image is earlier than the execution time of step 440, that is, it can be executed before, after, or simultaneously with step 420 or step 430.

[0347] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0348] The fifth translated image is the image generated by the trained image translation model when predicting the first modality image based on the second modality image. The trained image translation model is obtained according to the training method of any of the above image translation models.

[0349] Step 430: Input the fifth translated image into the image semantic segmentation model to obtain the third prediction probability, and input the second modality image into the image semantic segmentation model to obtain the fourth prediction probability;

[0350] The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modality image belongs to the target organization.

[0351] For example, the target tissue is typically at least one of the classification labels; for the heart region, the predicted probability is the probability that the image content belongs to at least one of AA, MYO, LVC, and LAC.

[0352] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0353] The classification error is used to represent the difference between the third and fourth predicted probabilities and the classification label, respectively; for example, the classification error of an image semantic segmentation model includes, but is not limited to, at least one of the following: cross-entropy loss function, 0-1 loss function, and Descein loss function.

[0354] Step 450: Based on the classification error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0355] The purpose of backpropagation training on an image semantic segmentation model is to minimize the error between the third predicted probability and the classification label.

[0356] In summary, the method provided in this embodiment provides a high-quality translated image for training the image semantic segmentation model by using a trained image translation model, calculating the error between the predicted probability and the classification label obtained by the image semantic segmentation model, and training the image semantic segmentation model to achieve domain-adaptive semantic segmentation in the image.

[0357] Figure 18 A flowchart illustrating a training method for an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0358] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0359] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0360] The details of steps 410 and 420 have been described above; please refer to the above text. Figure 17 Steps 410 and 420 in the illustrated embodiment.

[0361] Step 432: Input the fifth translated image and the second modality image into the anatomical encoder respectively to obtain the seventh feature representation of the fifth translated image and the eighth feature representation of the second modality image respectively;

[0362] The image semantic segmentation model includes an anatomical encoder and a classifier. The anatomical encoder is used to encode the anatomical structure of the image. It should be noted that the anatomical encoder in this embodiment and the anatomical encoder of the image translation model in the above embodiment usually have the same structure, but it is not excluded that the structures are different. The parameters of the anatomical encoder in this embodiment and the anatomical encoder of the image translation model in the above embodiment are usually different, but it is not excluded that the parameters are the same.

[0363] Step 434: Input the seventh feature representation and the eighth feature representation into the classifier respectively to obtain the third prediction probability and the fourth prediction probability;

[0364] The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modality image belongs to the target organization.

[0365] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0366] For example, the classification error includes a second cross-entropy loss function and / or a second Dessell loss function;

[0367] The second cross-entropy loss function includes the third cross-entropy loss function between the predicted probability and the classification label, and the fourth cross-entropy loss function between the predicted probability and the classification label;

[0368] For example, the cross-entropy loss function between the third predicted probability and the classification label is:

[0369]

[0370] Among them, L ce (p s2t ,y s2t ) represents the cross-entropy loss function between the third predicted probability and the classification label, p s2t p represents the third prediction probability. s2t =C(E) a (x s2t C represents the classifier, E a Indicates the anatomical encoder, x s2t The fifth translation image is represented by y. s2t The category label for the fifth translated image, y s2t =y s y s The image represents the classification label of the second modality, where H represents the height of the image, W represents the width of the image, and T represents the classification label of the image.

[0371] For example, the cross-entropy loss function between the fourth predicted probability and the classification label is:

[0372]

[0373] Among them, L ce (p s ,y s ) represents the cross-entropy loss function between the fourth predicted probability and the classification label, p s p represents the fourth prediction probability. s =C(E) a (x s C represents the classifier, E a Indicates the anatomical encoder, x s Represents the second modality image, y s The image represents the classification label of the second modality, where H represents the height of the image, W represents the width of the image, and T represents the classification label of the image.

[0374] The second Dessian loss function includes the third Dessian loss function between the predicted probability and the classification label, and the fourth Dessian loss function between the predicted probability and the classification label.

[0375] For example, the Dess loss function between the third predicted probability and the classification label is:

[0376]

[0377] Among them, L dice (p s2t ,y s2t ) represents the Desais loss function between the third predicted probability and the classification label, p s2t p represents the third prediction probability. s2t =C(E) a (x s2t C represents the classifier, E a Indicates the anatomical encoder, x s2t The fifth translation image is represented by y. s2t The category label for the fifth translated image, y s2t =y s y s The classification label represents the second modality image, and |||| represents the norm operation.

[0378] For example, the Dess loss function between the fourth predicted probability and the classification label is:

[0379]

[0380] Among them, L dice (p s ,y s ) represents the Desais loss function between the fourth predicted probability and the classification label, ps p represents the fourth prediction probability. s =C(E) a (x s C represents the classifier, E a Indicates the anatomical encoder, x s Represents the second modality image, y s Let |||| represent the classification label, and |||| represent the norm operation. For example, when the classification error includes the second cross-entropy loss function and the second Dessian loss function, the classification error is:

[0381] L S =L ce (p s ,y s )+L dice (p s ,y s )+L ce (p s2t ,y s2t )+L dice (p s2t ,y s2t );

[0382] Step 450: Based on the classification error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0383] The details of step 450 have been described above; please refer to the above text. Figure 17 Step 450 in the illustrated embodiment.

[0384] In summary, the method provided in this embodiment provides high-quality translated images for training an image semantic segmentation model by using a trained image translation model, and constructs the image semantic segmentation model through an anatomical encoder and classifier. The error between the predicted probability obtained by the image semantic segmentation model and the classification label is calculated to train the image semantic segmentation model, achieving domain-adaptive semantic segmentation in images.

[0385] To further improve the translation accuracy of image semantic segmentation models, more errors can be added to the classification error during training. The following examples will introduce the training method for image semantic segmentation models.

[0386] Figure 19 A flowchart illustrating a training method for an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0387] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0388] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0389] Step 430: Input the fifth translated image into the image semantic segmentation model to obtain the third prediction probability, and input the second modality image into the image semantic segmentation model to obtain the fourth prediction probability;

[0390] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0391] The details of steps 410, 420, 430, and 440 have been described above. Please refer to the above text. Figure 17 Steps 410, 420, 430, and 440 are shown in the illustrated embodiment.

[0392] Step 430a: Rotate the first modality image and input the rotated first modality image into the image semantic segmentation model to obtain the fifth prediction probability;

[0393] The fifth prediction probability is the probability that the image content of the rotated first modality belongs to the target tissue.

[0394] For example, the fifth prediction probability is:

[0395]

[0396] in, Represents a semantic segmentation model, R i Indicates rotation operation, x t This represents the first modality image.

[0397] Step 440a: Calculate the weighted entropy error based on the fifth prediction probability;

[0398] For example, the weighted entropy error is:

[0399]

[0400] Where H represents the height of the image, W represents the width of the image, T represents the classification label of the image, and p t This represents the fifth prediction probability, and 'a' represents the balancing parameter. For example, the value of 'a' is set empirically; for instance, a = 10. -8 .

[0401] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and the second branch, that is, the first branch can be executed before, after or simultaneously with the second branch; wherein, the first branch includes steps 420, 430 and 440, and the second branch includes steps 430a and 440a.

[0402] Step 450a: Based on the classification error and weighted entropy error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0403] The purpose of backpropagation training on an image semantic segmentation model is to minimize the error between the third predicted probability and the classification label.

[0404] In summary, the method provided in this embodiment provides a high-quality translated image for training the image semantic segmentation model by using a trained image translation model, calculating the weighted entropy error based on the fifth prediction probability, calculating the classification error between the prediction probability obtained by the image semantic segmentation model and the classification label, and training the image semantic segmentation model based on the classification error and the weighted entropy error, thereby achieving domain-adaptive semantic segmentation in the image.

[0405] Figure 20 A flowchart illustrating a training method for an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0406] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0407] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0408] Step 430: Input the fifth translated image into the image semantic segmentation model to obtain the third prediction probability, and input the second modality image into the image semantic segmentation model to obtain the fourth prediction probability;

[0409] The details of steps 410, 420, and 430 have been described above. Please refer to the above text. Figure 17 Steps 410, 420, and 430 are shown in the illustrated embodiment.

[0410] Step 430b: Rotate the first modality image and input the rotated first modality image into the image semantic segmentation model to obtain the fifth prediction probability;

[0411] The fifth prediction probability is the probability that the image content of the rotated first modality belongs to the target tissue.

[0412] For example, the fifth prediction probability is:

[0413]

[0414] in, Represents a semantic segmentation model, R i Indicates rotation operation, x t This represents the first modality image.

[0415] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and step 430b, that is, the first branch can be executed before, after or simultaneously with step 430b; wherein, the first branch includes steps 420 and 430.

[0416] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0417] The details of step 440 have been described above; please refer to the above text. Figure 17 Step 440 in the illustrated embodiment.

[0418] Step 435b: Input the fourth prediction probability and the fifth prediction probability into the discriminator respectively to obtain the first discrimination result of the fourth prediction probability and the second discrimination result of the fifth prediction probability;

[0419] The discriminator is used to distinguish whether the image corresponding to the predicted probability originates from the first mode or the second mode;

[0420] Step 440b: Calculate the adversarial consistency error based on the first and second discrimination results;

[0421] For example, the anti-consistency error is:

[0422]

[0423] Where E represents the mathematical expectation operation, D seg p represents the discriminator. s p represents the fifth prediction probability. s This represents the fourth prediction probability.

[0424] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between step 440 and the second branch, that is, step 440 can be executed before, after or simultaneously with the second branch; wherein, the second branch includes step 435b and step 440b.

[0425] Step 450b: Based on the classification error and the adversarial consistency error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0426] The purpose of backpropagation training on an image semantic segmentation model is to minimize the error between the third predicted probability and the classification label.

[0427] In summary, the method provided in this embodiment provides high-quality translated images for training the image semantic segmentation model by using a trained image translation model. By introducing a discriminator to calculate the adversarial consistency error, the method calculates the classification error between the predicted probability obtained by the image semantic segmentation model and the classification label. Based on the classification error and the adversarial consistency error, the method trains the image semantic segmentation model, thereby achieving domain-adaptive semantic segmentation in the image.

[0428] Figure 21 A flowchart illustrating a training method for an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0429] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0430] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0431] Step 430: Input the fifth translated image into the image semantic segmentation model to obtain the third prediction probability, and input the second modality image into the image semantic segmentation model to obtain the fourth prediction probability;

[0432] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0433] The details of steps 410, 420, 430, and 440 have been described above. Please refer to the above text. Figure 17 Steps 410, 420, 430, and 440 are shown in the illustrated embodiment.

[0434] Step 430b: Rotate the first modality image and input the rotated first modality image into the image semantic segmentation model to obtain the fifth prediction probability;

[0435] The fifth prediction probability is the probability that the image content of the rotated first modality belongs to the target tissue.

[0436] For example, the fifth prediction probability is:

[0437]

[0438] in, R represents an image semantic segmentation model. i Indicates rotation operation, xt This represents the first modality image.

[0439] Step 430c: Input the first modality image into the teacher image semantic segmentation model, rotate the predicted probability of the first modality image output by the teacher image semantic segmentation model to obtain the sixth predicted probability;

[0440] For example, the structure of the teacher image semantic segmentation model is the same as that of the image semantic segmentation model, and the parameters of the teacher image semantic segmentation model are determined based on the parameters of the image semantic segmentation model.

[0441] For example, the parameters of the teacher image semantic segmentation model are determined based on the parameters of the image semantic segmentation model using the Exponential Moving Average (EMA) algorithm. For instance, the t-th parameter of the teacher image semantic segmentation model is:

[0442]

[0443] Where, θ s The parameters represent the image semantic segmentation model, where α represents the attenuation factor. For example, the value of α is greater than or equal to 0 and less than or equal to 1.

[0444] For example, the sixth prediction probability is:

[0445]

[0446] in, R represents the teacher image semantic segmentation model. i Indicates rotation operation, x t This represents the first modality image.

[0447] Step 435c: Based on the sixth prediction probability, determine the predicted classification label through the confidence threshold;

[0448] For example, the predicted category label is:

[0449]

[0450] in, This represents the predicted classification label, and k* represents the confidence threshold. arg represents the argument operation of complex numbers. For example, the value of k* is greater than or equal to 0 and less than or equal to 1; optionally, k* = 0.9.

[0451] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the first branch and step 430b, that is, the first branch can be executed before, after or simultaneously with step 430b; wherein, the first branch includes steps 430c and 435c.

[0452] Step 440c: Calculate the rotation consistency error based on the predicted classification label and the fifth predicted probability;

[0453] For example, rotational consistency error includes the cross-entropy loss function between the fifth predicted probability and the predicted class label and / or the Dess loss function between the fifth predicted probability and the predicted class label.

[0454] For example, the cross-entropy loss function between the fifth predicted probability and the predicted class label is:

[0455]

[0456] in, p represents the cross-entropy loss function between the fifth predicted probability and the predicted class label. t Indicates the fifth prediction probability. The image is defined by the predicted classification label, where H represents the height of the image, W represents the width of the image, and T represents the classification label of the image.

[0457] For example, the Dess loss function between the fifth predicted probability and the predicted class label is:

[0458]

[0459] in, p represents the Descein loss function between the fifth predicted probability and the predicted class label. t Indicates the fifth prediction probability. This indicates the predicted category label, and |||| represents the norm operation.

[0460] For example, when the rotational consistency error includes the cross-entropy loss function between the fifth predicted probability and the predicted class label and the Dess loss function between the fifth predicted probability and the predicted class label, the rotational consistency error is:

[0461]

[0462] It should be noted that in this embodiment, no restrictive provisions are made on the timing relationship between the second branch and the third branch, that is, the second branch can be executed before, after or simultaneously with the third branch; wherein, the second branch includes steps 420, 430 and 440; the third branch includes steps 430b, 430c, 435c and 440c.

[0463] Step 450c: Based on the classification error and rotation consistency error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0464] The purpose of backpropagation training on an image semantic segmentation model is to minimize the error between the third predicted probability and the classification label.

[0465] In summary, the method provided in this embodiment provides high-quality translated images for training the image semantic segmentation model by using a trained image translation model. It calculates the rotation consistency error by introducing a teacher image semantic segmentation model, calculates the classification error between the predicted probability and the classification label obtained by the image semantic segmentation model, and trains the image semantic segmentation model based on the classification error and rotation consistency error, thereby achieving domain-adaptive semantic segmentation in the image.

[0466] Figure 22 A flowchart illustrating a training method for an image semantic segmentation model provided in an exemplary embodiment of this application is shown.

[0467] The first modality image x t Second mode image x s Input the trained image translation model 230 to obtain the fifth translated image x. s2t x s2t It is the trained image translation model 230 based on x s Predict x t The image generated at that time;

[0468] x s2t and x s Input image semantic segmentation model 240, obtain the third prediction probability p s2t and the fourth prediction probability p s p s2t It is x s2t The probability, p, that the image content belongs to the target organization. s It is x s The probability that the image content belongs to the target organization;

[0469] x t Rotate R i The rotated first modality image is then input into the image semantic segmentation model 240 to obtain the fifth prediction probability p. t ;

[0470] p s and p t Input to discriminator D respectively seg This yields the first discrimination result for the fourth prediction probability and the second discrimination result for the fifth prediction probability;

[0471] x t Input the teacher image semantic segmentation model 250, and rotate the predicted probability of the first modality image output by the teacher image semantic segmentation model 250 by R. i The sixth prediction probability is obtained; based on the sixth prediction probability, the predicted classification label is determined through a confidence threshold.

[0472] Based on p s2t p s and category tags y s The classification error was calculated to be 312; based on p t The weighted entropy error of 318 is calculated; based on the first and second discrimination results, the adversarial consistency error of 316 is calculated; based on... and p t The rotational consistency error is calculated to be 314.

[0473] For example, the total loss function of the image semantic segmentation model 240 is:

[0474] L = L S +L RC +λ AC L AC +λ ent L ent ;

[0475] Among them, L S L represents a classification error of 312. ent The weighted entropy error is 318, L. RC This indicates a rotational consistency error of 314, L. AC The adversarial consistency error is 316, λ ent This represents the balance factor for the weighted entropy error of 318. For example, λ ent The value range of λ is greater than or equal to 0 and less than or equal to 1; optionally, λ ent =0.01. λ AC This represents the balance factor against the consistency error of 316. For example, λ AC The value range of λ is greater than or equal to 0 and less than or equal to 1; optionally, λ AC =0.01.

[0476] Figure 23 A schematic diagram of a trained image semantic segmentation model 260 provided in an exemplary embodiment of this application is shown.

[0477] The trained image semantic segmentation model 260 includes: anatomical encoder E a And classifier C.

[0478] The image to be segmented x i Enter Ea , to obtain x i Feature representation z a i ; will z a i Input C, get x i The predicted probability p i Based on x i p i The predicted classification label y of the image to be segmented is determined by predicting a confidence threshold. i .

[0479] Figure 24 A flowchart illustrating a method for training and using an image semantic segmentation model according to this application is shown. This method can be performed by a computer device. The method includes:

[0480] Step 410: Obtain the classification labels for the first modality image, the second modality image, and the second modality image;

[0481] Step 420: Input the first modality image and the second modality image into the trained image translation model to obtain the fifth translated image;

[0482] Step 430: Input the fifth translated image into the image semantic segmentation model to obtain the third prediction probability, and input the second modality image into the image semantic segmentation model to obtain the fourth prediction probability;

[0483] Step 440: Calculate the classification error based on the third predicted probability, the fourth predicted probability, and the classification label;

[0484] Step 450: Based on the classification error, perform backpropagation training on the image semantic segmentation model to obtain the trained image semantic segmentation model;

[0485] The details of steps 410 to 450 have been described above. Please refer to the above text. Figure 17 Steps 410 to 450 in the illustrated embodiment.

[0486] Step 460: Obtain the image to be segmented;

[0487] For example, the image to be segmented includes the same modal information as the first modal image in the above embodiments; taking a medical image as an example, the image to be segmented and the first modal image in the above embodiments are obtained using the same technology.

[0488] Step 470: Input the image to be segmented into the trained image semantic segmentation model to obtain the predicted probability of the image to be segmented; for example, the predicted probability of the image to be segmented is the probability that the image content of the image to be segmented belongs to the target organization.

[0489] Step 480: Based on the predicted probability of the image to be segmented, determine the predicted classification label of the image to be segmented by using the prediction confidence threshold;

[0490] The prediction confidence threshold ranges from 0 to 1; optionally, the prediction confidence threshold is 0.9.

[0491] In summary, the method provided in this embodiment achieves domain-adaptive semantic segmentation by using a trained image semantic segmentation model. The segmentation performance of the trained image semantic segmentation model is validated using a public dataset from the Multi-Modality Whole Heart Segmentation competition. Figure 25 This paper illustrates the semantic segmentation of images using the trained Image Semantic Segmentation Model (EMCR), the Synergistic Image and Feature Adaptation (SIFA) model, the Supervised Model, and the Bidirectional Unsupervised Domain Adaptation (BiUDA) model. Table 1 shows a comparison of the quantitative results of semantic segmentation using different models. The Average Symmetric Surface Distance (ASD) represents the average surface distance at symmetrical positions, measuring the difference between the segmentation result and the full standard; the Dice coefficient is a set similarity metric function used to calculate the similarity between two samples. The trained Image Semantic Segmentation Model achieves better image semantic segmentation results than other models.

[0492] Table 1

[0493]

[0494] Those skilled in the art will understand that the above embodiments can be implemented independently, or the above embodiments can be freely combined to create new embodiments to implement the model training method of this application.

[0495] Figure 26 A block diagram of a training apparatus for an image translation model provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0496] The acquisition module 510 is used to acquire the first modality image, the second modality image, and the classification label of the second modality image;

[0497] The input module 520 is used to input the first modal image and the second modal image into the image translation model to obtain a first translated image, wherein the first translated image is an image generated by the image translation model when predicting the first modal image based on the second modal image;

[0498] The input module 520 is further configured to input the first translated image into the classification network to obtain a first prediction probability, and input the second modal image into the classification network to obtain a second prediction probability, wherein the first prediction probability is the probability that the image content of the first translated image belongs to the target organization, and the second prediction probability is the probability that the image content of the second modal image belongs to the target organization;

[0499] The calculation module 530 is used to calculate the structural consistency error based on the first predicted probability, the second predicted probability and the classification label, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively.

[0500] The training module 540 is used to perform backpropagation training on the image translation model based on the structural consistency error to obtain the trained image translation model.

[0501] In an optional design of this embodiment, the image translation model includes an anatomical encoder, a first modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in the first modal domain of the image.

[0502] The input module 520 is further configured to: input the second modal image into the anatomical encoder to obtain a first feature representation, wherein the first feature representation is a feature representation obtained by anatomical encoding of the second modal image; input the first modal image into the first modal encoder to obtain a second feature representation, wherein the second feature representation is a feature representation obtained by modal encoding of the first modal image; and input the first feature representation and the second feature representation into the generator to obtain the first translated image.

[0503] In an optional design of this embodiment, the classification network includes an anatomical encoder and a classifier;

[0504] The input module 520 is further configured to: input the first translated image and the second modality image into the anatomical encoder respectively to obtain the fifth feature representation of the first translated image and the sixth feature representation of the second modality image respectively; input the fifth feature representation and the sixth feature representation into the classifier respectively to obtain the first prediction probability and the second prediction probability respectively.

[0505] In an optional design of this embodiment, the structural consistency error includes a first cross-entropy loss function and / or a first Dessian loss function; wherein, the first cross-entropy loss function includes a cross-entropy loss function between the first predicted probability and the classification label, and a cross-entropy loss function between the second predicted probability and the classification label; the first Dessian loss function includes a Dessian loss function between the first predicted probability and the classification label, and a Dessian loss function between the second predicted probability and the classification label. In an optional design of this embodiment, the input module 520 is further configured to: input the first translated image and the first modality image into a first modality discriminator respectively to obtain a first discrimination result, wherein the first modality discriminator is used to discriminate whether the input image belongs to the first modality image, and the first discrimination result includes the first modality discrimination result of the first translated image and the first modality discrimination result of the first modality image;

[0506] The calculation module 530 is further configured to: calculate the first mode discrimination error based on the first discrimination result;

[0507] The training module 540 is further configured to: perform backpropagation training on the image translation model based on the structural consistency error and the first modality discrimination error, so as to obtain the trained image translation model.

[0508] In an optional design of this embodiment, the first modality discrimination error is an adversarial loss function of the first modality discrimination result.

[0509] In an optional design of this embodiment, the device further includes:

[0510] The calling module 550 is used to call the image translation model to process the first modal image and the second modal image to obtain the second translated image;

[0511] The input module 520 is further configured to: input the second translated image and the second modal image into the second modality discriminator respectively to obtain a second discrimination result, wherein the second modality discriminator is used to discriminate whether the input image belongs to the second modality image, and the second discrimination result includes the second modality discrimination result of the second translated image and the second modality discrimination result of the second modality image;

[0512] The calculation module 530 is further configured to: calculate the second mode discrimination error based on the second discrimination result;

[0513] The training module 540 is further configured to: perform backpropagation training on the image translation model based on the structural consistency error and the second modality discrimination error, so as to obtain the trained image translation model.

[0514] In an optional design of this embodiment, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0515] The calling module 550 is further configured to: input the second modal image into the second modal encoder to obtain a third feature representation, wherein the third feature representation is a feature representation obtained by modal encoding of the second modal image; input the first modal image into the anatomical encoder to obtain a fourth feature representation, wherein the fourth feature representation is a feature representation obtained by anatomical encoding of the first modal image; input the third feature representation and the fourth feature representation into the generator to obtain the second translated image, wherein the second translated image is an image generated by the image translation model when predicting the second modal image based on the first modal image.

[0516] In an optional design of this embodiment, the second modality discrimination error is an adversarial loss function of the second modality discrimination result.

[0517] In an optional design of this embodiment, the calling module 550 is further configured to: call the image translation model to process the first modal image and the second modal image to obtain a third translated image;

[0518] The calculation module 530 is further configured to: calculate the reconstruction loss error based on the third translated image and the first modality image;

[0519] The training module 540 is further configured to: perform backpropagation training on the image translation model based on the structural consistency error and the reconstruction loss error, so as to obtain the trained image translation model.

[0520] In an optional design of this embodiment, the image translation model includes an anatomical encoder, a first modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in the first modal domain of the image.

[0521] The calling module 550 is further configured to: input the first modal image into the first modal encoder to obtain a second feature representation, wherein the second feature representation is a feature representation obtained by modal encoding of the first modal image; input the first modal image into the anatomical encoder to obtain a fourth feature representation, wherein the fourth feature representation is a feature representation obtained by anatomical encoding of the first modal image; input the second feature representation and the fourth feature representation into the generator to obtain the third translated image, wherein the third translated image is an image generated by the image translation model when predicting the first modal image based on the first modal image.

[0522] In an optional design of this embodiment, the calling module 550 is further configured to: call the image translation model to process the first modal image and the second modal image to obtain a fourth translated image;

[0523] The calculation module 530 is further configured to: calculate the reconstruction loss error based on the fourth translated image and the second modality image;

[0524] The training module 540 is further configured to: perform backpropagation training on the image translation model based on the structural consistency error and the reconstruction loss error, so as to obtain the trained image translation model.

[0525] In an optional design of this embodiment, the image translation model includes an anatomical encoder, a first modal encoder, a second modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, the first modal encoder is used to encode modal information in a first modal domain of the image, and the second modal encoder is used to encode modal information in a second modal domain of the image.

[0526] The calling module 550 is further configured to: input the second modality image into the anatomical encoder to obtain a first feature representation, wherein the first feature representation is a feature representation obtained by anatomical encoding of the second modality image; input the second modality image into the second modality encoder to obtain a third feature representation, wherein the third feature representation is a feature representation obtained by modality encoding of the second modality image; input the first feature representation and the third feature representation into the generator to obtain the fourth translated image, wherein the fourth translated image is an image generated by the image translation model when predicting the second modality image based on the second modality image.

[0527] In an optional design of this embodiment, the reconstruction loss error is an L1 norm loss function.

[0528] Figure 27 A block diagram of a training apparatus for an image semantic segmentation model provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0529] The acquisition module 610 is used to acquire the first modality image, the second modality image, and the classification label of the second modality image;

[0530] The input module 620 is used to input the first modal image and the second modal image into the trained image translation model to obtain a fifth translated image. The fifth translated image is an image generated by the trained image translation model when predicting the first modal image based on the second modal image. The trained image translation model is obtained according to the training method of any of the above image translation models.

[0531] The input module 620 is further configured to input the fifth translated image into the image semantic segmentation model to obtain a third prediction probability, and to input the second modal image into the image semantic segmentation model to obtain a fourth prediction probability. The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modal image belongs to the target organization.

[0532] The calculation module 630 is used to calculate the classification error based on the third prediction probability, the fourth prediction probability and the classification label, wherein the classification error is used to represent the difference between the third prediction probability and the fourth prediction probability and the classification label, respectively;

[0533] The training module 640 is used to perform backpropagation training on the image semantic segmentation model based on the classification error, so as to obtain the trained image semantic segmentation model.

[0534] In an optional design of this embodiment, the image semantic segmentation model includes an anatomical encoder and a classifier, wherein the anatomical encoder is used to encode the anatomical structure of the image;

[0535] The input module 620 is further configured to: input the fifth translated image and the second modality image into the anatomical encoder respectively to obtain the seventh feature representation of the fifth translated image and the eighth feature representation of the second modality image respectively; input the seventh feature representation and the eighth feature representation into the classifier respectively to obtain the third prediction probability and the fourth prediction probability respectively.

[0536] In an optional design of this embodiment, the classification error includes a second cross-entropy loss function and / or a second Dessian loss function; wherein the second cross-entropy loss function includes a cross-entropy loss function between the third predicted probability and the classification label, and a cross-entropy loss function between the fourth predicted probability and the classification label; the second Dessian loss function includes a Dessian loss function between the third predicted probability and the classification label, and a Dessian loss function between the fourth predicted probability and the classification label. In an optional design of this embodiment, the input module 620 is further configured to: rotate the first modality image, and input the rotated first modality image into the image semantic segmentation model to obtain a fifth predicted probability, wherein the fifth predicted probability is the probability that the image content of the rotated first modality image belongs to the target organization;

[0537] The calculation module 630 is further configured to: calculate the weighted entropy error based on the fifth prediction probability;

[0538] The training module 640 is further configured to: perform backpropagation training on the image semantic segmentation model based on the classification error and the weighted entropy error, so as to obtain the trained image semantic segmentation model.

[0539] In an optional design of this embodiment, the input module 620 is further configured to: rotate the first modal image and input the rotated first modal image into the image semantic segmentation model to obtain a fifth prediction probability, wherein the fifth prediction probability is the probability that the image content of the rotated first modal image belongs to the target organization; input the fourth prediction probability and the fifth prediction probability into a discriminator respectively to obtain a first discrimination result of the fourth prediction probability and a second discrimination result of the fifth prediction probability, wherein the discriminator is used to distinguish whether the image corresponding to the prediction probability originates from the first modality or the second modality;

[0540] The calculation module 630 is further configured to: calculate the adversarial consistency error based on the first discrimination result and the second discrimination result;

[0541] The training module 640 is further configured to: perform backpropagation training on the image semantic segmentation model based on the classification error and the adversarial consistency error, so as to obtain the trained image semantic segmentation model.

[0542] In an optional design of this embodiment, the adversarial consistency error is the adversarial loss function between the first discrimination result and the second discrimination result.

[0543] In an optional design of this embodiment, the input module 620 is further configured to: rotate the first modal image and input the rotated first modal image into the image semantic segmentation model to obtain a fifth prediction probability, wherein the fifth prediction probability is the probability that the image content of the rotated first modal image belongs to the target organization;

[0544] The first modality image is input into the teacher image semantic segmentation model, and the predicted probability of the first modality image output by the teacher image semantic segmentation model is rotated to obtain the sixth predicted probability. The parameters of the teacher image semantic segmentation model are determined based on the parameters of the image semantic segmentation model.

[0545] The determination module 650 is used to: determine the predicted classification label based on the sixth prediction probability and through a confidence threshold;

[0546] The calculation module 630 is further configured to: calculate the rotation consistency error based on the predicted classification label and the fifth predicted probability;

[0547] The training module 640 is further configured to: perform backpropagation training on the image semantic segmentation model based on the classification error and the rotation consistency error, so as to obtain the trained image semantic segmentation model.

[0548] In an optional design of this embodiment, the rotational consistency error includes the cross-entropy loss function between the fifth predicted probability and the predicted classification label and / or the Dessian loss function between the fifth predicted probability and the predicted classification label.

[0549] In an optional design of this embodiment, the acquisition module 610 is further configured to: acquire the image to be segmented;

[0550] The input module 620 is further configured to: input the image to be segmented into the trained image semantic segmentation model to obtain the predicted probability of the image to be segmented;

[0551] The determining module 650 is further configured to: determine the predicted classification label of the image to be segmented based on the predicted probability of the image to be segmented by using a prediction confidence threshold.

[0552] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0553] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0554] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the model training method provided in the above-described method embodiments.

[0555] Alternatively, the computer device is a server. For example, Figure 28 This is a structural block diagram of a server provided in an exemplary embodiment of this application. Typically, the server 2300 includes a processor 2301 and a memory 2302.

[0556] Processor 2301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 2301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake-up state, also known as a Central Processing Unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, processor 2301 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 2301 may also include an Artificial Intelligence (AI) processor for handling computational operations related to machine learning. Memory 2302 may include one or more computer-readable storage media, which may be non-transitory. The memory 2302 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 2302 is used to store at least one instruction, which is executed by the processor 2301 to implement the video data tracking method provided in the method embodiments of this application. In some embodiments, the server 2300 may also optionally include an input interface 2303 and an output interface 2304. The processor 2301, the memory 2302, and the input interface 2303 and output interface 2304 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interface 2303 and output interface 2304 via a bus, signal lines, or circuit boards. The input interface 2303 and output interface 2304 can be used to connect at least one input / output (I / O) related peripheral device to the processor 2301 and the memory 2302. In some embodiments, the processor 2301, memory 2302, and input interface 2303 and output interface 2304 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2301, memory 2302, and input interface 2303 and output interface 2304 can be implemented on separate chips or circuit boards, and this application does not limit this. Those skilled in the art will understand that... Figure 28The structure shown does not constitute a limitation on server 2300 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0557] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is run on a computer device, are used to implement the model training method described above.

[0558] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model training methods provided in the above-described method embodiments.

[0559] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code that, when loaded and executed by a processor of a computer device, implements the model training method provided in the above-described method embodiments.

[0560] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0561] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0562] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for an image translation model, characterized in that, The method includes: Obtain the classification labels of the first modality image, the second modality image, and the second modality image; The first modal image and the second modal image are input into the image translation model to obtain a first translated image, a second translated image, a third translated image, and a fourth translated image. The first translated image is generated by the image translation model when predicting the first modal image based on the second modal image, and the first translated image has the same modal information as the first modal image. The second translated image is generated by the image translation model when predicting the second modal image based on the first modal image. The third translated image is generated by the image translation model when predicting the first modal image based on the first modal image. The fourth translated image is generated by the image translation model when predicting the second modal image based on the second modal image. The first translated image is input into the classification network to obtain a first predicted probability, and the second modal image is input into the classification network to obtain a second predicted probability. The first predicted probability is the probability that the image content of the first translated image belongs to the target organization, and the second predicted probability is the probability that the image content of the second modal image belongs to the target organization. Based on the first predicted probability, the second predicted probability, and the classification label, a structural consistency error is calculated, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively. The first translated image and the first modal image are respectively input into a first modality discriminator to obtain a first discrimination result. The first modality discriminator is used to discriminate whether the input image belongs to the first modality. The first discrimination result includes the first modality discrimination result of the first translated image and the first modality discrimination result of the first modality image. Based on the first discrimination result, the first modality discrimination error is calculated. The second translated image and the second modal image are respectively input into the second modality discriminator to obtain a second discrimination result. The second modality discriminator is used to discriminate whether the input image belongs to the second modality. The second discrimination result includes the second modality discrimination result of the second translated image and the second modality discrimination result of the second modality image. Based on the second discrimination result, the second modality discrimination error is calculated. Based on the third translated image and the first modal image, as well as the fourth translated image and the second modal image, the reconstruction loss error is calculated; Based on the structural consistency error, the first modality discrimination error, the second modality discrimination error, and the reconstruction loss error, the image translation model is trained by backpropagation to obtain the trained image translation model.

2. The method according to claim 1, characterized in that, The image translation model includes an anatomical encoder, a first modal encoder, and a generator. The anatomical encoder is used to encode the anatomical structure of the image, and the first modal encoder is used to encode modal information in the first modal domain of the image. The step of inputting the first modality image and the second modality image into the image translation model to obtain the first translated image includes: The second modality image is input into the anatomical encoder to obtain a first feature representation, which is a feature representation obtained by anatomical encoding of the second modality image; The first modal image is input into the first modal encoder to obtain a second feature representation, which is a feature representation obtained by modal encoding of the first modal image; The first feature representation and the second feature representation are input into the generator to obtain the first translated image.

3. The method according to claim 1, characterized in that, The classification network includes an anatomical encoder and a classifier; The step of inputting the first translated image into the classification network to obtain a first predicted probability, and inputting the second modality image into the classification network to obtain a second predicted probability, includes: The first translated image and the second modal image are respectively input into the anatomical encoder to obtain the fifth feature representation of the first translated image and the sixth feature representation of the second modal image. The fifth feature representation and the sixth feature representation are respectively input into the classifier to obtain the first prediction probability and the second prediction probability, respectively.

4. The method according to any one of claims 1 to 3, characterized in that, The structural consistency error includes a first cross-entropy loss function and / or a first Dessian loss function; wherein, the first cross-entropy loss function includes a cross-entropy loss function between the first predicted probability and the classification label, and a cross-entropy loss function between the second predicted probability and the classification label; the first Dessian loss function includes a Dessian loss function between the first predicted probability and the classification label, and a Dessian loss function between the second predicted probability and the classification label.

5. A training method for an image semantic segmentation model, characterized in that, The method includes: Obtain the classification labels of the first modality image, the second modality image, and the second modality image; The first modality image and the second modality image are input into the trained image translation model to obtain a fifth translated image. The fifth translated image is an image generated by the trained image translation model when predicting the first modality image based on the second modality image. The trained image translation model is obtained by the training method of the image translation model according to any one of claims 1-4. The fifth translated image is input into the image semantic segmentation model to obtain a third prediction probability, and the second modality image is input into the image semantic segmentation model to obtain a fourth prediction probability. The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modality image belongs to the target organization. Based on the third predicted probability, the fourth predicted probability, and the classification label, a classification error is calculated, wherein the classification error is used to represent the difference between the third predicted probability and the fourth predicted probability and the classification label, respectively; Based on the classification error, the image semantic segmentation model is trained by backpropagation to obtain the trained image semantic segmentation model.

6. The method according to claim 5, characterized in that, The image semantic segmentation model includes an anatomical encoder and a classifier, wherein the anatomical encoder is used to encode the anatomical structure of the image; The step of inputting the fifth translated image into the image semantic segmentation model to obtain a third prediction probability, and inputting the second modality image into the image semantic segmentation model to obtain a fourth prediction probability, includes: The fifth translated image and the second modal image are respectively input into the anatomical encoder to obtain the seventh feature representation of the fifth translated image and the eighth feature representation of the second modal image, respectively; The seventh feature representation and the eighth feature representation are respectively input into the classifier to obtain the third prediction probability and the fourth prediction probability.

7. The method according to claim 5, characterized in that, The classification error includes a second cross-entropy loss function and / or a second Dessian loss function; wherein the second cross-entropy loss function includes a cross-entropy loss function between the third predicted probability and the classification label, and a cross-entropy loss function between the fourth predicted probability and the classification label; the second Dessian loss function includes a Dessian loss function between the third predicted probability and the classification label, and a Dessian loss function between the fourth predicted probability and the classification label.

8. The method according to claim 5, characterized in that, The method further includes: The first modal image is rotated, and the rotated first modal image is input into the image semantic segmentation model to obtain a fifth prediction probability, which is the probability that the image content of the rotated first modal image belongs to the target organization; Based on the fifth predicted probability, calculate the weighted entropy error; The step of training the image semantic segmentation model through backpropagation based on the classification error to obtain the trained image semantic segmentation model includes: Based on the classification error and the weighted entropy error, the image semantic segmentation model is trained by backpropagation to obtain the trained image semantic segmentation model.

9. The method according to claim 5, characterized in that, The method further includes: The first modal image is rotated, and the rotated first modal image is input into the image semantic segmentation model to obtain a fifth prediction probability, which is the probability that the image content of the rotated first modal image belongs to the target organization; The fourth prediction probability and the fifth prediction probability are respectively input into the discriminator to obtain the first discrimination result of the fourth prediction probability and the second discrimination result of the fifth prediction probability. The discriminator is used to distinguish whether the image corresponding to the prediction probability comes from the first mode or the second mode. Based on the first discrimination result and the second discrimination result, calculate the adversarial consistency error; The step of training the image semantic segmentation model through backpropagation based on the classification error to obtain the trained image semantic segmentation model includes: Based on the classification error and the adversarial consistency error, the image semantic segmentation model is trained by backpropagation to obtain the trained image semantic segmentation model.

10. The method according to claim 5, characterized in that, The method further includes: The first modal image is rotated, and the rotated first modal image is input into the image semantic segmentation model to obtain a fifth prediction probability, which is the probability that the image content of the rotated first modal image belongs to the target organization; The first modality image is input into the teacher image semantic segmentation model, and the predicted probability of the first modality image output by the teacher image semantic segmentation model is rotated to obtain the sixth predicted probability. The parameters of the teacher image semantic segmentation model are determined based on the parameters of the image semantic segmentation model. Based on the sixth prediction probability, the predicted classification label is determined through a confidence threshold. Based on the predicted classification label and the fifth predicted probability, the rotational consistency error is calculated; The step of training the image semantic segmentation model through backpropagation based on the classification error to obtain the trained image semantic segmentation model includes: Based on the classification error and the rotation consistency error, the image semantic segmentation model is trained by backpropagation to obtain the trained image semantic segmentation model.

11. The method according to any one of claims 5 to 10, characterized in that, The method further includes: Obtain the image to be segmented; The image to be segmented is input into the trained image semantic segmentation model to obtain the predicted probability of the image to be segmented; Based on the predicted probability of the image to be segmented, the predicted classification label of the image to be segmented is determined by using a prediction confidence threshold.

12. A training device for an image translation model, characterized in that, The device includes: The acquisition module is used to acquire the first modality image, the second modality image, and the classification label of the second modality image; An input module is used to input the first modal image and the second modal image into the image translation model to obtain a first translated image, a second translated image, a third translated image, and a fourth translated image. The first translated image is an image generated by the image translation model when predicting the first modal image based on the second modal image, and the first translated image has the same modal information as the first modal image. The second translated image is an image generated by the image translation model when predicting the second modal image based on the first modal image. The third translated image is an image generated by the image translation model when predicting the first modal image based on the first modal image. The fourth translated image is an image generated by the image translation model when predicting the second modal image based on the second modal image. The input module is further configured to input the first translated image into the classification network to obtain a first prediction probability, and input the second modal image into the classification network to obtain a second prediction probability, wherein the first prediction probability is the probability that the image content of the first translated image belongs to the target organization, and the second prediction probability is the probability that the image content of the second modal image belongs to the target organization; The calculation module is used to calculate the structural consistency error based on the first predicted probability, the second predicted probability, and the classification label, wherein the structural consistency error is used to represent the difference between the first predicted probability and the second predicted probability and the classification label, respectively. The input module is further configured to input the first translated image and the first modal image into the first modality discriminator respectively to obtain a first discrimination result. The first modality discriminator is used to discriminate whether the input image belongs to the first modality. The first discrimination result includes the first modality discrimination result of the first translated image and the first modality discrimination result of the first modality image. The calculation module is further configured to calculate the first modality discrimination error based on the first discrimination result. The input module is further configured to input the second translated image and the second modal image into the second modality discriminator respectively to obtain a second discrimination result. The second modality discriminator is used to discriminate whether the input image belongs to the second modality. The second discrimination result includes the second modality discrimination result of the second translated image and the second modality discrimination result of the second modality image. The calculation module is further configured to calculate the second modality discrimination error based on the second discrimination result. The calculation module is also used to calculate the reconstruction loss error based on the third translated image and the first modal image, as well as the fourth translated image and the second modal image; The training module is used to perform backpropagation training on the image translation model based on the structural consistency error, the first modality discrimination error, the second modality discrimination error, and the reconstruction loss error, so as to obtain the trained image translation model.

13. A training device for an image semantic segmentation model, characterized in that, The device includes: The acquisition module is used to acquire the first modality image, the second modality image, and the classification label of the second modality image; An input module is used to input the first modality image and the second modality image into a trained image translation model to obtain a fifth translated image. The fifth translated image is an image generated by the trained image translation model when predicting the first modality image based on the second modality image. The trained image translation model is obtained by the training method of the image translation model according to any one of claims 1-4. The input module is further configured to input the fifth translated image into the image semantic segmentation model to obtain a third prediction probability, and to input the second modal image into the image semantic segmentation model to obtain a fourth prediction probability. The third prediction probability is the probability that the image content of the fifth translated image belongs to the target organization, and the fourth prediction probability is the probability that the image content of the second modal image belongs to the target organization. The calculation module is used to calculate the classification error based on the third prediction probability, the fourth prediction probability, and the classification label, wherein the classification error is used to represent the difference between the third prediction probability and the fourth prediction probability and the classification label, respectively; The training module is used to perform backpropagation training on the image semantic segmentation model based on the classification error, so as to obtain the trained image semantic segmentation model.

14. A computer device, characterized in that, The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is configured to execute the at least one program in the memory to implement the training method of the image translation model as described in any one of claims 1 to 4, or the training method of the image semantic segmentation model as described in any one of claims 5 to 11.

15. A computer-readable storage medium, characterized in that, The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the training method of the image translation model as described in any one of claims 1 to 4, or the training method of the image semantic segmentation model as described in any one of claims 5 to 11.

16. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the training method of the image translation model as described in any one of claims 1 to 4, or the training method of the image semantic segmentation model as described in any one of claims 5 to 11.

Images