Image segmentation method and apparatus, and radiotherapy plan generation method and system

Abstract

An image segmentation method and apparatus, and a radiotherapy plan generation method and system. The image segmentation method comprises: acquiring an image to be processed of a target object; inputting into a pre-trained image segmentation model the image to be processed, and outputting a segmented image, the segmented image comprising a target region and the category of the target region, wherein the image segmentation model is obtained by using a training sample set comprising first training images carrying labels and second training images not carrying a label to pre-train a neural network model comprising an encoder, a first decoder and a second decoder; and the pre-training comprises: the first decoder performing parameter updating by means of the first training images carrying the labels and labels generated by the second decoder on the basis of the second training images not carrying a label, and the second decoder performing parameter updating by means of the first training images carrying the labels. By means of the method, the efficiency and accuracy of image segmentation can be improved.

Description

Image segmentation method and device, radiotherapy plan generation method and system TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image segmentation method and device, a radiotherapy plan generation method and system. BACKGROUND

[0002] Radiotherapy is a common cancer treatment method. When a cancer patient is treated by radiotherapy, a doctor needs to delineate a tumor and its surrounding tissue to evaluate the safety of radiotherapy. This process usually first requires the doctor to manually delineate each medical image, segment the target region of the tissue organ, and label the category, and then develop a treatment plan accordingly.

[0003] Medical images such as CT (Computed Tomography), PET (Positron Emission Tomography), PET-CT, MRI (Magnetic Resonance Imaging), NMRI (Nuclear Magnetic Resonance Imaging), etc. have the concept of window width and window level. For example, a CT image can identify 2000 different gray scale density differences in the human body, but the human eye can only distinguish 16 gray scale degrees. Therefore, the CT value that the human eye can distinguish on the CT image is 125HU (2000 / 16), that is, the CT values of different tissues in the human body only differ by more than 125HU to be recognized by the human eye. The CT value of human soft tissue changes between 20-50HU, which is difficult for the human eye to recognize. Therefore, the method of windowing is usually used for segmented observation. The CT value range observed is called the window width, and the center CT value observed is the window level or window center. The width of the window width directly affects the clarity and contrast of the image. If a narrow window width is used, the displayed CT value range is small, the contrast is strong, and it is suitable for observing tissue structures with close densities (such as brain tissue); on the contrary, if a wide window width is used, the displayed CT value range is large, and the image contrast is poor, but the density is uniform, and it is suitable for observing structures with large density differences (such as bone and soft tissue). In principle, the window width should reflect the CT value range of the tissue or lesion, and the window level should be equal to or close to the CT value to be observed. When a doctor develops a treatment plan, some tissues that are not easy to identify need to adjust the window width and window level of the CT respectively, and then delineate them one by one, which is time-consuming and laborious. At present, the segmentation of medical images depends on medical personnel, and the image segmentation efficiency is low and the segmentation accuracy is also low. SUMMARY

[0004] The purpose of the embodiments of the present specification is to provide an image segmentation method and device, a radiotherapy plan generation method and system to solve the problems of low efficiency and accuracy of medical image segmentation.

[0005] To solve the above technical problems, the first aspect of the present specification provides an image segmentation method, comprising:

[0006] Obtaining a to-be-processed image of a target object;

[0007] Inputting the to-be-processed image into a pre-trained image segmentation model to output a segmentation image, the segmentation image comprising a target region and a category of the target region; wherein the image segmentation model is pre-trained by using a training sample set comprising a first training image carrying a label and a second training image not carrying a label, and a neural network model comprising an encoder, a first decoder and a second decoder; the pre-training comprises: the first decoder performs parameter updating by using the first training image carrying the label and a label generated by the second decoder based on the second training image not carrying the label, and the second decoder performs parameter updating by using the first training image carrying the label.

[0008] In some embodiments of the present specification, the pre-training comprises:

[0009] Parameter updating the first decoder based on the label carried by the first training image, a first output and a third output, and / or: parameter updating the first decoder based on a second output and a fourth output;

[0010] Parameter updating the second decoder based on the third output and the label carried by the first training image;

[0011] The first output is an output of the first decoder corresponding to the first training image;

[0012] The second output is an output of the first decoder corresponding to the second training image;

[0013] The third output is an output of the second decoder corresponding to the first training image;

[0014] The fourth output is an output of the second decoder corresponding to the second training image.

[0015] In some embodiments of the present specification, the pre-training comprises:

[0016] In response to the second training image: obtaining an anchor sample, adjusting the anchor sample based on a random display parameter to obtain a first random sample;

[0017] The encoder processes the anchor sample and the first random sample to obtain first anchor information and first random information, respectively;

[0018] The first decoder processes the first anchor information and the first random information to obtain a first anchor label and a first random label, respectively; and the second decoder processes the first anchor information to obtain a second anchor label.

[0019] Based on the first anchor label, the first random label and the second anchor label, a first loss of the first decoder is determined.

[0020] The first decoder is updated in parameters based on the first loss.

[0021] In some embodiments of the present specification, the first loss is determined by the following formula:

[0022]

[0023] wherein Loss a represents the first loss, and σ and υ represent weight parameters, respectively, and L o represents a loss between the first anchor label and the second anchor label, L p represents a loss between the first anchor label and the first random label, L n represents a loss between the first anchor label and a first negative sample label, which is obtained based on an output of the neural network model corresponding to other training images.

[0024] In some embodiments of the present specification, the anchor sample is obtained by:

[0025] At least one image enhancement processing is performed on the current training image, and at least one image obtained by processing is taken as an anchor sample.

[0026] In some embodiments of the present specification, the pre-training comprises:

[0027] In response to the first training image, an anchor sample is obtained, the anchor sample is adjusted based on a reference display parameter corresponding to a label carried by the anchor sample to obtain a first reference sample;

[0028] The encoder processes the first reference sample to obtain first reference information;

[0029] The second decoder processes the first reference information to obtain a first reference label;

[0030] Based on the label carried by the anchor sample and the first reference label, a second loss of the second decoder is determined.

[0031] perform parameter updating on the second decoder based on the second loss.

[0032] In some embodiments of the present specification, the pre-training comprises:

[0033] In response to the first training image: obtaining an anchor sample, adjusting the anchor sample based on a random display parameter to obtain a second random sample;

[0034] The encoder processes the anchor sample and the second random sample to obtain third anchor information and second random information, respectively;

[0035] The first decoder processes the third anchor information to obtain a third anchor label;

[0036] The first decoder or the second decoder processes the second random information to obtain a second random label;

[0037] Based on the label carried by the anchor sample, the third anchor label, the second random label and the first reference label, a third loss of the first decoder is determined;

[0038] Perform parameter updating on the first decoder based on the third loss.

[0039] In some embodiments of the present specification, the neural network model further comprises a third decoder, and the third decoder is updated based on the parameters of the first decoder;

[0040] The output of the third decoder corresponding to the negative sample is a second negative sample label, and the first decoder is further updated based on the second negative sample label.

[0041] In some embodiments of the present specification, the pre-training comprises:

[0042] In response to the first training image: obtaining an anchor sample, adjusting the anchor sample based on a random display parameter to obtain a third random sample, and adjusting the anchor sample based on a second reference display parameter corresponding to the label carried by the anchor sample to obtain a second reference sample;

[0043] The encoder processes the anchor sample, the third random sample and the second reference sample to obtain fourth anchor information, third random information and second reference information, respectively;

[0044] The first decoder processes the fourth anchor information to obtain a fourth anchor label;

[0045] The second decoder processes the second reference information to obtain a second reference label;

[0046] The first decoder, the second decoder or the third decoder processes the third random information to obtain a third random label;

[0047] determine a fourth loss of the first decoder based on the label carried by the anchor sample, the fourth anchor label, the third random label, the second reference label and the second negative sample label;

[0048] perform parameter update on the first decoder based on the fourth loss.

[0049] In some embodiments of the present specification, the fourth loss is determined by the following formula: Loss a = aL p + b(8L ce + eL fl + hL cc + L n );

[0050] wherein Loss a represents the fourth loss, a, b, 8, e and h respectively represent weight parameters; L ce represents the cross entropy of the fourth anchor label and the label carried by the anchor sample; L fl represents the focal loss of the fourth anchor label and the label carried by the anchor sample; L cc represents the loss between the fourth anchor label and the second reference label; L p represents the loss between the fourth anchor label and the third random label; L n represents the loss between the fourth anchor label and the second negative sample label.

[0051] In some embodiments of the present specification, the third decoder is updated by the following formula: 8 k = m8 k + (1-m)8 a ;

[0052] wherein 8 k represents the parameter of the third decoder, 8 a represents the parameter of the first decoder, and m represents an association parameter.

[0053] In some embodiments of the present specification, the first loss of the first decoder is determined based on the first anchor label, the first random label and the second anchor label, comprising:

[0054] fuse the first anchor label with the first anchor information to obtain a fused first anchor label;

[0055] determine the first loss of the first decoder based on the fused first anchor label, the first random label and the second anchor label.

[0056] In some embodiments of the present specification, the segmented image includes a plurality of target regions and categories of the plurality of target regions.

[0057] The label carried by the first training image includes a specific region in the first training image and a category of the specific region.

[0058] The category of all specific regions contained in the training sample set corresponds to at least the category of the plurality of target regions.

[0059] The second aspect of the present specification provides an image segmentation device, comprising:

[0060] An image acquisition module is configured to acquire a to-be-processed image of a target object.

[0061] An image segmentation module is configured to input the to-be-processed image into a pre-trained image segmentation model, and output a segmented image, wherein the segmented image includes a target region and a category of the target region; wherein the image segmentation model is obtained by pre-training a neural network model including an encoder, a first decoder and a second decoder using a training sample set including a first training image carrying a label and a second training image not carrying a label; the pre-training includes: the first decoder updates parameters through the first training image carrying the label, and the second decoder updates parameters through the label generated based on the second training image not carrying the label.

[0062] The third aspect of the present specification provides a radiotherapy plan generation method, comprising:

[0063] Acquiring a to-be-processed image of a target object.

[0064] Inputting the to-be-processed image into a pre-trained image segmentation model, and outputting a segmented image, wherein the segmented image includes a target region and a category of the target region.

[0065] Determining material information of the target region based on the category of the target region, and establishing a three-dimensional voxel model based on the target region and the material information of the target region.

[0066] Performing tissue dose simulation calculation based on the three-dimensional voxel model, and generating a treatment plan.

[0067] The fourth aspect of the present specification provides a radiotherapy system, comprising:

[0068] A beam irradiation device is configured to generate a radiotherapy beam.

[0069] A treatment plan module is configured to generate a treatment plan.

[0070] a control module configured to control the beam irradiation device to generate a radiation beam according to the treatment plan;

[0071] The treatment planning module comprises:

[0072] an image acquisition unit configured to acquire a to-be-processed image of a target object;

[0073] an image segmentation unit configured to input the to-be-processed image into a pre-trained image segmentation model, and output a segmented image, wherein the segmented image comprises a target region and a category of the target region;

[0074] a model establishing unit configured to determine material information of the target region based on the category of the target region, and establish a three-dimensional voxel model based on the target region and the material information of the target region;

[0075] a calculation unit configured to perform tissue dose simulation calculation based on the three-dimensional voxel model, and generate a treatment plan.

[0076] The fifth aspect of the present specification provides an electronic device, comprising a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the foregoing method.

[0077] The sixth aspect of the present specification provides a computer-readable storage medium, which stores a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the foregoing method.

[0078] The image segmentation method provided by the embodiment of the present specification comprises the following steps: obtaining a to-be-processed image of a target object; inputting the to-be-processed image into a pre-trained image segmentation model to output a segmented image, wherein the segmented image comprises a target region and a category of the target region; wherein the image segmentation model is obtained by pre-training a neural network model comprising an encoder, a first decoder and a second decoder using a training sample set comprising a first training image carrying a label and a second training image not carrying a label; the pre-training comprises: the first decoder performs parameter updating based on the first training image carrying the label and a label generated by the second decoder based on the second training image not carrying the label, and the second decoder performs parameter updating based on the first training image carrying the label. By the above method, weakly supervised training of the neural network model can be realized only by a small amount of images with labels; the unlabelled training of the first decoder can be guided by the pseudo label output by the second decoder, the dependence on training samples is low, and the model adaptability is high. Furthermore, the image segmentation model is obtained by training the neural network model using the training sample set comprising the first training image carrying the label and the second training image not carrying the label, and the model can automatically and accurately segment the image without adjusting the window width and window level of CT, performing individual partitioning, marking, saving a large amount of time and labor cost, and improving the efficiency and accuracy of image segmentation. On this basis, the radiotherapy plan is generated more efficiently, accurately and reasonably, the pressure on medical staff is reduced, and the effect of subsequent treatment is improved. BRIEF DESCRIPTION OF DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0080] Fig. 1 is a schematic diagram of the image segmentation method provided by the embodiment of the present specification;

[0081] Fig. 2 is a schematic diagram in response to the first training image provided by the embodiment of the present specification;

[0082] Fig. 3 is a schematic diagram in response to the second training image provided by the embodiment of the present specification;

[0083] Fig. 4 is an architectural schematic diagram provided by the embodiment of the present specification;

[0084] Fig. 5 is a schematic diagram of the image segmentation device provided by the embodiment of the present specification;

[0085] Fig. 6 is a schematic diagram of the radiotherapy plan generation method provided by the embodiment of the present specification;

[0086] FIG. 7 is a schematic diagram of a radiotherapy system according to an embodiment of the present specification;

[0087] FIG. 8 is a schematic diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0088] In order to enable persons skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0089] At present, the image segmentation methods mainly include traditional segmentation methods and deep learning-based methods.

[0090] The traditional segmentation methods mainly include threshold method, region growing method, edge detection method, clustering method, model matching method, etc. However, such segmentation methods often need a large amount of manual intervention, and cannot meet the demand of people on the calculation precision and efficiency of medical image segmentation. With the rapid development of deep learning technology, especially the wide application of Convolutional Neural Networks (CNN) in the field of image processing, the medical image segmentation technology has also been improved. For example, the U-Net deep learning method introduces skip connection and deconvolution, which can preserve image details while realizing efficient pixel-level classification. Such semantic-based image segmentation is a pixel-level classification task, which can detect biological structures and quantify morphological features, and can also quantitatively capture the shape of the object, providing a means for high-resolution spatial statistics. Fully Convolutional Networks (FCN) is a completely automatic segmentation technology, which does not need field experts to select Region of interest (ROI). The above deep learning methods involve supervised learning and need training data, such as deep neural networks, shape models and random forests, etc. However, in reality, it is often an expensive and time-consuming task to obtain large-scale labeled training data, especially in the field of medical images which is data scarce, the performance of deep neural network model is limited. If unsupervised learning method is used, its performance is largely dependent on the quality and features of the input data. If the data has noise, missing values or outliers, it will have a negative impact on the performance of the model. The method based on FCN needs to face the problems of image size, texture, shape and ROI selection in low-resolution images during training and use.

[0091] It can be seen that the current medical image segmentation method relies on medical personnel to draw a large number of target regions in detail according to resources. In order to ensure the accuracy of the above method, the determined target region needs to be accurate to the pixel level. The above method has relatively high requirements for the data set, and often requires interaction between medical personnel and the system during the process, such as selecting a preliminary range of ROI and then performing detailed segmentation, which is time-consuming and tedious. In addition, since it is for medical image segmentation, the image will change after changing the window width and window level during actual use, and the selection of the window width and window level will affect the segmentation effect of the image.

[0092] The image segmentation model in the embodiment of the present specification is obtained by pre-training a neural network model comprising an encoder, a first decoder and a second decoder using a training sample set comprising first training images carrying labels and second training images not carrying labels; the pre-training comprises: the first decoder updates parameters through the first training images carrying labels and the second decoder updates parameters through the labels generated based on the second training images not carrying labels. Further, after obtaining a to-be-processed image of a target object, the to-be-processed image can be input into the pre-trained image segmentation model to output a segmented image, which can include a target region and a category of the target region.

[0093] The image segmentation method provided in the embodiment of the present specification only needs a small amount of images with labels to realize weakly supervised training of the neural network model; the unlabelled training of the first decoder can be guided by the pseudo labels output by the second decoder, the dependence on training samples is low, and the model has high adaptability. Moreover, the neural network model is trained using a training sample set comprising first training images carrying labels and second training images not carrying labels to obtain an image segmentation model, using the model, without adjusting the window width and window level of the CT, performing individual partitioning, and labeling, the image can be accurately segmented automatically, a large amount of time and labor cost is saved, and the efficiency and accuracy of image segmentation can be improved. On this basis, generating a radiotherapy plan is also more efficient, accurate and reasonable, which can reduce the pressure on medical personnel and improve the effect of subsequent treatment.

[0094] The execution subject of each step of the image segmentation method and the radiotherapy plan generation method provided in the embodiments of the present application can be an electronic device, which refers to an electronic device with data calculation, processing and storage capabilities. The electronic device can be a terminal such as a personal computer (PC), a tablet computer, a smart phone, a wearable device, a smart robot, etc.; or a server. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0095] First, the image segmentation method provided in the embodiments of the present application will be introduced in combination with the accompanying drawings.

[0096] FIG. 1 shows a schematic diagram of the image segmentation method provided in the embodiments of the present application. Although the present specification provides the method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units can be included in the method or device based on conventional or non-creative labor. The execution order of the steps or the module structure of the device is not limited to the execution order or the module structure shown in the embodiments or drawings of the present specification in the absence of necessary causal relationship in logic. When the method or module structure is applied to actual devices, servers or terminal products, it can be sequentially executed or executed in parallel (for example, parallel processors or multi-threaded processing environments, even including distributed processing, server cluster implementation environments) according to the method or module structure shown in the embodiments or drawings. As shown in FIG. 1, the method can include:

[0097] S101: obtaining a to-be-processed image of a target object.

[0098] It can be understood that the to-be-processed image can be a medical image to be segmented, such as a CT image, a PET image, a PET-CT image, an MRI image, an NMRI image, etc.; the to-be-processed image can also be an image to be segmented in other fields, which is not limited in the present specification.

[0099] S102: inputting the to-be-processed image into a pre-trained image segmentation model to output a segmented image, wherein the segmented image includes a target region and a category of the target region.

[0100] The image segmentation model is obtained by pre-training a neural network model including an encoder, a first decoder and a second decoder using a training sample set including first training images carrying labels and second training images not carrying labels; the pre-training includes: the first decoder performing parameter updating by the first training images carrying labels, and the second decoder performing parameter updating by the labels generated based on the second training images not carrying labels.

[0101] It can be understood that the image segmentation model can be used to perform image segmentation on the image to be processed to obtain a segmentation image including a target region and a category of the target region. In some embodiments, the segmentation image output by the image segmentation model can be the output of the first decoder, that is, when training the image segmentation model, the first decoder and the second decoder can be trained using the training sample set, and when the model is actually applied to perform image segmentation, the first decoder can be used, and the second decoder plays a role of assisting the training of the first decoder.

[0102] The segmentation image output by the image segmentation model can include a target region and a category of the target region, and can also include multiple target regions and categories of the multiple target regions. In some embodiments of the present specification, the segmentation image can include a target region and a category of the target region, so that the user can quickly determine the target region in the image to be processed based on the segmented image; in some embodiments of the present specification, the segmentation image can include multiple target regions and categories of the multiple target regions, so that the user can quickly determine each target region in the image to be processed and the category of each target region, thereby facilitating the user to further analyze each target region in the image to be processed or quickly determine a specific region required by the user based on the user's demand.

[0103] Specifically, the label carried by the first training image in the training sample set can be a specific region in the image and a corresponding category. In some embodiments, the first training image can include multiple specific regions, and the label carried by the first training image can be multiple. The image segmentation model segments the to-be-processed image, that is, determines a target region in the to-be-processed image that matches the specific region in the first training image and the second training image, and determines the category corresponding to the target region based on the category of the matched specific region. In some embodiments of the present specification, the categories of all specific regions contained in the training sample set correspond to at least the categories of the multiple target regions. It can be understood that the categories of all specific regions can be more than or equal to the categories of the target regions of the to-be-processed image. The implementation can include only one category of a specific region in a single first training image, the categories of the specific regions contained in multiple first training images are not all the same, and the categories of the specific regions obtained by aggregation correspond to at least the categories of the multiple target regions; or the implementation can include multiple categories of specific regions in a single first training image, the categories of the specific regions contained in multiple first training images can be different or the same, and the categories of the specific regions obtained by aggregation correspond to at least the categories of the multiple target regions.

[0104] In some embodiments of the present specification, a corresponding image segmentation sub-model can be trained for each specific region, and different sub-models can simultaneously or sequentially perform image segmentation processing on the to-be-processed image, and then the output results of multiple sub-models can be fused; or an image segmentation model for identifying and segmenting multiple specific regions can be trained, which can segment all categories at one time, has higher segmentation efficiency, and has more extensive application scenarios.

[0105] In some embodiments of the present specification, the pre-training can include:

[0106] The first decoder is updated in parameters based on the label carried by the first training image, the first output, and the third output, and / or: the first decoder is updated in parameters based on the second output and the fourth output; the second decoder is updated in parameters based on the third output and the label carried by the first training image; wherein the first output is the output of the first decoder corresponding to the first training image; the second output is the output of the first decoder corresponding to the second training image; the third output is the output of the second decoder corresponding to the first training image; and the fourth output is the output of the second decoder corresponding to the second training image.

[0107] It can be understood that the output of the decoder can include the pseudo label output by the decoder corresponding to the training image and / or the pseudo label output by the decoder corresponding to the transformed training image. For example, for the first training image, during actual training, the first training image can be transformed, such as data augmentation, and can also be adjusted based on random display parameters to obtain a data-augmented sample or an adjusted sample, both of which are positive samples. These positive samples can be input into the first decoder together with the corresponding original first training image, and both obtain corresponding outputs, i.e., pseudo labels. Therefore, the first decoder corresponding to the first training image can refer to one pseudo label corresponding to the first training image, and can also be understood to further include multiple pseudo labels corresponding to other positive samples. Of course, the output of the decoder can be one or more pseudo labels for other decoders or for other training images, which is not limited in the present specification. It can be understood that the fourth output can include the label (i.e., the pseudo label) generated by the second decoder based on the second training image without carrying a label, and can also include the label (i.e., the pseudo label) generated by the second decoder based on the other image transformed from the second training image without carrying a label.

[0108] It can be understood that for the first decoder and the second decoder, the updating manner of the parameters of the decoder is different when the input training image is the first training image and the second training image, respectively. Specifically, when the first training image is input, the parameters of the second decoder can be updated using the first training image, i.e., the parameters of the second decoder are updated based on the third output and the label carried by the first training image; and the parameters of the first decoder can be updated using the first training image and the pseudo label output by the second decoder corresponding to the first training image, i.e., the parameters of the first decoder are updated based on the label carried by the first training image, the first output and the third output. When the second training image is input, since the second training image does not carry a label, the second decoder does not update the parameters, the second training image can be input into the second decoder to obtain the pseudo label output by the second decoder corresponding to the second training image, i.e., the fourth output, and then the parameters of the first decoder can be updated based on the fourth output and the second output.

[0109] The process of model training will be further introduced in combination with an embodiment.

[0110] In some embodiments of the present specification, the pre-training can include:

[0111] In response to the second training image: an anchor sample is obtained, the anchor sample is adjusted based on a random display parameter to obtain a first random sample; the encoder processes the anchor sample and the first random sample to obtain first anchor information and first random information, respectively; the first decoder processes the first anchor information and the first random information to obtain a first anchor label and a first random label, respectively; the second decoder processes the first anchor information to obtain a second anchor label; the first loss of the first decoder is determined based on the first anchor label, the first random label and the second anchor label; and the first decoder is updated in parameters based on the first loss.

[0112] It can be understood that the second anchor label output by the second decoder is the fourth output, which can be used as the pseudo label of the second training image without carrying a label, and the first decoder can be updated in combination with the pseudo label during parameter updating.

[0113] It can be understood that when the input training image is the second training image, it can be used as an anchor sample, and if multiple second training images are input at a time, one of them can be used as an anchor sample for training. In some embodiments, the random display parameter can be a parameter adjusted when performing random data augmentation on the anchor sample, and the random data augmentation can include rotation, translation, noise addition, cropping, etc., and the random display parameter can include at least one of the rotation angle parameter, the translation direction and compensation, the position parameter after translation, the noise, the size parameter after cropping, etc. The present specification does not limit this. In other embodiments, the random display parameter can also be a parameter for adjusting the window width and window level of the image, which can be a pre-specified window width or window level, or a randomly generated window width or window level. The present specification does not limit this. The anchor sample can be adjusted based on the random display parameter in the above single or multiple ways to obtain a first random sample; or the anchor sample can be adjusted based on multiple random display parameters in the above single or multiple ways to obtain multiple first random samples, each of which is trained by the foregoing method.

[0114] In some embodiments, the image sample is input to the encoder, and the processing by the encoder is a feature extraction operation, which can be implemented by conventional technical means in the art, for example, by downsampling the sample to extract the image features of the sample, i.e., by downsampling the anchor sample and the first random sample to extract the image features of the anchor sample and the first random sample as the first anchor information and the first random information. Of course, in other embodiments, the encoder can also use other means to extract features, such as semantic feature extraction, and can also perform other processing on the anchor sample and the first random sample, such as data or image conversion, to obtain the first anchor information and the first random information. The present specification does not limit this.

[0115] In some embodiments of the present specification, the first loss can be determined by the following formula:

[0116] wherein Loss a represents the first loss, σ and υ represent weight parameters respectively, L o represents the loss between the first anchor point label and the second anchor point label, L p represents the loss between the first anchor point label and the first random label, L n represents the loss between the first anchor point label and the first negative sample label, which is obtained based on the output of the neural network model corresponding to other training images, L f is an intermediate quantity, which can not be defined because it can be substituted into the formula and does not appear in the formula, and its meaning is the loss between the first anchor point label and the positive and negative samples, that is, the loss between the first anchor point label and the first random label and the first negative sample label.

[0117] In some embodiments of the present specification, determining the first loss of the first decoder based on the first anchor point label, the first random label and the second anchor point label can include:

[0118] fusing the first anchor point information and the first anchor point label to obtain a fused first anchor point label; and determining the first loss of the first decoder based on the fused first anchor point label, the first random label and the second anchor point label.

[0119] It can be understood that, for the first negative sample label, the label can be for the output of the first decoder, or for the output of the second decoder, or for the label output by any decoder selected from the queue composed of the outputs of the first decoder and the second decoder. Specifically, the output of the first decoder can be stored as a queue, and when it is needed to use negative samples to participate in the decoder parameter optimization, the label can be extracted from the queue as a negative sample label; or the output of the second decoder can be stored as a queue, and when it is needed to use negative samples to participate in the decoder parameter optimization, the label can be extracted from the queue as a negative sample label; or the outputs of the first decoder and the second decoder can be stored together as a queue, and when it is needed to use negative samples to participate in the decoder parameter optimization, the label can be extracted from the queue as a negative sample label. Moreover, other decoders, such as the third decoder in the following text, can be used to process other training images, and loss calculation can be performed based on the processed negative sample labels. Here, before the negative sample labels are further processed by using other decoders, the negative sample labels can be fused with the corresponding sample features and then processed. It can be understood that, in other embodiments, whether to directly select a training image from the training sample set and use the negative sample label obtained by processing the training image by using the encoder and the decoder to perform loss calculation can be determined based on the current memory usage. That is, the negative sample label can be obtained in multiple ways, and the present specification does not limit this.

[0120] In some embodiments, when the first loss is calculated, the first negative sample can also not be set, and thus the loss between the first anchor point label and the first negative sample label also does not need to be calculated. That is, the first loss can include the loss between the weighted first anchor point label and the second anchor point label and the loss between the weighted first anchor point label and the first random label. In this case, formula (1) can be replaced by: Loss a = σL o + υL p

[0121] In some embodiments of the present specification, in response to the first training image or the second training image, when the anchor point sample is obtained, at least one image enhancement processing can be performed on the current training image, and the processed at least one image and the current training image are respectively taken as anchor point samples. The image enhancement processing can be, for example, the random rotation, translation, noise addition, cropping, etc. mentioned in the foregoing, and the present specification does not limit this.

[0122] In some embodiments of the present specification, the pre-training can comprise: in response to the first training image: obtaining an anchor point sample, adjusting the anchor point sample based on a reference display parameter corresponding to a label carried by the anchor point sample to obtain a first reference sample; an encoder processing the first reference sample to obtain first reference information; a second decoder processing the first reference information to obtain a first reference label; determining a second loss of the second decoder based on the label carried by the anchor point sample and the first reference label; and updating parameters of the second decoder based on the second loss.

[0123] It can be understood that the reference display parameter can be a display parameter corresponding to the category of a specific region contained in the training image, and can include a specified window width and window level. Taking a medical image as an example, the window width and window level recommended for different tissue categories are different, and the difference in window width and window level can affect the contrast and brightness of the tissue image, and further affect the quality of the displayed image. Based on the reference display parameter, the image can be adjusted to a suitable window width and window level, so that the tissue image can be displayed more clearly, and the accuracy of subsequent feature extraction and segmentation processing of the image can be improved.

[0124] In some embodiments of the present specification, the pre-training can comprise: in response to the first training image: obtaining an anchor point sample, adjusting the anchor point sample based on a reference display parameter corresponding to a label carried by the anchor point sample to obtain a first reference sample; an encoder processing the first reference sample to obtain first reference information; a second decoder processing the first reference information to obtain a first reference label; determining a second loss of the second decoder based on the label carried by the anchor point sample and the first reference label; and updating parameters of the second decoder based on the second loss.

[0125] It can be understood that the adjustment of the anchor point sample obtained for the first training image can refer to the enhancement of the anchor point sample corresponding to the second training image as described above, and will not be repeated here. The second random label can also be obtained by processing the second random information by the third decoder below, that is, the second random information corresponding to the second random sample can be processed by any one of the first decoder, the second decoder or the third decoder to obtain the second random label, and the present specification does not limit this.

[0126] In some embodiments of the present specification, the determination of the third loss of the first decoder based on the label carried by the anchor point sample, the third anchor point label, the second random label and the first reference label comprises:

[0127] The third anchor point information and the third anchor point label are fused to obtain a fused third anchor point label; and a first loss of the first decoder is determined based on the fused third anchor point label, the first random label, and the second anchor point label.

[0128] In some embodiments of the present specification, determining the third loss of the first decoder based on the label carried by the anchor point sample, the third anchor point label, the second random label, and the first reference label can include: calculating the cross-entropy of the third anchor point label and the label carried by the anchor point sample; calculating the focal loss of the third anchor point label and the label carried by the anchor point sample; calculating the loss between the third anchor point label and the first reference label; calculating the loss between the third anchor point label and the second random label, and then performing weighted summation on the losses obtained by the above calculations to obtain the third loss. In other embodiments, a negative sample can also be set, the loss between the negative sample label and the third anchor point label is calculated, and the weighted summation of the cross-entropy, the focal loss, and the like is used as the third loss. Wherein the negative sample is other training images except the current training image, and the negative sample can be obtained in the manner of the negative sample corresponding to the first negative sample label. Wherein the loss between the third anchor point label and the second random label can be a mean square error loss, and the loss between the third anchor point label and the negative sample label can be a noise ratio estimation loss value between the third anchor point label and the negative sample label.

[0129] The above third anchor point label can be replaced by the fused third anchor point label to calculate the loss. The anchor point label is fused with the anchor point information, so that the anchor point label also carries the anchor point information, and the first decoder can also consider the original sample features when processing the image for classification. In this way, even if the training sample is transformed with a random window width and window position, the original anchor point information F1 can be used for comparison learning, so that the trained image segmentation model does not need to manually set a suitable window width and window position, and can adaptively perform accurate image segmentation.

[0130] In some embodiments of the present specification, the neural network model can further include a third decoder, which can be updated based on the parameters of the first decoder; the output of the third decoder corresponding to the negative sample can be a second negative sample label, and the first decoder can also update the parameters based on the second negative sample label.

[0131] It can be understood that the negative sample is other training images or samples transformed based on other training images except the current training image (i.e., the anchor point sample or the positive sample transformed based on the anchor point sample).

[0132] In some embodiments of the present disclosure, a training image with a difference from the anchor sample exceeding a preset threshold can be screened out from the training sample set as a negative sample. Further, the difference from the anchor sample can be represented by image similarity matching or by a distance between the anchor sample and the anchor sample collection location. The distance between the anchor sample and the anchor sample collection location can be a sequence distance along a collection image travel path when the image is collected.

[0133] In some embodiments of the present disclosure, the third decoder can update the parameters by the following formula: θ k = mθ k + (1-m)θ a Formula (2)

[0134] wherein θ k represents the parameters of the third decoder, θ a represents the parameters of the first decoder, and m represents an association parameter. In some embodiments, m can take [0.8, 1), preferably 0.99.

[0135] It can be understood that the third decoder can slowly update its own parameters according to the momentum conservation of the parameters of the first decoder, so that the third decoder is less affected by the first decoder. This ensures that the third decoder outputs a result that is basically consistent for the same input, and ensures the consistency of the features in the entire queue.

[0136] In some embodiments of the present disclosure, the pre-training can include: in response to the first training image: obtaining an anchor sample, adjusting the anchor sample based on a random display parameter to obtain a third random sample, adjusting the anchor sample based on a second reference display parameter corresponding to a label carried by the anchor sample to obtain a second reference sample; an encoder processes the anchor sample, the third random sample and the second reference sample to obtain fourth anchor information, third random information and second reference information, respectively; a first decoder processes the fourth anchor information to obtain a fourth anchor label; a second decoder processes the second reference information to obtain a second reference label; the first decoder, the second decoder or a third decoder processes the third random information to obtain a third random label; based on the label carried by the anchor sample, the fourth anchor label, the third random label, the second reference label and the second negative sample label, a fourth loss of the first decoder is determined; and the first decoder is updated based on the fourth loss.

[0137] It can be understood that the process of generating the third random sample and the second reference sample can refer to the description of the first random sample and the first reference sample in the foregoing, which will not be repeated here.

[0138] In some embodiments of the present specification, the fourth loss can be determined by the following formula: Loss a = aL p + b(6L ce + eL fl + hL cc + L n ) Formula (3)

[0139] Wherein, Loss a represents the fourth loss, a, b, 6, e and h respectively represent weight parameters; L ce represents the cross entropy of the fourth anchor label and the label carried by the anchor sample; L fl represents the focal loss of the fourth anchor label and the label carried by the anchor sample; L cc represents the loss between the fourth anchor label and the second reference label; L p represents the loss between the fourth anchor label and the third random label; L n represents the loss between the fourth anchor label and the second negative sample label. The loss between the fourth anchor label and the third random label can be mean square error loss, and the loss between the fourth anchor label and the second negative sample label can be the noise ratio estimation loss value between the fourth anchor label and the second negative sample label.

[0140] In some embodiments of the present specification, based on the first anchor label, the first random label and the second anchor label, the first loss of the first decoder can be determined, which can include: fusing the first anchor label with the first anchor information to obtain a fused first anchor label; based on the fused first anchor label, the first random label and the second anchor label, determining the first loss of the first decoder. Specifically, when calculating the loss value, the fused anchor label can be used to participate in the calculation of the loss value L n , L p When calculating L o , the anchor label before fusion can be used for loss value calculation, and other loss values, such as L ce , L fl , L cc , L o , etc., can also use the anchor label after fusion for loss value calculation.

[0141] Next, taking the training image as a medical image, combining FIG. 2 and FIG. 3, the parameter updating process of the decoder in the case of carrying the label and not carrying the label is further introduced.

[0142] Step 1, extract a batch of image data from the data pile as anchor I oand get all data I in the queue k , the queue is empty at the first training, and the queue is used to store the labels output by the decoder for the anchor sample set.

[0143] wherein the images in each batch are randomly extracted, and the size of the data in a batch is set according to the memory of the actual training environment, and there is no mutual operation between the data in a batch; the step of training a batch is a step, and after a step is completed, the next batch is extracted from the remaining data stack for the next step of training.

[0144] In the case of non-first training, the data in the queue is not empty, and in an embodiment, the data in the queue is screened, and the data with a larger distance from the anchor image in the queue is selected as I2, which is used as a negative sample of the anchor sample in training. This screening operation can avoid the interference of negative samples with small distances.

[0145] In an embodiment, the training image data includes medical images of at least one patient, and a plurality of images of each patient form a sequence of images according to the collection position or sequence, so that the images with a sequence distance greater than 3 from the anchor image in the queue are marked as I k In this embodiment, the sequence distance 3 is obtained according to the spacing between the Z axes of the sequence images, and those skilled in the art can also set the screening condition to a sequence distance of 4, 5, 10 or other reasonable minimum sequence distance, which is not limited here.

[0146] Step 2, adjust the window width and window position of the anchor I o to obtain an image I'5 (i.e. the reference sample in the foregoing description).

[0147] In an embodiment, adjusting the window width and window position is specifically to determine the parameter adjustment of the window width and window position of the image according to the label carried by the current sample, i.e. the tissue type represented by the label, to obtain an image I'5 (such as selecting an image I'5 with a suggested window width and window position corresponding to the bone window when the label is bone); and not according to the label to determine the parameter, randomly adjust the window width and window position of the image to obtain a random window width and window position image I'3 (i.e. the random sample in the foregoing description), wherein I o , I'3 and I'5 can be referred to as positive samples. Wherein, the random data augmentation and random adjustment of the window width and window position of the image can increase the image data, improve the generalization and robustness of the model, and avoid network overfitting. The suggested window width and window position of each tissue can use the public information in the field, and in some embodiments, the following table 1 is used as the suggestion:

[0148] Table 1

[0149] In the embodiments of the present specification, the neural network model can adopt an Actor-Critic architecture, as shown in FIG. 4. The critic decoder (i.e., the second decoder in the foregoing) and the actor-critic decoder (i.e., the first decoder in the foregoing) are set, and the role of the critic network is mainly to improve the segmentation ability of the actor-critic decoder on the basis of ensuring its generalization ability. Since the network will diverge in various directions without carrying labels, the critic decoder can constrain the network to develop in a reasonable direction. It can be understood that other architectures such as student-teacher networks can also be used, the key point being that two decoders are needed, and there is a guiding relationship between the two decoders. The output feature F'5 of the image I'5 processed by the encoder is used as the input of the critic decoder, because it is not enough to rely on the presence or absence of labels as the difference between the two decoders, and a suitable window width and window position have a great influence on the segmentation result, and F'3 is the feature corresponding to the image after random window width and window position, which is used as the input of the actor-critic decoder.

[0150] Step 3, using the critic decoder and the actor-critic decoder to process the input features respectively, and output corresponding labels.

[0151] In the present embodiment, a weakly supervised self-evaluation network structure is adopted in the encoder-decoder part to realize the use of unlabeled data to promote the learning of the decoder and obtain reliable pseudo-labels from the decoder output. That is, different classification predictions are made from different perspectives, and then the critic network guides the training of the actor network. Specifically, the encoder first extracts deep features in the image through downsampling, and the number of downsampling layers is determined according to the area of the smallest ROI. Deep downsampling is easy to make the features of ROIs with small areas such as the lens disappear. In some embodiments, three to four layers of downsampling are used according to the resolution and other conditions of the training image. Correspondingly, upsampling needs to be done in the decoder. It can be understood that if the encoder does not use upsampling but uses other ways to extract features, the operations in the decoder also need to be adjusted accordingly. The adjustment method can refer to conventional technical means in the art; the encoder downsampling is mainly to expand the global perception and reduce the computational amount. In the present embodiment, the encoder can be constructed by using the residual block of the ResNet network, and in some embodiments, a structure of three decoders is used. The three decoders can all use the self-attention mechanism of the Transformer network, or can be replaced by other networks. Among them, the encoder corresponds to I o, I'3, I'5, I2, the corresponding results can include: anchor point features F1, positive sample features F'3, specified window width bit image features F'5, negative sample features F2. It can be understood that only the first training image carrying the label has I'5 and F'5, and the second training image without carrying the label cannot obtain the specified window width and window bit because of the unknown category, so there is no I'5 and F'5 data. The critic network only performs full-supervised training on the data with labeled specified window width and window bit to obtain accurate mask (i.e. pseudo label) segmentation capability, and the actor network is responsible for semi-supervised training on all data (including labeled and unlabeled), which has strong generalization capability but weaker mask segmentation capability than the critic network.

[0152] Step 4, for the labeled data, the anchor point features F1 are taken as the input of the actor decoder to obtain the mask O1, and the specified window width bit image features F'5 are taken as the input of the critic decoder to obtain the mask O'5. The positive sample features F'3 are input into the decoder K to obtain O'3, and the negative sample features F2 are input into the decoder K to obtain O'3 and O2. The masks O1, O'3, O2 and O'5 are all the output labels of the decoder, i.e. pseudo labels, and the essence is the predicted category of each pixel point.

[0153] Referring to FIG. 2, the loss of the actor decoder includes the loss value Lcc of the mask O1 and the output O'5 of the critic decoder; the cross-entropy and focal loss value Lce+Lfl of the real label (i.e. label in FIG. 2); the mean square error loss value L p of the new O1 obtained by feature fusion of the mask O1 and the original features F1 and the positive sample output O'3; and the noise contrastive estimation (NCE) loss value Ln of the negative sample output O2 (the specific feature fusion method is prior art). The specific calculation formula of the loss of the actor decoder can refer to formula (3) in the foregoing. The loss of the critic decoder is the loss value L c .

[0154] For the unlabeled data, the actor network mainly relies on the critic network to form the loss, the input of the actor decoder is the anchor point features F1 and the positive sample features F'3, and the corresponding output masks O a1 and O3 after processing, i.e. corresponding to the first anchor point label and the first random label in the foregoing, the input of the critic decoder is F1, and the corresponding output masks O c1 and O a1 after processing, i.e. corresponding to the second anchor point label in the foregoing, by comparing the outputs O c1 and O o of the two networks, the loss value L f, complete the parameter update of the actor home network. The loss calculation process of the actor home decoder can refer to formula (1) in the foregoing, and the update process can refer to FIG. 3.

[0155] The decoder K updates its own parameters slowly according to the momentum conservation of the parameters of the decoder Actor, to ensure the consistency of the features in the entire queue. The specific update formula of the parameters of the decoder K can refer to formula (2) in the foregoing.

[0156] Step 5, store the output feature O1 in the queue.

[0157] When storing specifically, the first-in first-out principle can be used, and the number of queues can be determined according to the training environment. Ideally, the larger the better, because of the limited memory, the first-in first-out way is used to maintain a queue of image features.

[0158] In the embodiment, referring to FIG. 2, when the entire queue is full of 10 batches of features, the newly generated feature O1 enters the queue, and the first feature b1 in the queue leaves the queue, b2 to b10 move one place, and O1 occupies the position of b10. The role of the queue is to provide sufficient negative samples for the original image for contrast learning. When the queue is empty, there is only the original image for learning without the queue image, that is, at the beginning, there is only positive sample contrast learning. Although this way will have a certain impact on the training result, that is, only one batch is affected, as the queue is full of features, the features before the queue are removed from the queue, and the impact on the final training result can be minimized.

[0159] Step 6, after training, the trained image segmentation model is used to segment the medical image, and the range, contour and category of the tissue in the medical image can be obtained.

[0160] In some embodiments, as shown in FIG. 4, the first training image includes a plurality of specific regions, and the first training image carries a plurality of labels. The categories of all specific regions contained in the training sample set correspond to at least the categories of the plurality of target regions. An image segmentation model for identifying and segmenting a plurality of specific regions is trained. The segmentation image output by the image segmentation model in FIG. 4 automatically fuses a plurality of target regions, and the segmentation result output contains a plurality of target regions after fusion and their respective categories, which are superimposed and displayed. The image segmentation model can segment all categories at a time, has higher segmentation efficiency, and has more extensive application scenarios. For example, a plurality of images of the same patient can be segmented for a plurality of regions of interest, and then fused features are operated, different tissues are classified and labeled with different colors, and finally displayed on a CT image, and the tissue category names represented by different colors are marked.

[0161] The embodiments of the present specification can realize improvement of a medical image segmentation algorithm, accelerate model convergence through a weakly supervised self-evaluation network structure, make different classification predictions from different perspectives, and then guide the actor home decoder through the critic decoder for unlabeled training, which has low dependence on data sets and good adaptability; using the method for segmentation, it is not necessary to adjust the window width and window level one by one, and accurate segmentation images can be obtained regardless of whether the window width and window level of the input image are appropriate, because the labeled and unlabeled training images are trained according to special rules when the model is trained, the influence of the window width and window level on the image features and image segmentation is penetrated into the model using the method, so that the model has stable and excellent image segmentation capability that is not affected by the window width and window level, and full-area automatic segmentation can be realized, and the stability of the segmentation result is improved. It can be understood that the image segmentation of the self-adaptive window width and window level of the medical image is only an embodiment of the beneficial effects of the method, and is not the whole value of the method. The method can also produce beneficial effects of low dependence on training samples, high model adaptability and accurate segmentation results in other image segmentation fields. Moreover, the contrast learning in the model is in units of pixels, and the pixel-level contrast learning can improve the cosine similarity between pixels of the same class (i.e. the use of the loss value Ln in the foregoing), and the parameter updating of the image segmentation model through the method can reduce the cosine similarity of different classes, optimize the decoder parameters (especially the actor network), improve the segmentation capability, and make the segmentation result accurate.

[0162] The image segmentation method in the present embodiment can not only segment CT images, but also be applied to the segmentation of PET-CT images. By training a deep learning model, different structures and tissues in the image can be automatically segmented (especially including the segmentation of tumor target regions), so as to realize accurate target region delineation. In view of the influence of the window width and window level on the feature recognition of the deep learning model in the medical image data, and the limited CT, PET and other medical image data that can be used for deep learning model training, the medical image is subjected to deep learning and intelligent analysis through an artificial intelligence algorithm, the model is trained using PET-CT fused original image data, the image feature extraction deviation caused by different window width and window level settings is avoided, and the training data can be effectively reduced. Ultimately, the treatment target region in the image is automatically identified under the condition of few samples, the doctor is assisted to delineate, and the treatment plan is formulated quickly. It can be understood that the image segmentation method in the present specification can not be limited to the CT or PET-CT medical image described above, and the method provided in the present application can also be used for segmentation processing of other types of medical images such as MRI.

[0163] Based on the image segmentation method described above, one or more embodiments of the present specification also provide an image segmentation device. The device can include a device (including a distributed system), software (application), module, plug-in, server, client, etc. using the method described in the embodiments of the present specification, and a device combined with the necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided by the embodiments of the present specification is described in the following embodiments. Since the implementation scheme of the device solves the problem similar to the method, the implementation of the specific device of the embodiments of the present specification can refer to the implementation of the foregoing method, and the repeated parts will not be described herein. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived. As shown in FIG. 5, the image segmentation device 500 can include:

[0164] An image acquisition module 501 is configured to acquire a to-be-processed image of a target object.

[0165] An image segmentation module 502 is configured to input the to-be-processed image into a pre-trained image segmentation model, and output a segmentation image, wherein the segmentation image includes a target region and a category of the target region; wherein the image segmentation model is obtained by pre-training a neural network model including an encoder, a first decoder, and a second decoder using a training sample set including a first training image carrying a label and a second training image not carrying a label; the pre-training includes: the first decoder performs parameter updating through the first training image carrying the label, and the second decoder performs parameter updating through the label generated based on the second training image not carrying the label.

[0166] The description and functions of the above modules can be understood with reference to the content of the image segmentation method part, which will not be described herein.

[0167] The image acquisition module 501 and the image segmentation module 502 can be an independent execution unit respectively, or can share an execution unit. The execution unit can include one or more processors, or can include one or more processors and one or more memories, and the memory stores instructions executable by the processor.

[0168] Based on the same innovative concept of the image segmentation method described above, one or more embodiments of the present specification also provide a radiotherapy plan generation method. As shown in FIG. 6, the method can include:

[0169] S601: Acquire a to-be-processed image of a target object.

[0170] S602: input the image to be processed into a pre-trained image segmentation model, and output a segmented image, wherein the segmented image comprises a target region and a category of the target region.

[0171] It can be understood that in the radiotherapy plan generation method, this step can adopt the image segmentation method described in the above embodiments of the present specification, and can be implemented by referring to the foregoing image segmentation method. The repeated parts will not be described again. In addition, the image segmentation model can also be trained by using other model training methods in the related art.

[0172] S603: determining material information of the target region based on the category of the target region, and establishing a three-dimensional voxel model based on the target region and the material information of the target region.

[0173] S604: performing tissue dose simulation calculation based on the three-dimensional voxel model, and generating a treatment plan. The method for generating the treatment plan in this step can refer to the related art in the field, and the target region and the category of the target region obtained by the foregoing image segmentation method are more accurate, so that the generated treatment plan is also more accurate and has better effect.

[0174] In the field of radiotherapy, the treatment target region of BNCT (boron neutron capture therapy) is different from that of traditional radiotherapy, and generally does not define the characteristics of GTV (gross tumor volume), CTV (clinical target volume), PTV (planning target volume) and other target regions alone. The present method can be applied in BNCT treatment, and can segment the PET-CT image obtained by fusing the PET image containing functional information such as F-BPA drug distribution and the CT image. By training a deep learning model, different structures and tissues in the image (especially the tumor target region) are automatically segmented, so as to realize accurate target region delineation.

[0175] At present, BNCT has been carried out for a short time, and the medical image data such as CT and PET that can be used for deep learning model training is limited, it is difficult to obtain a large amount of training data, and the data with labels are even more scarce. The present method is aimed at the above problems, and considers the influence of window width and window level in medical image data on feature recognition of the deep learning model. Through artificial intelligence algorithm, the medical image is subjected to deep learning and intelligent analysis. The model can be trained by using the original image data of PET-CT fusion, so as to avoid the deviation of image feature extraction caused by different window width and window level settings, effectively reduce the required training data, and finally realize automatic, accurate and reliable identification of the treatment target region under the condition of few samples, assist doctors in delineation, and speed up the preparation of treatment plan.

[0176] PET image with F-BPA drug distribution information is a BNCT specific image, which uses high contrast to display the distribution information of boron drug concentration in the patient's body. Although there may be misalignment and limited resolution between F-BPA and CT, F-BPA can still provide very useful information to help manually delineate the treatment target area on CT, and the complementary advantages of F-BPA and CT are crucial for accurate target delineation. The method can use PET-CT fusion images and initial CT, PET as multi-modal image data as the input of the image segmentation model, and integrate the semantic features specific to PET and CT for deep training to automatically identify the treatment target area and assist the doctor in delineation. The method can realize the segmentation of the 9 main organs-at-risks (OARs) most relevant to head and neck BNCT, including brain stem, chiasm, mandible, left optic nerve, right optic nerve, left parotid gland, right parotid gland, left mandible, and right mandible.

[0177] Based on the same innovative idea of the above radiotherapy plan generation method, one or more embodiments of the present specification also provide a radiotherapy system 700, since the implementation scheme of the system solves the problem is similar to the radiotherapy plan generation method, therefore the specific system implementation of the embodiments of the present specification can refer to the implementation of the foregoing method, and the repeated parts will not be described herein. As shown in FIG. 7, the radiotherapy system 700 can include:

[0178] A beam irradiation device 701 for generating a radiation beam.

[0179] A treatment planning module 702 for generating a treatment plan.

[0180] A control module 703 for controlling the beam irradiation device 701 to generate a radiation beam according to the treatment plan.

[0181] The treatment planning module 702 can include:

[0182] An image acquisition unit 7021 for acquiring a to-be-processed image of a target object.

[0183] An image segmentation unit 7022 for inputting the to-be-processed image into a pre-trained image segmentation model and outputting a segmentation image, wherein the segmentation image includes a target region and a category of the target region.

[0184] A model establishing unit 7023 for determining material information of the target region based on the category of the target region, and establishing a three-dimensional voxel model based on the target region and the material information of the target region.

[0185] A calculation unit 7024 for performing tissue dose simulation calculation based on the three-dimensional voxel model and generating a treatment plan.

[0186] The treatment planning module 702, the control module 703, the image acquisition unit 7021, the image segmentation unit 7022, the model establishing unit 7023, and the calculation unit 7024 can be independent execution units respectively, or can share an execution unit with one or more modules, the execution unit can include one or more processors, or can include one or more processors and one or more memories, and the memories store instructions executable by the processors.

[0187] In some embodiments, the radiotherapy system can include a treatment planning system (TPS), in other embodiments, the radiotherapy system can further include a mechanical arm control system, and in other embodiments, the radiotherapy system can further include a data management system for interaction and data processing of other systems and modules. Correspondingly, one or more of the treatment planning module 702, the control module 703, the image acquisition unit 7021, the image segmentation unit 7022, the model establishing unit 7023, and the calculation unit 7024 can be arranged in the TPS, the mechanical arm control system, or the data management system, or in a separately arranged data processing device. The specific arrangement can be set according to actual needs and conditions, and is not limited herein.

[0188] The embodiments of the present application also provide an electronic device, as shown in FIG. 8, which can include a processor 801 and a memory 802, wherein the processor 801 and the memory 802 can be connected through a bus or other means, and in FIG. 8, the connection through the bus is taken as an example.

[0189] The processor 801 can be a central processing unit (CPU). The processor 801 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0190] The memory 802, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the image segmentation method and / or the radiotherapy plan generation method in the embodiments of the present application. The processor 801 executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 802, that is, implements the image segmentation method and / or the radiotherapy plan generation method in the above method embodiments.

[0191] The memory 802 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created by the processor 801 and the like. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 802 can optionally include a memory disposed remotely with respect to the processor 801, and these remote memories can be connected to the processor 801 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0192] The one or more modules are stored in the memory 802, and in some embodiments, when executed by the processor 801, perform the image segmentation method in the embodiment as shown in FIG. 1. In some embodiments, when executed by the processor 801, perform the radiotherapy plan generation method as described above.

[0193] The above electronic device specific details can be understood by referring to the corresponding related descriptions and effects in the above method embodiments, which will not be described here.

[0194] The present specification also provides a computer storage medium storing computer program instructions, which, when executed, implement the steps of the above image segmentation method in some embodiments. In some embodiments, the computer program instructions, when executed, implement the above radiotherapy plan generation method.

[0195] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above-mentioned types of memories.

[0196] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions.

[0197] For the convenience of description, the above device is described as various units by function respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0198] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disc, an optical disc, etc., and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of some parts of the embodiments of the present application.

[0199] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, etc.

[0200] The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0201] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the system embodiments are described simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the specification. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0202] The above only describes the embodiments of the embodiments of the specification and does not limit the embodiments of the specification. The embodiments of the specification can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the specification shall be included in the scope of claims of the embodiments of the specification.

Claims

1. A method of image segmentation, characterized by, The method comprises: obtaining a to-be-processed image of a target object; inputting the to-be-processed image into a pre-trained image segmentation model to output a segmented image, the segmented image comprising a target region and a category of the target region; wherein the image segmentation model is obtained by pre-training a neural network model comprising an encoder, a first decoder and a second decoder using a training sample set comprising first training images carrying labels and second training images not carrying labels; the pre-training comprises: updating parameters of the first decoder based on the first training images carrying labels and labels generated by the second decoder based on the second training images not carrying labels, and updating parameters of the second decoder based on the first training images carrying labels.

2. The method of claim 1, wherein, The pre-training comprises: updating parameters of the first decoder based on the labels carried by the first training images, first output and third output, and / or updating parameters of the first decoder based on second output and fourth output; updating parameters of the second decoder based on the third output and the labels carried by the first training images; wherein the first output is output of the first decoder corresponding to the first training images; the second output is output of the first decoder corresponding to the second training images; the third output is output of the second decoder corresponding to the first training images; the fourth output is output of the second decoder corresponding to the second training images.

3. The method of claim 1, wherein, The pre-training comprises: in response to the second training images: obtaining an anchor sample, adjusting the anchor sample based on random display parameters to obtain a first random sample; the encoder processes the anchor sample and the first random sample to obtain first anchor information and first random information, respectively; the first decoder processes the first anchor information and the first random information to obtain first anchor labels and first random labels, respectively; the second decoder processes the first anchor information to obtain second anchor labels; determining a first loss of the first decoder based on the first anchor labels, the first random labels and the second anchor labels; updating parameters of the first decoder based on the first loss.

4. The method of claim 3, wherein, The first loss is determined by the following formula: wherein Loss a represents the first loss, σ and υ represent weight parameters respectively, L o represents the loss between the first anchor label and the second anchor label, L p represents the loss between the first anchor label and the first random label, L n represents the loss between the first anchor label and a first negative sample label, the first negative sample label being obtained based on the output of the neural network model corresponding to other training images.

5. The method of claim 3, wherein, The obtaining of the anchor sample comprises: performing at least one image enhancement processing on the current training image, and taking the processed at least one image and the current training image as anchor samples, respectively.

6. The method of claim 1, wherein, The pre-training comprises: in response to the first training images: obtaining an anchor sample, adjusting the anchor sample based on reference display parameters corresponding to the labels carried by the anchor sample to obtain a first reference sample; the encoder processes the first reference sample to obtain first reference information; the second decoder processes the first reference information to obtain first reference labels; determining a second loss of the second decoder based on the labels carried by the anchor sample and the first reference labels; updating parameters of the second decoder based on the second loss.

7. The method of claim 6, wherein, The pre-training comprises: in response to the first training images: obtaining an anchor sample, adjusting the anchor sample based on random display parameters to obtain a second random sample; The encoder processes the anchor sample and the second random sample to obtain third anchor information and second random information, respectively; The first decoder processes the third anchor information to obtain a third anchor label; The first decoder or the second decoder processes the second random information to obtain a second random label; Based on the label carried by the anchor sample, the third anchor label, the second random label and the first reference label, a third loss of the first decoder is determined; The parameters of the first decoder are updated based on the third loss.

8. The method of claim 1, wherein, The neural network model further comprises a third decoder, which is updated based on the parameters of the first decoder; The output of the third decoder corresponding to the negative sample is a second negative sample label, and the parameters of the first decoder are further updated based on the second negative sample label.

9. The method of claim 3, wherein, The first loss of the first decoder is determined based on the first anchor label, the first random label and the second anchor label, comprising: The first anchor label is fused with the first anchor information to obtain a fused first anchor label; The first loss of the first decoder is determined based on the fused first anchor label, the first random label and the second anchor label.

10. The method of claim 1, wherein, The segmented image includes a plurality of target regions and categories of the plurality of target regions; The label carried by the first training image includes a specific region in the first training image and a category of the specific region; The categories of all specific regions contained in the training sample set correspond to at least the categories of the plurality of target regions.

11. An image segmentation apparatus characterized by comprising: Comprising: An image acquisition module for acquiring a to-be-processed image of a target object; An image segmentation module for inputting the to-be-processed image into a pre-trained image segmentation model to output a segmented image, the segmented image including a target region and a category of the target region; wherein the image segmentation model is obtained by pre-training a neural network model comprising an encoder, a first decoder and a second decoder using a training sample set comprising a first training image carrying a label and a second training image not carrying a label; the pre-training comprises parameter updating of the first decoder based on the first training image carrying a label and parameter updating of the second decoder based on a label generated by the second decoder based on the second training image not carrying a label.

12. A radiation therapy plan generation method, characterized by, Comprising: Acquire a to-be-processed image of a target object; Input the to-be-processed image into a pre-trained image segmentation model to output a segmented image, the segmented image including a target region and a category of the target region; Determine material information of the target region based on the category of the target region, and establish a three-dimensional voxel model based on the target region and the material information of the target region; Perform organizational dose simulation calculation based on the three-dimensional voxel model and generate a treatment plan.

13. A radiotherapy system, characterized by, Comprising: A beam irradiation device for generating a radiation beam; A treatment planning module for generating a treatment plan; A control module for controlling the beam irradiation device to generate a radiation beam according to the treatment plan; The treatment planning module comprises: An image acquisition unit for acquiring a to-be-processed image of a target object; An image segmentation unit is configured to input the to-be-processed image into a pre-trained image segmentation model, and output a segmented image, wherein the segmented image comprises a target region and a category of the target region; A model establishing unit is configured to determine material information of the target region based on the category of the target region, and establish a three-dimensional voxel model based on the target region and the material information of the target region; A calculation unit is configured to perform an organ dose simulation calculation based on the three-dimensional voxel model, and generate a treatment plan.

14. An electronic device comprising a processor and a memory for storing processor-executable instructions, the electronic device characterized by: The processor executes the instructions to implement the steps of the method in any one of claims 1-10 or 12.

15. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method in any one of claims 1-10 or 12.

Images