Image processing method, device, computer equipment and storage medium
By fusing the feature information of multimodal images in image processing, efficient outline and dose distribution images are generated, the problem of low dose distribution image acquisition efficiency in the prior art is solved, and the efficiency and accuracy of radiotherapy plan design is improved.
Patent Information
- Application Number
- CN202210867623.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-07-22
AI Technical Summary
Existing image processing methods are inefficient in obtaining dose distribution images of radiotherapy targets and organ-threatening images, resulting in limited radiotherapy planning design efficiency.
By adopting an image processing method, by obtaining images of at least one modality and inputting an identification model, the feature information is extracted using an encoder and a decoder, and the feature map is fused to generate outlined images and dose distribution images, improving the efficiency and accuracy of image processing. sex.
It effectively improves the acquisition efficiency and accuracy of dose distribution images, simplifies the radiotherapy planning design process, and shortens the prediction time.
Smart Images

Figure CN115100185B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to an image processing method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] Radiotherapy is one of the important treatment methods for cancer. Radiotherapy requires that a specific dose be obtained in the tumor target area while the dose to normal tissues and organs is as low as possible. Before planning the dose distribution, the doctor needs to outline the treatment target area and organs at risk on the image, and give a dose distribution prescription to the outlined area in order to carry out the next radiotherapy plan.
[0003] In order to obtain the dose distribution image corresponding to the treatment target area and organs at risk on the image, the existing image processing method is to first process the image to obtain a contour image, and then obtain a predicted dose distribution image based on the given contour image. Since the acquisition of the predicted dose distribution image can only be performed after the contour image is acquired, the current image processing method has the problem of low efficiency in acquiring the predicted dose distribution image. Summary of the invention
[0004] Based on this, it is necessary to provide an image processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of dose distribution image acquisition in order to address the technical problem of low efficiency of dose distribution image acquisition in traditional image processing methods.
[0005] In a first aspect, the present application provides an image processing method. The method comprises:
[0006] acquiring images of at least one modality;
[0007] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0008] In one embodiment, the recognition model includes at least one encoder, a delineation decoder, and a dose distribution decoder;
[0009] Input at least one modality of image into the recognition model to obtain delineation images and dose distribution images, including:
[0010] Inputting images of at least one modality into at least one encoder respectively to obtain feature maps corresponding to the images of at least one modality;
[0011] When there is more than one type of modality, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map;
[0012] Input the fused feature map into the delineation decoder to obtain the delineation image;
[0013] Extract features from the outlined image to obtain a feature map corresponding to the outlined image;
[0014] The fused feature map and the feature map corresponding to the outline image are input into the dose distribution decoder to obtain the dose distribution image.
[0015] In one embodiment, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map, including:
[0016] Performing weighted sum processing on the feature maps corresponding to the images of at least one modality to obtain a fused feature map;
[0017] Alternatively, feature maps of multiple channels are extracted from images of each modality, and the feature maps corresponding to images of at least one modality under the same channel are weighted and processed to obtain weighted and feature maps of multiple channels, and the weighted and feature maps of multiple channels are used as fused feature maps.
[0018] In one embodiment, the step of training the recognition model includes:
[0019] Determine a first loss function corresponding to the encoder, a second loss function corresponding to the delineation decoder, and a third loss function corresponding to the dose distribution decoder;
[0020] Determine a model loss function according to the first loss function, the second loss function and the third loss function;
[0021] Based on the model loss function, the initial model is trained to obtain the recognition model.
[0022] In one embodiment, the initial model is trained based on the model loss function to obtain a recognition model, including:
[0023] Acquire image samples of at least one modality, wherein the image samples of each modality include an image of the corresponding modality, a standard delineation image, and a standard dose distribution image, wherein the standard delineation images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same;
[0024] Inputting images from at least one modality of image samples into an initial model to obtain delineation prediction results and dose distribution prediction results;
[0025] Based on the outline prediction results, standard outline images, dose distribution prediction results, standard dose distribution images and model loss function, the model loss is calculated, and based on the model loss, the parameters of the initial model are updated to obtain the recognition model.
[0026] In one embodiment, the image processing method further includes:
[0027] Obtaining an initial delineated image of the target area;
[0028] comparing the outlined image with an initial outlined image of the target area;
[0029] If the similarity between the outline image and the initial outline image of the target area exceeds a preset range, the initial outline image of the target area is subjected to feature extraction to obtain a feature map corresponding to the initial outline image of the target area; the fused feature map and the feature map corresponding to the initial outline image of the target area are input into a dose distribution decoder to obtain an updated dose distribution image.
[0030] In a second aspect, the present application also provides an image processing device. The device comprises:
[0031] An acquisition module, used for acquiring images of at least one modality;
[0032] The processing module is used to input images of at least one modality into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0034] acquiring images of at least one modality;
[0035] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0036] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0037] acquiring images of at least one modality;
[0038] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0039] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0040] acquiring images of at least one modality;
[0041] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0042] The above-mentioned image processing method, apparatus, computer equipment, storage medium and computer program product obtain images of at least one modality and input the images of at least one modality into a recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area. The feature information of the images under each modality can be effectively utilized to improve the accuracy of the delineation image and the dose distribution image; the simultaneous processing of images of at least one modality can improve the acquisition efficiency of the dose distribution image; the delineation image and the dose distribution image are obtained together after the images are input into the recognition model, which further improves the acquisition efficiency of the dose distribution image. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;
[0044] Figure 2 is a schematic flow chart of an image processing method in one embodiment;
[0045] Figure 3 A flowchart of a possible implementation method of step S400 in an embodiment;
[0046] Figure 4 A schematic diagram of a flow chart of a training step of a recognition model in one embodiment;
[0047] Figure 5 A schematic diagram of a sub-flow diagram of a possible implementation method of step S160 in an embodiment;
[0048] Figure 6 is a flowchart of an image processing method in yet another embodiment;
[0049] Figure 7 is a flowchart of an image processing method in a specific embodiment;
[0050] Figure 8 It is a flowchart of an image processing method when there are M encoders in a specific embodiment;
[0051] Fig. 9 is a structural block diagram of an image processing device in one embodiment;
[0052] Fig.10FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0054] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the terminal 102 needs to process. The data storage system can be integrated on the terminal 102, or it can be placed on the cloud or other network servers. The terminal 102 acquires an image of at least one modality; the image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image, the delineation image is used to indicate the outline of the target area, and the dose distribution image is used to indicate the dose at each position of the target area. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.
[0055] In one embodiment, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:
[0056] S200: Acquire an image of at least one modality.
[0057] Among them, images refer to medical images collected for treatment targets and organs at risk, and images of at least one modality refer to images from at least one data source, including single-modality images or multi-modality images. Images from the same data source are images of the same modality, and images of each modality have different morphological manifestations. Single-modality images cannot fully subdivide the relevant areas in the image, so there is a problem of insufficient information in the processing of treatment targets and organs at risk in the image, while multi-modality images can complement information and effectively improve the accuracy of image processing. Images of at least one modality include but are not limited to CT (Computed Tomography, electronic computed tomography) images, MRI (Magnetic Resonance Imaging, magnetic resonance imaging) images, and PET (Positron Emission Computed Tomography, positron emission computed tomography) images. Images of at least one modality can be three-dimensional images or two-dimensional images.
[0058] S400, inputting images of at least one modality into a recognition model to obtain a delineation image and a dose distribution image, wherein the delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0059] Among them, the recognition model refers to the machine learning model obtained by model training of the initial machine learning model, which can be a common image processing model in the medical field. The type of the recognition model is usually a convolutional neural network model. Optionally, the types of recognition models include but are not limited to U-Net, Res-U-Net, and nnUnet. The image is usually a medical image. The target area outlined in the image often belongs to the same treatment target area or organ at risk. Different doses need to be set for different positions of the treatment target area or organ at risk. The outline image is an image used to indicate the outline of the target area in the image. The dose distribution image is used to indicate the dose at each position of the target area. The target area includes at least one of the following: treatment target area or organ at risk. Preferably, the outline image can be a binary image. The pixel points with a pixel value of 1 in the outline image belong to the points in the target area, and the pixel points with a pixel value of 0 in the outline image belong to the points outside the target area. The dose distribution image can be a thermal map. The blocks of different colors in the dose distribution image can reflect the size of the dose and the location of the dose distribution. Outline images and dose distribution images can be used for radiotherapy plan design. Radiotherapy plan design mainly optimizes machine parameters through target constraints to obtain a radiotherapy plan that meets clinical goals. Clinical goals include but are not limited to target coverage or DVH (Dose Volume Histogram) of organs at risk. Outline images and dose distribution are used for radiotherapy planning. It means that by establishing a database of past radiotherapy plans, a recognition model is used to predict the dose distribution image corresponding to the best radiotherapy plan that can be achieved with images and outline images, thereby extracting three-dimensional point dose distribution, two-dimensional point dose distribution or DVH from the dose distribution image. DVH contains indicators such as the average dose and maximum dose of the treatment target or organ at risk, which can be used as constraints in the radiotherapy plan optimization process.
[0060] In the above-mentioned image processing method, by acquiring images of at least one modality, the images of at least one modality are input into a recognition model to obtain an outline image and a dose distribution image. The outline image is used to indicate the outline of the target area, and the dose distribution image is used to indicate the dose at each position of the target area. The characteristic information of the images under each modality can be effectively utilized to improve the accuracy of the outline image and the dose distribution image; the simultaneous processing of images of at least one modality can improve the acquisition efficiency of the dose distribution image; the outline image and the dose distribution image are obtained together after the images are input into the recognition model, which further improves the acquisition efficiency of the dose distribution image.
[0061] In one embodiment, Figure 3 As shown, the recognition model includes at least one encoder, a delineation decoder and a dose distribution decoder; an image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image, including:
[0062] S410, inputting images of at least one modality into at least one encoder to obtain feature maps corresponding to the images of at least one modality;
[0063] S420, when there is more than one type of modality, fusion of feature maps corresponding to images of at least one modality to obtain a fused feature map;
[0064] S430, inputting the fused feature map into a delineation decoder to obtain a delineation image;
[0065] S440, extracting features from the outline image to obtain a feature map corresponding to the outline image;
[0066] S450, inputting the fused feature map and the feature map corresponding to the delineated image into a dose distribution decoder to obtain a dose distribution image.
[0067] In this embodiment, the recognition model includes at least one encoder, each encoder has a similar structure, and each encoder has different input data types, the input data is an image of each modality, and each modality image corresponds to an encoder, and the recognition model also includes a delineation decoder and a dose distribution decoder, the delineation decoder is a decoder for outputting a delineation image, and the dose distribution decoder is a decoder for outputting a dose distribution. The encoder and the decoder are model frameworks in deep learning. The encoder encodes the input image of at least one modality into an intermediate expression form, and mainly uses ordinary convolution layers and downsampling layers to reduce the size of the feature map of the image of at least one modality, so that the feature map of the image of at least one modality becomes a lower dimensional representation, the purpose is to extract as many low-level features and high-level features of the image of at least one modality as possible, so as to accurately segment the image of at least one modality using the extracted spatial information and global information. The decoder decodes the intermediate expression output by the encoder into an output image, which is mainly composed of ordinary convolution layers, upsampling layers and fusion layers, and uses upsampling operations to gradually restore the spatial dimension, fuse the features extracted during the encoding process, and output an image of the same size as the input image while minimizing information loss.
[0068] The images of at least one modality are respectively input into at least one encoder to obtain the feature maps corresponding to the images of at least one modality. Specifically, the number of encoders is equal to the number of modal types that can be supported at most. When obtaining the images of at least one modality, the number of modal types of the images of at least one modality can also be obtained. The images of at least one modality include images of one modality or images of multiple modalities. If the images of at least one modality include images of one modality and the number of encoders is greater than 1, the images of one modality are input into the encoder corresponding to the image of the modality to obtain the feature map corresponding to the image of the modality; if the images of at least one modality include images of multiple modalities, and the number of modal types of the images of multiple modalities is less than or equal to the number of encoders, the images of each modality are respectively input into the encoder corresponding to the image of the modality to obtain the feature map corresponding to the image of each modality; in addition, if the images of at least one modality include images of multiple modalities, and the number of modal types of the images of multiple modalities is equal to the number of encoders, the images of multiple modalities can also be input into any encoder in series to obtain a feature map, which contains the features of the images of multiple modalities. This method of encoding with the encoder corresponding to the image of the modality under different numbers of modality types can ensure the accuracy of the obtained image feature map; in addition, if the modality type of the image is not missing, if the images of multiple modalities are input in series to any encoder, the efficiency of obtaining the dose distribution image can be improved.
[0069] When there are more than one type of modality, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map. Specifically, the fusion method refers to an image fusion method in a convolutional neural network, including but not limited to weighted sum (add) or concatenation (concat), and the feature maps corresponding to the images of at least one modality are weighted summed or concatenated to obtain a fused feature map.
[0070] The fused feature map is input into the outline decoder, and the image output by the outline decoder is the outline image. The outline image is subjected to feature extraction to obtain the feature map corresponding to the outline image. The fused feature map and the feature map corresponding to the outline image are input into the dose distribution decoder, and the image output by the dose distribution decoder is the dose distribution image. In addition, the fused feature map can also be input into the dose distribution decoder to obtain the dose distribution image. This method does not require feature extraction of the outline image, and directly uses the fused feature map to input into the dose distribution decoder, which can further improve the efficiency of obtaining the dose distribution image.
[0071] The scheme of the above-mentioned embodiment is a method of obtaining a delineation image and a dose distribution image by inputting images of at least one modality into at least one encoder respectively, and then fusing the feature maps, and inputting the obtained fused feature maps into a delineation decoder and a dose distribution decoder respectively. The image processing is performed using images of at least one modality, which can effectively utilize the feature information of the images under each modality and improve the accuracy of the delineation image and the dose distribution image; the simultaneous processing of images of at least one modality can improve the acquisition efficiency of the dose distribution image; under different numbers of modality types, the encoder corresponding to the image of the modality is used for encoding processing, which can ensure the accuracy of the obtained image feature map; in addition, in the case where the modality types of the image are not missing, if a method is used in which images of multiple modalities are input in series into any encoder, the acquisition efficiency of the dose distribution image can be improved; at least one encoder performs feature extraction and feature fusion on images of at least one modality, which can ensure the accuracy of the obtained image feature map; the delineation image and the dose distribution image are obtained together after the image is input into the recognition model, which further improves the acquisition efficiency of the dose distribution image.
[0072] In one embodiment, feature maps corresponding to images of at least one modality are fused to obtain a fused feature map, including: weighting and processing the feature maps corresponding to images of at least one modality to obtain a fused feature map; or, extracting feature maps of multiple channels from images of each modality, weighting and processing the feature maps corresponding to images of at least one modality under the same channel to obtain weighted and feature maps of multiple channels, and using the weighted and feature maps of multiple channels as the fused feature map.
[0073] In this embodiment, the weighted sum is given to the feature Figure 1 The feature maps corresponding to the images of at least one modality are weighted and processed to obtain a fused feature map. Specifically, the feature values at the same position in the feature maps corresponding to the images of at least one modality are multiplied by preset weights, and the feature values at different positions are multiplied by different preset weights to obtain the weighted feature values at the corresponding positions, and then the weighted feature values at the same position are added to obtain the weighted sum feature values at the corresponding positions, thereby obtaining the fused feature map. The feature values at each position in the fused feature map are equal to the weighted sum feature values at the corresponding positions.
[0074] Alternatively, the feature maps corresponding to the images of at least one modality may be averaged to obtain a fused feature map. Specifically, the feature values at the same position in the feature maps corresponding to the images of at least one modality are multiplied by a preset weight, and the feature values at different positions are multiplied by the same preset value to obtain the weighted feature values at the corresponding positions. The weighted feature values at the same positions are then added to obtain the weighted sum feature values at the corresponding positions, thereby obtaining a fused feature map. The feature values at each position in the fused feature map are equal to the weighted sum feature values at the corresponding positions.
[0075] Alternatively, feature maps of multiple channels can be extracted from the images of each modality; under the same channel, the feature values at the same position in the feature maps corresponding to the images of at least one modality are multiplied by preset weights, and the feature values at different positions are multiplied by different preset weights to obtain the weighted feature values at the corresponding positions, and then the weighted feature values at the same positions are added to obtain the weighted sum feature values at the corresponding positions, and then a fused feature map is obtained, and the feature values at each position in the fused feature map are equal to the weighted sum feature values at the corresponding positions; under multiple channels, fused feature maps corresponding to multiple channels are obtained respectively, that is, the weighted sum feature maps of multiple channels, and the weighted sum feature maps of multiple channels are used as the fused feature map.
[0076] The scheme of the above-mentioned embodiment is a method for obtaining a fused feature map by weighted sum processing or mean processing of the feature maps corresponding to the images of at least one modality. After the weighted sum or mean processing, the amount of information in the feature map is increased, but the dimension describing the image itself is not increased, only the amount of information in each dimension is increased, which is beneficial to improving the classification effect of the image and improving the accuracy of the subsequent delineation image; by extracting feature maps of multiple channels from the images of each modality, weighted sum processing is performed on the feature maps corresponding to the images of at least one modality under the same channel to obtain a multi-channel weighted sum feature map, which is a method for fusion feature maps. After the feature maps of different channels are merged, the information under each feature is not increased, while the number of eigenvalues in the feature map is increased, which is beneficial to improving the accuracy of the delineation image and the dose distribution image.
[0077] In one embodiment, Figure 4 As shown in the figure, the training steps of the recognition model include:
[0078] S120, determining a first loss function corresponding to the encoder, a second loss function corresponding to the delineation decoder, and a third loss function corresponding to the dose distribution decoder;
[0079] S140, determining a model loss function according to the first loss function, the second loss function, and the third loss function;
[0080] S160, based on the model loss function, the initial model is trained to obtain a recognition model.
[0081] In this embodiment, the loss function is an indicator reflecting the performance of the recognition model, that is, an indicator reflecting the extent to which the current recognition model is not fitted and the extent to which it is inconsistent. The loss functions corresponding to the encoder and the decoder can be the same or different, and the loss functions corresponding to the delineation decoder and the dose distribution decoder can be the same or different. The first loss function corresponding to the encoder, the second loss function corresponding to the delineation decoder, and the third loss function corresponding to the dose distribution decoder are determined. The model loss function is the weighted sum of the first loss function, the second loss function, and the third loss function. The weights corresponding to the first loss function, the second loss function, and the third loss function can be different. The initial model is a machine learning model, which can be a common image processing model in the medical field. The type of the initial model is usually a convolutional neural network model. Optionally, the types of the initial model include but are not limited to U-Net, Res-U-Net, and nnUnet. The initial model is trained using the model loss function to obtain a recognition model. The condition for stopping the training can be that the loss function converges to a preset threshold or the number of training rounds reaches a preset number of rounds.
[0082] The scheme of the above-mentioned embodiment, by giving the same or different weights to the first loss function corresponding to the encoder, the second loss function corresponding to the outline decoder and the third loss function corresponding to the dose distribution decoder, respectively, the method of weighted model loss can distribute the model loss to the encoder, the outline decoder and the dose distribution decoder in proportion. The recognition model obtained based on model loss training can reduce the error between the predicted value and the true value of the recognition model and improve the stability of the recognition model.
[0083] In one embodiment, Figure 5 As shown, based on the model loss function, the initial model is trained to obtain a recognition model, including:
[0084] S162, acquiring image samples of at least one modality, wherein the image samples of each modality include an image of the corresponding modality, a standard delineation image, and a standard dose distribution image, wherein the standard delineation images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same;
[0085] S164, inputting images in the image samples of at least one modality into the initial model respectively to obtain delineation prediction results and dose distribution prediction results;
[0086] S166, based on the outline prediction results, the standard outline image, the dose distribution prediction results, the standard dose distribution image and the model loss function, the model loss is calculated, and based on the model loss, the parameters of the initial model are updated to obtain the recognition model.
[0087] In this embodiment, the initial model includes at least one initial encoder, an initial delineation decoder and an initial dose distribution decoder, and image samples of at least one modality are obtained. The image samples of each modality include images of the corresponding modality, standard delineation images and standard dose distribution images. The standard delineation images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same. The image samples of each modality correspond to an initial encoder, and the number of modality types of the image samples of at least one modality is equal to the number of initial encoders. The images in the image samples of at least one modality are respectively input into the initial model to obtain delineation prediction results and dose distribution prediction results, including: inputting the images in the image samples of at least one modality into at least one initial encoder respectively to obtain feature maps corresponding to the image samples of at least one modality; when the modality types are more than one, the feature maps corresponding to the image samples of at least one modality are fused to obtain a fused feature sample map; inputting the fused feature sample map into the initial delineation decoder to obtain a delineation prediction result; extracting features from the delineation prediction result to obtain a feature map corresponding to the delineation prediction result; inputting the fused feature sample map and the feature map corresponding to the delineation prediction result into the initial dose distribution decoder to obtain a dose distribution prediction result.Among them, inputting images in image samples of at least one modality into at least one initial encoder respectively to obtain feature maps corresponding to each of the image samples of at least one modality includes inputting images in image samples of any N modalities in the image samples of at least one modality into the initial encoders corresponding to the image samples of the N modalities respectively to obtain feature maps corresponding to the image samples of the N modalities, where N is greater than or equal to 1 and less than or equal to the number of modal types of the image samples of at least one modality; the image samples of at least one modality can also be arranged and combined to obtain multiple groups of image samples, and images in any group of image samples in the multiple groups are respectively input into the initial encoders corresponding to the group of image samples to obtain feature maps corresponding to the multiple groups of image samples. For example, the image samples of at least one modality include image samples of three modalities, and the image samples of the three modalities are arranged and combined to obtain seven groups of image samples, which are three groups of single-modality image samples, three groups of two-modality image samples, and three groups of two-modality image samples. This invention relates to a group of image samples of three modalities, and inputs any group of image samples from the seven groups of image samples into the initial encoder corresponding to the group of image samples respectively, and obtains feature maps corresponding to the seven groups of image samples, wherein, for one group of image samples from the three groups of image samples of two modalities, the images in the two modal image samples in the group of image samples are respectively input into the initial encoder corresponding to the two modal image samples, and the obtained two groups of feature maps are respectively feature maps corresponding to the two modal image samples; for one group of image samples of three modalities, the images in the three modal image samples in the group of image samples are respectively input into the initial encoder corresponding to the three modal image samples, and the obtained three groups of feature maps are respectively feature maps corresponding to the three modal image samples; for one group of image samples of three modalities, the images in the three modal image samples in the group of image samples can also be input in series into any initial encoder, and the obtained one group of feature maps contains the features of the images of the three modalities. This method of arranging and combining image samples of at least one modality and then inputting them into the initial encoder to obtain a feature map can ensure that the features in the extracted image samples of multiple modalities are as similar as possible. The recognition model trained based on the feature maps corresponding to the image samples of multiple modalities ensures that images of multiple modalities can still be processed when the modality type is missing. In the case where the image modality type is not missing, if the method of using multiple modalities images to be serially input into any initial encoder is used, the efficiency of obtaining dose distribution images can be improved.In the case where there are more than one modality type, the feature maps corresponding to the image samples of at least one modality are fused to obtain a fused feature sample map. Specifically, the feature maps corresponding to the image samples of at least one modality are weighted and processed, averaged or serially processed to obtain a fused feature sample map; the fused feature sample map is input into the initial delineation decoder, and the result output by the initial delineation decoder is the delineation prediction result. The delineation prediction result is subjected to feature extraction to obtain the feature map corresponding to the delineation prediction result, and the feature map corresponding to the fused feature sample map and the delineation prediction result is input into the initial dose distribution decoder, and the result output by the initial dose distribution decoder is the dose distribution prediction result. The coding prediction result, the standard delineation image, the delineation prediction result, the standard dose distribution image and the dose distribution prediction result are brought into the model loss function, the model loss is calculated, and the parameters of the initial model are updated using the model loss until the loss function converges to a preset threshold, and the parameter update of the initial model is stopped to obtain the recognition model.
[0088] The scheme of the above-mentioned embodiment obtains image samples of at least one modality, inputs the images in the image samples of at least one modality into at least one initial encoder respectively, obtains the feature maps corresponding to the image samples of at least one modality, and after the feature maps are fused, inputs the fused feature sample maps into the initial delineation decoder and the initial dose distribution decoder respectively, to obtain the delineation prediction result and the dose distribution prediction result respectively. This method of arranging and combining the image samples of at least one modality and then inputting the feature maps into the initial encoder can ensure that the features extracted from the image samples of multiple modalities are as similar as possible. The recognition model trained based on the feature maps corresponding to the image samples of multiple modalities ensures that the images of multiple modalities can be processed even when the modality types are missing. In the case where the image modality types are not missing, if the method of inputting the images of multiple modalities in series into any initial encoder is adopted, the acquisition efficiency of the dose distribution image can be improved. The delineation prediction The results, standard delineation images, dose distribution prediction results and standard dose distribution images are brought into the model loss function, the model loss is calculated, the parameters of the initial model are updated based on the model loss, and the method of obtaining the recognition model is carried out. Image samples of at least one modality are used for image processing, which can effectively utilize the feature information of image samples under each modality and improve the accuracy of the recognition model; at least one initial encoder performs feature extraction and feature fusion on image samples of at least one modality, which can ensure the accuracy of the recognition model; the parameters of the initial model are updated based on the model loss to obtain the recognition model, which can reduce the error between the predicted value and the true value of the recognition model and improve the stability of the recognition model; the method of simultaneously obtaining the delineation prediction results and the dose distribution prediction results through the training of the recognition model, reduces the data volume of the recognition model through the common encoder, and simultaneously obtains the delineation prediction results and the dose distribution prediction results. The idea of multi-task learning is adopted to shorten the prediction time.
[0089] In one embodiment, determining a first loss function corresponding to the encoder, a second loss function corresponding to the delineator, and a third loss function corresponding to the dose distribution decoder comprises:
[0090] The mean absolute error (MAE), mean square error (MSE) or cross entropy is used as the first loss function; the dice loss, Hausdorff distance or intersection over union (IoU) is used as the second loss function; the mean absolute error (MAE) or mean square error (MSE) is used as the third loss function.
[0091] In this embodiment, the type of the first loss function corresponding to the encoder includes any one of the following: mean absolute error MAE, mean square error MSE or cross entropy; the type of the second loss function corresponding to the outline decoder includes any one of the following: dice loss, Hausdorff distance or intersection over Union (IoU); the type of the third loss function corresponding to the dose distribution decoder includes any one of the following: mean absolute error MAE or mean square error MSE.
[0092] The scheme of the above embodiment, by using the mean absolute error MAE, the mean square error MSE or the cross entropy as the first loss function; using the dice loss, the Hausdorff distance or the intersection-over-union ratio IoU as the second loss function; using the mean absolute error MAE or the mean square error MSE as the third loss function, is conducive to selecting different types of loss functions for the encoder, the outline decoder and the dose distribution decoder according to their actual loss conditions. Based on different loss functions, the accuracy of the recognition model can be improved.
[0093] In one embodiment, Figure 6 As shown, the image processing method also includes:
[0094] S620, obtaining an initial outline image of the target area;
[0095] S640, comparing the outlined image with the initial outlined image of the target area;
[0096] S660, if the similarity between the outline image and the initial outline image of the target area exceeds a preset range, the initial outline image of the target area is subjected to feature extraction to obtain a feature map corresponding to the initial outline image of the target area; the fused feature map and the feature map corresponding to the initial outline image of the target area are input into a dose distribution decoder to obtain an updated dose distribution image.
[0097] In this embodiment, the initial outline image of the target area can be an outline image obtained by the doctor outlining the current area. The initial outline image of the target area is compared with the outline image output by the recognition model. If the degree of similarity between the outline image and the initial outline image of the target area is within a preset range, the outline image is subjected to feature extraction to obtain a feature map corresponding to the outline image, and the fused feature map and the feature map corresponding to the outline image are input into the dose distribution decoder to obtain a dose distribution image. If the degree of similarity between the outline image and the initial outline image of the target area exceeds the preset range, it means that the outline image output by the recognition model is similar to the initial outline image of the target area. If the difference between the outline image and the target area is large, the initial outline image of the target area is subjected to feature extraction to obtain a feature map corresponding to the initial outline image of the target area, and the fused feature map and the feature map corresponding to the initial outline image of the target area are input into the dose distribution decoder to obtain an updated dose distribution image. That is, the updated dose distribution image is obtained based on the feature map corresponding to the initial outline image of the target area when the outline image output by the recognition model is large different from the initial outline image of the target area. When the outline image output by the recognition model is large different from the initial outline image of the target area, the updated dose distribution image obtained will have higher accuracy.
[0098] The scheme of the above-mentioned embodiment obtains an initial outline image of the target area, compares the outline image with the initial outline image of the target area, and when the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, an updated dose distribution image is obtained by using a feature map corresponding to the initial outline image of the target area. Since the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, if the dose distribution image is still obtained by using the feature map corresponding to the outline image, the dose distribution image will be inaccurate. In this case, using the feature map corresponding to the initial outline image of the target area to obtain an updated dose distribution image will have higher accuracy.
[0099] To illustrate the image processing method and effect in this solution in detail, a most detailed embodiment is used as an example below, wherein the initial model selects the U-Net neural network model:
[0100] The initial model includes three initial encoders, an initial outline decoder and an initial dose distribution decoder; the mean absolute error MAE, the mean square error MSE or the cross entropy is used as the first loss function corresponding to the initial encoder; the dice loss, the Hausdorff distance or the intersection over union (IoU) is used as the second loss function corresponding to the initial outline decoder; the mean absolute error MAE or the mean square error MSE is used as the third loss function corresponding to the initial dose distribution decoder; the first loss function, the second loss function and the third loss function are weighted averaged to determine the model loss function, and the weights corresponding to the first loss function, the second loss function and the third loss function may be different.
[0101] Image samples of at least one modality are input into the initial model for training to obtain delineation prediction results and dose distribution prediction results. Specifically, the image samples of each modality include images of the corresponding modality, standard delineation images and standard dose distribution images. The standard delineation images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same. The image samples of at least one modality include image samples of three modalities. The image samples of the three modalities are arranged and combined to obtain 7 groups of image samples, which are 3 groups of single-modality image samples, 3 groups of 2-modality image samples and 1 group of 3-modality image samples. Any group of image samples in the 7 groups of image samples is input into the initial encoder corresponding to the group of image samples to obtain feature maps corresponding to the 7 groups of image samples, wherein for one group of image samples in the three groups of 2-modality image samples, the image samples of the two modalities in the group of image samples are combined. For example, the initial encoders corresponding to the two modal image samples are respectively inputted, and the two groups of feature maps obtained are the feature maps corresponding to the two modal image samples respectively; for one group of image samples among the three groups of single-modal image samples, the images in the single-modal image samples in the group of image samples are inputted into the initial encoder corresponding to the modal image samples, and the one group of feature maps obtained is the feature map corresponding to the single-modal image samples; for a group of image samples with three modalities, the images in the three modal image samples in the group of image samples are respectively inputted into the initial encoders corresponding to the three modal image samples, and the three groups of feature maps obtained are the feature maps corresponding to the three modal image samples respectively; in addition, for a group of image samples with three modalities, the images in the three modal image samples in the group of image samples can also be inputted in series into the initial encoder corresponding to any one modal image sample, and the three groups of feature maps obtained are the feature maps corresponding to the three modal image samples respectively. The feature maps corresponding to the image samples of at least one modality are weighted and processed, averaged or connected in series to obtain a fused feature sample map, which is input into an initial delineation decoder. The result output by the initial delineation decoder is the delineation prediction result. The delineation prediction result is subjected to feature extraction to obtain a feature map corresponding to the delineation prediction result. The fused feature sample map and the feature map corresponding to the delineation prediction result are input into an initial dose distribution decoder. The result output by the initial dose distribution decoder is the dose distribution prediction result. The encoding prediction result, the standard delineation image, the delineation prediction result, the standard dose distribution image and the dose distribution prediction result are brought into a model loss function to calculate the model loss. The model loss is used to update the parameters of the initial model until the loss function converges to a preset threshold, and the parameter update of the initial model is stopped to obtain a recognition model. The recognition model includes three encoders, a delineation decoder and a dose distribution decoder.
[0102] like Figure 7FIG. 1 is a flow chart of an image processing method, wherein images of three modalities are obtained, and the images of the three modalities are CT images, MRI images and PET images taken of a target area, and the target area is a tumor treatment target area; the CT image is input into an encoder corresponding to the CT image in the recognition model, namely, encoder 1, to obtain a feature map corresponding to the CT image; the MRI image is input into an encoder corresponding to the MRI image, namely, encoder 2, to obtain a feature map corresponding to the MRI image; the PET image is input into an encoder corresponding to the PET image, namely, encoder 3, to obtain a feature map corresponding to the PET image; the feature maps corresponding to the images of the three modalities are fused by using a weighted sum method to obtain a fused feature map; if images of two modalities are obtained, namely, CT images and PET images taken of the target area, the CT image is input into an encoder corresponding to the MRI image, namely, encoder 2, to obtain a feature map corresponding to the MRI image; the PET image is input into an encoder corresponding to the PET image, namely, encoder 3, to obtain a feature map corresponding to the PET image; the feature maps corresponding to the images of the three modalities are fused by using a weighted sum method to obtain a fused feature map; In the image input recognition model, the encoder corresponding to the CT image, i.e., encoder 1, obtains the feature map corresponding to the CT image, and the PET image is input into the encoder corresponding to the PET image, i.e., encoder 3, to obtain the feature map corresponding to the PET image. The feature maps corresponding to the two modal images are fused by using the weighted sum method to obtain a fused feature map; if two modal images are obtained, i.e., MRI images and PET images taken of the target area, respectively, the MRI image is input into the encoder corresponding to the MRI image, i.e., encoder 2, to obtain the feature map corresponding to the MRI image, and the PET image is input into the encoder corresponding to the PET image, i.e., encoder 3, to obtain the feature map corresponding to the PET image, and the feature maps corresponding to the two modal images are fused by using the weighted sum method to obtain a fused feature map. The fused feature map is input into the delineation decoder to obtain a delineation image; the delineation image is subjected to feature extraction to obtain the feature map corresponding to the delineation image; the fused feature map and the feature map corresponding to the delineation image are input into the dose distribution decoder to obtain a dose distribution image, the delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0103] Furthermore, in the present application, the initial model may also include M initial encoders, where M is greater than or equal to 1, and the recognition model may also include M encoders. Accordingly, the image samples of at least one modality may include image samples of M modalities. In the training process of the recognition model, the image samples of M modalities may be arranged and combined to obtain multiple groups of image samples, and then the images in any group of image samples in the multiple groups of image samples are respectively input into the initial encoder corresponding to the group of image samples to obtain the outline prediction results and the dose distribution prediction results. Each group of image samples in the multiple groups of image samples may also be input in series into any initial encoder to obtain a feature map, which contains the features of the images of M modalities, and then the outline prediction results and the dose distribution prediction results are obtained. Based on the outline prediction results, standard outline images, dose distribution prediction results, standard dose distribution images and model loss function, the model loss is calculated, and based on the model loss, the parameters of the initial model are updated to obtain the recognition model. Figure 8 The figure shows a flow chart of an image processing method when the recognition model includes M encoders, and M kinds of modal images are obtained, wherein the M kinds of modal images are modal 1 images, modal 2 images and modal M images taken of the target area, respectively, and the M kinds of modal images are input into corresponding decoders, respectively, and feature maps corresponding to the M kinds of modal images are obtained, wherein the modal 1 image is input into decoder 1 to obtain a feature map corresponding to the modal 1 image, the modal 2 image is input into decoder 2 to obtain a feature map corresponding to the modal 2 image, and the modal M image is input into decoder M to obtain a feature map corresponding to the modal M image, and the feature maps corresponding to the M kinds of modal images are fused by using a weighted sum or a series method to obtain a fused feature map; the fused feature map is input into a delineation decoder to obtain a delineation image; the delineation image is subjected to feature extraction to obtain a feature map corresponding to the delineation image; the fused feature map and the feature map corresponding to the delineation image are input into a dose distribution decoder to obtain a dose distribution image, the delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0104] In addition, the outline image can also be modified to obtain a modified outline image, and determine whether the degree of similarity between the modified outline image and the outline image is within a preset range. If the outline image has not been modified, or the outline image has been modified and the degree of similarity between the modified outline image and the outline image is within the preset range, the outline image and the dose distribution image can be directly used for radiotherapy plan design; if the outline image has been modified and the degree of similarity between the modified outline image and the outline image exceeds the preset range, the modified outline image is subjected to feature extraction to obtain a feature map corresponding to the modified outline image, the fused feature map and the feature map corresponding to the modified outline image are input into the dose distribution decoder to obtain an updated dose distribution image, and the modified outline image and the updated dose distribution image are used for radiotherapy plan design.
[0105] The above-mentioned image processing method obtains images of at least one modality, inputs the images of at least one modality into a recognition model, and obtains a delineation image and a dose distribution image. The delineation image is used to indicate the outline of the target area, and the dose distribution image is used to indicate the dose at each position of the target area. The characteristic information of the images under each modality can be effectively utilized to improve the accuracy of the delineation image and the dose distribution image; the simultaneous processing of images of at least one modality can improve the acquisition efficiency of the dose distribution image; the delineation image and the dose distribution image are obtained together after the images are input into the recognition model, and the data amount of the recognition model is reduced through a common encoder. The delineation image and the dose distribution image are obtained at the same time, which can shorten the prediction time and further improve the acquisition efficiency of the dose distribution image.
[0106] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0107] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.
[0108] In one embodiment, Fig. 9 As shown, an image processing device 700 is provided, including: an acquisition module 720 and a processing module 740, wherein:
[0109] The acquisition module 720 is used to acquire images of at least one modality.
[0110] The processing module 740 is used to input images of at least one modality into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0111] The above-mentioned image processing device obtains images of at least one modality and inputs the images of at least one modality into a recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the outline of the target area, and the dose distribution image is used to indicate the dose at each position of the target area. The device can effectively utilize the characteristic information of the images under each modality to improve the accuracy of the delineation image and the dose distribution image. The simultaneous processing of images of at least one modality can improve the efficiency of obtaining the dose distribution image. The delineation image and the dose distribution image are obtained together after the images are input into the recognition model, which further improves the efficiency of obtaining the dose distribution image.
[0112] In one embodiment, the recognition model includes at least one encoder, a delineation decoder and a dose distribution decoder; the image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image, and the processing module 740 is also used to: input the image of at least one modality into at least one encoder respectively to obtain a feature map corresponding to each image of at least one modality; when there is more than one type of modality, fuse the feature maps corresponding to each image of at least one modality to obtain a fused feature map; input the fused feature map into the delineation decoder to obtain a delineation image; extract features from the delineation image to obtain a feature map corresponding to the delineation image; input the fused feature map and the feature map corresponding to the delineation image into the dose distribution decoder to obtain a dose distribution image.
[0113] In one embodiment, in terms of fusing the feature maps corresponding to the images of at least one modality to obtain a fused feature map, the processing module 740 is also used to: weight and process the feature maps corresponding to the images of at least one modality to obtain a fused feature map; or, extract feature maps of multiple channels from the images of each modality, weight and process the feature maps corresponding to the images of at least one modality under the same channel to obtain the weighted sum feature map of multiple channels, and use the weighted sum feature map of multiple channels as the fused feature map.
[0114] In one embodiment, in terms of training the recognition model, the image processing device 700 also includes a training module 710, which is also used to: determine a first loss function corresponding to the encoder, a second loss function corresponding to the outline decoder, and a third loss function corresponding to the dose distribution decoder; determine the model loss function based on the first loss function, the second loss function, and the third loss function; and train the initial model based on the model loss function to obtain a recognition model.
[0115] In one embodiment, the training module 710 is also used to: obtain image samples of at least one modality, the image samples of each modality include images of the corresponding modality, standard outline images and standard dose distribution images, the standard outline images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same; input the images in the image samples of at least one modality into the initial model to obtain outline prediction results and dose distribution prediction results; calculate the model loss based on the outline prediction results, standard outline images, dose distribution prediction results, standard dose distribution images and model loss function, and update the parameters of the initial model based on the model loss to obtain the recognition model.
[0116] In one embodiment, in determining the first loss function corresponding to the encoder, the second loss function corresponding to the outline decoder, and the third loss function corresponding to the dose distribution decoder, the training module 710 is also used to: use the mean absolute error MAE, the mean square error MSE or the cross entropy as the first loss function; use the dice loss, the Hausdorff distance or the intersection-over-union ratio IoU as the second loss function; use the mean absolute error MAE or the mean square error MSE as the third loss function.
[0117] In one embodiment, the image processing device 700 is also used to obtain an initial outline image of the target area; compare the outline image with the initial outline image of the target area; if the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, the initial outline image of the target area is subjected to feature extraction to obtain a feature map corresponding to the initial outline image of the target area; the fused feature map and the feature map corresponding to the initial outline image of the target area are input into a dose distribution decoder to obtain an updated dose distribution image.
[0118] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0119] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store images of at least one modality, recognition models, delineation images, dose distribution images, feature maps corresponding to each of the images of at least one modality, fusion feature maps, feature maps corresponding to the delineation images, initial models, image samples of at least one modality, fusion feature sample maps, delineation prediction results, feature maps corresponding to the delineation prediction results, dose distribution prediction results and model losses. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by a processor, an image processing method is implemented.
[0120] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0121] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0122] acquiring images of at least one modality;
[0123] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0124] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0125] The recognition model includes at least one encoder, a delineation decoder and a dose distribution decoder; images of at least one modality are respectively input into at least one encoder to obtain feature maps corresponding to the images of at least one modality; when there is more than one type of modality, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map; the fused feature map is input into the delineation decoder to obtain a delineation image; the delineation image is subjected to feature extraction to obtain a feature map corresponding to the delineation image; the fused feature map and the feature map corresponding to the delineation image are input into the dose distribution decoder to obtain a dose distribution image.
[0126] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0127] The feature maps corresponding to the images of at least one modality are weighted and processed to obtain a fused feature map; or, feature maps of multiple channels are extracted from the images of each modality, and the feature maps corresponding to the images of at least one modality under the same channel are weighted and processed to obtain weighted and feature maps of multiple channels, and the weighted and feature maps of multiple channels are used as the fused feature map.
[0128] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0129] Determine a first loss function corresponding to the encoder, a second loss function corresponding to the outline decoder, and a third loss function corresponding to the dose distribution decoder; determine a model loss function based on the first loss function, the second loss function, and the third loss function; based on the model loss function, train the initial model to obtain a recognition model.
[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0131] Obtain image samples of at least one modality, wherein the image samples of each modality include images of the corresponding modality, standard outline images, and standard dose distribution images, wherein the standard outline images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same; input the images in the image samples of at least one modality into the initial model to obtain outline prediction results and dose distribution prediction results; calculate the model loss based on the outline prediction results, the standard outline images, the dose distribution prediction results, the standard dose distribution images, and the model loss function; based on the model loss, update the parameters of the initial model to obtain the recognition model.
[0132] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0133] The mean absolute error (MAE), mean square error (MSE) or cross entropy is used as the first loss function; the dice loss, Hausdorff distance or intersection over union (IoU) is used as the second loss function; the mean absolute error (MAE) or mean square error (MSE) is used as the third loss function.
[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0135] Acquire an initial outline image of the target area; compare the outline image with the initial outline image of the target area; if the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, extract features from the initial outline image of the target area to obtain a feature map corresponding to the initial outline image of the target area; input the fused feature map and the feature map corresponding to the initial outline image of the target area into a dose distribution decoder to obtain an updated dose distribution image.
[0136] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0137] acquiring images of at least one modality;
[0138] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0139] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0140] The recognition model includes at least one encoder, a delineation decoder and a dose distribution decoder; images of at least one modality are respectively input into at least one encoder to obtain feature maps corresponding to the images of at least one modality; when there is more than one type of modality, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map; the fused feature map is input into the delineation decoder to obtain a delineation image; the delineation image is subjected to feature extraction to obtain a feature map corresponding to the delineation image; the fused feature map and the feature map corresponding to the delineation image are input into the dose distribution decoder to obtain a dose distribution image.
[0141] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0142] The feature maps corresponding to the images of at least one modality are weighted and processed to obtain a fused feature map; or, feature maps of multiple channels are extracted from the images of each modality, and the feature maps corresponding to the images of at least one modality under the same channel are weighted and processed to obtain weighted and feature maps of multiple channels, and the weighted and feature maps of multiple channels are used as the fused feature map.
[0143] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0144] Determine a first loss function corresponding to the encoder, a second loss function corresponding to the outline decoder, and a third loss function corresponding to the dose distribution decoder; determine a model loss function based on the first loss function, the second loss function, and the third loss function; based on the model loss function, train the initial model to obtain a recognition model.
[0145] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0146] Obtain image samples of at least one modality, wherein the image samples of each modality include images of the corresponding modality, standard outline images, and standard dose distribution images, wherein the standard outline images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same; input the images in the image samples of at least one modality into the initial model to obtain outline prediction results and dose distribution prediction results; calculate the model loss based on the outline prediction results, the standard outline images, the dose distribution prediction results, the standard dose distribution images, and the model loss function; based on the model loss, update the parameters of the initial model to obtain the recognition model.
[0147] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0148] The mean absolute error (MAE), mean square error (MSE) or cross entropy is used as the first loss function; the dice loss, Hausdorff distance or intersection over union (IoU) is used as the second loss function; the mean absolute error (MAE) or mean square error (MSE) is used as the third loss function.
[0149] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0150] Acquire an initial outline image of the target area; compare the outline image with the initial outline image of the target area; if the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, extract features from the initial outline image of the target area to obtain a feature map corresponding to the initial outline image of the target area; input the fused feature map and the feature map corresponding to the initial outline image of the target area into a dose distribution decoder to obtain an updated dose distribution image.
[0151] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0152] acquiring images of at least one modality;
[0153] An image of at least one modality is input into the recognition model to obtain a delineation image and a dose distribution image. The delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area.
[0154] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0155] The recognition model includes at least one encoder, a delineation decoder and a dose distribution decoder; images of at least one modality are respectively input into at least one encoder to obtain feature maps corresponding to the images of at least one modality; when there is more than one type of modality, the feature maps corresponding to the images of at least one modality are fused to obtain a fused feature map; the fused feature map is input into the delineation decoder to obtain a delineation image; the delineation image is subjected to feature extraction to obtain a feature map corresponding to the delineation image; the fused feature map and the feature map corresponding to the delineation image are input into the dose distribution decoder to obtain a dose distribution image.
[0156] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0157] The feature maps corresponding to the images of at least one modality are weighted and processed to obtain a fused feature map; or, feature maps of multiple channels are extracted from the images of each modality, and the feature maps corresponding to the images of at least one modality under the same channel are weighted and processed to obtain weighted and feature maps of multiple channels, and the weighted and feature maps of multiple channels are used as the fused feature map.
[0158] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0159] Determine a first loss function corresponding to the encoder, a second loss function corresponding to the outline decoder, and a third loss function corresponding to the dose distribution decoder; determine a model loss function based on the first loss function, the second loss function, and the third loss function; based on the model loss function, train the initial model to obtain a recognition model.
[0160] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0161] Obtain image samples of at least one modality, wherein the image samples of each modality include images of the corresponding modality, standard outline images, and standard dose distribution images, wherein the standard outline images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same; input the images in the image samples of at least one modality into the initial model to obtain outline prediction results and dose distribution prediction results; calculate the model loss based on the outline prediction results, the standard outline images, the dose distribution prediction results, the standard dose distribution images, and the model loss function; based on the model loss, update the parameters of the initial model to obtain the recognition model.
[0162] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0163] The mean absolute error (MAE), mean square error (MSE) or cross entropy is used as the first loss function; the dice loss, Hausdorff distance or intersection over union (IoU) is used as the second loss function; the mean absolute error (MAE) or mean square error (MSE) is used as the third loss function.
[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0165] Acquire an initial outline image of the target area; compare the outline image with the initial outline image of the target area; if the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, extract features from the initial outline image of the target area to obtain a feature map corresponding to the initial outline image of the target area; input the fused feature map and the feature map corresponding to the initial outline image of the target area into a dose distribution decoder to obtain an updated dose distribution image.
[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0169] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An image processing method, characterized in that: The method comprises: acquiring images of at least one modality; Inputting the image of the at least one modality into the recognition model to obtain a delineation image and a dose distribution image, wherein the delineation image is used to indicate the contour of the target area, and the dose distribution image is used to indicate the dose at each position of the target area; The recognition model includes at least one encoder, a delineation decoder, and a dose distribution decoder; The step of inputting the image of the at least one modality into a recognition model to obtain a delineation image and a dose distribution image comprises: Inputting the image of the at least one modality into any one of the same encoders respectively to obtain feature maps corresponding to the image of the at least one modality; Obtaining a fused feature map by fusing the feature maps, and inputting the fused feature maps corresponding to the images of the at least one modality into the delineation decoder to obtain the delineation image; The fusion feature maps corresponding to the images of the at least one modality are input into the dose distribution decoder to obtain the dose distribution image.
2. The method according to claim 1, characterized in that The step of obtaining a fused feature map by fusion of feature maps, and inputting the fused feature maps corresponding to the images of the at least one modality into the delineation decoder to obtain the delineation image comprises: In the case where there is more than one type of modality, the feature maps corresponding to the images of the at least one modality are fused to obtain the fused feature map; Inputting the fused feature map into the delineation decoder to obtain the delineation image; The step of inputting the fusion feature maps corresponding to the images of the at least one modality into the dose distribution decoder to obtain the dose distribution image comprises: Extracting features from the delineated image to obtain a feature map corresponding to the delineated image; The fused feature map and the feature map corresponding to the delineated image are input into the dose distribution decoder to obtain the dose distribution image.
3. The method according to claim 2, characterized in that The step of fusing the feature maps corresponding to the images of the at least one modality to obtain a fused feature map includes: Performing weighted sum processing on the feature maps corresponding to the images of the at least one modality to obtain the fused feature map; Alternatively, feature maps of multiple channels are extracted from the images of each modality, and the feature maps corresponding to the images of at least one modality under the same channel are weighted and processed to obtain weighted and feature maps of multiple channels, and the weighted and feature maps of the multiple channels are used as the fused feature map.
4. The method according to claim 2 or 3, characterized in that: The training step of the recognition model includes: Determining a first loss function corresponding to the encoder, a second loss function corresponding to the delineation decoder, and a third loss function corresponding to the dose distribution decoder; Determine a model loss function according to the first loss function, the second loss function and the third loss function; Based on the model loss function, the initial model is trained to obtain the recognition model.
5. The method according to claim 4, characterized in that The step of training the initial model based on the model loss function to obtain the recognition model includes: Acquire image samples of at least one modality, wherein the image samples of each modality include an image of the corresponding modality, a standard delineation image, and a standard dose distribution image, wherein the standard delineation images in the image samples of various modalities are the same, and the standard dose distribution images in the image samples of various modalities are the same; Inputting images in image samples of at least one modality into the initial model to obtain delineation prediction results and dose distribution prediction results; Based on the delineation prediction result, the standard delineation image, the dose distribution prediction result, the standard dose distribution image and the model loss function, the model loss is calculated, and based on the model loss, the parameters of the initial model are updated to obtain the recognition model.
6. The method according to claim 2, characterized in that The method further comprises: Acquiring an initial delineated image of the target area; comparing the delineated image with an initial delineated image of the target area; If the degree of similarity between the outline image and the initial outline image of the target area exceeds a preset range, the initial outline image of the target area is subjected to feature extraction to obtain a feature map corresponding to the initial outline image of the target area; the fused feature map and the feature map corresponding to the initial outline image of the target area are input into the dose distribution decoder to obtain an updated dose distribution image.
7. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring images of at least one modality; A processing module, used for inputting the image of the at least one modality into the recognition model to obtain a delineation image and a dose distribution image, wherein the delineation image is used for indicating the contour of the target area, and the dose distribution image is used for indicating the dose at each position of the target area; The recognition model includes at least one encoder, a delineation decoder, and a dose distribution decoder; The processing module is specifically used to: input the images of the at least one modality into any one of the same encoders to obtain the feature maps corresponding to the images of the at least one modality; obtain a fused feature map by fusing the feature maps, input the fused feature maps corresponding to the images of the at least one modality into the delineation decoder to obtain the delineation image; input the fused feature maps corresponding to the images of the at least one modality into the dose distribution decoder to obtain the dose distribution image.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
