Postoperative brain glioma image segmentation method and device

Abstract

The invention relates to a postoperative brain glioma image segmentation method and device, and relates to the technical field of image segmentation, and the method comprises the steps: obtaining modal vector information corresponding to a to-be-segmented nuclear magnetic sequence image; determining a primary coding feature tensor of the lowest image feature scale level of the nuclear magnetic sequence image to be segmented based on a SwinTransform module in an encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic sequence image to be segmented, further determining a final coding feature tensor of the highest image feature scale level; and finally determining a primary decoding feature tensor of the lowest image feature scale level through a SwinTransform module in a decoder of the same image feature scale level, and determining a segmentation result corresponding to the nuclear magnetic sequence image to be segmented based on the primary decoding feature tensor. The method has the effect of improving the flexibility of postoperative brain glioma image segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation, and in particular to a method and device for postoperative brain glioma image segmentation. Background Art

[0002] Gliomas are the primary malignant tumors with the highest incidence in the central nervous system of adults. They are highly malignant and difficult to treat, with high disability and recurrence rates, posing a serious threat to the life and health of patients. The growth of gliomas will lead to increased intracranial pressure, and symptoms such as headache, nausea, and vomiting will appear. Furthermore, postoperative radiotherapy for gliomas is a commonly used adjuvant treatment for gliomas. It is used after glioma surgery to kill or inhibit residual tumor cells, thereby prolonging the patient's survival. Among them, postoperative radiotherapy for gliomas mainly uses three-dimensional conformal radiotherapy and conformal intensity-modulated radiotherapy to improve the dose coverage, conformality and protection of normal tissues in the target area of ​​gliomas, reduce unnecessary irradiation volume, and reduce complications. Therefore, it is particularly important to effectively segment the glioma target area from the postoperative glioma images.

[0003] At present, the method commonly used to segment the glioma target area from postoperative glioma images is: through a segmentation model that has the ability to segment the glioma target area from sequence images with existing sequence labels, the sequence images to be segmented are input into the segmentation model to achieve automatic segmentation of the glioma target area, which is convenient for subsequent targeted postoperative radiotherapy of the glioma target area. However, this method only supports the input of sequence images with specified sequence labels. If it is necessary to segment sequence images with newly added sequence labels, the segmentation effect of the segmentation model is poor. The segmentation model needs to be retrained from scratch to better adapt to the new sequence labels, resulting in poor flexibility in the segmentation of postoperative glioma images. Summary of the invention

[0004] In order to improve the flexibility of postoperative brain glioma image segmentation, the present application provides a postoperative brain glioma image segmentation method, device, storage medium and electronic device.

[0005] In a first aspect of the present application, a postoperative brain glioma image segmentation method is provided, which specifically comprises: Acquire modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented, wherein the modal vector information includes a plurality of target modal vectors, wherein the target modal vector is a modal vector obtained by learning glioma-related image features at different image feature scale levels for nuclear magnetic resonance sequence images with the same sequence number as the nuclear magnetic resonance sequence images to be segmented, and the nuclear magnetic resonance sequence images to be segmented are sequence images with a completely new sequence number; Based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence images to be segmented, determining the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence images to be segmented, the encoder and decoder of each image feature scale level in the fine-tuned image segmentation model both include a SwinTransformer module that introduces a target modal vector of the same image feature scale level; Based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, determine the upper coding feature tensor corresponding to the previous image feature scale level, and determine the upper coding feature tensor as the primary coding feature tensor, and repeat the step of determining the upper coding feature tensor corresponding to the previous image feature scale level based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level until the final coding feature tensor of the highest image feature scale level is determined; The final encoded feature tensor is processed by introducing a global Transformer module of a target modal vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor, and according to the highest-level decoded feature tensor, a lower-level decoded feature tensor of the next image feature scale level is determined through a SwinTransformer module in a decoder of the same image feature scale level, wherein the highest-level decoded feature tensor is a decoded feature tensor corresponding to the highest image feature scale level; Determine the lower level decoding feature tensor as the highest level encoding feature tensor, and repeat the step of determining the lower level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest level decoding feature tensor until the primary decoding feature tensor of the lowest image feature scale level is determined; Based on the primary decoded feature tensor, a segmentation result corresponding to the to-be-segmented nuclear magnetic resonance sequence image is determined, wherein the segmentation result is a segmentation result of a brain glioma target area in the to-be-segmented nuclear magnetic resonance sequence image.

[0006] By adopting the above technical solution, after obtaining the modal vector information of the nuclear magnetic resonance sequence images to be segmented, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the nuclear magnetic resonance sequence images to be segmented are encoded from the lowest image feature scale level to the upper level in sequence, and the image features output by the encoder are optimized, so that the image features related to brain glioma in the nuclear magnetic resonance sequence images to be segmented can be more accurately extracted. Furthermore, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the highest-level decoding feature tensor is sequentially decoded from the highest image feature scale level to the lower level, so as to optimize the image features output by the decoder, and finally the fine-tuned image segmentation model can segment the brain glioma target area more accurately when performing image segmentation based on the primary decoding feature tensor, and can adapt to the segmentation of the nuclear magnetic resonance sequence images to be segmented without retraining the model.

[0007] Optionally, the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance image sequence to be segmented, determining the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance image sequence to be segmented, specifically includes: Stretching the to-be-segmented nuclear magnetic resonance sequence image to a preset size and performing normalization processing to obtain a processed image; Performing convolution processing on the processed image to obtain first image features with reduced size, and rearranging the first image features to obtain a corresponding image tensor; Adding a preset position tensor and the image tensor to obtain a corresponding initial feature tensor, wherein the number of rows and columns of elements in the image tensor is the same as that in the position tensor; The initial feature tensor is processed by introducing a SwinTransformer module of the target modal vector corresponding to the same image feature scale level into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented.

[0008] By adopting the above technical solution, after the initial feature tensor is obtained, the SwinTransformer module of the target modal vector corresponding to the same image feature scale level is introduced to process the initial feature tensor. The target modal vector here is learned to the encoder encoding stage of the lowest image feature scale level. The image features of the segmented nuclear magnetic resonance sequence images at the lowest image feature scale level are learned, so that after being processed by this SwinTransformer module, the output image features are optimized.

[0009] Optionally, the determining of the upper-level coded feature tensor corresponding to the previous image feature scale level based on the primary coded feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level specifically includes: Rearranging the primary encoding feature tensor to obtain the reduced-size rearranged image feature, wherein the number of channels of the rearranged image feature is the same as the number of channels of the first image feature; Dividing the rearranged image features into a plurality of sub-blocks of preset pixels, and performing vectorization processing on each of the sub-blocks to obtain a first vector; Performing a linear transformation on each of the first vectors to obtain a second vector after dimension reduction, and rearranging each of the second vectors to obtain a rearranged coded feature tensor of an image feature scale level previous to the lowest image feature scale level; The rearranged encoded feature tensor is processed by a SwinTransformer module that introduces a target modal vector corresponding to the same image feature scale level in an encoder of the previous image feature scale level to obtain an upper-level encoded feature tensor corresponding to the previous image feature scale level.

[0010] By adopting the above technical solution, the rearranged image features are divided into several sub-blocks of preset pixels, so as to realize the division of the local area of ​​the rearranged image features, so as to further perform local analysis and processing, and further, each sub-block is vectorized to obtain a first vector, and each first vector is linearly transformed to obtain a corresponding second vector, while keeping the number of channels unchanged, so as to realize the reduction of the image scale, which is convenient for the subsequent extraction of high-dimensional features. Finally, the rearranged coded feature tensor is processed by introducing the SwinTransformer module of the target modal vector corresponding to the same image feature scale level, so as to optimize the extraction process of higher-dimensional brain glioma-related image features for the segmented nuclear magnetic resonance sequence images.

[0011] Optionally, determining the lower-level decoded feature tensor of the next image feature scale level according to the highest-level decoded feature tensor through a SwinTransformer module in a decoder of the same image feature scale level in the fine-tuned image segmentation model specifically includes: Upsampling the highest-level decoded feature tensor to obtain a sampled result, and performing channel merging of the upsampled result with a rearranged encoded feature tensor at a previous image feature scale level of the highest image feature scale level to obtain a merged result; Performing convolution processing with a step size of 1 on the combined result and rearranging the result to obtain a rearranged decoded feature tensor of the next image feature scale level of the highest image feature scale level; The rearranged decoded feature tensor is processed by introducing a SwinTransformer module corresponding to the target modal vector of the same image feature scale level in the decoder of the next image feature scale level to obtain a lower-level decoded feature tensor corresponding to the next image feature scale level.

[0012] By adopting the above technical solution, the SwinTransformer module that introduces the target modal vector corresponding to the same image feature scale level in the decoder of the next image feature scale level (the next level of the highest image feature scale level) is used to process the rearranged decoded feature tensor to obtain the lower-level decoded feature tensor corresponding to the next image feature scale level of the highest image feature scale level, thereby optimizing the image features output by the decoder.

[0013] Optionally, the obtaining of modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented specifically includes: Acquire at least one sequence image sample having a sequence number identical to the target sequence number of the to-be-segmented nuclear magnetic resonance sequence image, wherein the sequence image sample is a sequence image with a brain glioma target region annotated thereon; Freeze the parameters of the encoder and decoder at each image feature scale level in the pre-trained image segmentation model and introduce the initial modal vector to obtain the image segmentation model to be fine-tuned; Input each of the sequence image samples into the image segmentation model to be fine-tuned for training, and optimize the parameters of the initial modal vector to obtain the target modal vector corresponding to the target sequence label and the fine-tuned image segmentation model and store them in a preset database; When the to-be-segmented nuclear magnetic resonance sequence images need to be segmented, corresponding modality vector information is extracted from the database according to the target sequence label.

[0014] By adopting the above technical solution, the parameters in the encoder and decoder corresponding to each image feature scale level in the pre-trained image segmentation model are frozen, and the corresponding initial modal vector is introduced into the encoder and decoder corresponding to each image feature scale level to obtain the image segmentation model to be fine-tuned, thereby not only ensuring that the MRI images of the original supported sequence or device can still obtain good segmentation results, but also the pre-trained image segmentation model can be fine-tuned only through sequence image samples, and the parameters in the initial modal vector are updated and optimized, so that the initial modal vector can learn the glioma-related image features of different image feature scale levels in the MRI images under the target sequence label, so as to adapt to the MRI images of a new sequence or the MRI images of a new device, without the need to retrain the model from scratch.

[0015] By adopting the above technical solution, the sequence image samples are input into the image segmentation model to be fine-tuned for training to obtain the image segmentation results. Then, the image segmentation results output by the training are minimized through the segmentation result loss function, so that the image segmentation results output during training can gradually approach the annotation information of the target area of ​​glioma, ensuring that the initial modal vector can better learn the image features related to glioma in the sequence images of the target sequence labels, thereby improving the accuracy of the trained model in segmenting the sequence images of the target sequence labels.

[0016] Optionally, the SwinTransformer module of the target modal vector corresponding to the same image feature scale level is introduced into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model to process the initial feature tensor to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented, specifically including: The initial feature tensor is sub-divided into sub-blocks by a SwinTransformer module in an encoder at the lowest image feature scale level to obtain a block result, and a target modal vector corresponding to the same image feature scale level is introduced into the SwinTransformer module in the encoder at the lowest image feature scale level; Merging the block-divided result and the target modal vector corresponding to the encoder at the lowest image feature scale level to obtain a first merged result; Performing autocorrelation processing on the first merged result by a SwinTransformer module in an encoder at the lowest image feature scale level to obtain a first processed result; The output results corresponding to the initial feature tensor are split from the first processed results and rearranged to obtain a primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented.

[0017] By adopting the above technical scheme, the target modal vector of the same image feature scale level is simultaneously introduced into the SwinTransformer module in the encoder of the lowest image feature scale level, that is, the modal vector of the nuclear magnetic resonance image sequence to be segmented at the same image feature scale level corresponding to the nuclear magnetic resonance image sequence to be segmented and the initial feature tensor are learned and autocorrelation processing is performed, so that the SwinTransformer module can better segment the image features related to the brain glioma at the lowest image feature scale level from the initial feature tensor, thereby optimizing the image features output from this SwinTransformer module.

[0018] Optionally, the process of processing the final encoded feature tensor by introducing a global Transformer module of a target modality vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor specifically includes: Merging the final encoded feature tensor with a target modality vector corresponding to the same image feature scale level introduced by a global Transformer module in a decoder of the highest image feature scale level to obtain a second merged result; performing autocorrelation processing on the second merged result through the global Transformer module to obtain a second processed result; The final encoded feature tensor is split from the second processed result and input into the output result after the global Transformer module is processed to obtain the corresponding highest-level decoded feature tensor.

[0019] By adopting the above technical solution, the image features corresponding to the MRI sequence images to be segmented are merged and assembled with the corresponding target modality vectors, and then processed through the Transformer module, so that the decoder can more accurately obtain the features related to brain glioma under different modalities.

[0020] Optionally, determining the segmentation result corresponding to the to-be-segmented nuclear magnetic resonance sequence image based on the primary decoded feature tensor specifically includes: Rearranging the primary decoded feature tensor to obtain the reduced-size second image feature, where the second image feature has the same number of channels as the first image feature; Inputting the second image feature into the output head of the fine-tuned image segmentation model for processing and performing upsampling processing to obtain a restored image of the preset size; The restored image is processed by a Sigmoid activation function to obtain an initial segmentation result of the brain glioma target area; The initial segmentation result of the brain glioma target area is stretched to the original size of the nuclear magnetic resonance image sequence to be segmented, so as to obtain the segmentation result corresponding to the nuclear magnetic resonance image sequence to be segmented.

[0021] By adopting the above technical solution, the primary decoded feature tensor is rearranged, sent to the output head for processing and up-sampled, and restored to the same image size as the input when it was encoded, so as to better restore the original quality and details of the MRI sequence image to be segmented. Furthermore, the restored image is processed by the Sigmoid activation function to achieve the purpose of segmenting the glioma target area. Finally, the initial segmentation result of the glioma target area is stretched to the same size as the MRI sequence image to be segmented, and finally a more accurate segmentation result is obtained.

[0022] In a second aspect of the present application, a postoperative brain glioma image segmentation device is provided, which specifically comprises: An information acquisition module, used for acquiring modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented, wherein the modal vector information includes a plurality of target modal vectors, wherein the target modal vectors are modal vectors obtained by learning glioma-related image features at different image feature scale levels for nuclear magnetic resonance sequence images with the same sequence number as the nuclear magnetic resonance sequence images to be segmented, and the nuclear magnetic resonance sequence images to be segmented are sequence images with completely new sequence numbers; A first encoding module is used to determine the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence images to be segmented based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence images to be segmented, wherein the encoder and decoder of each image feature scale level in the fine-tuned image segmentation model both include a SwinTransformer module that introduces a target modal vector of the same image feature scale level; The second encoding module determines the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, and determines the upper-level encoding feature tensor as the primary encoding feature tensor, and repeatedly executes the step of determining the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level until the final encoding feature tensor of the highest image feature scale level is determined; A first decoding module is used to process the final encoded feature tensor by introducing a global Transformer module of a target modal vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor, and determine a lower-level decoded feature tensor of the next image feature scale level according to the highest-level decoded feature tensor through a SwinTransformer module in a decoder of the same image feature scale level, wherein the highest-level decoded feature tensor is a decoded feature tensor corresponding to the highest image feature scale level; A second decoding module is used to determine the lower-level decoding feature tensor as the highest-level encoding feature tensor, and repeatedly perform the step of determining the lower-level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest-level decoding feature tensor, until the primary decoding feature tensor of the lowest image feature scale level is determined; The image segmentation module is used to determine the segmentation result corresponding to the nuclear magnetic resonance image sequence to be segmented based on the primary decoding feature tensor, and the segmentation result is the segmentation result of the brain glioma target area in the nuclear magnetic resonance image sequence to be segmented.

[0023] By adopting the above technical solution, the information acquisition module obtains the target modal vector, the first encoding module determines the primary encoding feature tensor, the second encoding module gradually determines the upper encoding feature tensor of the previous image feature scale level until the final encoding feature tensor is determined, then the first decoding module determines the lower decoding feature tensor, the second decoding module determines the primary decoding feature tensor of the lowest image feature scale level, and finally the image segmentation module determines the segmentation result corresponding to the nuclear magnetic resonance sequence image to be segmented based on the primary decoding feature tensor.

[0024] In a third aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps described in any one of the first aspects are performed.

[0025] In a fourth aspect of the present application, an electronic device is provided, specifically comprising: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the processor is used to load and execute the computer program stored in the memory so that the electronic device performs the method as described in any one of the first aspects.

[0026] In summary, the present application includes at least one of the following beneficial technical effects: after obtaining the modal vector information of the nuclear magnetic resonance sequence images to be segmented, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the nuclear magnetic resonance sequence images to be segmented are encoded step by step from the lowest image feature scale level upwards, and the image features output by the encoder are optimized, so that the image features related to brain glioma in the nuclear magnetic resonance sequence images to be segmented can be more accurately extracted. Furthermore, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the highest-level decoded feature tensor is decoded from the highest image feature scale level downwards in sequence, so as to optimize the image features output by the decoder, and finally the fine-tuned image segmentation model can segment the brain glioma target area more accurately when performing image segmentation based on the primary decoded feature tensor, and the segmentation of the nuclear magnetic resonance sequence images to be segmented can be adapted without retraining the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of a postoperative brain glioma image segmentation method provided in an embodiment of the present application; Figure 2 It is a flowchart of another postoperative brain glioma image segmentation method provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a postoperative brain glioma image segmentation device provided in an embodiment of the present application.

[0028] Explanation of the accompanying drawings: 11. Information acquisition module; 12. First encoding module; 13. Second encoding module; 14. First decoding module; 15. Second decoding module; 16. Image segmentation module. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0030] In the description of the embodiments of the present application, words such as "illustrative", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "illustrative", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "illustrative", "for example" or "for example" is intended to present related concepts in a concrete way.

[0031] In the description of the embodiments of the present application, the term "and / or" is only a kind of association relationship describing the associated objects, indicating that there may be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0032] See also Figure 1The present application embodiment discloses a flow chart of a postoperative brain glioma image segmentation method, which can be implemented by a computer program or run on a postoperative brain glioma image segmentation device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application, specifically including: S101: Obtaining modal vector information corresponding to the nuclear magnetic resonance image sequence to be segmented.

[0033] Specifically, nuclear magnetic images are medical imaging data obtained from magnetic resonance imaging (MRI) scans, including features such as morphology, signal intensity, and tissue structure. Nuclear magnetic images are divided into nuclear magnetic images of different sequence types. These sequence types are mainly based on the differences in pulse sequences in magnetic resonance imaging technology. Each sequence has its specific application scenarios and advantages, which are used to enhance specific features of the image or highlight certain tissue structures, including but not limited to T1 sequence nuclear magnetic images, T2 sequence nuclear magnetic images, (Diffusion Weighted Imaging, DWI) sequence nuclear magnetic images, etc. Further, nuclear magnetic sequence images refer to nuclear magnetic images of a single sequence type. In an embodiment of the present application, the nuclear magnetic sequence image to be segmented refers to the nuclear magnetic image of a newly added sequence or the nuclear magnetic image of a new device that needs to segment out the target area of ​​glioma, that is, the sequence image with a completely new sequence number, which can be a 2D type image. In other embodiments, it can also be a 2.5D type image or a 3D type image.

[0034] Among them, the glioma target area refers to the location of the tumor tissue in the brain, and it is also the area that needs to be focused on during radiotherapy and surgical treatment. In the embodiment of the present application, the glioma target area includes but is not limited to two types of glioma target areas: gross tumor target volume (GrossTarget Volume, GTV) and clinical target volume (Clinical Target Volume, CTV). Among them, the GTV target area is the macroscopic tumor tissue and the area that may be invaded; the CTV target area is based on the GTV, taking into account the possible local spread of tumor cells and surrounding normal tissues, and is appropriately expanded to obtain the area. In other embodiments, the glioma target area may also include an internal target volume (Internal Target Volume, ITV), etc.

[0035] Further, the modal vector information can be understood as a set of modal vectors of the MRI sequence images to be segmented, which includes multiple target modal vectors, specifically modal vectors completed by learning image features related to gliomas at different image feature scale levels of MRI sequence images with the same sequence number as the MRI sequence images to be segmented. Among them, the image feature scale level refers to the representation and use of image features of different sizes in image processing and computer vision, which means extracting image features at different scales or levels. The modal vector refers to a learnable variable in the encoder and decoder in a deep learning model, which can learn and extract features from multimodal input data, and can continuously optimize parameters as the model is trained. In an embodiment of the present application, the target modal vector is capable of accurately extracting features related to gliomas in MRI sequence images to be segmented.

[0036] Furthermore, a feasible way to obtain the modal vector information is to obtain the target sequence number of the MRI sequence image to be segmented through a Picture Archiving and Communication System (PACS) connected to the medical imaging device. Exemplarily, the sequence number corresponding to the T1 sequence MRI image is T1. Furthermore, the modal vector information corresponding to the target sequence number is matched from the modal vector matching table preset in the database, and the modal vector matching table includes different sequence numbers and corresponding modal vector information.

[0037] S102: Based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance image sequence to be segmented, determine the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance image sequence to be segmented.

[0038] Specifically, in an embodiment of the present application, the sequence label of the nuclear magnetic resonance sequence image to be segmented is determined as the target sequence label, and the fine-tuned image segmentation model is a deep learning model that can automatically segment the target area of ​​glioma from a sequence image with multiple sequence labels including the target sequence label. The image feature scale level is divided into 5 levels from low to high. In other embodiments, it can also be other reasonable scale levels. In addition, in deep learning, features are usually extracted at different scales through convolution kernels or pooling operations of different sizes. The lower the image feature scale level, the lower the dimensional features are extracted at a larger scale, and the higher the image feature scale level, the higher the dimensional features are extracted at a smaller scale. In addition, the encoders and decoders of each image feature scale level in the fine-tuned image segmentation model include a SwinTransformer module that introduces a target modal vector of the same image feature scale level.

[0039] Further, the nuclear magnetic resonance sequence image to be segmented is stretched to a preset size and normalized to obtain a processed image. Exemplarily, the preset size is N*N. Then, the processed image is input into the fine-tuned image segmentation model. Based on the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence image to be segmented, the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented is determined. The specific process is: convolution processing is performed on the processed image. A 2D convolution processing with a kernel size of k*k and a stride of K can be used to extract image features in the processed image. Finally, the first image feature with a reduced size is obtained. The reduced size is Characterization, further expressed as the C channel In the embodiment of the present application, C takes the value of 4 and k takes the value of 4. In other embodiments, C also takes the value of 1 and k takes the value of 2. Further, the obtained first image features are rearranged to obtain Then, the preset position tensor, specifically The position tensor of row C and column is added to the image tensor to obtain the corresponding initial feature tensor, and then the initial feature tensor is input into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model, and the initial feature tensor is processed by introducing the SwinTransformer module of the target modal vector of the same image feature scale level, and finally the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented is obtained, so as to optimize the output image features, so as to more accurately obtain the image features related to brain glioma at the scale corresponding to the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented. Among them, the specific process of obtaining the primary encoding feature tensor through the SwinTransformer module is: at the odd level, the initial feature tensor is divided into normal sub-blocks through the SwinTransformer module, and at the even level, the initial feature tensor needs to be cyclically shifted right by half a window and then processed in blocks. Furthermore, the block-divided result and the corresponding target modal vector are merged to obtain the first merged result. Then, the first merged result is autocorrelatedly processed by the SwinTransformer module in the encoder at the lowest image feature scale level to obtain the first processed result. Finally, the output result of this initial feature tensor after passing through the SwinTransformer module is split from the first processed result. At the odd level, the primary coding feature tensor is obtained after rearrangement; at the even level, the rearrangement is cyclically shifted to the left by half a window for restoration, and finally the primary coding feature tensor is obtained, thereby achieving more accurate acquisition of brain glioma-related features at the lowest image feature scale level. Among them, the SwinTransformer module is a new type of Transformer architecture with hierarchical feature representation and self-attention mechanism of displacement window, which realizes efficient image processing capabilities.

[0040] S103: Based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, determine the upper-level coding feature tensor corresponding to the previous image feature scale level, and determine the upper-level coding feature tensor as the primary coding feature tensor, and repeat the step of determining the upper-level coding feature tensor corresponding to the previous image feature scale level based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level until the final coding feature tensor of the highest image feature scale level is determined.

[0041] Specifically, after the primary coding feature tensor is determined, in the embodiment of the present application, a feasible way to determine the upper coding feature tensor corresponding to the upper image feature scale level is: rearrange the primary coding feature tensor into a rearranged image feature with a reduced size, that is, the C channel The image feature has the same size and number of channels as those corresponding to the first image feature. Further, the rearranged image feature is divided into several sub-blocks of preset pixels. In the embodiment of the present application, the preset pixels are 2*2, which can be understood as each sub-block containing a 2*2 pixel area in the rearranged image feature, thereby realizing the division of the local area of ​​the rearranged image feature, and further local analysis and processing. Further, each sub-block is vectorized to form a vector with a length of 2*2*C, that is, the corresponding first vector, thereby realizing the reduction of the image scale and facilitating the subsequent extraction of high-dimensional features. Further, each first vector is linearly transformed to obtain the corresponding second vector, which is specifically expressed as Proj 2*2*C→2*C , that is, the mapping of 2*2*C dimensional vector to 2*C dimensional vector, while keeping the number of channels unchanged, so as to better extract features. Further, each second vector is rearranged to obtain the rearranged coded feature tensor of the previous image feature scale level of the lowest image feature scale level, specifically The rearranged encoded feature tensor of 2 rows and C columns. It should be noted that the previous image feature scale level refers to the image feature scale level one level higher than the lowest image feature scale level. In this image feature scale level, it means that features with higher dimensions can be extracted at a smaller scale.

[0042] Further, by introducing the target modal vector corresponding to the same image feature scale level (the modal vector that completes the learning of the image features of the same image feature scale level in the encoder encoding stage) into the encoder of the previous image feature scale level, the SwinTransformer module is processed to obtain the upper-level coded feature tensor corresponding to the previous image feature scale level. The processing process can be specifically referred to step S102, which will not be repeated here. Thereby, the extraction of higher-dimensional brain glioma-related image features for the segmented nuclear magnetic resonance sequence image is realized. Further, the upper-level coded feature tensor is re-determined as a primary coded feature tensor, and then the steps of the encoder based on the primary coded feature tensor and the previous image feature scale level are repeatedly executed, that is, according to the upper-level coded feature tensor and the encoder of the previous image feature scale level, the coded feature tensor corresponding to the upper-level image feature scale level of the upper-level coded feature tensor is continued to be determined, so as to extract higher-dimensional image features until the final coded feature tensor of the highest image feature scale level is determined.

[0043] S104: The final encoded feature tensor is processed by introducing a global Transformer module of the target modal vector corresponding to the same image feature scale level in the decoder of the highest image feature scale level to obtain the corresponding highest-level decoded feature tensor, and based on the highest-level decoded feature tensor, the lower-level decoded feature tensor of the next image feature scale level is determined through the SwinTransformer module in the decoder of the same image feature scale level.

[0044] Specifically, after the final encoding feature tensor is determined, the decoder decoding stage is entered, and then the final encoding feature tensor is processed by the decoder of the highest image feature scale level in the fine-tuned image segmentation model, and specifically, the final encoding feature tensor is processed by the global Transformer module that introduces the target modal vector corresponding to the same image feature scale level, and the corresponding highest-level decoding feature tensor is obtained, that is, the decoding feature tensor corresponding to the highest image feature scale level. The specific process is: the final encoding feature tensor is merged with the target modal vector corresponding to the same image feature scale level introduced by the global Transformer module in the decoder of the highest image feature scale level to obtain a second merged result, and then the second merged result is autocorrelatedly processed by the global Transformer module to obtain a second processed result, and finally the final encoding feature tensor is split from the second processed result and input into the output result after processing by the global Transformer module to obtain the corresponding highest-level decoding feature tensor, thereby optimizing the image features output by the global Transformer module.

[0045] Among them, the global Transformer module is a neural network architecture that relies on the attention mechanism to establish a global dependency relationship between input and output. Further, based on the decoder of the same image feature scale level in the fine-tuned image segmentation model, that is, the decoder of the next image feature scale level of the highest image feature scale level and the highest-level decoded feature tensor, the lower-level decoded feature tensor of the next image feature scale level is determined. The specific process is: up-sampling the highest-level decoded feature tensor through the decoder corresponding to the next image feature scale level of the highest image feature scale level to obtain the corresponding sampled result, and further, the sampled result is channel-merged with the rearranged encoded feature tensor of the previous image feature scale level of the highest image feature scale level, specifically through the Concate operation to obtain the merged result.

[0046] Further, the combined result is subjected to convolution processing with a step size of 1, specifically, a convolution processing with a convolution kernel of 1*1 and a step size of 1, to obtain a rearranged decoded feature tensor of the next image feature scale level of the highest image feature scale level. Then, the rearranged decoded feature tensor is processed by the SwinTransformer module that introduces the target modal vector corresponding to the same image feature scale level in the decoder of the next image feature scale level (the next level of the highest image feature scale level), to obtain a lower-level decoded feature tensor corresponding to the next image feature scale level of the highest image feature scale level.

[0047] It should be noted that the SwinTransformer modules in the encoders and decoders of different image feature scale levels and the global Transformer module in the decoder of the highest image feature scale level in the present application are different from the classic SwinTransformer modules and global Transformer modules in the prior art. The present application introduces the target modal vector corresponding to the same image feature scale level and the image feature information for autocorrelation processing, and finally extracts the encoding feature tensor and decoding feature tensor results of the corresponding image feature scale category from the obtained processing results.

[0048] S105: Determine the lower-level decoding feature tensor as the highest-level encoding feature tensor, and repeat the steps of determining the lower-level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest-level decoding feature tensor, until the primary decoding feature tensor of the lowest image feature scale level is determined.

[0049] Specifically, after the lower-level decoding feature tensor is determined, the lower-level decoding feature tensor is re-determined as the highest-level encoding feature tensor, and then the steps of determining the lower-level decoding feature tensor of the next image feature scale level based on the decoder and the highest-level decoding feature tensor at the same image feature scale level in the fine-tuned image segmentation model are repeated, thereby determining the decoding feature tensor of the next image feature scale level of this lower-level decoding feature tensor, and so on, until the primary decoding feature tensor of the lowest image feature scale level is determined.

[0050] S106: Determine the segmentation result corresponding to the nuclear magnetic resonance sequence image to be segmented based on the primary decoded feature tensor.

[0051] Specifically, after the primary decoded feature tensor is determined, the primary decoded feature tensor is rearranged into a second image feature with the same scaled size as the first image feature and the same number of channels, represented as C channel The second image feature is obtained, and then the second image feature is input into the output head of the fine-tuned image segmentation model for processing, and the C channel of the second image feature is transformed into an output category channel, wherein the convolution kernel contained in the output head is 1*1 and the step size is 1. Further, the processed result of the output head is upsampled, specifically the upsampling with the convolution kernel being k*k and the step size being k is adopted, and the second image feature is restored to a preset size to obtain a restored image. Finally, the restored image is processed by the Sigmoid activation function in the fine-tuned image segmentation model to obtain the initial segmentation result of the glioma target area, that is, an image of a preset size containing the GTV target area and an image of a preset size containing the CTV target area. Finally, the initial segmentation result of the glioma target area is stretched to the original size of the nuclear magnetic resonance sequence image to be segmented, and finally the segmentation result corresponding to the nuclear magnetic resonance sequence image to be segmented is obtained.

[0052] See also Figure 2 The present application embodiment discloses a flowchart of another method for postoperative glioma image segmentation, which can be implemented by a computer program or run on a postoperative glioma image segmentation device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application, specifically including: S201: Acquire at least one sequence image sample whose sequence number is the same as the target sequence number of the nuclear magnetic resonance sequence image to be segmented, wherein the sequence image sample is a sequence image with a brain glioma target area marked thereon.

[0053] S202: Freeze the parameters in the encoder and decoder of each image feature scale level in the pre-trained image segmentation model and introduce an initial modal vector to obtain the image segmentation model to be fine-tuned.

[0054] Specifically, if the MRI sequence images to be segmented are MRI images of a newly added sequence or MRI images of a new device, that is, MRI sequence images to be segmented, the pre-trained image segmentation model may not be able to perform glioma target area segmentation well for such MRI sequence images, because the sequence image samples used to pre-train the image segmentation model do not include sequence images with the target sequence number of the MRI sequence images to be segmented. In order to avoid retraining the image segmentation model to adapt to a completely new sequence of MRI images or MRI images of a completely new device, at least one sequence image of the same type with the same sequence number as the target sequence number is retrieved from the PACS system, and then the doctor manually outlines or marks the glioma target area in the same type of sequence images to obtain the corresponding sequence image samples.

[0055] Furthermore, since the encoder and decoder in the pre-trained image segmentation model already have the ability to adapt to sequence images with multiple sequence labels, that is, the ability to segment glioma target areas from sequence images with multiple sequence labels, the encoder and decoder corresponding to each image feature scale level in the pre-trained image segmentation model are frozen, and the corresponding initial modal vector is introduced into the encoder and decoder corresponding to each image feature scale level to obtain the image segmentation model to be fine-tuned, wherein the initial modal vector is a modal vector that has not yet been learned for glioma-related image features. This not only ensures that the nuclear magnetic resonance images of the original supported sequence or device can still obtain good segmentation results, but also the pre-trained image segmentation model can be fine-tuned only through sequence image samples in the future, so that the initial modal vector learns the glioma-related image features of different image feature scale levels in the nuclear magnetic resonance images under the target sequence label, so as to adapt to the nuclear magnetic resonance images of a new sequence or the nuclear magnetic resonance images of a new device, without the need to retrain the model from scratch.

[0056] S203: Input each sequence image sample into the image segmentation model to be fine-tuned for training, and optimize the parameters of the initial modal vector to obtain the target modal vector corresponding to the target sequence label and the fine-tuned image segmentation model and store them in a preset database.

[0057] S204: When the MRI sequence image to be segmented needs to be segmented, corresponding modality vector information is extracted from the database according to the target sequence label.

[0058] Specifically, after the image segmentation model to be fine-tuned is determined, each sequence image sample is input into the image segmentation model to be fine-tuned for training. In the embodiment of the present application, the training process is as follows: a single sequence image sample is input into the image segmentation model to be fine-tuned to obtain the corresponding image segmentation result. The intermediate process can refer to steps S101-S106, which will not be repeated here. Furthermore, the image segmentation result is substituted into the preset segmentation result loss function to obtain the corresponding loss value, and then the loss value is minimized until the image segmentation model to be fine-tuned converges. During the process, the parameters of the introduced initial modal vector are optimized by optimization algorithms such as back propagation algorithm and gradient descent. This is a prior art and will not be repeated here. Finally, the learned target modal vector corresponding to the target sequence label and the fine-tuned image segmentation model are obtained and stored in a preset database. Among them, the segmentation result loss function is: ; In the formula, Loss represents the loss value, nClass represents the number of types of glioma target areas that need to be segmented, S represents the number of sequence image samples, i represents the i-th type of glioma target area, s represents the s-th sequence image sample, represents the labeling information of the sth sequence image sample for the i-th type of glioma target area, represents the image segmentation result of the i-th type of glioma target area determined by the image segmentation model to be fine-tuned for the s-th sequence image sample, and the parameter and is a non-negative parameter, BCE represents the binary cross entropy function, and DSC represents the Dyss similarity coefficient loss function.

[0059] Finally, when it is necessary to perform image segmentation on the segmented MRI sequence image and determine the target area of ​​glioma, the corresponding target modal vectors are directly extracted from the database according to the corresponding target sequence label and summarized into modal vector information.

[0060] S205: Based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance image sequence to be segmented, determine the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance image sequence to be segmented.

[0061] S206: Based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, determine the upper-level coding feature tensor corresponding to the previous image feature scale level, and determine the upper-level coding feature tensor as the primary coding feature tensor, and repeat the step of determining the upper-level coding feature tensor corresponding to the previous image feature scale level based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level until the final coding feature tensor of the highest image feature scale level is determined.

[0062] S207: The final encoded feature tensor is processed by introducing a global Transformer module of the target modal vector corresponding to the same image feature scale level in the decoder of the highest image feature scale level to obtain the corresponding highest-level decoded feature tensor, and based on the highest-level decoded feature tensor, the lower-level decoded feature tensor of the next image feature scale level is determined through the SwinTransformer module in the decoder of the same image feature scale level.

[0063] S208: Determine the lower-level decoding feature tensor as the highest-level encoding feature tensor, and repeat the steps of determining the lower-level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest-level decoding feature tensor, until the primary decoding feature tensor of the lowest image feature scale level is determined.

[0064] S209: Determine the segmentation result corresponding to the nuclear magnetic resonance sequence image to be segmented based on the primary decoded feature tensor.

[0065] Specifically, please refer to steps S101-S106, which will not be repeated here. It should be noted that a postoperative brain glioma image segmentation method disclosed in an embodiment of the present application introduces a target modal vector in the SwinTransformer module and the Transformer module to describe the image features related to brain glioma in the sequence images of each newly added sequence, thereby achieving to a certain extent the decoupling of the image features of the nuclear magnetic resonance images that need to be segmented from the sequence image features of the newly added sequence. When it is necessary to segment the sequence images of the newly added sequence, the network parameters in the encoder and decoder of the pre-trained image segmentation model are frozen, and only the sequence images of the newly added sequence and the annotation information about the target area of ​​the brain glioma are needed to train the pre-trained image segmentation model, and optimize the parameter information of the modal vector corresponding to the corresponding newly added sequence, thereby paying a smaller model training cost, ensuring the sequence image segmentation effect of the original sequence, and realizing the sequence image segmentation requirements of the newly added sequence more quickly. Among them, the sequence image can be regarded as a sequence nuclear magnetic resonance image.

[0066] The implementation principle of a postoperative brain glioma image segmentation method in an embodiment of the present application is as follows: after obtaining the modal vector information of the nuclear magnetic resonance sequence image to be segmented, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the nuclear magnetic resonance sequence image to be segmented is sequentially encoded from the lowest image feature scale level upwards, and the image features output by the encoder are optimized, so that the image features related to the brain glioma in the nuclear magnetic resonance sequence image to be segmented can be more accurately extracted. Furthermore, the SwinTransformer module of the target modal vector of the same image feature scale level is introduced into the encoder of the same image feature scale level, and the highest-level decoded feature tensor is sequentially decoded from the highest image feature scale level downwards, so as to optimize the image features output by the decoder, and finally the fine-tuned image segmentation model can segment the brain glioma target area more accurately when performing image segmentation based on the primary decoded feature tensor, and the segmentation of the nuclear magnetic resonance sequence image to be segmented can be adapted without retraining the model.

[0067] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0068] See also Figure 3, is a schematic diagram of the structure of a postoperative brain glioma image segmentation device provided in an embodiment of the present application. The postoperative brain glioma image segmentation device can be implemented as all or part of the device through software, hardware or a combination of both. The device includes an information acquisition module 11, a first encoding module 12, a second encoding module 13, a first decoding module 14, a second decoding module 15 and an image segmentation module 16.

[0069] The information acquisition module 11 is used to acquire the modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented, the modal vector information including a plurality of target modal vectors, the target modal vector being a modal vector obtained by learning the image features related to glioma at different image feature scale levels of the nuclear magnetic resonance sequence images with the same sequence number as the nuclear magnetic resonance sequence images to be segmented, wherein the nuclear magnetic resonance sequence images to be segmented are sequence images with a completely new sequence number; A first encoding module 12 is used to determine the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence images to be segmented based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence images to be segmented, and the encoder and decoder of each image feature scale level in the fine-tuned image segmentation model both include a SwinTransformer module that introduces a target modal vector of the same image feature scale level; The second encoding module 13 determines the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, and determines the upper-level encoding feature tensor as the primary encoding feature tensor, and repeatedly executes the step of determining the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, until the final encoding feature tensor of the highest image feature scale level is determined; The first decoding module 14 is used to process the final encoded feature tensor by introducing a global Transformer module of a target modal vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor, and determine a lower-level decoded feature tensor of the next image feature scale level according to the highest-level decoded feature tensor through a SwinTransformer module in a decoder of the same image feature scale level, wherein the highest-level decoded feature tensor is a decoded feature tensor corresponding to the highest image feature scale level; A second decoding module 15 is used to determine the lower-level decoding feature tensor as the highest-level encoding feature tensor, and repeatedly perform the steps of determining the lower-level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest-level decoding feature tensor, until the primary decoding feature tensor of the lowest image feature scale level is determined; The image segmentation module 16 is used to determine the segmentation result corresponding to the nuclear magnetic resonance image sequence to be segmented based on the primary decoded feature tensor, and the segmentation result is the segmentation result of the brain glioma target area in the nuclear magnetic resonance image sequence to be segmented.

[0070] Optionally, the first encoding module 12 is specifically configured to: The nuclear magnetic resonance image to be segmented is stretched to a preset size and normalized to obtain a processed image; Performing convolution processing on the processed image to obtain first image features with reduced size, and rearranging the first image features to obtain a corresponding image tensor; Add the preset position tensor and image tensor to obtain the corresponding initial feature tensor. The number of rows and columns of elements in the image tensor and the position tensor are the same. The initial feature tensor is processed by introducing the SwinTransformer module of the target modal vector corresponding to the same image feature scale level into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance image sequence to be segmented.

[0071] Optionally, the second encoding module 13 is specifically used for: Rearranging the primary encoding feature tensor to obtain a rearranged image feature of reduced size, wherein the number of channels of the rearranged image feature is the same as the number of channels of the first image feature; Dividing the rearranged image features into a plurality of sub-blocks of preset pixels, and performing vectorization processing on each sub-block to obtain a first vector; Performing a linear transformation on each first vector to obtain a second vector after dimension reduction, and rearranging each second vector to obtain a rearranged coded feature tensor of an image feature scale level previous to the lowest image feature scale level; The rearranged encoded feature tensor is processed by the SwinTransformer module that introduces the target modal vector corresponding to the same image feature scale level in the encoder of the previous image feature scale level to obtain the upper-level encoded feature tensor corresponding to the previous image feature scale level.

[0072] Optionally, the first decoding module 14 is specifically configured to: Upsampling the highest level decoded feature tensor to obtain a sampled result, and merging the upsampled result with the rearranged encoded feature tensor of the previous image feature scale level of the highest image feature scale level to obtain a merged result; The combined results are convolved with a step size of 1 and rearranged to obtain a rearranged decoded feature tensor of the next image feature scale level of the highest image feature scale level; The rearranged decoded feature tensor is processed by introducing the SwinTransformer module corresponding to the target modal vector of the same image feature scale level in the decoder of the next image feature scale level to obtain the lower-level decoded feature tensor corresponding to the next image feature scale level.

[0073] Optionally, the information acquisition module 11 is specifically used for: Acquire at least one sequence image sample whose sequence number is the same as the target sequence number of the nuclear magnetic resonance sequence image to be segmented, wherein the sequence image sample is a sequence image with a brain glioma target area marked; Freeze the parameters of the encoder and decoder at each image feature scale level in the pre-trained image segmentation model and introduce the initial modal vector to obtain the image segmentation model to be fine-tuned; Input each sequence image sample into the image segmentation model to be fine-tuned for training, and optimize the parameters of the initial modal vector to obtain the target modal vector corresponding to the target sequence label and the fine-tuned image segmentation model and store them in a preset database; When it is necessary to segment the MRI sequence images to be segmented, the corresponding modal vector information is extracted from the database according to the target sequence label.

[0074] Optionally, the first encoding module 12 is specifically used for: The initial feature tensor is sub-divided into sub-blocks through the SwinTransformer module in the encoder of the lowest image feature scale level to obtain the block-divided result, and the target modal vector corresponding to the same image feature scale level is introduced into the SwinTransformer module in the encoder of the lowest image feature scale level; Merging the block-divided result with the target modality vector corresponding to the encoder at the lowest image feature scale level to obtain a first merged result; Performing autocorrelation processing on the first merged result by a SwinTransformer module in an encoder at the lowest image feature scale level to obtain a first processed result; The output results corresponding to the initial feature tensor are split from the first processed result and rearranged to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented.

[0075] Optionally, the first decoding module 14 is specifically configured to: Merge the final encoded feature tensor with the target modality vector corresponding to the same image feature scale level introduced by the global Transformer module in the decoder of the highest image feature scale level to obtain the merged result; The merged result is processed through the global Transformer module to obtain the processed result; The final encoded feature tensor is split from the processed result and input into the output result after processing by the global Transformer module to obtain the corresponding highest-level decoded feature tensor.

[0076] Optionally, the image segmentation module 16 is specifically used for: Rearranging the primary decoded feature tensor to obtain a second image feature with a reduced size, wherein the second image feature has the same number of channels as the first image feature; Inputting the second image feature into the output head of the fine-tuned image segmentation model for processing and performing upsampling processing to obtain a restored image of a preset size; The restored image is processed by Sigmoid activation function to obtain the initial segmentation result of the brain glioma target area; The initial segmentation result of the brain glioma target area is stretched to the original size of the nuclear magnetic resonance image sequence to be segmented, and the segmentation result corresponding to the nuclear magnetic resonance image sequence to be segmented is obtained.

[0077] It should be noted that the postoperative brain glioma image segmentation device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the postoperative brain glioma image segmentation method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the postoperative brain glioma image segmentation device provided in the above embodiment and the postoperative brain glioma image segmentation method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0078] The embodiment of the present application further discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, a postoperative brain glioma image segmentation method of the above embodiment is adopted.

[0079] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0080] Among them, through this computer-readable storage medium, a postoperative brain glioma image segmentation method of the above embodiment is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0081] An embodiment of the present application further discloses an electronic device, wherein a computer program is stored in a computer-readable storage medium, and when the computer program is loaded and executed by a processor, the above-mentioned postoperative brain glioma image segmentation method is adopted.

[0082] The electronic device may be a desktop computer, a laptop computer, a cloud server or other electronic device, and the electronic device includes but is not limited to a processor and a memory. For example, the electronic device may also include input and output devices, a network access device, and a bus.

[0083] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0084] Among them, the memory can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device, or it can be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the electronic device. Moreover, the memory can also be a combination of an internal storage unit and an external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0085] Among them, through this electronic device, a postoperative brain glioma image segmentation method of the above embodiment is stored in the memory of the electronic device, and is loaded and executed on the processor of the electronic device for easy use.

[0086] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A postoperative brain glioma image segmentation method, characterized in that: The method comprises: Acquire modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented, wherein the modal vector information includes a plurality of target modal vectors, wherein the target modal vector is a modal vector obtained by learning glioma-related image features at different image feature scale levels for nuclear magnetic resonance sequence images with the same sequence number as the nuclear magnetic resonance sequence images to be segmented, and the nuclear magnetic resonance sequence images to be segmented are sequence images with a completely new sequence number; Based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence images to be segmented, determining the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence images to be segmented, the encoder and decoder of each image feature scale level in the fine-tuned image segmentation model both include a SwinTransformer module that introduces a target modal vector of the same image feature scale level; Based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, determine the upper coding feature tensor corresponding to the previous image feature scale level, and determine the upper coding feature tensor as the primary coding feature tensor, and repeat the step of determining the upper coding feature tensor corresponding to the previous image feature scale level based on the primary coding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level until the final coding feature tensor of the highest image feature scale level is determined; The final encoded feature tensor is processed by introducing a global Transformer module of a target modal vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor, and according to the highest-level decoded feature tensor, a lower-level decoded feature tensor of the next image feature scale level is determined through a SwinTransformer module in a decoder of the same image feature scale level, wherein the highest-level decoded feature tensor is a decoded feature tensor corresponding to the highest image feature scale level; Determine the lower level decoding feature tensor as the highest level encoding feature tensor, and repeat the steps of determining the lower level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest level decoding feature tensor until the primary decoding feature tensor of the lowest image feature scale level is determined; Based on the primary decoded feature tensor, a segmentation result corresponding to the to-be-segmented nuclear magnetic resonance sequence image is determined, wherein the segmentation result is a segmentation result of a brain glioma target area in the to-be-segmented nuclear magnetic resonance sequence image.

2. The postoperative brain glioma image segmentation method according to claim 1, characterized in that: The SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance image sequence to be segmented, determining the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance image sequence to be segmented, specifically includes: Stretching the to-be-segmented nuclear magnetic resonance sequence image to a preset size and performing normalization processing to obtain a processed image; Performing convolution processing on the processed image to obtain first image features with reduced size, and rearranging the first image features to obtain a corresponding image tensor; Adding a preset position tensor and the image tensor to obtain a corresponding initial feature tensor, wherein the number of rows and columns of elements in the image tensor is the same as that in the position tensor; The initial feature tensor is processed by introducing a SwinTransformer module of the target modal vector corresponding to the same image feature scale level into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented.

3. The postoperative brain glioma image segmentation method according to claim 1, characterized in that: The SwinTransformer module in the encoder based on the primary coding feature tensor and the previous image feature scale level determines the upper coding feature tensor corresponding to the previous image feature scale level, specifically including: Rearranging the primary encoding feature tensor to obtain the reduced-size rearranged image feature, wherein the number of channels of the rearranged image feature is the same as the number of channels of the first image feature; Dividing the rearranged image features into a plurality of sub-blocks of preset pixels, and performing vectorization processing on each of the sub-blocks to obtain a first vector; Performing a linear transformation on each of the first vectors to obtain a second vector after dimension reduction, and rearranging each of the second vectors to obtain a rearranged coded feature tensor of an image feature scale level previous to the lowest image feature scale level; The rearranged encoded feature tensor is processed by a SwinTransformer module that introduces a target modal vector corresponding to the same image feature scale level in an encoder of the previous image feature scale level to obtain an upper-level encoded feature tensor corresponding to the previous image feature scale level.

4. The postoperative brain glioma image segmentation method according to claim 3, characterized in that: The step of determining the lower level decoded feature tensor of the next image feature scale level according to the highest level decoded feature tensor through the SwinTransformer module in the decoder of the same image feature scale level specifically includes: Upsampling the highest-level decoded feature tensor to obtain a sampled result, and performing channel merging of the upsampled result with a rearranged encoded feature tensor at a previous image feature scale level of the highest image feature scale level to obtain a merged result; Performing convolution processing with a step size of 1 on the combined result and rearranging the result to obtain a rearranged decoded feature tensor of the next image feature scale level of the highest image feature scale level; The rearranged decoded feature tensor is processed by introducing a SwinTransformer module corresponding to the target modal vector of the same image feature scale level in the decoder of the next image feature scale level to obtain a lower-level decoded feature tensor corresponding to the next image feature scale level.

5. The postoperative brain glioma image segmentation method according to claim 1, characterized in that: The obtaining of the modal vector information corresponding to the nuclear magnetic resonance sequence image to be segmented specifically includes: Acquire at least one sequence image sample having a sequence number identical to the target sequence number of the to-be-segmented nuclear magnetic resonance sequence image, wherein the sequence image sample is a sequence image with a brain glioma target region annotated thereon; Freeze the parameters of the encoder and decoder at each image feature scale level in the pre-trained image segmentation model and introduce the initial modal vector to obtain the image segmentation model to be fine-tuned; Input each of the sequence image samples into the image segmentation model to be fine-tuned for training, and optimize the parameters of the initial modal vector to obtain the target modal vector corresponding to the target sequence label and the fine-tuned image segmentation model and store them in a preset database; When the to-be-segmented nuclear magnetic resonance sequence images need to be segmented, corresponding modality vector information is extracted from the database according to the target sequence label.

6. The postoperative brain glioma image segmentation method according to claim 2, characterized in that: The SwinTransformer module of the target modal vector corresponding to the same image feature scale level is introduced into the encoder of the lowest image feature scale level in the fine-tuned image segmentation model to process the initial feature tensor, so as to obtain the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented, specifically including: The initial feature tensor is sub-divided into sub-blocks by a SwinTransformer module in an encoder at the lowest image feature scale level to obtain a block result, and a target modal vector corresponding to the same image feature scale level is introduced into the SwinTransformer module in the encoder at the lowest image feature scale level; Merging the block-divided result and the target modal vector corresponding to the encoder at the lowest image feature scale level to obtain a first merged result; Performing autocorrelation processing on the first merged result by a SwinTransformer module in an encoder at the lowest image feature scale level to obtain a first processed result; The output results corresponding to the initial feature tensor are split from the first processed results and rearranged to obtain a primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence image to be segmented.

7. The postoperative brain glioma image segmentation method according to claim 1, characterized in that: The global Transformer module of the target modality vector corresponding to the same image feature scale level is introduced into the decoder of the highest image feature scale level to process the final encoded feature tensor to obtain the corresponding highest-level decoded feature tensor, specifically including: Merging the final encoded feature tensor with a target modality vector corresponding to the same image feature scale level introduced by a global Transformer module in a decoder of the highest image feature scale level to obtain a second merged result; performing autocorrelation processing on the second merged result through the global Transformer module to obtain a second processed result; The final encoded feature tensor is split from the second processed result and input into the output result after the global Transformer module is processed to obtain the corresponding highest-level decoded feature tensor.

8. The postoperative brain glioma image segmentation method according to claim 2, characterized in that: Determining the segmentation result corresponding to the to-be-segmented nuclear magnetic resonance sequence image based on the primary decoded feature tensor specifically includes: Rearranging the primary decoded feature tensor to obtain the reduced-size second image feature, where the second image feature has the same number of channels as the first image feature; Inputting the second image feature into the output head of the fine-tuned image segmentation model for processing and performing upsampling processing to obtain a restored image of the preset size; The restored image is processed by a Sigmoid activation function to obtain an initial segmentation result of the brain glioma target area; The initial segmentation result of the brain glioma target area is stretched to the original size of the nuclear magnetic resonance image sequence to be segmented, so as to obtain the segmentation result corresponding to the nuclear magnetic resonance image sequence to be segmented.

9. A postoperative brain glioma image segmentation device, characterized in that: include: An information acquisition module (11) is used to acquire modal vector information corresponding to the nuclear magnetic resonance sequence images to be segmented, wherein the modal vector information includes a plurality of target modal vectors, wherein the target modal vectors are modal vectors obtained by learning brain glioma-related image features at different image feature scale levels for nuclear magnetic resonance sequence images with the same sequence number as the nuclear magnetic resonance sequence images to be segmented, and the nuclear magnetic resonance sequence images to be segmented are sequence images with completely new sequence numbers; A first encoding module (12) is used to determine the primary encoding feature tensor of the lowest image feature scale level of the nuclear magnetic resonance sequence images to be segmented based on the SwinTransformer module in the encoder of the lowest image feature scale level in the fine-tuned image segmentation model and the nuclear magnetic resonance sequence images to be segmented, wherein the encoder and decoder of each image feature scale level in the fine-tuned image segmentation model both include a SwinTransformer module that introduces a target modal vector of the same image feature scale level; The second encoding module (13) determines the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, and determines the upper-level encoding feature tensor as the primary encoding feature tensor, and repeatedly executes the step of determining the upper-level encoding feature tensor corresponding to the previous image feature scale level based on the primary encoding feature tensor and the SwinTransformer module in the encoder of the previous image feature scale level, until the final encoding feature tensor of the highest image feature scale level is determined; A first decoding module (14) is used to process the final encoded feature tensor by introducing a global Transformer module of a target modal vector corresponding to the same image feature scale level in a decoder of the highest image feature scale level to obtain a corresponding highest-level decoded feature tensor, and determine a lower-level decoded feature tensor of the next image feature scale level according to the highest-level decoded feature tensor through a SwinTransformer module in a decoder of the same image feature scale level, wherein the highest-level decoded feature tensor is a decoded feature tensor corresponding to the highest image feature scale level; a second decoding module (15), configured to determine the lower level decoding feature tensor as the highest level encoding feature tensor, and to repeatedly perform the step of determining the lower level decoding feature tensor of the next image feature scale level through the SwinTransformer module in the decoder of the same image feature scale level according to the highest level decoding feature tensor, until the primary decoding feature tensor of the lowest image feature scale level is determined; An image segmentation module (16) is used to determine a segmentation result corresponding to the to-be-segmented nuclear magnetic resonance image sequence based on the primary decoded feature tensor, wherein the segmentation result is a segmentation result of a brain glioma target area in the to-be-segmented nuclear magnetic resonance image sequence.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 8 is adopted.

Images