A mode missing medical image segmentation method and device, terminal and storage medium

Abstract

The application discloses a kind of modal missing medical image segmentation method, device, terminal and storage medium, belong to medical image processing technical field, including obtaining three-dimensional medical image, the three-dimensional medical image regularization obtains the three-dimensional medical image after regularization;The three-dimensional medical image after regularization is extracted and integrated by encoder to obtain high-dimensional image feature output data;The high-dimensional image feature output data is further enriched by multistage knowledge distillation method and obtains high-dimensional image feature;The further enriched high-dimensional image feature is input into segmentation decoder to obtain final segmentation result.The application extracts more potential information from single mode input, finally improves the segmentation performance in single mode situation, and obtains high-quality segmentation result.

Description

Technical Field

[0001] This invention discloses a method, device, terminal, and storage medium for medical image segmentation with modal defects, belonging to the field of medical image processing technology. Background Technology

[0002] Image segmentation can be broadly categorized into semantic segmentation and instance segmentation, depending on the specific requirements. Medical image segmentation, as an important field of image processing, aims to depict specific biological structures, such as the outlines of organs or the shapes of lesions, from complex medical images, thereby assisting doctors in clinical diagnosis and improving the accuracy of diagnosis and treatment.

[0003] Medical image segmentation methods can be broadly categorized into two types based on different methods and techniques: edge-based methods, threshold-based methods, mathematical methods based on traditional machine learning, and deep learning-based methods. Among these, deep learning-based methods have received widespread attention and development due to their powerful ability to extract multi-level image features and contextual information.

[0004] On the other hand, the synergistic effect of complementary information between different modalities of medical images can significantly improve the efficiency of medical diagnosis, thereby improving the performance of medical image segmentation. Unfortunately, due to differences in scanning equipment, lengthy scanning times, and other adverse factors, obtaining a comprehensive set of modal data in actual clinical scenarios is often quite difficult. For example, some modalities require the injection of contrast agents, which may harm the patient's health. Therefore, in clinical practice, situations often arise where the required medical image modalities are insufficient.

[0005] Several methods have been proposed to address the problem of missing modalities. These include synthesizing missing modalities, but this approach is far from adequate in scenarios with extreme modality loss. Another category of methods focuses on mapping available modalities to a shared latent space and recovering missing information based on the constructed latent representations; however, these methods often perform poorly when multiple modalities are missing. A third category of methods attempts to transfer knowledge from a multimodal teacher network to a unimodal student network through knowledge distillation; these methods achieve better results when only one modality is available.

[0006] In view of the above situation, there is an urgent need to study a robust segmentation method in the context of medical image modality loss, so as to overcome the shortcomings in current practical applications. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a modality-deficient medical image segmentation method, device, terminal, and storage medium, which solves the problems existing in current medical image segmentation methods.

[0008] The technical solution of the present invention is as follows:

[0009] According to a first aspect of the present invention, a method for segmenting medical images with modal defects is provided, comprising:

[0010] A three-dimensional medical image is acquired, and the three-dimensional medical image is regularized to obtain a regularized three-dimensional medical image.

[0011] The regularized 3D medical image is extracted and integrated by an encoder to obtain high-dimensional image feature output data;

[0012] The high-dimensional image feature output data is further enriched by a multi-level knowledge distillation method to obtain high-dimensional image features.

[0013] The further enriched high-dimensional image features are input into the segmentation decoder to obtain the final segmentation result.

[0014] Preferably, the step of regularizing the three-dimensional medical image to obtain a regularized three-dimensional medical image includes:

[0015] The medical image with height H, width W, and depth D is divided into H / P×W / P×D / P non-overlapping tiles, where the size of each tile is P×P×P.

[0016] The H / P×W / P×D / P disjoint patches are used to regularize the input image using formula (1) to obtain a regularized 3D medical image:

[0017]

[0018] Among them, y p，c It is the label value of the c-th category, y p，c This is the output of the tile selection subtask module, L bce This represents the binary cross-entropy loss function.

[0019] Preferably, the step of extracting and integrating the regularized 3D medical image through an encoder to obtain high-dimensional image feature output data includes:

[0020] The high-level abstract features of the regularized 3D medical image are extracted through multi-layer convolutional blocks in the encoder;

[0021] The high-level abstract features are integrated by a fully connected layer to obtain high-dimensional image feature output data.

[0022] Preferably, the multi-layer convolutional block includes: a two-dimensional convolutional module, a BatchNorm module, and a ReLU activation function module.

[0023] Preferably, the step of obtaining further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method includes:

[0024] The high-dimensional image feature output data is passed through a projection head, and further enriched high-dimensional image features are obtained through formulas (2)-(4):

[0025] L kd (p s ,pt)=L bce (σ(p s / T1), σ(p t / T1)) (2)

[0026] Where, p s and p t Let T1 and L represent the logits outputs of the student model and teacher model, respectively. T1 is the temperature hyperparameter of pixel-level knowledge distillation, σ is the sigmoid operation, and L represents the logits outputs of the student model and teacher model, respectively. bce Let L be the binary cross-entropy loss function. kd This represents the generalized pixel-level knowledge distillation loss function;

[0027]

[0028] in, and These represent the logits outputs of the tile selection heads for the student and teacher models, respectively. T2 is the temperature hyperparameter of tile-level knowledge distillation, σ is the sigmoid operation, and L... bce Let L be the binary cross-entropy loss function. pskd This represents the loss function for knowledge distillation at the tile level.

[0029]

[0030] In the formula, N represents the number of tiles. Let represent the i-th embedding vector of the student model. It is embedded in the teacher model The only positive sample It is embedded in the teacher model Negative samples, NS represents The set of negative samples, sim represents the cosine similarity function, τ is the temperature scaling parameter, and L cont This represents the knowledge distillation loss function for plot-level comparisons.

[0031] According to a second aspect of the present invention, a medical image segmentation apparatus for modality loss is provided, comprising:

[0032] The regularization module is used to acquire three-dimensional medical images and regularize the three-dimensional medical images to obtain regularized three-dimensional medical images.

[0033] The extraction and integration module is used to extract and integrate the regularized three-dimensional medical image through an encoder to obtain high-dimensional image feature output data.

[0034] The knowledge distillation module is used to obtain further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method.

[0035] The target result segmentation module is used to input the further enriched high-dimensional image features into the segmentation decoder to obtain the final segmentation result.

[0036] According to a third aspect of the present invention, a terminal is provided, comprising:

[0037] One or more processors;

[0038] Memory for storing the one or more processor-executable instructions;

[0039] Wherein, the one or more processors are configured as follows:

[0040] Perform the method described in the first aspect of the embodiments of the present invention.

[0041] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the method described in the first aspect of the present invention.

[0042] According to a fifth aspect of the present invention, an application product is provided that, when the application product is running on a terminal, causes the terminal to execute the method described in the first aspect of the present invention.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention provides a method, apparatus, terminal, and storage medium for segmenting medical images with modal absence. Regularization is achieved through a subtask oriented towards patch selection, enabling the model to identify patches with pixels of specific categories. Furthermore, a contrastive learning method is employed, defining positive and negative sample pairs in the embedding layer based on the indications of patch-level labels. This encourages the student model to learn the similarities and differences with the teacher model in the representation, thereby extracting more potential information from the unimodal input and ultimately improving the segmentation performance in unimodal scenarios, resulting in high-quality segmentation results.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a modality-deficient medical image segmentation method according to an exemplary embodiment.

[0047] Figure 2 This is a schematic block diagram illustrating the structure of a modality-deficient medical image segmentation device according to an exemplary embodiment.

[0048] Figure 3 This is a schematic block diagram of a terminal structure according to an exemplary embodiment. Detailed Implementation

[0049] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0051] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0052] This invention provides a method for segmenting medical images with modal defects. This method is implemented by a terminal, which can be a desktop computer or a laptop computer, etc., and the terminal includes at least a CPU.

[0053] Example 1

[0054] Figure 1 and 2 This is a flowchart illustrating a modality-deficient medical image segmentation method according to an exemplary embodiment. The method is used in a terminal and includes the following steps:

[0055] Step 101: Obtain a three-dimensional medical image, and then regularize the three-dimensional medical image to obtain a regularized three-dimensional medical image. The specific details are as follows:

[0056] A three-dimensional medical image is acquired. The three-dimensional medical image is either a grayscale image or a color image. The medical image with height H, width W, and depth D is divided into H / P×W / P×D / P non-overlapping patches, where the size of each patch is P×P×P.

[0057] The H / P×W / P×D / P disjoint patches are used to regularize the input image using formula (1) to obtain a regularized 3D medical image:

[0058]

[0059] Among them, y p，c It is the label value of the c-th category, y′ p，c This is the output of the tile selection subtask module, L bce This represents the binary cross-entropy loss function.

[0060] Step 102: The regularized 3D medical image is extracted and integrated using an encoder to obtain high-dimensional image feature output data, the details of which are as follows:

[0061] The high-level abstract features of the regularized 3D medical image are extracted through multi-layer convolutional blocks in the encoder, and then integrated by fully connected layers to obtain high-dimensional image feature output data. These multi-layer convolutional blocks include: a 2D convolutional module, a BatchNorm module, and a ReLU activation function module.

[0062] Step 103: The high-dimensional image feature output data is further enriched with high-dimensional image features through a multi-level knowledge distillation method, as detailed below:

[0063] Multi-level knowledge distillation includes: generalized pixel-level knowledge distillation, tile-level knowledge distillation, and tile-level comparative knowledge distillation. The high-dimensional image feature output data is passed through a projection head, and further enriched high-dimensional image features are obtained through formulas (2)-(4):

[0064] L kd (p s p t ) = L bce (σ(ps / T1), σ(pt / T1)) (2)

[0065] Where, p s and p t Let T1 and L represent the logits outputs of the student model and teacher model, respectively. T1 is the temperature hyperparameter of pixel-level knowledge distillation, σ is the sigmoid operation, and L represents the logits outputs of the student model and teacher model, respectively.bce Let L be the binary cross-entropy loss function. kd This represents the generalized pixel-level knowledge distillation loss function;

[0066]

[0067] in, and These represent the logits output of the tile selection heads for the student and teacher models, respectively. T2 is the temperature hyperparameter of tile-level knowledge distillation, σ is the sigmoid operation, and L... bce Let L be the binary cross-entropy loss function. pskd This represents the loss function for knowledge distillation at the tile level.

[0068]

[0069] In the formula, N represents the number of tiles. Let represent the i-th embedding vector of the student model. It is embedded in the teacher model The only positive sample It is embedded in the teacher model Negative samples, NS represents The set of negative samples, sim represents the cosine similarity function, τ is the temperature scaling parameter, and L cont This represents the knowledge distillation loss function for plot-level comparisons.

[0070] Step 104: Input the further enriched high-dimensional image features into the segmentation decoder to obtain the final segmentation result.

[0071] Specific examples based on the above steps are as follows: Experiments were conducted on the Brain Tumor Segmentation Challenge (BraTS 2018) dataset, with all cases divided into 70%, 10%, and 20% for training, validation, and testing, and the results were compared with those obtained from five other unimodal medical image segmentation methods.

[0072] To objectively evaluate the comparison results, the average objective evaluation index of each segmentation method was calculated, and the results are shown in Table 1. A higher objective evaluation index value indicates better segmentation performance. Table 1 shows a significant improvement on the single-modal benchmark. For the four test modalities, the average Dice coefficient increased by 4.60%, 3.04%, 5.01%, and 3.49%, respectively, exceeding the performance of the compared methods. The results indicate that introducing tile-level distillation can provide students with more knowledge. Furthermore, the process of learning similarities and differences in representations has proven beneficial, helping students extract more potential information from single-modal data.

[0073] Table 1

[0074]

[0075] Example 2

[0076] Figure 2 This is a schematic block diagram illustrating the structure of a modality-deficient medical image segmentation apparatus according to an exemplary embodiment. The apparatus includes:

[0077] Regularization module 210 is used to acquire a three-dimensional medical image and regularize the three-dimensional medical image to obtain a regularized three-dimensional medical image.

[0078] The extraction and integration module 220 is used to extract and integrate the regularized three-dimensional medical image through an encoder to obtain high-dimensional image feature output data.

[0079] The knowledge distillation module 230 is used to obtain further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method.

[0080] The target result segmentation module 240 is used to input the further enriched high-dimensional image features into the segmentation decoder to obtain the final segmentation result.

[0081] This application achieves regularization through a tile selection subtask, enabling the model to recognize tiles with pixels of specific categories. It also employs a contrastive learning approach, defining positive and negative sample pairs in the embedding layer based on tile-level labels. This encourages the student model to learn the similarities and differences with the teacher model in the representation, thereby extracting more latent information from the unimodal input and ultimately improving the segmentation performance in unimodal scenarios, resulting in high-quality segmentation results.

[0082] Example 3

[0083] Figure 3 This is a structural block diagram of a terminal provided in an embodiment of this application. The terminal can be the terminal in the above embodiments. The terminal 300 can be a portable mobile terminal, such as a smartphone or tablet computer. The terminal 300 may also be referred to as user equipment, portable terminal, or other names.

[0084] Typically, terminal 300 includes a processor 301 and a memory 302.

[0085] Processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0086] The memory 302 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement a modality-deficient medical image segmentation method provided in this application.

[0087] In some embodiments, the terminal 300 may also optionally include: a peripheral device interface 303 and at least one peripheral device. Specifically, the peripheral device includes at least one of: a radio frequency circuit 304, a touch display screen 305, a camera 306, an audio circuit 307, a positioning component 308, and a power supply 309.

[0088] The peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0089] The radio frequency (RF) circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 304 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0090] The touch display screen 305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. The touch display screen 305 also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to the processor 301 for processing. The touch display screen 305 is used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one touch display screen 305, which is located on the front panel of the terminal 300; in other embodiments, there may be at least two touch display screens, respectively located on different surfaces of the terminal 300 or in a folded design; in still other embodiments, the touch display screen 305 may be a flexible display screen, located on a curved or folded surface of the terminal 300. Furthermore, the touch display screen 305 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The touch display screen 305 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0091] Camera assembly 306 is used to acquire images or videos. Optionally, camera assembly 306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is used for video calls or selfies, and the rear-facing camera is used for taking photos or videos. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, and a wide-angle camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, and panoramic shooting and VR (Virtual Reality) shooting by fusion of the main camera and the wide-angle camera. In some embodiments, camera assembly 306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0092] Audio circuit 307 provides an audio interface between the user and terminal 300. Audio circuit 307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to processor 301 for processing, or input to radio frequency circuit 304 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of terminal 300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 301 or radio frequency circuit 304 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, audio circuit 307 may also include a headphone jack.

[0093] The positioning component 308 is used to determine the current geographic location of the terminal 300 in order to enable navigation or LBS (Location Based Service). The positioning component 308 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0094] The power supply 309 is used to power the various components in the terminal 300. The power supply 309 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 309 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired connection, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0095] In some embodiments, the terminal 300 further includes one or more sensors 310. The one or more sensors 310 include, but are not limited to: an accelerometer 311, a gyroscope 312, a pressure sensor 313, a fingerprint sensor 314, an optical sensor 315, and a proximity sensor 316.

[0096] Accelerometer 311 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established with terminal 300. For example, accelerometer 311 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 301 can control touchscreen 305 to display the user interface in landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 311. Accelerometer 311 can also be used for games or for acquiring user motion data.

[0097] The gyroscope sensor 312 can detect the orientation and rotation angle of the terminal 300. The gyroscope sensor 312, in conjunction with the accelerometer sensor 311, can collect the user's 3D (3D) movements on the terminal 300. Based on the data collected by the gyroscope sensor 312, the processor 301 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0098] The pressure sensor 313 can be disposed on the side bezel of the terminal 300 and / or on the lower layer of the touch display screen 305. When the pressure sensor 313 is disposed on the side bezel of the terminal 300, it can detect the user's grip signal on the terminal 300 and perform left / right hand recognition or quick operation based on the grip signal. When the pressure sensor 313 is disposed on the lower layer of the touch display screen 305, it can control the operable controls on the UI interface based on the user's pressure operation on the touch display screen 305. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0099] The fingerprint sensor 314 is used to collect a user's fingerprint to identify the user's identity. When the user's identity is identified as trusted, the processor 301 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 314 can be located on the front, back, or side of the terminal 300. When the terminal 300 has physical buttons or a manufacturer's logo, the fingerprint sensor 314 can be integrated with the physical buttons or manufacturer's logo.

[0100] An optical sensor 315 is used to collect ambient light intensity. In one embodiment, the processor 301 can control the display brightness of the touch screen 305 based on the ambient light intensity collected by the optical sensor 315. Specifically, when the ambient light intensity is high, the display brightness of the touch screen 305 is increased; when the ambient light intensity is low, the display brightness of the touch screen 305 is decreased. In another embodiment, the processor 301 can also dynamically adjust the shooting parameters of the camera assembly 306 based on the ambient light intensity collected by the optical sensor 315.

[0101] The proximity sensor 316, also known as a distance sensor, is typically located on the front of the terminal 300. The proximity sensor 316 is used to detect the distance between the user and the front of the terminal 300. In one embodiment, when the proximity sensor 316 detects that the distance between the user and the front of the terminal 300 is gradually decreasing, the processor 301 controls the touchscreen display 305 to switch from a screen-on state to a screen-off state; when the proximity sensor 316 detects that the distance between the user and the front of the terminal 300 is gradually increasing, the processor 301 controls the touchscreen display 305 to switch from a screen-off state to a screen-on state.

[0102] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on terminal 300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0103] Example 4

[0104] In an exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements a modality-deficient medical image segmentation method as provided in all embodiments of the present application.

[0105] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0106] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0107] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0108] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0109] Example 5

[0110] In an exemplary embodiment, an application product is also provided, including one or more instructions that can be executed by the processor 301 of the aforementioned device to complete the aforementioned modality-deficient medical image segmentation method.

[0111] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for the present invention. Other modifications can be readily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and examples shown and described herein.

Claims

1. A method for segmenting medical images with missing modalities, characterized in that, include: A three-dimensional medical image is acquired, and the three-dimensional medical image is regularized to obtain a regularized three-dimensional medical image. The regularized 3D medical image is extracted and integrated by an encoder to obtain high-dimensional image feature output data; The high-dimensional image feature output data is further enriched by a multi-level knowledge distillation method to obtain high-dimensional image features. The further enriched high-dimensional image features are input into the segmentation decoder to obtain the final segmentation result; The step of obtaining further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method includes: The high-dimensional image feature output data is passed through a projection head, and further enriched high-dimensional image features are obtained through formulas (1)-(3): （1） in, p s and p t The outputs of the student and teacher models are represented by logits, respectively, and T1 represents the temperature hyperparameter of knowledge distillation at the pixel level. It's a sigmoid operation. The binary cross-entropy loss function is... This represents the generalized pixel-level knowledge distillation loss function; （2） in, and These represent the logits outputs of the tile selection headers for the student model and the teacher model, respectively. It is the temperature hyperparameter of tile-level knowledge distillation. It's a sigmoid operation. The binary cross-entropy loss function is... This represents the loss function for knowledge distillation at the tile level. （3） In the formula, Indicates the number of tiles. Let represent the i-th embedding vector of the student model. It is embedded in the teacher model The only positive sample It is embedded in the teacher model negative samples, express The negative sample set, Represents the cosine similarity function. It is a temperature scaling parameter. This represents the knowledge distillation loss function for plot-level comparisons.

2. The modality-deficient medical image segmentation method according to claim 1, characterized in that, The step of regularizing the three-dimensional medical image to obtain a regularized three-dimensional medical image includes: The medical image with height H, width W, and depth D is divided into H / P×W / P×D / P non-overlapping patches, where the size of each patch is P×P×P. The H / P×W / P×D / P disjoint patches are used to regularize the input image using formula (4) to obtain a regularized 3D medical image: （4） in, It is the label value of the c-th category. This is the output of the tile selection subtask module. Let C represent the binary cross-entropy loss function, where C is the total number of target categories in the classification task.

3. The modality-deficient medical image segmentation method according to claim 1, characterized in that, The step of extracting and integrating high-dimensional image feature output data from the regularized 3D medical image using an encoder includes: The high-level abstract features of the regularized 3D medical image are extracted through multi-layer convolutional blocks in the encoder; The high-level abstract features are integrated by a fully connected layer to obtain high-dimensional image feature output data.

4. The modality-deficient medical image segmentation method according to claim 1, characterized in that, The multi-layer convolutional block includes: a two-dimensional convolution module, a BatchNorm module, and a ReLU activation function module.

5. A medical image segmentation device for modality loss, characterized in that, include: The regularization module is used to acquire three-dimensional medical images and regularize the three-dimensional medical images to obtain regularized three-dimensional medical images. The extraction and integration module is used to extract and integrate the regularized three-dimensional medical image through an encoder to obtain high-dimensional image feature output data. The knowledge distillation module is used to obtain further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method. The target result segmentation module is used to input the further enriched high-dimensional image features into the segmentation decoder to obtain the final segmentation result; The step of obtaining further enriched high-dimensional image features from the high-dimensional image feature output data through a multi-level knowledge distillation method includes: The high-dimensional image feature output data is passed through a projection head, and further enriched high-dimensional image features are obtained through formulas (1)-(3): （1） in, p s and p t The outputs of the student and teacher models are represented by logits, respectively, and T1 represents the temperature hyperparameter of knowledge distillation at the pixel level. It's a sigmoid operation. The binary cross-entropy loss function is... This represents the generalized pixel-level knowledge distillation loss function; （2） in, and These represent the logits outputs of the tile selection headers for the student model and the teacher model, respectively. It is the temperature hyperparameter of tile-level knowledge distillation. It's a sigmoid operation. The binary cross-entropy loss function is... This represents the loss function for knowledge distillation at the tile level. （3） In the formula, Indicates the number of tiles. Let represent the i-th embedding vector of the student model. It is embedded in the teacher model The only positive sample It is embedded in the teacher model negative samples, express The negative sample set, Represents the cosine similarity function. It is a temperature scaling parameter. This represents the knowledge distillation loss function for plot-level comparisons.

6. A terminal, characterized in that, include: One or more processors; Memory for storing the one or more processor-executable instructions; Wherein, the one or more processors are configured as follows: Perform the modality-deficient medical image segmentation method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the modality-deficient medical image segmentation method as described in any one of claims 1 to 4.

Images