Medical image segmentation method and device, equipment and medium

Abstract

The invention provides a medical image segmentation method and device, equipment and a medium, and the method comprises the steps: receiving a to-be-segmented medical image, and inputting the to-be-segmented medical image into a trained medical image segmentation model; performing multi-scale feature extraction on the to-be-segmented medical image based on a visual encoder to obtain a plurality of visual features of different scales; performing mask segmentation on the medical image to be segmented based on a prior mask generator to obtain a segmentation mask; based on a prior fusion coefficient generator, generating a plurality of fusion coefficients for controlling the fusion of other scale visual features and the segmentation mask according to the minimum scale visual feature in the plurality of visual features; performing cross-modal fusion on the plurality of visual features and the segmentation mask according to the plurality of fusion coefficients based on a priori guidance decoder to generate a multi-modal feature; and detecting the multi-modal features based on a detection head to generate an image segmentation result. According to the invention, the segmentation accuracy and robustness of the medical image are improved.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, specifically to a medical image segmentation method, apparatus, device, and medium. Background Technology

[0002] In recent years, deep learning technology has greatly promoted the development of medical image segmentation, making it a core component of computer-aided diagnosis, surgical planning, and navigation systems. However, medical images are typically characterized by complex tissue structures, blurred lesion boundaries, diverse imaging modalities, and strong noise interference, which places higher demands on the segmentation model's ability to understand spatial details and structures.

[0003] Meanwhile, significant breakthroughs have been achieved in the field of natural image segmentation using large-scale pre-trained models. Among them, the Segment Anything Model (SAM), a general-purpose segmentation foundation model, has been trained on a massive dataset of over 11 million images, demonstrating powerful zero-shot segmentation capabilities and a deep understanding of object structure. SAM can generate high-quality segmentation masks based on given cue points or bounding boxes, providing a new approach for medical image segmentation: utilizing the masks generated by SAM as prior knowledge to guide the segmentation process. However, directly applying SAM to medical image segmentation faces significant challenges.

[0004] First, SAM is trained on natural images. Its learned texture, color, and object shape priors do not match the grayscale characteristics, tissue texture, and specific lesion morphology of medical images, resulting in masks that are often inaccurate on medical images and have limited effectiveness when used directly. Second, to utilize the prior knowledge of SAM, existing techniques typically attempt to concatenate the SAM-generated mask with the original image channels or use it as additional input to attention mechanisms. While simple, these methods suffer from rigid prior utilization: introducing priors in a fixed and unchanging manner fails to dynamically adjust the level of reliance on SAM prior information based on the specific content of the current medical image. When the SAM prior itself contains errors, this rigid fusion method can even amplify these errors, leading to decreased segmentation performance. Furthermore, medical image segmentation requires the fusion of multi-scale information to simultaneously capture global semantics and local details. Using a single-scale prior mask fails to establish an adaptive correlation between prior information and features at different scales, resulting in poor fusion quality and consequently, inaccurate medical image segmentation results.

[0005] Therefore, there is an urgent need to provide a medical image segmentation method, device, equipment, and medium that can effectively adapt the general segmentation prior knowledge of SAM to the medical image field and can adaptively and on-demand integrate it into the multi-scale feature decoding process according to the image content, thereby achieving more accurate and robust segmentation results. Summary of the Invention

[0006] In view of this, it is necessary to provide a medical image segmentation method, apparatus, device and medium to solve the technical problem in the prior art that directly adapts a general fixed, single-scale prior mask to the field of medical images without considering the differences between medical images and general images, resulting in inaccurate image segmentation results.

[0007] To address the aforementioned technical problems, in a first aspect, the present invention provides a medical image segmentation method, comprising: The system receives a medical image to be segmented and inputs it into a trained medical image segmentation model. The medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head. Based on the visual encoder, multi-scale feature extraction is performed on the medical image to be segmented to obtain multiple visual features at different scales; Based on the prior mask generator, the medical image to be segmented is segmented using a mask to obtain a segmentation mask; Based on the prior fusion coefficient generator, multiple fusion coefficients are generated according to the smallest scale visual feature among the multiple visual features to control the fusion of the remaining scale visual features with the segmentation mask; Based on the prior guided decoder, the multiple visual features and the segmentation mask are fused across modally according to the multiple fusion coefficients to generate multimodal features; The multimodal features are detected using the detection head to generate image segmentation results.

[0008] In one possible implementation, the visual encoder includes a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block, and a fourth visual feature extraction block; The first visual feature extraction block is used to extract features from the medical image to be segmented to obtain first visual features; The second visual feature extraction block is used to extract features from the first visual feature to obtain the second visual feature; The third visual feature extraction block is used to extract features from the second visual features to obtain the third visual features; The fourth visual feature extraction block is used to extract features from the third visual features to obtain the fourth visual features; The scales of the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature gradually decrease.

[0009] In one possible implementation, the prior mask generator includes a mask segmentation module and a mask format conversion module; The mask segmentation module is used to perform mask segmentation on the medical image to be segmented to obtain an initial segmentation mask; The mask format conversion module is used to convert the initial segmentation mask, which has a shape of height H × width W × number of masks N, into a segmentation mask with a shape of L × N by flattening the spatial dimensions of the initial segmentation mask, where L = H × W.

[0010] In one possible implementation, the prior fusion coefficient generator includes a global average pooling layer, a multilayer perceptron, and an activation function layer; The global average pooling layer is used to perform global average pooling on the fourth visual feature to obtain a global feature vector. The multilayer perceptron is used to perform a nonlinear transformation on the global feature vector and output a multidimensional vector, which corresponds to the first visual feature, the second visual feature, and the third visual feature, respectively. The activation function layer is used to normalize the multidimensional vector to obtain the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient.

[0011] In one possible implementation, the prior guided decoder includes a projection module, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module; The projection module is used to perform a projection transformation on the segmentation mask to obtain projection features; The multi-head self-attention module is used to perform multi-head self-attention learning on the visual features to obtain visual enhancement features; The multi-head cross-attention module is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features based on the first fusion coefficient, the second fusion coefficient and the third fusion coefficient to obtain initial multimodal features; The upsampling module is used to reshape and upsample the initial multimodal features to obtain the multimodal features.

[0012] In one possible implementation, the multi-head cross-attention module includes a multi-head cross-attention learning unit, a normalization processing layer, and a multimodal fusion layer; The multi-head cross-attention learning unit is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features to obtain attention learning features; The normalization processing layer is used to normalize the attention learning features to obtain normalized features; The multimodal fusion layer is used to take the product of the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient with the normalized feature and the feature sum of the visual enhancement feature as the initial multimodal feature.

[0013] In one possible implementation, before inputting the medical image to be segmented into a trained medical image segmentation model, the method further includes: Obtain a medical image dataset, and train the constructed initial image segmentation model based on the medical image dataset to obtain the medical image segmentation model; The loss function used during the training of the initial image segmentation model is the cross-entropy loss function.

[0014] Secondly, the present invention also provides a medical image segmentation apparatus, comprising: An image receiving unit is used to receive a medical image to be segmented and input the medical image to be segmented into a trained medical image segmentation model; the medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head; The visual feature extraction unit is used to perform multi-scale feature extraction on the medical image to be segmented based on the visual encoder to obtain multiple visual features at different scales. A mask segmentation unit is used to perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain a segmentation mask. The fusion coefficient determination unit is used to generate multiple fusion coefficients based on the prior fusion coefficient generator according to the smallest scale visual feature among the multiple visual features, for controlling the fusion of the remaining scale visual features with the segmentation mask; A cross-modal fusion unit is used to perform cross-modal fusion of the multiple visual features and the segmentation mask based on the prior guided decoder according to the multiple fusion coefficients to generate multimodal features; The image segmentation unit is used to detect the multimodal features based on the detection head and generate image segmentation results.

[0015] Thirdly, the present invention also provides a medical image segmentation device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the medical image segmentation method described in any of the above possible implementations.

[0016] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the medical image segmentation method described in any of the above possible implementations.

[0017] The beneficial effects of this invention are as follows: The medical image segmentation method provided by this invention does not directly use the segmentation mask obtained by the prior mask generator as the image segmentation result. Instead, it uses the segmentation mask obtained by the prior mask generator as prior knowledge and performs cross-modal fusion with multiple visual features of different scales. This compensates for the differences between medical images and general images, and adaptively transfers the mask segmentation learned in general images to the medical image segmentation task, thereby significantly improving the segmentation accuracy of complex and irregular lesions and solving the problem of insufficient performance of general models in professional medical fields. Simultaneously, to overcome the rigidity caused by fixed priors, this invention designs a prior fusion coefficient generator. This generator can generate multiple fusion coefficients based on the smallest scale visual feature among multiple visual features, which are used to control the fusion of visual features of other scales with the segmentation mask. This dynamically determines the fusion coefficients for different scales of visual features, achieving dynamic multimodal fusion. This improves the adaptability of the generated multimodal features to medical images, thereby improving the accuracy of image segmentation results.

[0018] Furthermore, the prior fusion coefficient generator generates multiple fusion coefficients to control the fusion of visual features at other scales with the segmentation mask, thereby generating a fusion implementation for the visual features of the medical image at each scale, realizing the fusion of the prior mask at different scales, and further improving the accuracy of the medical image segmentation results. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart of an embodiment of the medical image segmentation method provided by the present invention; Figure 2 A schematic diagram of an embodiment of the medical image segmentation model provided by the present invention; Figure 3 A schematic diagram of an embodiment of the visual encoder provided by the present invention; Figure 4 A schematic diagram of an embodiment of the multi-head cross-attention module provided by the present invention; Figure 5 A schematic diagram of an embodiment of the medical image segmentation device provided by the present invention; Figure 6 This is a schematic diagram of an embodiment of the medical image segmentation device provided by the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] This invention provides a medical image segmentation method, apparatus, device, and medium, which are described below.

[0025] Figure 1 This is a schematic flowchart of an embodiment of the medical image segmentation method provided by the present invention. Figure 2 This is a schematic diagram of an embodiment of the medical image segmentation model provided by the present invention, as shown below. Figure 1 and Figure 2 As shown, medical image segmentation methods include: S101. Receive the medical image to be segmented and input it into the trained medical image segmentation model; the medical image segmentation model includes a visual encoder, a priori mask generator, a priori fusion coefficient generator, a priori guided decoder, and a detection head; S102. Based on the visual encoder, perform multi-scale feature extraction on the medical image to be segmented to obtain multiple visual features at different scales. S103. Perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain the segmentation mask.

[0026] Specifically, the prior mask generator is a general SAM model.

[0027] S104. Based on the prior fusion coefficient generator, multiple fusion coefficients are generated from the smallest scale visual feature among multiple visual features to control the fusion of the remaining scale visual features with the segmentation mask. S105. Based on the prior guided decoder, multiple visual features and segmentation masks are fused across modally according to multiple fusion coefficients to generate multimodal features; S106. Detect multimodal features based on the detection head and generate image segmentation results.

[0028] Specifically, the detection head first fuses multimodal features and then performs detection to generate image segmentation results. The image segmentation results include, but are not limited to, pixel-level segmentation masks, target contours, or region category information.

[0029] It should be understood that the medical image segmentation method in this embodiment of the invention can be implemented in any device based on a medical image segmentation method, such as a medical image segmentation device. Specifically, the medical image segmentation method is stored in the device as a pre-programmed program, and when the device is started, the program is invoked and the medical image segmentation method is implemented.

[0030] Compared with existing technologies, the medical image segmentation method provided in this invention does not directly use the segmentation mask obtained by the prior mask generator as the image segmentation result. Instead, it uses the segmentation mask obtained by the prior mask generator as prior knowledge and performs cross-modal fusion with multiple visual features of different scales. This compensates for the differences between medical images and general images, adaptively transferring the mask segmentation learned in general images to the medical image segmentation task. This significantly improves the segmentation accuracy of complex and irregular lesions and solves the problem of insufficient performance of general models in professional medical fields. Furthermore, to overcome the rigidity caused by fixed priors, this invention designs a prior fusion coefficient generator. This generator generates multiple fusion coefficients based on the smallest scale visual feature among multiple visual features, controlling the fusion of other scale visual features with the segmentation mask. It dynamically determines the fusion coefficients for different scale visual features, achieving dynamic multimodal fusion. This improves the adaptability of the generated multimodal features to medical images, thereby improving the accuracy of image segmentation results.

[0031] Furthermore, the prior fusion coefficient generator generates multiple fusion coefficients to control the fusion of visual features at other scales with the segmentation mask, thereby generating a fusion implementation for the visual features of the medical image at each scale, realizing the fusion of the prior mask at different scales, and further improving the accuracy of the medical image segmentation results.

[0032] In some embodiments of the present invention, the visual encoder is ConvNeXt-Tiny, such as... Figure 3 As shown, the visual encoder includes a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block, and a fourth visual feature extraction block; The first visual feature extraction block is used to extract features from the medical image to be segmented to obtain the first visual features; The second visual feature extraction block is used to extract features from the first visual features to obtain the second visual features; The third visual feature extraction block is used to extract features from the second visual features to obtain the third visual features; The fourth visual feature extraction block is used to extract features from the third visual features to obtain the fourth visual features; Among them, the scales of the first visual feature, second visual feature, third visual feature and fourth visual feature gradually decrease.

[0033] It should be understood that the fourth visual feature can characterize the global semantic features of the medical image to be segmented. Therefore, the fusion coefficients of the other three visual features can be determined based on the fourth visual feature.

[0034] Specifically, the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature can be represented as follows: , , , , For feature size, and The height and width of the medical image to be segmented.

[0035] It should be noted that ConvNeXt-Tiny is a mature architecture, and the specific structures of the first visual feature extraction block, the second visual feature extraction block, the third visual feature extraction block, and the fourth visual feature extraction block, as well as other structures, will not be elaborated here.

[0036] In specific embodiments of the present invention, such as Figure 2 As shown, the prior mask generator includes a mask segmentation module and a mask format conversion module; The mask segmentation module is used to perform mask segmentation on the medical image to be segmented to obtain the initial segmentation mask.

[0037] Specifically, the initial segmentation mask can be represented as: . This indicates the operation of masking segmentation of the medical image to be segmented based on the SAM model.

[0038] The mask format conversion module is used to convert an initial segmentation mask with shape H×width W×mask number N into a segmentation mask with shape L×N by flattening the spatial dimensions of the initial segmentation mask, where L=H×W.

[0039] This invention converts the initial segmentation mask into a two-dimensional matrix segmentation mask, which is adaptable to subsequent processing. Furthermore, matrix operations are more efficient in hardware than three-dimensional tensor operations, and since visual features are two-dimensional, the two-dimensional matrix segmentation mask can match the dimension of the visual features, thereby reducing the difficulty of subsequent cross-modal fusion, achieving more efficient information interaction, and improving segmentation efficiency.

[0040] In some embodiments of the present invention, such as Figure 2 As shown, the prior fusion coefficient generator includes a global average pooling layer, a multilayer perceptron, and an activation function layer; The global average pooling layer is used to perform global average pooling on the fourth visual features to obtain a global feature vector. A multilayer perceptron is used to perform nonlinear transformations on global feature vectors and output multidimensional vectors, which correspond to the first visual feature, the second visual feature, and the third visual feature, respectively. The activation function layer is used to normalize the multidimensional vector, resulting in the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient.

[0041] Specifically, the process of determining each fusion coefficient can be expressed as follows:

[0042] In the formula, It can be the first fusion coefficient, the second fusion coefficient, or the third fusion coefficient; This is a global average pooling operation; For multilayer sensor operation; For activation processing; It is the fourth visual feature.

[0043] The multilayer perceptron consists of two fully connected layers and a ReLU activation function.

[0044] In a specific embodiment of the present invention, the activation process is performed using the Sigmoid function.

[0045] Specifically, Divided into , , These are used to achieve cross-modal fusion of the first visual feature, the second visual feature, and the third visual feature with the prior mask, respectively.

[0046] In some embodiments of the present invention, such as Figure 2 As shown, the priori guided decoder includes a projection module, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module; The projection module is used to perform projection transformation on the segmentation mask to obtain projection features.

[0047] The purpose of projection transformation is to align the dimensions of the segmentation mask with the dimensions of the visual features.

[0048] Specifically, projection features It can be represented as:

[0049] In the formula, For segmentation mask; It is a learnable matrix; For convolution processing; For activation processing, the activation function used is the ReLU function.

[0050] This represents the output projection feature, where... It is the length of the projected feature. It is the number of channels for the projection feature. Consistent with the number of channels for visual features, among which, .

[0051] The multi-head self-attention module is used to learn visual features through multi-head self-attention to obtain visually enhanced features.

[0052] Specifically, visual enhancement features It can be represented as:

[0053] In the formula, Visual features; For normalization processing; For multi-head self-attention learning.

[0054] The multi-head cross-attention module is used to learn the visual enhancement features and projection features through multi-head cross-attention based on the first fusion coefficient, the second fusion coefficient and the third fusion coefficient, so as to obtain the initial multimodal features.

[0055] The upsampling module is used to reshape and upsample the initial multimodal features to obtain multimodal features.

[0056] Specifically, multimodal features It can be represented as:

[0057] In the formula, For reshaping; For upsampling processing; These are the initial multimodal features.

[0058] In specific embodiments of the present invention, such as Figure 4 As shown, the multi-head cross-attention module includes a multi-head cross-attention learning unit, a normalization processing layer, and a multimodal fusion layer; The multi-head cross-attention learning unit is used to perform multi-head cross-attention learning on visual enhancement features and projection features to obtain attention learning features; The normalization layer is used to normalize the attention learning features to obtain normalized features; The multimodal fusion layer is used to take the product of the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient with the normalized feature and the sum of the visual enhancement feature as the initial multimodal feature.

[0059] Specifically, initial multimodal features It can be represented as:

[0060] In the formula, This is for multi-head cross-attention learning.

[0061] As is well known to those skilled in the art, deep learning models need to be trained before use. Therefore, in some embodiments of the present invention, before inputting the medical image to be segmented into the trained medical image segmentation model in step S101, the method further includes: Obtain a medical image dataset and train the initial image segmentation model based on the medical image dataset to obtain a medical image segmentation model; The loss function used during the initial image segmentation model training is the cross-entropy loss function.

[0062] Specifically, training stops when the loss value of the loss function converges or the number of training iterations reaches the set maximum, thus obtaining a medical image segmentation model.

[0063] In a specific embodiment of the present invention, the expression for the cross-entropy loss function is as follows:

[0064] In the formula, Indicates the number of pixels in an image; The true category label for the i-th pixel; This represents the predicted probability of the i-th pixel as predicted by the model.

[0065] In summary, the medical image segmentation method proposed in this invention uses the segmentation mask generated by SAM as prior knowledge and combines it with visual features to obtain more accurate segmentation results. Simultaneously, a cross-modal fusion mechanism is introduced, enabling the SAM mask prior information to interact efficiently with the original visual features through a cross-attention mechanism. Furthermore, to further optimize the adaptability of the SAM mask prior information, a prior fusion coefficient based on visual features is used. This coefficient is calculated based on the smallest-scale visual features of the original medical image and is used to dynamically adjust the influence of the SAM mask prior information on different decoding layers. This allows the model to perform finer-grained control over modal fusion under different conditions, ensuring that the model can adaptively adjust the contribution weight of the SAM mask prior information when fusing features at different scales, ultimately improving the accuracy of medical image segmentation.

[0066] On the other hand, embodiments of the present invention also provide a medical image segmentation device, such as... Figure 5 As shown, the medical image segmentation device 500 includes: The image receiving unit 501 is used to receive the medical image to be segmented and input the medical image to be segmented into the trained medical image segmentation model; the medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head; The visual feature extraction unit 502 is used to extract multi-scale features from the medical image to be segmented based on the visual encoder, and obtain multiple visual features at different scales. The mask segmentation unit 503 is used to perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain the segmentation mask; The fusion coefficient determination unit 504 is used to generate multiple fusion coefficients based on the smallest scale visual feature among multiple visual features, which are used to control the fusion of the remaining scale visual features with the segmentation mask. The cross-modal fusion unit 505 is used to perform cross-modal fusion of multiple visual features and segmentation masks based on multiple fusion coefficients according to the prior guided decoder to generate multimodal features; Image segmentation unit 506 is used to detect multimodal features based on the detection head and generate image segmentation results.

[0067] The medical image segmentation device 500 provided in the above embodiments can realize the technical solutions described in the above medical image segmentation method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above medical image segmentation method embodiments, and will not be repeated here.

[0068] like Figure 6 As shown, the present invention also provides a medical image segmentation device 600. The medical image segmentation device 600 includes a processor 601, a memory 602, and a display 603. Figure 6 Only some components of the medical image segmentation device 600 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead.

[0069] In some embodiments, processor 601 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 602 or process data, such as the medical image segmentation method of the present invention.

[0070] In some embodiments, memory 602 may be an internal storage unit of the medical image segmentation device 600, such as a hard disk or memory of the medical image segmentation device 600. In other embodiments, memory 602 may also be an external storage device of the medical image segmentation device 600, such as a pluggable hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the medical image segmentation device 600.

[0071] In some embodiments, display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 603 is used to display information from the medical image segmentation device 600 and to display a user interface for visualization. Components 601-603 of the medical image segmentation device 600 communicate with each other via a device bus.

[0072] In some embodiments of the present invention, when the processor 601 executes the medical image segmentation program in the memory 602, the following steps may be performed: The system receives a medical image to be segmented and inputs it into a trained medical image segmentation model. The medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head. Multi-scale feature extraction is performed on the medical image to be segmented based on the visual encoder to obtain multiple visual features at different scales; The medical image to be segmented is segmented using a priori mask generator to obtain the segmentation mask. Based on the prior fusion coefficient generator, multiple fusion coefficients are generated from the smallest scale visual feature among multiple visual features to control the fusion of the remaining scale visual features with the segmentation mask. Based on the prior-guided decoder, multiple visual features and segmentation masks are fused across modally according to multiple fusion coefficients to generate multimodal features; The detection head detects multimodal features and generates image segmentation results.

[0073] It should be understood that when the processor 601 executes the medical image segmentation program in the memory 602, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0074] Furthermore, this embodiment of the invention does not specifically limit the type of the medical image segmentation device 600 mentioned. The medical image segmentation device 600 can be a portable medical image segmentation device such as a tablet computer, personal digital assistant (PDA), wearable device, or laptop computer. In some other embodiments of the invention, the medical image segmentation device 600 may not be a portable magnetic resonance imaging device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0075] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions of the medical image segmentation method provided in the above-described method embodiments.

[0076] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0077] The above provides a detailed description of a medical image segmentation method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A medical image segmentation method, characterized in that, include: Receive the medical image to be segmented and input the medical image to be segmented into the trained medical image segmentation model; The medical image segmentation model includes a visual encoder, a priori mask generator, a priori fusion coefficient generator, a priori guided decoder, and a detection head. Based on the visual encoder, multi-scale feature extraction is performed on the medical image to be segmented to obtain multiple visual features at different scales; Based on the prior mask generator, the medical image to be segmented is segmented using a mask to obtain a segmentation mask; Based on the prior fusion coefficient generator, multiple fusion coefficients are generated according to the smallest scale visual feature among the multiple visual features to control the fusion of the remaining scale visual features with the segmentation mask; Based on the prior guided decoder, the multiple visual features and the segmentation mask are fused across modally according to the multiple fusion coefficients to generate multimodal features; The multimodal features are detected using the detection head to generate image segmentation results.

2. The medical image segmentation method according to claim 1, characterized in that, The visual encoder includes a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block, and a fourth visual feature extraction block; The first visual feature extraction block is used to extract features from the medical image to be segmented to obtain first visual features; The second visual feature extraction block is used to extract features from the first visual feature to obtain the second visual feature; The third visual feature extraction block is used to extract features from the second visual features to obtain the third visual features; The fourth visual feature extraction block is used to extract features from the third visual features to obtain the fourth visual features; The scales of the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature gradually decrease.

3. The medical image segmentation method according to claim 1, characterized in that, The prior mask generator includes a mask segmentation module and a mask format conversion module; The mask segmentation module is used to perform mask segmentation on the medical image to be segmented to obtain an initial segmentation mask; The mask format conversion module is used to convert the initial segmentation mask, which has a shape of height H × width W × number of masks N, into a segmentation mask with a shape of L × N by flattening the spatial dimensions of the initial segmentation mask, where L = H × W.

4. The medical image segmentation method according to claim 2, characterized in that, The prior fusion coefficient generator includes a global average pooling layer, a multilayer perceptron, and an activation function layer; The global average pooling layer is used to perform global average pooling on the fourth visual feature to obtain a global feature vector. The multilayer perceptron is used to perform a nonlinear transformation on the global feature vector and output a multidimensional vector, which corresponds to the first visual feature, the second visual feature, and the third visual feature, respectively. The activation function layer is used to normalize the multidimensional vector to obtain the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient.

5. The medical image segmentation method according to claim 4, characterized in that, The priori guided decoder includes a projection module, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module; The projection module is used to perform a projection transformation on the segmentation mask to obtain projection features; The multi-head self-attention module is used to perform multi-head self-attention learning on the visual features to obtain visual enhancement features; The multi-head cross-attention module is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features based on the first fusion coefficient, the second fusion coefficient and the third fusion coefficient to obtain initial multimodal features; The upsampling module is used to reshape and upsample the initial multimodal features to obtain the multimodal features.

6. The medical image segmentation method according to claim 5, characterized in that, The multi-head cross-attention module includes a multi-head cross-attention learning unit, a normalization processing layer, and a multimodal fusion layer; The multi-head cross-attention learning unit is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features to obtain attention learning features; The normalization processing layer is used to normalize the attention learning features to obtain normalized features; The multimodal fusion layer is used to take the product of the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient with the normalized feature and the feature sum of the visual enhancement feature as the initial multimodal feature.

7. The medical image segmentation method according to claim 1, characterized in that, Before inputting the medical image to be segmented into the trained medical image segmentation model, the method further includes: Obtain a medical image dataset, and train the constructed initial image segmentation model based on the medical image dataset to obtain the medical image segmentation model; The loss function used during the training of the initial image segmentation model is the cross-entropy loss function.

8. A medical image segmentation device, characterized in that, include: An image receiving unit is used to receive a medical image to be segmented and input the medical image to be segmented into a trained medical image segmentation model; the medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head; The visual feature extraction unit is used to perform multi-scale feature extraction on the medical image to be segmented based on the visual encoder to obtain multiple visual features at different scales. A mask segmentation unit is used to perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain a segmentation mask. The fusion coefficient determination unit is used to generate multiple fusion coefficients based on the prior fusion coefficient generator according to the smallest scale visual feature among the multiple visual features, for controlling the fusion of the remaining scale visual features with the segmentation mask; A cross-modal fusion unit is used to perform cross-modal fusion of the multiple visual features and the segmentation mask based on the prior guided decoder according to the multiple fusion coefficients to generate multimodal features; The image segmentation unit is used to detect the multimodal features based on the detection head and generate image segmentation results.

9. A medical image segmentation device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the medical image segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can perform the steps in the medical image segmentation method according to any one of claims 1 to 7.

Images