A medical image segmentation method, apparatus, device and medium

Abstract

The application provides a medical image segmentation method, device, equipment and medium. The method comprises the following steps: receiving a medical image to be segmented, and inputting the medical image to be segmented into a trained medical image segmentation model; performing multi-scale feature extraction on the medical image to be segmented based on a visual encoder to obtain multiple visual features of different scales; performing mask segmentation on the medical image to be segmented based on a prior mask generator to obtain a segmentation mask; generating multiple fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask based on a prior fusion coefficient generator according to the smallest scale visual feature in the multiple visual features; performing cross-modal fusion on the multiple visual features and the segmentation mask based on a prior guided decoder according to the multiple fusion coefficients to generate multi-modal features; and performing detection on the multi-modal features based on a detection head to generate an image segmentation result. The application improves the segmentation accuracy and robustness of the medical image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, in particular to a medical image segmentation method, device, equipment and medium. BACKGROUND

[0002] In recent years, deep learning technology has greatly promoted the development of medical image segmentation, making it a core link in computer-aided diagnosis, surgical planning and navigation systems. However, medical images usually have complex organizational structure, blurred lesion boundary, diverse imaging modalities and strong noise interference, etc., which puts higher requirements on the ability of the segmentation model in spatial details and structural understanding.

[0003] At the same time, in the field of natural image segmentation, large-scale pre-training models have made breakthrough progress. Among them, Segment Anything Model (SAM) as a general segmentation base model is trained on a massive dataset of more than 11 million images, showing strong zero-shot segmentation ability and deep understanding of object structure. SAM can generate high-quality segmentation masks according to given prompt points or boxes, which provides a new idea for medical image segmentation: using the masks generated by SAM as prior knowledge to guide the segmentation process of medical images. However, direct application of SAM to medical image segmentation faces serious challenges.

[0004] Firstly, SAM is trained on natural images, and its learned texture, color and object shape priori do not match the gray characteristics, organizational texture and specific morphology of medical images, resulting in inaccurate masks generated by SAM on medical images, and direct use has limited effect. Secondly, in order to utilize the prior knowledge of SAM, existing technologies usually try to concatenate the masks generated by SAM with the original images in the channel, or use them as additional inputs for attention mechanisms. Although this method is simple, it has the problem of rigid utilization of priori: introducing priori in a fixed and unchanging way, which cannot dynamically adjust the degree of reliance on SAM priori information according to the specific content of the current medical image. When SAM priori itself has errors, this rigid fusion method can even amplify the errors, leading to a decrease in segmentation performance. In addition, medical image segmentation needs to integrate multi-scale information to capture both global semantics and local details. By using a single-scale priori mask for priori, it is impossible to establish an adaptive relationship between priori information and features of different scales, resulting in poor fusion quality and inaccurate medical image segmentation results.

[0005] Therefore, it is urgent to provide a medical image segmentation method, device, equipment and medium, which can effectively adapt the general segmentation priori knowledge of SAM to the medical image field, and can adaptively and on-demand fuse it into the multi-scale feature decoding process according to the image content, so as to achieve more accurate and robust segmentation results. SUMMARY

[0006] Therefore, it is necessary to provide a medical image segmentation method, device, equipment and medium to solve the technical problem that the prior art directly adapts a general solidified and single-scale prior mask to the medical image field without considering the difference between medical images and general images, resulting in inaccurate image segmentation results.

[0007] To solve the above technical problems, in a first aspect, the present application provides a medical image segmentation method, comprising:

[0008] receiving a to-be-segmented medical image and inputting the to-be-segmented medical image into a trained medical image segmentation model; the medical image segmentation model comprises a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder and a detection head;

[0009] performing multi-scale feature extraction on the to-be-segmented medical image based on the visual encoder to obtain a plurality of visual features of different scales;

[0010] performing mask segmentation on the to-be-segmented medical image based on the prior mask generator to obtain a segmentation mask;

[0011] generating a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask based on the smallest scale visual feature in the plurality of visual features by the prior fusion coefficient generator;

[0012] performing cross-modal fusion of the plurality of visual features and the segmentation mask based on the plurality of fusion coefficients by the prior guided decoder to generate multi-modal features;

[0013] detecting the multi-modal features based on the detection head to generate an image segmentation result.

[0014] In a possible implementation manner, the visual encoder comprises a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block and a fourth visual feature extraction block;

[0015] The first visual feature extraction block is configured to extract features from the to-be-segmented medical image to obtain a first visual feature;

[0016] The second visual feature extraction block is configured to extract features from the first visual feature to obtain a second visual feature;

[0017] The third visual feature extraction block is configured to extract features from the second visual feature to obtain a third visual feature;

[0018] The fourth visual feature extraction block is configured to perform feature extraction on the third visual feature to obtain a fourth visual feature.

[0019] The first visual feature, the second visual feature, the third visual feature, and the fourth visual feature have gradually decreasing scales.

[0020] In a possible implementation, the prior mask generator comprises a mask segmentation module and a mask format conversion module.

[0021] The mask segmentation module is configured to perform mask segmentation on the medical image to be segmented to obtain an initial segmentation mask.

[0022] The mask format conversion module is configured to convert the initial segmentation mask with a shape of height H x width W x mask number N into a segmentation mask with a shape of L x N by flattening the spatial dimension of the initial segmentation mask, where L = H x W.

[0023] In a possible implementation, the prior fusion coefficient generator comprises a global average pooling layer, a multi-layer perceptron, and an activation function layer.

[0024] The global average pooling layer is configured to perform global average pooling processing on the fourth visual feature to obtain a global feature vector.

[0025] The multi-layer perceptron is configured to perform nonlinear transformation on the global feature vector to output a multi-dimensional vector, which corresponds to the first visual feature, the second visual feature, and the third visual feature, respectively.

[0026] The activation function layer is configured to perform normalization processing on the multi-dimensional vector to obtain a first fusion coefficient, a second fusion coefficient, and a third fusion coefficient.

[0027] In a possible implementation, the prior guided decoder comprises a projection module, a multi-head self-attention module, a multi-head cross-attention module, and an up-sampling module.

[0028] The projection module is configured to perform projection transformation on the segmentation mask to obtain a projection feature.

[0029] The multi-head self-attention module is configured to perform multi-head self-attention learning on the visual feature to obtain a visual enhanced feature.

[0030] The multi-head cross-attention module is configured to perform multi-head cross-attention learning on the visual enhanced feature and the projection feature based on the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient to obtain an initial multi-modal feature.

[0031] The upsampling module is configured to reshape and upsample the initial multi-modal feature to obtain the multi-modal feature.

[0032] In a possible implementation, the multi-head cross attention module comprises a multi-head cross attention learning unit, a normalization processing layer, and a multi-modal fusion layer.

[0033] The multi-head cross attention learning unit is configured to perform multi-head cross attention learning on the visual enhanced feature and the projection feature to obtain an attention learning feature.

[0034] The normalization processing layer is configured to perform normalization processing on the attention learning feature to obtain a normalized feature.

[0035] The multi-modal fusion layer is configured to multiply the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient with the normalized feature respectively, and multiply the product with the feature of the visual enhanced feature and the projection feature as the initial multi-modal feature.

[0036] In a possible implementation, before inputting the medical image to be segmented into the trained medical image segmentation model, the method further comprises:

[0037] obtaining a medical image dataset, and training an initial image segmentation model based on the medical image dataset to obtain the medical image segmentation model.

[0038] The loss function used when training the initial image segmentation model is a cross-entropy loss function.

[0039] In a second aspect, the present application further provides a medical image segmentation device, comprising:

[0040] an image receiving unit configured to receive a medical image to be segmented and input the medical image to be segmented into a trained medical image segmentation model; the medical image segmentation model comprises a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head.

[0041] a visual feature extraction unit configured to perform multi-scale feature extraction on the medical image to be segmented based on the visual encoder to obtain a plurality of visual features of different scales.

[0042] a mask segmentation unit configured to perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain a segmentation mask.

[0043] a fusion coefficient determination unit configured to generate a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features with the segmentation mask based on the minimum scale visual feature in the plurality of visual features according to the prior fusion coefficient generator.

[0044] a cross-modal fusion unit, configured to perform cross-modal fusion on the plurality of visual features and the segmentation mask according to the plurality of fusion coefficients based on the prior guided decoder, to generate a multi-modal feature;

[0045] an image segmentation unit, configured to perform detection on the multi-modal feature based on the detection head, to generate an image segmentation result.

[0046] In a third aspect, the present application also provides a medical image segmentation device, comprising a memory and a processor, wherein,

[0047] The memory is configured to store a program.

[0048] The processor is coupled to the memory and configured to execute the program stored in the memory, so as to implement the steps of the medical image segmentation method in any possible implementation manner described above.

[0049] In a fourth aspect, the present application also provides a computer readable storage medium for storing computer readable programs or instructions, which can implement the steps of the medical image segmentation method in any possible implementation manner described above when executed by a processor.

[0050] The medical image segmentation method provided by the present application does not directly use the segmentation mask obtained by the prior mask generator as the image segmentation result, but uses the segmentation mask obtained by the prior mask generator as prior knowledge, and performs cross-modal fusion on the segmentation mask and visual features of different scales, so as to make up for the difference between medical images and general images, adaptively migrate the mask segmentation learned in general images to the medical image segmentation task, and thus significantly improve the segmentation accuracy of complex and irregular shape lesions, and solve the problem of insufficient performance of general models in professional medical fields. At the same time, in order to overcome the rigid problem caused by fixed prior, the present application designs a prior fusion coefficient generator, which can generate a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask according to the smallest scale visual feature in the plurality of visual features, dynamically determines the fusion coefficients for visual features of different scales, realizes dynamic multi-modal fusion, and thus improves the adaptability of the generated multi-modal feature to the medical image, and achieves the purpose of improving the accuracy of the image segmentation result.

[0051] Further, the prior fusion coefficient generator generates a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask, that is, a fusion implementation is generated for the visual features of each scale of the medical image, the fusion of the prior mask of different scales is realized, and the accuracy of the medical image segmentation result is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments description. Obviously, the drawings in the following description only show some embodiments of the present application, and all other drawings obtained by those skilled in the art based on these drawings without creative effort should be within the protection scope of the present application.

[0053] Figure 1 An embodiment flow diagram of the medical image segmentation method provided by the present application is shown in the following figure.

[0054] Figure 2 An embodiment structure diagram of the medical image segmentation model provided by the present application is shown in the following figure.

[0055] Figure 3 An embodiment structure diagram of the visual encoder provided by the present application is shown in the following figure.

[0056] Figure 4 An embodiment structure diagram of the multi-head cross attention module provided by the present application is shown in the following figure.

[0057] Figure 5 An embodiment structure diagram of the medical image segmentation device provided by the present application is shown in the following figure.

[0058] Figure 6 An embodiment structure diagram of the medical image segmentation device provided by the present application is shown in the following figure. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should be within the protection scope of the present application.

[0060] It should be understood that the schematic drawings are not drawn to scale. The flowcharts in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or removed from the flowcharts by those skilled in the art under the guidance of the content of the present application. Some block diagrams shown in the drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0061] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, nor are they necessarily all mutually exclusive or alternative embodiments. It is expressly understood that any of the embodiments described herein can be incorporated in a combination of embodiments.

[0062] The application provides a medical image segmentation method, device, equipment and medium, which are described below respectively.

[0063] Figure 1 An embodiment flowchart of a medical image segmentation method provided by the application is shown in the following figure, Figure 2 An embodiment structure diagram of a medical image segmentation model provided by the application is shown in the following figure, Figure 1 and Figure 2 The medical image segmentation method comprises:

[0064] S101, receiving a medical image to be segmented, and inputting the medical image to be segmented into a trained medical image segmentation model; the medical image segmentation model comprises a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder and a detection head;

[0065] S102, performing multi-scale feature extraction on the medical image to be segmented based on the visual encoder to obtain a plurality of visual features of different scales;

[0066] S103, performing mask segmentation on the medical image to be segmented based on the prior mask generator to obtain a segmentation mask.

[0067] Specifically, the prior mask generator is a general SAM model.

[0068] S104, generating a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask based on the smallest scale visual feature in the plurality of visual features based on the prior fusion coefficient generator;

[0069] S105, performing cross-modal fusion of the plurality of visual features and the segmentation mask based on the plurality of fusion coefficients based on the prior guided decoder to generate multi-modal features;

[0070] S106, detecting the multi-modal features based on the detection head to generate an image segmentation result.

[0071] Specifically, the detection head first fuses and then detects the multi-modal features to generate an image segmentation result. The image segmentation result includes but is not limited to a pixel-level segmentation mask, a target contour or region class information.

[0072] It should be understood that the medical image segmentation method in the embodiments of the present application can be implemented in any device based on the medical image segmentation method, for example, in a medical image segmentation device. Specifically, the medical image segmentation method is stored in the above-mentioned device in the form of a prepared program, and when the device is started, the program is called, and the medical image segmentation method is implemented.

[0073] Compared with the prior art, the medical image segmentation method provided by the embodiments of the present application does not directly use the segmentation mask obtained by the prior mask generator as the image segmentation result, but uses the segmentation mask obtained by the prior mask generator as prior knowledge, and performs cross-modal fusion with multiple visual features of different scales, thereby making up for the difference between medical images and general images, adaptively migrating the mask segmentation learned in general images to the medical image segmentation task, thereby significantly improving the segmentation accuracy of complex and irregular shape lesions, and solving the problem of insufficient performance of general models in professional medical fields. At the same time, in order to overcome the rigid problem caused by fixed prior, the prior fusion coefficient generator is designed in the embodiments of the present application, which can generate multiple fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask according to the smallest scale visual feature in the multiple visual features, dynamically determine the fusion coefficients for visual features of different scales, realize dynamic multi-modal fusion, and thereby improve the adaptability of the generated multi-modal features to the medical images, and achieve the purpose of improving the accuracy of the image segmentation result.

[0074] Further, the prior fusion coefficient generator generates multiple fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask, that is, a fusion implementation is generated for the visual features of each scale of the medical image, the fusion of the prior mask of different scales is realized, and the accuracy of the medical image segmentation result is further improved.

[0075] In some embodiments of the present application, the visual encoder is ConvNeXt-Tiny, as shown in Figure 3 The visual encoder includes a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block and a fourth visual feature extraction block.

[0076] The first visual feature extraction block is used for feature extraction on the medical image to be segmented to obtain the first visual feature.

[0077] The second visual feature extraction block is used for feature extraction on the first visual feature to obtain the second visual feature.

[0078] The third visual feature extraction block is used for feature extraction on the second visual feature to obtain the third visual feature.

[0079] The fourth visual feature extraction block is used for feature extraction on the third visual feature to obtain the fourth visual feature.

[0080] wherein the scales of the first visual feature, the second visual feature, the third visual feature and the fourth visual feature gradually decrease.

[0081] It should be understood that the fourth visual feature can represent the global semantic feature of the medical image to be segmented, and therefore the fusion coefficients of the other three visual features can be determined based on the fourth visual feature.

[0082] Specifically, the first visual feature, the second visual feature, the third visual feature and the fourth visual feature can be respectively represented as: , , , , is a feature size, and is the height and the width of the medical image to be segmented.

[0083] It should be noted that ConvNeXt-Tiny is a mature architecture, and the specific structures of the first visual feature extraction block, the second visual feature extraction block, the third visual feature extraction block and the fourth visual feature extraction block and other structures are not described here.

[0084] In specific embodiments of the present application, as shown in Figure 2 the prior mask generator includes a mask segmentation module and a mask format conversion module;

[0085] The mask segmentation module is configured to perform mask segmentation on the medical image to be segmented to obtain an initial segmentation mask.

[0086] Specifically, the initial segmentation mask can be represented as: . represents the operation of performing mask segmentation on the medical image to be segmented based on the SAM model.

[0087] The mask format conversion module is configured to convert the initial segmentation mask with a shape of height H x width W x number of masks N into a segmentation mask with a shape of L x N by flattening the spatial dimensions of the initial segmentation mask, where L = H x W.

[0088] The embodiment of the present application converts the initial segmentation mask into a two-dimensional matrix form of the segmentation mask, which can adapt to the subsequent processing process. At the same time, matrix operation is more efficient than three-dimensional tensor operation in hardware, and the visual feature is two-dimensional, the two-dimensional matrix form of the segmentation mask can be consistent with the dimension of the visual feature, thereby reducing the difficulty of subsequent cross-modal fusion, realizing more efficient information interaction, and improving the segmentation efficiency.

[0089] In some embodiments of the present application, as shown in Figure 2As shown, the prior fusion coefficient generator comprises a global average pooling layer, a multilayer perceptron and an activation function layer;

[0090] The global average pooling layer is configured to perform global average pooling processing on the fourth visual feature to obtain a global feature vector;

[0091] The multilayer perceptron is configured to perform nonlinear transformation on the global feature vector to output a multi-dimensional vector, which corresponds to the first visual feature, the second visual feature and the third visual feature respectively;

[0092] The activation function layer is configured to perform normalization processing on the multi-dimensional vector to obtain the first fusion coefficient, the second fusion coefficient and the third fusion coefficient.

[0093] Specifically, the determination process of each fusion coefficient can be represented as:

[0094]

[0095] In the formula, is the first fusion coefficient, the second fusion coefficient or the third fusion coefficient; is a global average pooling operation; is a multilayer perceptron operation; is an activation processing; is the fourth visual feature.

[0096] The multilayer perceptron is composed of two fully connected layers and a ReLU activation function.

[0097] In specific embodiments of the present application, the activation processing is performed using a Sigmoid function.

[0098] Specifically, is divided into , , , which are respectively configured to realize the cross-modal fusion of the first visual feature, the second visual feature and the third visual feature with the prior mask.

[0099] In some embodiments of the present application, as shown in Figure 2 , the prior guided decoder comprises a projection module, a multi-head self-attention module, a multi-head cross-attention module and an up-sampling module;

[0100] The projection module is configured to perform projection transformation on the segmentation mask to obtain a projection feature.

[0101] The purpose of the projection transformation is to align the dimensions of the segmentation mask with the dimensions of the visual feature.

[0102] Specifically, the projection feature can be represented as:

[0103]

[0104] wherein, is a segmentation mask; is a learnable matrix; is a convolution process; is an activation process, wherein an activation function used by the activation process is a ReLU function.

[0105] represents an output projection feature, wherein, is a length of the projection feature, is a channel number of the projection feature, is consistent with a channel number of the visual feature, wherein, .

[0106] The multi-head self-attention module is configured to perform multi-head self-attention learning on the visual feature to obtain a visual enhanced feature.

[0107] Specifically, the visual enhanced feature may be represented as:

[0108]

[0109] wherein, is a visual feature; is a normalization process; is a multi-head self-attention learning.

[0110] The multi-head cross-attention module is configured to perform multi-head cross-attention learning on the visual enhanced feature and the projection feature based on a first fusion coefficient, a second fusion coefficient and a third fusion coefficient to obtain an initial multi-modal feature.

[0111] The up-sampling module is configured to perform reshaping and up-sampling processes on the initial multi-modal feature to obtain a multi-modal feature.

[0112] Specifically, the multi-modal feature may be represented as:

[0113]

[0114] wherein, is a reshaping process; is an up-sampling process; is an initial multi-modal feature.

[0115] In specific embodiments of the present application, as shown in Figure 4 the multi-head cross-attention module includes a multi-head cross-attention learning unit, a normalization processing layer and a multi-modal fusion layer;

[0116] The multi-head cross-attention learning unit is configured to perform multi-head cross-attention learning on the visual enhanced feature and the projected feature to obtain an attention learning feature.

[0117] The normalization processing layer is configured to perform normalization processing on the attention learning feature to obtain a normalized feature.

[0118] The multi-modal fusion layer is configured to multiply the first fusion coefficient, the second fusion coefficient and the third fusion coefficient respectively with the normalized feature, and multiply the product with the feature of the visual enhanced feature and the feature of the attention learning feature as an initial multi-modal feature.

[0119] Specifically, the initial multi-modal feature may be represented as:

[0120]

[0121] In the formula, is the multi-head cross-attention learning.

[0122] As known by those skilled in the art, a deep learning model needs to be trained before use, therefore, in some embodiments of the present application, before the medical image to be segmented is input to the trained medical image segmentation model in step S101, the method further comprises:

[0123] obtaining a medical image dataset, and training the constructed initial image segmentation model based on the medical image dataset to obtain the medical image segmentation model;

[0124] The loss function used when training the initial image segmentation model is a cross-entropy loss function.

[0125] Specifically, when the loss value of the loss function converges or the number of training reaches a set maximum number, the training is stopped, and the medical image segmentation model is obtained.

[0126] In specific embodiments of the present application, the expression of the cross-entropy loss function is as follows:

[0127]

[0128] In the formula, represents the number of pixels in the image; represents the true class label of the i-th pixel; represents the predicted probability of the i-th pixel predicted by the model.

[0129] To sum up, the medical image segmentation method provided by the embodiment of the present application uses the segmentation mask generated by the SAM as prior knowledge, and combines the visual features to obtain more accurate segmentation results. At the same time, the cross-modal fusion mechanism is introduced, so that the SAM mask prior information can be efficiently interacted with the original visual features through the cross-attention mechanism. In addition, in order to further optimize the adaptability of the SAM mask prior information, the prior fusion coefficient is guided based on the visual features, that is, the fusion coefficient is calculated based on the visual features of the smallest scale of the original medical image, which is used to dynamically regulate the influence of the SAM mask prior information on different decoding layers, so that the model can control the modal fusion in different situations in a more fine-grained manner, and ensure that the model can adaptively adjust the contribution weight of the SAM mask prior information when fusing features of different scales, thereby improving the accuracy of medical image segmentation.

[0130] On the other hand, the embodiment of the present application also provides a medical image segmentation device, as shown in the following Figure 5 The medical image segmentation device 500 comprises:

[0131] The image receiving unit 501 is configured to receive a medical image to be segmented and input the medical image to be segmented into the trained medical image segmentation model; the medical image segmentation model comprises a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder and a detection head.

[0132] The visual feature extraction unit 502 is configured to perform multi-scale feature extraction on the medical image to be segmented based on the visual encoder, and obtain a plurality of visual features of different scales.

[0133] The mask segmentation unit 503 is configured to perform mask segmentation on the medical image to be segmented based on the prior mask generator, and obtain a segmentation mask.

[0134] The fusion coefficient determination unit 504 is configured to generate a plurality of fusion coefficients for controlling the fusion of the remaining scale visual features and the segmentation mask based on the minimum scale visual feature in the plurality of visual features based on the prior fusion coefficient generator.

[0135] The cross-modal fusion unit 505 is configured to perform cross-modal fusion on the plurality of visual features and the segmentation mask based on the plurality of fusion coefficients based on the prior guided decoder, and generate a multi-modal feature.

[0136] The image segmentation unit 506 is configured to perform detection on the multi-modal feature based on the detection head, and generate an image segmentation result.

[0137] The medical image segmentation device 500 provided by the above embodiment can realize the technical solutions described in the above medical image segmentation method embodiments, and the principles of the implementation of the above modules or units can be referred to the corresponding contents in the above medical image segmentation method embodiments, which will not be described here.

[0138] As Figure 6 shown, the present application also correspondingly provides a medical image segmentation device 600. The medical image segmentation device 600 comprises a processor 601, a memory 602 and a display 603. Figure 6 Only part of the components of the medical image segmentation device 600 are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0139] The processor 601 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, for running program codes or processing data stored in the memory 602, such as the medical image segmentation method in the present application.

[0140] The memory 602 can be an internal storage unit of the medical image segmentation device 600 in some embodiments, such as a hard disk or a memory of the medical image segmentation device 600. The memory 602 can also be an external storage device of the medical image segmentation device 600 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card and the like equipped on the medical image segmentation device 600.

[0141] The display 603 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch and the like in some embodiments. The display 603 is used to display information of the medical image segmentation device 600 and to display a visualized user interface. The components 601-603 of the medical image segmentation device 600 communicate with each other through a device bus.

[0142] In some embodiments of the present application, when the processor 601 executes the medical image segmentation program in the memory 602, the following steps can be implemented:

[0143] A medical image to be segmented is received, and the medical image to be segmented is input into a trained medical image segmentation model; the medical image segmentation model comprises a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder and a detection head;

[0144] Based on the visual encoder, multi-scale feature extraction is performed on the medical image to be segmented to obtain a plurality of visual features of different scales;

[0145] Based on the prior mask generator, mask segmentation is performed on the medical image to be segmented to obtain a segmentation mask;

[0146] The priori fusion coefficient generator generates a plurality of fusion coefficients for controlling fusion of the rest of the scale visual features with the segmentation mask according to the smallest scale visual feature in the plurality of visual features;

[0147] The priori guided decoder performs cross-modal fusion of the plurality of visual features and the segmentation mask according to the plurality of fusion coefficients to generate a multi-modal feature;

[0148] The detection head detects the multi-modal feature to generate an image segmentation result.

[0149] It should be understood that, in addition to the above functions, the processor 601 can also implement other functions when executing the medical image segmentation program in the memory 602. For details, refer to the description of the corresponding method embodiments.

[0150] Further, the type of the medical image segmentation device 600 referred to in the embodiments of the present application is not specifically limited, and the medical image segmentation device 600 can be a portable medical image segmentation device such as a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. In some other embodiments of the present application, the medical image segmentation device 600 can also not be a portable magnetic resonance device, but a desktop computer having a touch-sensitive surface (such as a touch panel).

[0151] Correspondingly, the embodiments of the present application also provide a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the medical image segmentation method steps or functions provided by the above method embodiments.

[0152] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware (such as a processor, a controller, etc.) to complete. The computer program can be stored in a computer-readable storage medium. The computer-readable storage medium includes a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0153] The above provides a detailed description of the medical image segmentation method, device, equipment and medium provided by the present application. The principle and implementation mode of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A medical image segmentation method, characterized in that, include: Receive the medical image to be segmented and input the medical image to be segmented into the trained medical image segmentation model; The medical image segmentation model includes a visual encoder, a priori mask generator, a priori fusion coefficient generator, a priori guided decoder, and a detection head. Based on the visual encoder, multi-scale feature extraction is performed on the medical image to be segmented to obtain multiple visual features at different scales; Based on the prior mask generator, the medical image to be segmented is segmented using a mask to obtain a segmentation mask; Based on the prior fusion coefficient generator, multiple fusion coefficients are generated according to the smallest scale visual feature among the multiple visual features to control the fusion of the remaining scale visual features with the segmentation mask; Based on the prior guided decoder, the multiple visual features and the segmentation mask are fused across modally according to the multiple fusion coefficients to generate multimodal features; The multimodal features are detected using the detection head to generate image segmentation results.

2. The medical image segmentation method according to claim 1, characterized in that, The visual encoder includes a first visual feature extraction block, a second visual feature extraction block, a third visual feature extraction block, and a fourth visual feature extraction block; The first visual feature extraction block is used to extract features from the medical image to be segmented to obtain first visual features; The second visual feature extraction block is used to extract features from the first visual feature to obtain the second visual feature; The third visual feature extraction block is used to extract features from the second visual features to obtain the third visual features; The fourth visual feature extraction block is used to extract features from the third visual features to obtain the fourth visual features; The scales of the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature gradually decrease.

3. The medical image segmentation method according to claim 1, characterized in that, The prior mask generator includes a mask segmentation module and a mask format conversion module; The mask segmentation module is used to perform mask segmentation on the medical image to be segmented to obtain an initial segmentation mask; The mask format conversion module is used to convert the initial segmentation mask, which has a shape of height H × width W × number of masks N, into a segmentation mask with a shape of L × N by flattening the spatial dimensions of the initial segmentation mask, where L = H × W.

4. The medical image segmentation method according to claim 2, characterized in that, The prior fusion coefficient generator includes a global average pooling layer, a multilayer perceptron, and an activation function layer; The global average pooling layer is used to perform global average pooling on the fourth visual feature to obtain a global feature vector. The multilayer perceptron is used to perform a nonlinear transformation on the global feature vector and output a multidimensional vector, which corresponds to the first visual feature, the second visual feature, and the third visual feature, respectively. The activation function layer is used to normalize the multidimensional vector to obtain the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient.

5. The medical image segmentation method according to claim 4, characterized in that, The priori guided decoder includes a projection module, a multi-head self-attention module, a multi-head cross-attention module, and an upsampling module; The projection module is used to perform a projection transformation on the segmentation mask to obtain projection features; The multi-head self-attention module is used to perform multi-head self-attention learning on the visual features to obtain visual enhancement features; The multi-head cross-attention module is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features based on the first fusion coefficient, the second fusion coefficient and the third fusion coefficient to obtain initial multimodal features; The upsampling module is used to reshape and upsample the initial multimodal features to obtain the multimodal features.

6. The medical image segmentation method according to claim 5, characterized in that, The multi-head cross-attention module includes a multi-head cross-attention learning unit, a normalization processing layer, and a multimodal fusion layer; The multi-head cross-attention learning unit is used to perform multi-head cross-attention learning on the visual enhancement features and the projection features to obtain attention learning features; The normalization processing layer is used to normalize the attention learning features to obtain normalized features; The multimodal fusion layer is used to take the product of the first fusion coefficient, the second fusion coefficient, and the third fusion coefficient with the normalized feature and the feature sum of the visual enhancement feature as the initial multimodal feature.

7. The medical image segmentation method according to claim 1, characterized in that, Before inputting the medical image to be segmented into the trained medical image segmentation model, the method further includes: Obtain a medical image dataset, and train the constructed initial image segmentation model based on the medical image dataset to obtain the medical image segmentation model; The loss function used during the training of the initial image segmentation model is the cross-entropy loss function.

8. A medical image segmentation device, characterized in that, include: An image receiving unit is used to receive a medical image to be segmented and input the medical image to be segmented into a trained medical image segmentation model; the medical image segmentation model includes a visual encoder, a prior mask generator, a prior fusion coefficient generator, a prior guided decoder, and a detection head; The visual feature extraction unit is used to perform multi-scale feature extraction on the medical image to be segmented based on the visual encoder to obtain multiple visual features at different scales. A mask segmentation unit is used to perform mask segmentation on the medical image to be segmented based on the prior mask generator to obtain a segmentation mask. The fusion coefficient determination unit is used to generate multiple fusion coefficients based on the prior fusion coefficient generator according to the smallest scale visual feature among the multiple visual features, for controlling the fusion of the remaining scale visual features with the segmentation mask; A cross-modal fusion unit is used to perform cross-modal fusion of the multiple visual features and the segmentation mask based on the prior guided decoder according to the multiple fusion coefficients to generate multimodal features; The image segmentation unit is used to detect the multimodal features based on the detection head and generate image segmentation results.

9. A medical image segmentation device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the medical image segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can perform the steps in the medical image segmentation method according to any one of claims 1 to 7.

Images